Data Engineer (Databricks)

About the role

We are seeking a skilled Data Engineer with deep expertise in Databricks to design, build, and maintain robust data pipelines for our enterprise data platform. You will work on a fast‑paced project environment, collaborating with cross‑functional technical teams to deliver high‑quality data solutions.

Key responsibilities

• Design and implement production‑grade data pipelines using Databricks, Apache Spark, and PySpark.
• Ensure pipelines include comprehensive logging, error handling, and recovery mechanisms.
• Work with Delta Lake and layered architectures (Bronze/Silver/Gold) to manage data lifecycle.
• Integrate AWS services such as S3, IAM, Glue, Lambda, and Kinesis for storage and processing.
• Optimize data models for quality, performance, and scalability.
• Develop and maintain Databricks Workflows, Delta Live Tables, Unity Catalog, Auto Loader, and Structured Streaming jobs.
• Implement CI/CD pipelines and infrastructure automation using Git‑based tools.
• Collaborate with BI teams to deliver Tableau dashboards and integrate with data warehouses.

Required profile

• 4+ years of experience in data engineering or enterprise data‑platform delivery.
• Hands‑on experience with Databricks, Apache Spark, PySpark, and SQL.
• Proven experience with Delta Lake and layered data architecture.
• Strong knowledge of AWS services (S3, IAM, Glue, Lambda, Kinesis) and data modelling.
• Experience building reliable pipelines with logging, error handling, and recovery.
• Familiarity with CI/CD, Git, and automation tools.
• Relevant certifications in Databricks, AWS, data engineering, AI/ML, or GenAI are a plus.

Required skills

• Databricks
• Apache Spark
• PySpark
• SQL
• Delta Lake
• Bronze/Silver/Gold data architecture
• AWS S3, IAM, Glue, Lambda, Kinesis
• Data modelling and performance optimisation
• Databricks Workflows, Delta Live Tables, Unity Catalog
• Auto Loader, Structured Streaming
• APIs, XML/JSON ingestion
• CI/CD, Git
• Tableau and BI integration