Senior Databricks Consultant / Architect

About the role

An experienced Senior Databricks Consultant / Architect is sought to lead the design, architecture, development and implementation of enterprise-scale data engineering and analytics solutions on Azure Databricks. The role centres on Lakehouse architecture, PySpark, Delta Lake, cloud data platforms and data governance.

Key responsibilities

• Design and architect scalable, secure, high-performance Lakehouse solutions on Databricks
• Lead end-to-end data engineering and data platform implementation
• Define enterprise data architecture, standards, best practices and technical roadmaps
• Build and optimise ETL/ELT pipelines for batch and streaming data, including Medallion Architecture (Bronze, Silver, Gold) and Delta Live Tables
• Use Unity Catalog for data governance, security, access control and lineage
• Integrate Databricks with Azure services such as ADLS Gen2, Azure Data Factory, Event Hubs and Azure Key Vault
• Optimise Spark workloads, clusters, SQL queries and platform performance
• Set up CI/CD pipelines, automated deployments and DevOps practices
• Provide technical leadership, run design reviews and mentor data engineering teams
• Translate business requirements into technical solutions with stakeholders and engineering teams

Required profile

• Strong hands-on experience with Azure Databricks and Apache Spark
• Strong experience in enterprise data architecture and large-scale data engineering projects
• Able to drive architecture decisions and technical delivery independently
• Excellent communication, stakeholder management and problem-solving skills
• Ability to work with cross-functional teams and give technical direction

Required skills

• PySpark, Spark SQL, Python and SQL
• Delta Lake, Lakehouse architecture and Medallion Architecture
• Unity Catalog, Delta Live Tables and Auto Loader
• Azure Data Lake Storage Gen2 and Azure Data Factory
• Databricks Workflows, cluster management and performance tuning
• Data security, governance and access management
• Git, Azure DevOps and CI/CD
• Good to have: AWS or GCP Databricks, Kafka, Databricks SQL and monitoring, Terraform, MLflow, Databricks Professional certification