Data Engineering Consultant – Senior Level

About the role

We are looking for a highly skilled and experienced Senior Data Engineer to design, develop and optimize scalable data engineering solutions using Big Data, cloud and modern AI technologies. The successful candidate will be responsible for developing high‑performance data pipelines, data ingestion frameworks and data processing solutions, while working closely with architects, business stakeholders, data scientists and engineering teams.

Key responsibilities

• Design, develop and maintain scalable batch and real‑time data pipelines using Apache Spark/PySpark, SQL and Python.
• Develop and optimise ETL/ELT pipelines for large‑scale data processing.
• Build data ingestion solutions from databases, APIs, files and streaming platforms.
• Work with AWS services such as S3, Glue, EMR, Redshift, Kinesis, Lambda and DynamoDB.
• Implement data models, data warehouses and data lake solutions.
• Create CI/CD pipelines and support containerised deployments with Docker, Kubernetes and OpenShift.
• Integrate Generative AI and NLP capabilities into enterprise data applications.

Required profile

• Bachelor’s or Master’s degree in Computer Science, IT, Engineering or related field.
• Minimum 8 years of experience in Data Engineering / Big Data with large‑scale data processing.
• Strong programming experience in Python, Java and/or Scala.
• Hands‑on experience with Apache Spark/PySpark, SQL, Hadoop and Hive.
• Experience with cloud platforms, particularly AWS.
• Familiarity with Kafka, Airflow, Jenkins, Git and CI/CD practices.

Required skills

• Python
• Java
• Scala
• PySpark / Apache Spark
• SQL
• Hadoop
• Hive
• Kafka
• AWS (S3, Glue, EMR, Redshift, Kinesis, Lambda, DynamoDB)
• Docker
• Kubernetes / OpenShift
• Airflow
• Jenkins
• Databricks
• Snowflake
• MongoDB
• Oracle
• PostgreSQL
• Generative AI technologies