AI Data Engineer – Cloudera to AWS Migration Consultant

About the role

We are looking for a senior AI & Data Engineering consultant to lead the migration of enterprise data workloads from Cloudera/Hadoop to AWS. The role focuses on designing and delivering scalable AI, Machine Learning and Generative AI solutions that drive business innovation.

Key responsibilities

• Design, develop and deploy enterprise‑scale AI, ML and Generative AI solutions on cloud and hybrid environments.
• Build and optimise data processing pipelines using Python, SQL, Spark and PySpark.
• Implement Retrieval‑Augmented Generation (RAG) solutions with embeddings, vector databases and large language models.
• Migrate and modernise Cloudera/Hadoop workloads to AWS services such as SageMaker, Glue, EMR, Athena, Redshift and S3.
• Develop MLOps frameworks for CI/CD, model lifecycle management, monitoring and governance.
• Collaborate with stakeholders, architects and engineering teams to translate business needs into AI‑driven solutions.

Required profile

• Senior‑level professional with strong experience in data engineering, machine learning and generative AI.
• Hands‑on expertise in Python, SQL and PySpark/Spark.
• Proven track record migrating Cloudera/Hadoop workloads to AWS.
• Ability to design secure, high‑performance AI architectures.

Required skills

• Python
• SQL
• PySpark / Spark
• Cloudera, Hadoop, HDFS, Hive, Impala, CDP
• AWS (S3, Glue, EMR, Athena, Redshift, SageMaker)
• Generative AI, LLM integration, RAG, embeddings, vector databases
• LangChain, Bedrock, OpenAI, Azure OpenAI, Pinecone, Weaviate, OpenSearch
• Kubernetes

What we offer

• Competitive contract rate (SGD 5,500‑8,250 per month)
• Opportunity to work on cutting‑edge AI projects in a leading financial services environment
• Collaborative team with strong technical expertise