AI Engineer – LLM Evaluation Specialist

About the role

AI Singapore, hosted at Nanyang Technological University, is looking for an AI Engineer to design and run evaluation frameworks for large language models. You will work with a multidisciplinary team to push the boundaries of multilingual, multicultural, and multimodal AI capabilities.

Key responsibilities

• Develop and maintain evaluation pipelines that measure the performance of Large Language Models (LLMs).
• Stay current with research on multilingual, multicultural, and multimodal LLM evaluation methods, including the LLM‑as‑a‑Judge paradigm.
• Collaborate with partners to collect, translate, and verify evaluation datasets.
• Perform data preparation, AI modelling, coding, testing, validation, and deployment of scalable AI solutions.
• Work with cross‑functional AI product teams to design solutions and resolve issues.
• Maintain code repositories and documentation standards.
• Engage with the community through technical sessions, article write‑ups, and discussion forums.

Required profile

• Graduate degree in Computer Science, AI, Data Science or a related field, or equivalent practical experience.
• Deep understanding of LLM evaluation experimental design and the strengths/limitations of various evaluation methods.
• Strong written and verbal communication skills in English and at least one Southeast Asian language.
• Independent learner capable of reading and interpreting research papers.

Required skills

• Python programming.
• Version control with Git.
• Experience with LLM inference frameworks such as vLLM.
• Proficiency in deep‑learning frameworks, especially PyTorch.
• Design and execution of LLM evaluation experiments.