AI Engineer – Evaluation of Large Language Models

About the role

AI Singapore, hosted at Nanyang Technological University, is looking for an AI Engineer focused on evaluation. The role will join a multidisciplinary team of AI scientists, data engineers and software engineers to design and run rigorous assessments of large language models, especially their multilingual, multicultural and multimodal capabilities.

Key responsibilities

• Develop and maintain evaluation frameworks and pipelines for Large Language Models (LLMs).
• Stay current with research on multilingual, multicultural and multimodal LLM evaluation, including the LLM‑as‑a‑Judge paradigm.
• Collaborate with partners to collect, translate and verify evaluation datasets.
• Perform data preparation, AI modelling, coding, testing, validation and deployment of scalable evaluation solutions.
• Work with cross‑functional AI product teams to design experiments and resolve technical issues.
• Maintain code repositories and documentation standards.
• Contribute to community activities such as technical meet‑ups, article write‑ups and discussion forums.

Required profile

• Graduate degree in Computer Science, AI, Data Science or a related field, or equivalent practical experience.
• Deep understanding of LLM evaluation experimental design and the strengths/limitations of various methods.
• Experience with LLM inference frameworks such as vLLM and deep‑learning frameworks like PyTorch.
• Proficiency in writing production‑level Python code and using Git for version control.
• Strong written and verbal communication skills.
• Ability to work independently and quickly grasp research papers.
• Fluent in English and one additional Southeast Asian language.

Required skills

• Python programming
• vLLM inference framework
• PyTorch deep‑learning library
• Git version control