AI Engineer (Evaluation)

About the role

This position is an AI Engineer (Evaluation) vacancy hosted at Nanyang Technological University (NTU) under the office of the Vice President (Artificial Intelligence & Digital Economy). It sits within the AI Singapore initiative, which brings together research institutions, AI start-ups and companies to support impactful research, develop talent and advance Singapore's AI efforts. The role focuses on evaluating Large Language Models (LLMs).

Key responsibilities

• Develop and maintain evaluation frameworks and pipelines to measure the capabilities of Large Language Models (LLMs)
• Stay up to date with and experiment with the latest research in multilingual, multicultural and multimodal LLM evaluation, including the LLM-as-a-Judge paradigm
• Work with partners to collect, translate and verify evaluation datasets
• Perform data preparation and analysis, AI modelling, coding, testing, validation and deployment to deliver reliable and scalable AI solutions
• Collaborate with cross-functional teams within AI Products to design and resolve issues
• Maintain code repository and documentation standards
• Contribute to community engagement such as technical session meet-ups, article write-ups and discussion forums

Required profile

• Degree in computer science, AI, data science or a related field (or equivalent practical experience)
• Strong understanding of LLM evaluation experimental design and the advantages and limitations of different evaluation methods
• Experience with LLM inference frameworks such as vLLM and AI/deep learning frameworks such as PyTorch
• Ability to read and understand research papers
• Fluent in English and one other Southeast Asian language, to build quality evaluations from a multilingual and multicultural perspective

Required skills

• Production-level Python coding
• Version control with Git
• LLM inference frameworks (e.g. vLLM) and deep learning frameworks (e.g. PyTorch)

Only shortlisted candidates will be notified.