Research Engineer, Benchmarks

About the role

A small, technical team in the AI evaluation sector is looking for a Research Engineer, Benchmarks to design and build rigorous benchmarks for frontier AI agents. The work helps ensure evaluations reflect realistic workflows and give credible insight into real-world agent performance.

Key responsibilities

• Design, implement and maintain internal benchmarks for evaluating AI agents on domain-specific tasks.
• Work with subject-matter experts to turn real workflows into evaluation tasks and criteria.
• Build reliable infrastructure to run models and agents against benchmark tasks at scale.
• Develop metrics and analyses to assess benchmark difficulty, reliability and failure modes.
• Validate how benchmark results relate to real-world evaluations and user needs.
• Write clear technical documentation and benchmark reports.

Required profile

• 2 to 4 years of experience in software engineering, machine learning engineering or research, including at least 2 years building AI benchmarks, evaluation infrastructure or agent environments.
• Degree in a technical field.
• Experience designing and operating benchmark or evaluation environments for AI agents or large language models.
• Experience building infrastructure to run AI models or agents reliably, and developing metrics, statistical analyses or validation studies.
• Experience collaborating with subject-matter experts to model workflows and create benchmark tasks.
• Strong technical writing, attention to detail and ability to reason from first principles in an early-stage, unstructured environment.
• Published research or technical writing on AI benchmarking, model evaluation or failure modes is valued; reinforcement learning experience is also useful.

Required skills

• Python
• Docker
• Linux

What we offer

• Base salary of $100,000 to $170,000 USD annually.
• Visa sponsorship available.
• On-site location in Singapore.