Research Engineer, Benchmarks

About the role

This is a core technical role on a small, high‑caliber team building rigorous benchmarks to evaluate frontier AI agents on realistic, domain‑specific workflows. You will own the design and implementation of evaluations that frontier labs and enterprise customers rely on to understand real‑world agent performance. The work is critical to the credibility and impact of the company’s benchmark platform.

Key responsibilities

• Design, implement, and maintain internal benchmarks for evaluating frontier agents on domain‑specific tasks.
• Partner with subject‑matter experts to define realistic workflows and translate them into well‑scoped evaluation tasks.
• Build reliable infrastructure to run models and agents against benchmark tasks at scale.
• Develop metrics and statistical analyses that measure benchmark difficulty, reliability, and failure modes.
• Validate that benchmark performance correlates with real‑world evaluations and customer expectations.
• Write clear technical documentation and benchmark reports for research and engineering audiences.

Required profile

• 2 to 4 years of experience in research engineering or machine learning engineering, focusing on AI benchmarks or evaluation infrastructure.
• Strong proficiency in Python, Docker, and Linux for building research or production infrastructure.
• Hands‑on experience designing and running benchmarks or evaluation environments for AI agents or large language models.
• Experience developing metrics and validation studies to assess benchmark difficulty, reliability, and real‑world correlation.
• Comfort working independently in fast‑paced, early‑stage environments with unstructured problem spaces.
• Sharp attention to detail and ability to reason from first principles about task design, scoring, and edge cases.

Required skills

• Python
• Docker
• Linux
• Reinforcement learning training pipelines
• Data generation
• RL agent evaluation

What we offer

• Salary range: $150,000 to $250,000 USD annually.
• Visa sponsorship is available.