Large Language Model (LLM) Evaluation Engineer

About the role

We are looking for a Senior Executive to design, build and maintain an automated evaluation pipeline for large language models. The role focuses on assessing general, agent and persona capabilities across multiple benchmarks and ensuring reliable, reproducible results.

Key responsibilities

• Develop and operate a one‑click LLM evaluation pipeline covering general, agent and persona/role‑playing dimensions, with historical comparison and regression testing.
• Run benchmark suites such as MMLU, C‑Eval, HumanEval, GSM8K, MATH, IFEval, BFCL, τ‑bench and GAIA, handling deployment, execution and result analysis.
• Design persona/role‑playing evaluation frameworks measuring identity recognition, role compatibility, multi‑turn stability and style consistency.
• Record, analyze and report evaluation outcomes, perform comparative and anomaly detection, and generate checkpoint evaluation reports.
• Conduct intermediate evaluations during pre‑training to monitor model capability evolution.

Required profile

• Bachelor’s degree or higher in Computer Science, Artificial Intelligence or a related field.
• Familiarity with mainstream LLM evaluation benchmarks and frameworks such as lm‑eval‑harness, OpenCompass and EvalPlus.
• Strong proficiency in Python and ability to build end‑to‑end evaluation pipelines.
• Experience with LLM inference frameworks like vLLM and SGLang for batch inference and evaluation.
• Proven experience in evaluation data analysis and visualization.
• Detail‑oriented, rigorous and committed to reproducibility of results.

Required skills

• Python programming
• lm‑eval‑harness
• OpenCompass
• EvalPlus
• vLLM
• SGLang
• Data analysis and visualization tools
• CI/CD integration for evaluation automation