Large Language Model Pre-training Engineer

About the role

We are seeking a skilled Large Language Model Pre‑training Engineer to lead the design and execution of cutting‑edge LLM training pipelines. You will work on end‑to‑end pre‑training projects, ensuring high‑performance, scalable, and reliable model development.

Key responsibilities

• Define and implement pre‑training strategies and architecture.
• Develop and maintain large‑scale data engineering pipelines for LLM training.
• Execute distributed training on clusters with 64+ GPUs.
• Implement long‑context training techniques such as RoPE scaling, NTK‑aware interpolation, and YaRN.
• Monitor training metrics, optimize performance, and iterate on model improvements.
• Evaluate model outputs and drive iterative optimization cycles.

Required profile

• Bachelor's degree or higher in Computer Science, AI, NLP, Machine Learning, Distributed Systems, or related field.
• Hands‑on experience delivering end‑to‑end LLM pre‑training projects, including models of 7B+ parameters.
• Proven ability to troubleshoot and resolve issues in large‑scale distributed training environments.
• Experience with long‑context training and context‑extension methods.

Required skills

• Megatron‑LM, DeepSpeed, FSDP frameworks.
• Large‑scale GPU clusters (64+ GPUs) and distributed training systems.
• Data cleaning, deduplication, quality filtering, tokenization, data mixing, and data quality optimization.
• Analysis of training loss, gradients, convergence, and stability.
• RoPE scaling, NTK‑aware interpolation, YaRN techniques.
• Mixture‑of‑Experts (MoE) model pre‑training.

What we offer

• Competitive monthly salary (SGD 5,000‑10,000).
• Opportunity to work on state‑of‑the‑art LLM projects.
• Collaborative environment with leading AI researchers.