Backend Inference Runtime Engineer Graduate (AML Inference) - 2027 Start

About the role

Data AML is ByteDance's machine learning mid-platform, providing training and inference systems for recommendation and advertising for businesses such as Douyin, Toutiao and Xigua Video. It supplies machine learning computing power to internal business units and researches general and innovative algorithms. ByteDance is recruiting a graduate for its Backend Inference Runtime Engineer position (AML Inference) starting in 2027. Successful candidates must be able to commit to an onboarding date by the end of 2027.

Key responsibilities

• Iterate on the underlying architecture of the large model inference engine and optimize end-to-end GPU performance through operator fusion and compilation optimization, improving single-card inference throughput and reducing latency.
• Adapt the inference engine to all GPU/NPU hardware architectures and build a high-performance, low-loss base for large model inference.
• Design and develop distributed parallel solutions for large model inference, covering tensor parallelism, pipeline parallelism, sequence parallelism and MoE expert parallelism, and addressing multi-card deployment, communication overhead, load imbalance and parallel efficiency.
• Follow cutting-edge technologies such as global large model inference, GPU high-performance computing and cache optimization, benchmark against vLLM and TensorRT-LLM, and build the team's core technological barriers.

Required profile

• Completing or recently completed a Bachelor's or Master's degree in computing or a related discipline.
• Solid foundation in computer low-level knowledge, with good cross-team collaboration, communication, presentation and document-writing skills, and strong stress tolerance.
• Preferred: thorough understanding of large model inference principles and experience implementing distributed inference solutions.

Required skills

• C/C++ and Python, with CUDA programming and familiarity with GPU hardware and memory models.
• Deep learning operator development, including matrix operations, normalization and activation functions.
• Compilation technologies such as operator fusion, constant folding, memory reuse, scheduling optimization and quantization.
• GPU profiling with tools such as Nsight and Profiler.
• Preferred: experience customizing vLLM, SGLang or TensorRT-LLM.