Senior Software Engineer - LLM Inference Optimization

About the role

Rakuten Asia Pte. Ltd., the Asia regional headquarters of Rakuten, is hiring an LLM Inference Optimization Engineer within its AI & Data Division. The engineer focuses on maximizing the performance, efficiency and scalability of LLM inference workloads on Rakuten's GPU clusters, optimizing inference engines and their underlying GPU kernels and runtimes so that AI models are served at peak efficiency.

Key responsibilities

• Profile, optimize and extend modern LLM inference engines (e.g. vLLM, SGLang, TensorRT-LLM) to improve throughput, latency and GPU utilization.
• Design and implement inference-time optimizations such as quantization, KV-cache management, continuous batching, speculative decoding and parallelism strategies (tensor, expert, pipeline).
• Develop and tune GPU kernels (CUDA, Triton) for critical operators such as attention, GEMM and MoE, targeting the latest NVIDIA architectures.
• Build benchmarking and load-testing frameworks to measure end-to-end serving performance under realistic workloads and SLOs.
• Partner with infrastructure teams to deploy and scale inference on Kubernetes-based GPU clusters, including autoscaling, fault tolerance and observability.
• Keep up with inference research and bring cutting-edge techniques into production.

Required profile

• 3+ years of hands-on experience optimizing deep learning inference on NVIDIA GPUs, preferably for large language models.
• Bachelor's or higher degree in Computer Science, Engineering or a related field.
• Experience with at least one mainstream inference serving engine (vLLM, SGLang, TensorRT-LLM, Triton Inference Server or equivalent).

Required skills

• Strong knowledge of LLM inference internals: attention mechanisms, KV cache, batching strategies, quantization and parallelism.
• Proficiency in CUDA and/or Triton kernel programming, with a solid understanding of modern GPU architecture.
• Strong software engineering fundamentals in Python and C++.
• Nice to have: open-source inference or ML projects, low-precision inference (FP8, INT4, FP4) and model compression, GPU profiling tools (Nsight Systems, PyTorch Profiler), Kubernetes deployment experience, SLO-driven capacity planning, and experience with training frameworks such as PyTorch, DeepSpeed, Megatron-LM or FSDP.