Staff ML Inference Engineer

About the role

Cloudera is looking for a Staff ML Inference Engineer in its Professional Services area, a hands-on ML engineering role focused on inference and performance. You will develop and optimize production model-serving solutions through Cloudera AI Inference, working alongside platform and applied AI engineers to turn customer requirements into reliable, efficient model endpoints.

Key responsibilities

• Deploy and optimize production model-serving solutions across on-premises, private-cloud and public-cloud environments, validating model, runtime and GPU compatibility, API behavior, concurrency and reliability.
• Develop repeatable benchmarks and recommend GPU types, counts and serving configurations based on model characteristics, memory, latency targets, throughput and availability needs.
• Diagnose bottlenecks in serving runtimes, GPU compute and memory, CPU, networking and I/O, and evaluate optimizations such as batching, KV-cache management, parallelism, quantization and replica scaling.
• Evaluate inference workloads across GPU generations, improve resource utilization and consider GPU sharing or partitioning.
• Build benchmarking, profiling, diagnostic and validation tooling to measure time to first token, inter-token latency, end-to-end latency, throughput and GPU memory consumption.
• Work with customer teams, platform engineers, applied AI engineers and core engineering to turn findings into product requirements, tested configurations and reusable deployment guidance.

Required profile

• Eight or more years of engineering experience in ML systems, software development, distributed systems or performance engineering, with evidence of technical leadership and independent ownership.
• Practical experience running GPU-based model-serving workloads in production with at least one inference stack such as NVIDIA NIM, vLLM, SGLang, TensorRT-LLM, Triton or equivalent.
• Ability to turn ambiguous requirements into testable hypotheses and validate solutions through experiments, profiling and measurement.
• Demonstrated experience benchmarking and improving inference performance, and sizing GPU resources against workload demand.

Required skills

• Strong Python programming.
• Solid foundation in Linux, containers, distributed services and Kubernetes.
• Understanding of model weights, runtime overhead, KV-cache requirements and latency-throughput trade-offs.
• Nice to have: open-weight LLM deployment, multi-GPU or multi-node inference, GPU sharing/partitioning (e.g. NVIDIA MIG), profiling with NVIDIA Nsight, and reusable benchmarking or deployment automation tools.

What we offer

• Generous PTO policy and Unplugged Days.
• Flexible WFH policy.
• Mental and physical wellness programs.
• Phone and internet reimbursement.
• Career development access, comprehensive benefits and competitive packages.
• Paid volunteer time and Employee Resource Groups.