Senior ML Engineer | Kimchi (LLM Inference Optimization)

📋 Job Overview

Throughput. Latency. KV cache utilization. Move those three numbers in the right direction, and two things happen: customers get faster, cheaper inference, and our margins improve. That's the entire thesis of this role. Every kernel you tune, every quantization scheme you ship, every scheduler tweak you land shows up directly in a customer's p99 and on our P&L. This is a high-impact seat with high autonomy to lead the technical direction of inference optimization at Kimchi.

🏢 About Cast AI

Cast AI is an automation platform that operates cloud-native and AI infrastructure at scale. By embedding autonomous decision-making directly into Kubernetes and cloud environments, Cast AI continuously optimizes performance, reliability, and efficiency in production. Over 2,100 companies already rely on Cast AI, including Akamai, BMW, Cisco, FICO, HuggingFace, NielsenIQ, Swisscom, and TGS. We're headquartered in Miami but operate across 34 countries with a global and intentional approach to diversity.

🎯 The Role

Kimchi is the system that makes decisions automatically - continuously matching workloads to the most cost-efficient, best-performing LLM and serving configuration on a customer's infrastructure. We're building the optimization layer between the model and the hardware, and we need engineers who understand both sides deeply.

✅ Key Responsibilities

• Push throughput through continuous batching, speculative decoding, chunked prefill, and kernel-level tuning.
• Cut latency by attacking TTFT and TPOT separately and fixing actual bottlenecks.
• Optimize KV cache utilization through paged attention, prefix caching, eviction policies, and cache reuse.
• Quantize without regressing quality using INT8, INT4, FP8 across weights, activations, and KV.
• Shrink cold starts and memory footprint through faster init, smarter weight loading, and tighter memory accounting.
• Scale across nodes with distributed inference topologies, network-aware placement, and checkpointing strategies.
• Set the technical direction for inference optimization.

📌 Required Qualifications

• 5+ years building real ML systems with depth in inference or training infrastructure.
• Strong Python for production services.
• Hands-on experience with vLLM, SGLang, or TensorRT-LLM.
• Fluency with quantization tradeoffs and quality regressions.
• Comfort with distributed systems: collective communication, sharding strategies, and failure modes.
• A bias toward measurement and self-direction.

🎁 Benefits

• Competitive salary (depending on the level of experience).
• Flexible, remote-first global environment.
• Collaborate with a global team of cloud experts and innovators.
• Equity options.
• Fast-paced workflow with quick feedback.
• 10% of work time for personal projects or self-improvement.
• Learning budget for professional and personal development.
• Annual hackathon to spark new ideas and strengthen team bonds.
• Team-building budget and company events.
• Equipment budget.
• Extra days off to help maintain a healthy work-life balance.

🛂 Visa & Eligibility

Cast AI does not provide any form of visa sponsorship/work permit.