Member of Technical Staff, Inference Systems
Job Description:
About the Role
Join a small AI systems team building a high-performance inference platform from the ground up. You will shape the runtime architecture and improve how large language models are served across GPUs and nodes.
What You'll Do
- Build a Rust inference runtime, owning batching, scheduling, request routing, and the serving stack.
- Design KV cache management, prefix caching, and other optimizations to reduce latency and cost per token.
- Scale inference across multiple GPUs and nodes.
- Profile and benchmark the inference pipeline, then deliver performance improvements.
- Collaborate with the team on core architectural decisions.
What We're Looking For
- Two to ten years of backend or distributed systems engineering experience.
- Hands-on experience building, operating, or optimizing LLM inference or serving systems, beyond using hosted APIs.
- Strong understanding of transformer inference, including attention, KV cache, batching, scheduling, and performance bottlenecks.
- Experience with production inference engines such as vLLM, SGLang, or TensorRT-LLM, and performance-critical systems where latency, throughput, and cost matter.
- Strong systems programming skills in Rust, C++, Go, or systems-level Python and PyTorch, with the ability to become productive in Rust quickly.
- Experience with multi-GPU or multi-node serving, CUDA or Triton, or techniques such as prefix caching, speculative decoding, and prefill/decode disaggregation is valuable.
- Strong computer science and systems fundamentals.
Compensation & Benefits
Base salary: $230,000 to $350,000 annually. Visa sponsorship is available.
Location
On-site in Palo Alto, California, United States.