Member of Technical Staff, Inference Systems

  • Palo Alto, California, United States
  • Full-Time
  • On-Site

Job Description:

About the Role

Join a small AI systems team building a high-performance inference platform from the ground up. You will shape the runtime architecture and improve how large language models are served across GPUs and nodes.

What You'll Do

  • Build a Rust inference runtime, owning batching, scheduling, request routing, and the serving stack.
  • Design KV cache management, prefix caching, and other optimizations to reduce latency and cost per token.
  • Scale inference across multiple GPUs and nodes.
  • Profile and benchmark the inference pipeline, then deliver performance improvements.
  • Collaborate with the team on core architectural decisions.

What We're Looking For

  • Two to ten years of backend or distributed systems engineering experience.
  • Hands-on experience building, operating, or optimizing LLM inference or serving systems, beyond using hosted APIs.
  • Strong understanding of transformer inference, including attention, KV cache, batching, scheduling, and performance bottlenecks.
  • Experience with production inference engines such as vLLM, SGLang, or TensorRT-LLM, and performance-critical systems where latency, throughput, and cost matter.
  • Strong systems programming skills in Rust, C++, Go, or systems-level Python and PyTorch, with the ability to become productive in Rust quickly.
  • Experience with multi-GPU or multi-node serving, CUDA or Triton, or techniques such as prefix caching, speculative decoding, and prefill/decode disaggregation is valuable.
  • Strong computer science and systems fundamentals.

Compensation & Benefits

Base salary: $230,000 to $350,000 annually. Visa sponsorship is available.

Location

On-site in Palo Alto, California, United States.