ML Infrastructure Engineer
About the Role
We're a seed-stage enterprise AI infrastructure company building the context layer that makes AI agents reliable, accurate, and secure for critical business operations — including highly regulated industries like insurance, banking, asset management, and healthcare. Our platform automatically constructs a governed, real-time domain model across all enterprise data, enabling AI agents to make confident, auditable decisions in production.
As an ML Infrastructure Engineer , you'll own the systems that keep our agents running reliably and fast at scale. This is a hands-on production engineering role — focused on real-world impact, not research. You'll design, build, and scale our inference and model-serving infrastructure as concurrency and customer demands grow.
What You'll Do
-
Own inference and model-serving infrastructure end to end — from initial design through production deployment and ongoing scaling.
-
Build and scale systems that enable AI agents to run reliably and efficiently under high and increasing concurrency.
-
Identify and resolve infrastructure bottlenecks in collaboration with ML and platform engineering teams.
-
Optimize systems for latency, throughput, and reliability across cloud-hosted production environments.
-
Drive observability, monitoring, and debugging practices across our production ML stack.
What We're Looking For
Dealbreakers — all required:
-
5+ years of hands-on experience building and operating machine learning inference systems, model-serving platforms, or ML infrastructure in production environments.
-
Demonstrated experience designing and scaling inference-serving infrastructure using tools such as TensorFlow Serving, TorchServe, Triton, KServe, or equivalent custom systems.
-
Proven ability to optimize production ML systems for latency, throughput, and reliability at scale.
Required skills:
-
Experience with containerization and orchestration (Docker, Kubernetes) for deploying and scaling ML workloads.
-
Background in distributed systems that handle high concurrency and dynamic resource allocation under load.
-
Proficiency with monitoring and observability tooling — e.g., Prometheus, Grafana, ELK stack, distributed tracing.
-
Experience deploying and managing ML systems on cloud platforms (AWS, GCP, or Azure).
-
Proficiency in at least one systems or backend language: Python, Go, Rust, C++, or Java.
Nice to have:
-
Experience with knowledge graphs, semantic search, or graph databases (e.g., Neo4j, Amazon Neptune, or similar).
-
Background in real-time inference or low-latency serving requirements.
-
Familiarity with agentic AI systems, autonomous agents, or multi-step reasoning pipelines.
-
Experience with enterprise data infrastructure, data pipelines, or data integration platforms.
Location & Work Arrangement
This is a full-time, on-site role based in San Mateo, CA . On-site collaboration is an important part of how this small, fast-moving team operates. Visa sponsorship is not available for this position.
Why Join
-
Early-stage opportunity with significant ownership and impact — you'll shape foundational infrastructure decisions.
-
Work on genuinely hard distributed systems problems in a production AI context.
-
Small, experienced team with deep ML and enterprise engineering backgrounds.
-
Well-funded at the seed stage with strong institutional backing and a clear enterprise customer focus.
Originally posted on Himalayas