Inference Infrastructure Architect

About Telnyx

Telnyx is an industry leader that's not just imagining the future of global connectivity—we're building it. From architecting and amplifying the reach of a private, global, multi-cloud IP network , to bringing hyperlocal edge technology right to your fingertips through intuitive APIs, we're shaping a new era of seamless interconnection between people, devices, and applications.

We're driven by a desire to transform and modernize what's antiquated, automate the manual, and solve real-world problems through innovative connectivity solutions. As a testament to our success, we're proud to stand as a financially stable and profitable company. Our robust profitability allows us not only to invest in pioneering technologies but also to foster an environment of continuous learning and growth for our team.

Our collective vision is a world where borderless connectivity fuels limitless innovation. By joining us, you can be part of laying the foundations for this interconnected future. We're currently seeking passionate individuals who are excited about the opportunity to contribute to an industry-shaping company while growing their own skills and careers.

Inference Infrastructure Architect
Senior / Staff · Remote — mainland China · Founding China team

Telnyx runs its own B300 fleet — our hardware, in our facilities, operated from the metal up, and expanding globally. This role exists to turn that fleet into useful throughput: Telnyx 's own AI-agent traffic on the voice and messaging network, and external customers' inference behind the token gateway, dedicated deployments and tuned models.
You have two mandates:

-
Operate and expand the fleet efficiently. Maximize useful inference throughput per GPU-dollar while meeting latency and reliability SLOs, and continuously reduce cost per token.

-
Make the product run on it. Serverless inference for the open-weight catalog, and dedicated, tuned model deployments for enterprises.

The stack is open source, bare metal to OpenAI-compatible endpoint. You work upstream in it.
What you'll build

-
Serverless serving pools. vLLM / SGLang engines with continuous batching, prefix caching, low-precision serving (FP8, FP4, INT4) and MoE expert parallelism, serving the head of the open-weight catalog at multi-tenant scale.

-
The fleet layer. llm-d / NVIDIA Dynamo on the Kubernetes Gateway API: KV-cache-aware routing, prefill/decode disaggregation, KV tiering (Mooncake), and multi-LoRA serving so hundreds of customer adapters share one base pool.

-
The platform under it. Kubernetes on bare metal: GPU Operator, topology-aware scheduling, LeaderWorkerSet, Kueue; Kata Containers for isolated dedicated tenants; bare-metal lifecycle with OpenStack Ironic.

-
Weight logistics and elasticity. P2P model distribution (Dragonfly, safetensors streaming), warm pools, and autoscaling driven by inference metrics — queue depth, KV occupancy, TTFT — never CPU.

-
The dedicated tier. Per-tenant pools, GPU-hour metering, latency SLOs, private networking; and the serving side of the model-tuning loop: adapter versioning, canary rollout, rollback on outcome regression.

-
Observability and economics. DCGM and engine metrics into Prometheus / OpenTelemetry; capacity planning grounded in roofline math — bandwidth-bound decode, batching curves, utilization versus cost per token.

The stack you'll work in

Some of this is committed direction: Kubernetes on bare metal, vLLM / SGLang, an OpenAI-compatible endpoint. Much of the rest is candidates you'll evaluate. You'll select, benchmark and integrate the components that earn their place in production. We value depth in the core serving stack and sound architectural judgment, not prior experience with every project listed.

-
Engines: vLLM · SGLang

-
Serving techniques: prefill/decode disaggregation · wide expert parallelism · speculative decoding (MTP, EAGLE-3) · low precision (FP8, FP4, INT4)

-
MoE & attention libraries: