Technical Lead
You'll be the founding technical lead for Veris EvalOps, building the platform that answers the two questions every AI-deploying business needs answered: is this system safe to launch, and is it still working correctly a month later. You'll take it from first line of code to first paying clients in 6–7 months.
Role Summary
This is a zero-to-one build, not a maintenance role. You'll architect and ship two commercial modules — a pre-production Release Gate that turns “looks good” into a reproducible readiness score, and a Knowledge Health Monitor that continuously audits the knowledge base an AI draws from — while hiring and leading the engineers who build them alongside you. There's no principal architect above you to escalate to: you make the calls and live with them, with the first pilot client live by month 3.
Key Responsibilities
• Own the evaluation engine. LLM-as-judge scoring, rule-based checks, groundedness verification, hallucination detection, and regression comparison — every readiness score comes from here.
• Build the tracing and observability layer. Distributed tracing across LLM calls, RAG retrievals, and agent workflows, built on OpenTelemetry, capturing every token, tool call, cost, and latency metric.
• Ship the knowledge health pipeline. Ingestion and continuous analysis of enterprise knowledge sources — stale-content detection, contradiction analysis, and coverage-gap mapping.
• Own platform core and integrations. Multi-tenant architecture, RBAC, API connectors, dashboards, and the CI/CD hooks that let the Release Gate plug into client engineering workflows.
• Build and lead the team. Hire and run 5–7 engineers across three streams — Platform Core, Release Gate, Knowledge Health — and own every architecture decision end to end.
Must-have
• 5+ years in engineering. Including 2+ years leading a team of 3–8 through a complete build cycle — architecture to shipping to paying users. Not a first-time lead role.
• LLM evaluation methodology. LLM-as-judge design, RAGAS/DeepEval-style metrics, golden dataset construction, regression testing for AI systems, and hallucination detection — not just the library calls, the mechanics behind them.
• LLM observability and tracing. OpenTelemetry-based tracing across LLM calls, RAG retrievals, and multi-step agent trajectories; cost/latency attribution; drift and anomaly detection.
• RAG system architecture. Production experience across the full pipeline — chunking, embeddings, a vector store (Pinecone, Weaviate, Qdrant, or pgvector), retrieval, re-ranking.
• AI agent systems. Production experience with agent patterns (ReAct, Plan-and-Execute, supervisor/sub-agent), tool-call evaluation, and guardrails.
• Backend platform engineering. Production-grade async Python (FastAPI, Celery), multi-tenant SaaS architecture, PostgreSQL/Redis, and CI/CD integration.
• Build-vs-integrate judgment and client-facing comfort. Can weigh integrating Langfuse/Braintrust vs. building from scratch, and work directly with pilot clients during onboarding and results review.
Nice-to-have
• LLM APIs and model ecosystem. Multi-provider experience (OpenAI, Anthropic, Azure OpenAI, Bedrock) and routing/prompt-management at scale.
• MLOps and experiment tracking. Background with MLflow, Weights & Biases, or equivalent experiment-tracking tooling.
• Security, compliance, and AI governance. EU AI Act and NIST AI RMF awareness, PII handling in AI pipelines, and red-teaming basics — increasingly a qualification question in enterprise security reviews.
SmartDev is an AI-powered software development company headquartered in Vietnam, part of the Verysell Group. We help global businesses deliver faster and build smarter — combining AI-driven development practices with deep expertise across fintech, healthcare, retail, and enterprise technology . Our team of engineers, architects, and AI specialists works across the full stack: from custom software and cloud solutions to generative AI, MLOps,