AI Operations Specialist - LLM & Conversation Systems

AI Operations Specialist – LLM & Conversation Systems

Position Type: Full-Time, Remote
Working Hours: U.S. Business Hours
About the Role

We are looking for a hands-on AI Operations Specialist to own and improve our production AI conversation systems.

We are an AI-powered SMS lead recovery platform serving telehealth brands, processing millions of messages each month. Our AI systems interact directly with customers, and their performance directly impacts client conversions.

In this role, you'll onboard new clients, configure AI conversation systems, monitor live conversations, diagnose issues, improve prompts and system behavior, and measure whether your changes actually improve conversion.

This is not a theoretical AI role. We're looking for someone who has run LLM systems in production and understands how to continuously monitor, troubleshoot, test, and improve them.
Responsibilities
Client Onboarding & AI Setup

- Onboard new clients into the AI conversation platform.

- Set up webhook integrations and client knowledge bases.

- Configure campaigns and conversation workflows.

- Ensure new client systems are properly tested before going live.

- Troubleshoot integration and configuration issues during onboarding.

Production AI Operations

- Monitor live AI-generated conversations and customer interactions.

- Review conversations to identify errors, inconsistencies, and missed opportunities.

- Diagnose AI failure modes and determine their root causes.

- Identify issues affecting conversation quality and conversion.

- Escalate technical issues when necessary and follow through until resolved.

Prompt & System Optimization

- Write, edit, and improve prompts that generate customer-facing messages.

- Adjust system instructions and conversation logic based on real-world performance.

- Test changes systematically before and after deployment.

- Maintain documentation of prompt and system changes.

- Continuously improve AI behavior while maintaining accuracy and consistency.

Performance & Conversion Analysis

- Analyze conversation transcripts and performance data.

- Measure whether system and prompt changes improve conversion.

- Identify patterns across conversations that can inform future improvements.

- Track key performance outcomes and communicate findings to leadership.

- Use data and real customer interactions to guide optimization decisions.

Founder Collaboration

- Work directly with the founder on daily AI operations and improvements.

- Communicate issues, findings, and recommendations clearly.

- Take ownership of problems from identification through resolution.

- Move quickly while maintaining high standards for customer-facing AI output.

Required Experience & Skills
Must-Have

- Proven experience running LLM or AI systems in production .

- Hands-on experience monitoring and improving production AI outputs.

- Experience with prompt engineering and system prompt optimization.

- Strong understanding of LLM behavior, failure modes, and iterative testing.

- Experience analyzing AI-generated conversations or customer-facing outputs.

- Ability to work with webhooks, APIs, knowledge bases, or similar integrations.

- Strong analytical and problem-solving skills.

- Excellent written English — required for editing customer-facing AI messages and documenting system changes.

- Strong attention to detail.

- Ability to work independently and take ownership of production systems.

What Makes You a Strong Fit

- You've actually operated AI systems in production rather than only experimenting with ChatGPT.

- You are comfortable reviewing hundreds or thousands of AI interactions to identify patterns and problems.

- You think in terms of outcomes and conversion , not just whether the AI response sounds good.

- You are methodical about testing changes and measuring their impact.

- You can quickly identify why an AI system is behaving incorrectly and determine how to improve it.

- You communic