AI Workflow Evaluator - Fully Remote | Upto $190/hr

About the job

Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .

Position: MCP & Plug In Connectors Expert
Type: Contract
Compensation: $50–$190/hour
Location: Remote
Commitment: 20+ hours/week
Role Responsibilities

- Evaluate AI systems on complex personal workflows, including personal health , travel , and activity planning .

- Design realistic prompts for complex personal-life tasks using Google Drive , Expedia , Notion , and other plugins/connectors.

- Execute tasks while recording your screen and using personal plugins/connectors.

- Write clear explanations of AI successes and failures to improve AI model performance .

- Judge AI outputs for practicality, personalization, and reasoning.

- Identify where models miss context, overreach, or fail to use tools correctly.

Qualifications

Must-Have

- US-based only.

- Strong MCP experience and plugin/connector usage.

- Experience using LLM plugins/connectors like Google Drive , Expedia , and Notion .

- Heavy personal usage of LLM products.

- Active LLM account with 6+ months of history.

- Willingness to sign a data-share consent form via DocuSign .

- Experience using AI for multi-step planning, research, and decision-making.

- Strong written judgment and attention to detail.

- Ability to evaluate AI outputs and write against rubrics.

Preferred
- Extensive rubric experience with 100+ hours on prior projects.

Compensation & Legal

- Hourly contractor .

- Experts added to the project will begin in a trial period.

Application Process (Takes 20–30 mins to complete)

- Upload resume

- AI interview based on your resume

- Submit form

Resources & Support

- For details about the interview process and platform information, please check:

- For any help or support, reach out to:

PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.

Originally posted on Himalayas