E-commerce Annotation Specialist

About the job

Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .

Position: Shopify Shop App Agent Annotator
Type: Contract
Compensation: $40–$50/hour
Location: Remote
Commitment: ~5 hours/week
Role Responsibilities

- Evaluate full shopper-agent conversations, including tool calls and their outcomes.

- Identify the specific turn that received a thumbs-down from the shopper.

- Classify the issue with a primary domain tag and a specific secondary tag.

- Write concise comments explaining the shopper's intent, the agent's actions, and the rationale for the tags.

- Work independently and asynchronously to maintain consistency in tagging across conversations.

Qualifications

Must-Have

- Rule discipline to apply a fixed taxonomy consistently.

- Comfort reading structured traces to understand agent actions.

- Restraint under ambiguity to use the unknown label appropriately.

- Clear, concise written English for explaining labeling decisions.

Preferred

- Consumer e-commerce fluency in orders, tracking, and platform responsibilities.

- Prior annotation, labelling, or model-evaluation work with a defined rubric.

- Experience with the Shop app or other AI shopping assistants.

- Familiarity with customer-support operations and escalation paths.

Application Process (Takes 20–30 mins to complete)

- Upload resume

- AI interview based on your resume

- Submit form

Resources & Support

- For details about the interview process and platform information, please check:

- For any help or support, reach out to:

PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.

Originally posted on Himalayas