Shopify Expert - Fully Remote | Upto $50/hr

About the job

Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .

Position: Shopify Storefront Agent Evaluator
Type: Contract
Compensation: $40–$50/hour
Location: Remote
Commitment: up to 5 hours/week
Role Responsibilities

- Evaluate the Storefront Agent by simulating real shopper interactions, including product discovery, comparisons, and inquiries about sizing, stock, shipping, and returns.

- Grade responses based on Relevance , Accuracy , and Safety to ensure they meet shopper expectations and adhere to store policies.

- Provide concise feedback on any failures to ensure clarity for team members not present during evaluation.

- Work independently to deliver precise and repeatable judgments on agent performance.

Qualifications

Must-Have

- Consumer e-commerce fluency — you shop online regularly and understand effective product recommendations.

- Careful, consistent judgment — apply the rubric uniformly across responses.

- Clear, concise written English — articulate reasons for response failures in a sentence or two.

Preferred

- Hands-on Shopify merchant experience — familiarity with shopper behavior on Shopify stores.

- Experience using AI shopping assistants or chat agents — understand common failure points.

- Prior work evaluating, red-teaming, or annotating AI model outputs.

- Familiarity with retail operations: catalog and variant structure, inventory, shipping, and returns policy.

Application Process (Takes 20–30 mins to complete)

- Upload resume

- AI interview based on your resume

- Submit form

Resources & Support

- For details about the interview process and platform information, please check:

- For any help or support, reach out to:

PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.

Originally posted on Himalayas