Shopify Shop App Agent Annotator at Mercor

Shopify Shop App Agent Annotator | $40–50/hr | Worldwide Remote

Help improve Shopify's AI-powered Shop App by diagnosing why shoppers were dissatisfied with agent responses — working approximately 5 hours per week as a part-time contractor.

About the Role

Shopify's Shop App Agent is an AI assistant embedded within the Shop app, helping buyers with product discovery, order tracking, account questions, and Shop Cash. When a shopper taps thumbs-down on a response, they leave no explanation — just a signal. Your job is to read the full conversation transcript and determine why that dissatisfaction occurred.

What You'll Do

• Read complete shopper ↔ agent conversation transcripts, including tool calls and their outputs
• Identify the specific turn that triggered the thumbs-down rating
• Classify the issue using a defined taxonomy: one primary domain tag and one specific secondary tag
• Write a concise 2–3 sentence comment explaining what the shopper wanted, what the agent did or failed to do, and why your chosen tags apply
• Apply an unknown label honestly when no clear cause is identifiable

What Makes a Great Candidate

• Rule discipline: Ability to apply a fixed taxonomy consistently across hundreds of conversations
• Comfort with structured traces: Confident reading tool calls, inputs, and raw outputs to understand agent behavior
• Restraint under ambiguity: Willingness to label something unknown rather than fabricate a plausible cause
• Clear written English: Concise, precise explanations of your labeling decisions

Preferred Qualifications

• Familiarity with consumer e-commerce: orders, tracking, returns, refunds, and platform vs. merchant responsibilities
• Prior annotation, labeling, or model-evaluation experience against a defined rubric
• Experience using the Shop app or similar AI shopping assistants
• Background in customer support operations or escalation workflows

Important Note

This is not a customer support or engineering role. You are not fixing the agent or replying to shoppers — you are providing precise, repeatable diagnostic labels that help the team measure and improve where the agent falls short.