This job post has expired on September 17, 2026. It is likely that the position has already been filled.
Shopify Shop App Agent Annotator at Mercor
posted 1 month agoShopify Shop App Agent Annotator | $40–50/hr | Worldwide Remote
Help improve Shopify's AI-powered Shop App by diagnosing why shoppers were dissatisfied with agent responses — working approximately 5 hours per week as a part-time contractor.
About the Role
Shopify's Shop App Agent is an AI assistant embedded within the Shop app, helping buyers with product discovery, order tracking, account questions, and Shop Cash. When a shopper taps thumbs-down on a response, they leave no explanation — just a signal. Your job is to read the full conversation transcript and determine why that dissatisfaction occurred.
What You'll Do
- Read complete shopper ↔ agent conversation transcripts, including tool calls and their outputs
- Identify the specific turn that triggered the thumbs-down rating
- Classify the issue using a defined taxonomy: one primary domain tag and one specific secondary tag
- Write a concise 2–3 sentence comment explaining what the shopper wanted, what the agent did or failed to do, and why your chosen tags apply
- Apply an unknown label honestly when no clear cause is identifiable
What Makes a Great Candidate
- Rule discipline: Ability to apply a fixed taxonomy consistently across hundreds of conversations
- Comfort with structured traces: Confident reading tool calls, inputs, and raw outputs to understand agent behavior
- Restraint under ambiguity: Willingness to label something unknown rather than fabricate a plausible cause
- Clear written English: Concise, precise explanations of your labeling decisions
Preferred Qualifications
- Familiarity with consumer e-commerce: orders, tracking, returns, refunds, and platform vs. merchant responsibilities
- Prior annotation, labeling, or model-evaluation experience against a defined rubric
- Experience using the Shop app or similar AI shopping assistants
- Background in customer support operations or escalation workflows
Important Note
This is not a customer support or engineering role. You are not fixing the agent or replying to shoppers — you are providing precise, repeatable diagnostic labels that help the team measure and improve where the agent falls short.
How to apply for this role
- Upload your resume — keep it up-to-date and in English. Mercor will auto-fill your profile from it.
- Complete the AI interview — a 15-minute conversation about your experience. Be ready to discuss specific projects and challenges you've solved.
- Submit your application — only about 20% of applicants finish all the steps, so completing yours puts you well ahead.