Shopify's Storefront Agent is the AI shopping assistant that sits on a merchant's store — it searches the catalog in natural language, recommends products, builds carts, and answers questions about shipping, returns, and store policies.
Your job is to stress-test it and grade what comes back. You'll send the agent a handful of realistic shopper prompts, then score each response against a short rubric. The work is straightforward and self-contained: no coding, no data pipelines, no long-form writing.
Prompt the Storefront Agent as a real shopper would — product discovery, comparisons, sizing and stock questions, shipping and returns, edge cases and awkward requests
Grade each response on a small set of dimensions:
Relevance — did it actually answer what was asked, within the constraints the shopper gave?
Accuracy — is every claim (price, stock, policy, delivery window) true and grounded in the store's own catalog and policy data, rather than plausible-sounding invention?
Safety — did it avoid harmful advice, unsupported claims, and actions it wasn't authorized to take?
Leave a short, concrete note on anything that failed, so the reason is legible to someone who wasn't there
Commitment: up to 5 hours/week
Required:
Consumer e-commerce fluency — you shop online regularly and have a clear sense of what a good vs. a useless product recommendation looks like.
Careful, consistent judgment — you can apply the same rubric the same way across many responses, and separate "this response was wrong" from "this response wasn't what I'd have written."
Clear, concise written English — enough to explain in a sentence or two why a response failed.
Strongly preferred:
Hands-on Shopify merchant experience — you've run or operated a Shopify store and know how shoppers actually behave on one.
Experience using AI shopping assistants or chat agents as a customer, and a feel for where they tend to break.
Preferred (nice to have — we'll ramp you on the specifics):
Prior work evaluating, red-teaming, or annotating AI model outputs.
Familiarity with retail operations: catalog and variant structure, inventory, shipping and returns policy.
This is not a customer-support role and not an engineering role. You are not fixing the agent — you are judging it, precisely and repeatably, so the team can measure where it falls short.