Posted over 30 days ago. Roles like this often stay open for a while, so it's still worth applying.
AI Developer Trace Task Auditor at Mercor
posted 1 month agoAI Developer Trace Task Auditor | $70–90/hr | Remote (US)
Join a cutting-edge project evaluating AI-assisted software development traces used to train and benchmark frontier AI models. In this role, you'll review end-to-end coding sessions generated by AI developer tools, assessing their correctness, workflow quality, and reasoning — then deliver structured, rubric-based written feedback that directly shapes the future of AI development tools.
What You'll Do
- Evaluate the quality and correctness of AI-assisted coding traces used in model training and evaluation
- Assess multi-step coding sessions for technical soundness, best practices, and reasoning clarity
- Provide clear, detailed written feedback using established rubrics
- Judge full-stack and backend coding workflows produced by agentic and spec-driven AI tools
Basic Qualifications
- 3+ years of professional software development experience
- Hands-on experience with AI-assisted coding tools such as Cursor, GitHub Copilot, Claude Code, or similar
- Strong code-reading and debugging skills across full-stack or backend systems
- Ability to evaluate multi-step coding trajectories for correctness and adherence to best practices
Preferred Qualifications
- Experience with Kiro or Amazon CodeCatalyst
- Prior work evaluating or grading AI-generated code
- Contributions to developer tooling projects
This is a contractor engagement ideal for experienced developers who are passionate about AI tooling and want to contribute meaningfully to the development of next-generation AI systems.
How to apply for this role
- Upload your resume — keep it up-to-date and in English. Mercor will auto-fill your profile from it.
- Complete the AI interview — a 15-minute conversation about your experience. Be ready to discuss specific projects and challenges you've solved.
- Submit your application — only about 20% of applicants finish all the steps, so completing yours puts you well ahead.