AI Developer Trace Task Auditor at Mercor

AI Developer Trace Task Auditor | $70–90/hr | Remote (US)

Join a cutting-edge project evaluating AI-assisted software development traces used to train and benchmark frontier AI models. In this role, you'll review end-to-end coding sessions generated by AI developer tools, assessing their correctness, workflow quality, and reasoning — then deliver structured, rubric-based written feedback that directly shapes the future of AI development tools.

What You'll Do

• Evaluate the quality and correctness of AI-assisted coding traces used in model training and evaluation
• Assess multi-step coding sessions for technical soundness, best practices, and reasoning clarity
• Provide clear, detailed written feedback using established rubrics
• Judge full-stack and backend coding workflows produced by agentic and spec-driven AI tools

Basic Qualifications

• 3+ years of professional software development experience
• Hands-on experience with AI-assisted coding tools such as Cursor, GitHub Copilot, Claude Code, or similar
• Strong code-reading and debugging skills across full-stack or backend systems
• Ability to evaluate multi-step coding trajectories for correctness and adherence to best practices

Preferred Qualifications

• Experience with Kiro or Amazon CodeCatalyst
• Prior work evaluating or grading AI-generated code
• Contributions to developer tooling projects

This is a contractor engagement ideal for experienced developers who are passionate about AI tooling and want to contribute meaningfully to the development of next-generation AI systems.