SWE-Bench Task Auditor at Mercor

SWE-Bench Task Auditor | $70–90/hr | Remote (US)

Join a frontier AI lab as a SWE-Bench Task Auditor, where you'll play a critical role in ensuring the quality, correctness, and reproducibility of software engineering benchmark tasks used to train and evaluate cutting-edge AI models. This is a high-impact contract role for experienced engineers with a strong open-source background.

What You'll Do

• Evaluate repository-level benchmark tasks for quality, correctness, and reproducibility
• Audit reference patches, test harnesses, and Docker isolation environments
• Detect issues such as answer leakage and reward hacking that could compromise benchmark integrity
• Provide clear, rubric-based written feedback on task grading and structure
• Assess test runners and grading pipelines to ensure rigorous evaluation standards

Basic Qualifications

• 3+ years of professional software engineering experience
• Demonstrated open-source contribution or maintainer experience (merged PRs, committer/maintainer roles)
• Strong ability to audit reference patches, test runners, and Docker-based isolation
• Proficiency in Python and at least one of: Java, Go, TypeScript, or C++

Preferred Qualifications

• Familiarity with SWE-Bench (Verified) or similar repository-level benchmarks
• Maintainer history on major Python OSS projects (e.g., Django, Flask, scikit-learn, sympy, pytest)
• Prior experience in code review or task-grading roles

This is a remote contract opportunity open to candidates based in the United States. Ideal for senior engineers passionate about AI evaluation quality and open-source software.