Senior Code Review Expert for AI Evaluation at Mercor

Code Review Expert | $40–125/hr | Worldwide Remote

Mercor is partnering with a leading AI research organization to evaluate and improve coding assistant AI systems. We're seeking experienced engineers and technical specialists to review and assess full transcripts of user–AI coding conversations. This flexible, remote engagement helps shape the future of developer-assisting AI.

Key Responsibilities

• Review comprehensive transcripts between users and AI coding assistants
• Analyze the AI's reasoning, execution logic, and stated actions in detail
• Score transcripts using a 10-point rubric across multiple evaluation criteria
• Write brief justifications citing specific examples from the dialogue
• Identify inconsistencies between AI claims and actual actions (e.g., stating "I'll run tests" without executing them)

Ideal Qualifications

Top candidates:

• Senior or Staff Engineers with extensive code review and execution analysis experience
• QA Engineers with strong verification and consistency-checking skills
• Technical Writers or Documentation Specialists adept at comparing instructions versus implementation

Also a strong fit:

• Backend or Full-Stack Developers comfortable with function calls, APIs, and testing workflows
• DevOps or SRE professionals experienced in tool orchestration and system behavior analysis

Technical Skills

• Proficiency in Python (most transcripts are Python-based)
• Familiarity with JavaScript, TypeScript, Java, C++, Go, Ruby, Rust, or Bash is a plus
• Experience with Git workflows, testing frameworks, and debugging tools

Work Structure

• Fully remote and asynchronous—complete tasks on your own schedule
• Each transcript batch must be completed within 5 hours of starting
• Unlimited tasks available with potential for recurring batches
• Flexible, task-based engagement

Compensation & Terms

• Competitive hourly rates ($40–125/hr) based on geography and experience
• Independent contractor classification
• Weekly payments via Stripe Connect

Application Process

Submit your resume to begin. Selected candidates will receive rubric documentation and platform access. Most applicants hear back within a few business days.

About Mercor

Mercor is a premier talent marketplace connecting top experts with leading AI labs and research organizations. Backed by Benchmark, General Catalyst, Adam D'Angelo, Larry Summers, and Jack Dorsey, we enable thousands of professionals to contribute to frontier AI projects.