Model Evaluator (DevOps) at Turing

Model Evaluator – DevOps | $25/task | Remote (Asia/Africa/LATAM) | 1-month contractor role

Turing is seeking experienced software engineers to evaluate AI coding agents for a frontier AI training and evaluation initiative. You will review and compare AI-generated coding solutions on real-world software engineering tasks and provide structured ratings and evidence-based feedback to help improve AI-assisted software development.

What you'll do

• Review and compare three AI coding agents working on the same real-world developer request.
• Analyze each agent's complete execution, including shell commands, files modified, code changes, test results and final responses.
• Evaluate each agent across 10 quality dimensions, including instruction following, code correctness, efficiency, safety, honesty and communication.
• Assign each agent an overall quality score from 1 to 5 and rank all three solutions.
• Identify coding errors, incomplete implementations, inefficient approaches, security concerns and incorrect assumptions.
• Write concise, structured, evidence-based rationales explaining your scores and rankings.

Requirements

• 3+ years of hands-on software development experience in at least one major language such as Python, JavaScript/TypeScript, Java, Go or PHP.
• Strong understanding of code review, debugging, testing and development best practices.
• Practical experience with shell commands, Git, Docker, build tools and automated test outputs.
• Ability to independently review complex code changes and judge whether they satisfy the developer's requirements.
• Experience interpreting terminal outputs, debugging logs, test failures and version-control changes.
• Strong analytical skills and attention to detail, plus clear written communication for objective technical evaluations.

Nice to have

• Experience with code reviews, software QA, technical assessments or automated testing.
• Familiarity with AI coding agents and tools such as Claude Code, Cursor or GitHub Copilot.
• Experience evaluating LLM-generated code, AI model outputs, or rubric-based structured assessments.

Selection process

Candidates complete three practice evaluation tasks, each requiring quality ratings, overall scores, comparative rankings and written rationales. A reviewer assesses each task for technical accuracy, consistency and quality of reasoning; onboarding happens only after all three tasks are approved.

Strict no-AI policy: do not use ChatGPT or any other AI tool to write your evaluation rationales, explanations or reasons. All reasoning must be written independently by you. Compliance is checked, and any use of AI-generated rationales results in automatic rejection.

Pay, schedule and eligibility

• Payment: $25 per approved task.
• Commitment: at least 8 hours per day and 40 hours per week, with 4 hours of overlap with PST.
• 1-month contractor assignment (no medical or paid leave); expected start next week.
• Fully remote. Open to applicants based in India, Pakistan, Nigeria, Kenya, Egypt, Ghana, Bangladesh, Turkey, Mexico and Latin America.