Model Evaluator – AI/ML at Turing

Model Evaluator – AI/ML | $25/task | Remote (Asia, Africa, Latin America)

About the role

Turing is seeking experienced software engineers to support a frontier AI training and evaluation initiative focused on improving the coding capabilities of AI agents. You will review and compare AI-generated coding solutions across real-world software engineering tasks, analyze complete agent execution workflows, and evaluate code correctness, efficiency, safety, and overall quality. Your structured ratings and evidence-based feedback will help improve AI-assisted software development. The work is fully remote, on AI projects with leading LLM companies.

What the day-to-day looks like

• Review three AI coding agents that were each given the same real developer request.
• Read each agent's full run: the commands it ran, the files it changed, what it checked, and its final message.
• Judge which run did the job best.
• For every task: rate each run on 10 quality areas, such as following instructions, whether the code works, efficiency, safety, honesty and communication; give each run an overall score from 1 to 5 and rank the three; and write a short rationale explaining your ranking, with evidence from the runs.

Requirements

• Minimum 3+ years of overall software engineering experience.
• Strong experience with at least one of: Python, JavaScript/TypeScript, Java, Go, PHP, or similar.
• Proficiency with Shell commands, Git, Docker, and basic software pipeline setup.
• Comfortable running, modifying, and testing real-world projects locally.

Pay, schedule and contract

• Payment model: $25 per approved task.
• Commitment: at least 8 hours per day and 40 hours per week, with 4 hours of overlap with PST.
• Employment type: contractor assignment (no medical/paid leave).
• Duration: 1-month contract; the expected start date is next week.

Where applicants can be

India, Pakistan, Nigeria, Kenya, Egypt, Ghana, Bangladesh, Turkey, Mexico, and Latin America.

Selection process

The evaluation process is assessment-based and takes approximately 75 minutes.

• Candidates complete three practice evaluation tasks: quality ratings, overall scores, comparative rankings, and written rationales.
• A reviewer assesses each task for technical accuracy, consistency, and quality of reasoning.
• Onboarding happens only after all three practice tasks have been approved.

Strict no-AI policy: do not use ChatGPT or any other AI tool to write your evaluation rationales, explanations, or reasons. All rationales and reasoning must be written independently by you. Compliance will be checked, and any use of AI-generated rationales or reasons will result in automatic rejection.