AI Model Evaluator (Java) at Turing

Java Model Evaluator | $25/task | Remote (Asia, Africa, LATAM)

About the role

Turing, an AI company, is hiring experienced software engineers for a 1-month contractor assignment supporting a frontier AI training and evaluation initiative focused on improving the coding capabilities of AI agents. You will review and compare AI-generated coding solutions across real-world software engineering tasks, analyze complete agent execution workflows, and provide structured ratings and evidence-based feedback.

Day-to-day work

• Review three AI coding agents that were each given the same real developer request.
• Read each agent's full run: the commands it ran, the files it changed, what it checked and its final message, then judge which run did the job best.
• Rate each run on 10 quality areas, such as following instructions, whether the code works, efficiency, safety, honesty and communication.
• Give each run an overall score from 1 to 5 and rank the three runs.
• Write a short rationale explaining your ranking, with evidence from the runs.

Requirements

• Minimum 3+ years of overall software engineering experience.
• Strong experience with Java.
• Proficiency with Shell commands, Git, Docker and basic software pipeline setup.
• Comfortable running, modifying and testing real-world projects locally.

Assessment and selection

• Selection is assessment-based; the evaluation process takes approximately 75 minutes.
• Candidates must complete three practice evaluation tasks, each requiring quality ratings, overall scores, comparative rankings and written rationales.
• A reviewer assesses each task for technical accuracy, consistency and quality of reasoning.
• Candidates are onboarded only after all three practice tasks have been approved.

Strict no-AI policy: Do not use ChatGPT or any other AI tool to write your evaluation rationales, explanations or reasons. All rationales and reasoning must be written independently by the candidate. Compliance will be checked, and any use of AI-generated rationales or reasons results in automatic rejection.

Pay, schedule and contract

• Payment: $25 per approved task.
• At least 8 hours per day and 40 hours per week, with 4 hours of overlap with PST.
• Contractor assignment (no medical or paid leave).
• Duration: 1 month; expected start date is next week.

Locations

Fully remote. Open to applicants in India, Pakistan, Nigeria, Kenya, Egypt, Ghana, Bangladesh, Turkey, Mexico and Latin America.