Python Model Evaluator (AI Coding) at Turing

Python Model Evaluator (AI Coding) | $25/task | Remote contractor

Turing, an AI company, is hiring experienced software engineers for an AI training and evaluation initiative focused on improving the coding capabilities of AI agents. You will review and compare AI-generated coding solutions across real-world software engineering tasks, analyzing complete agent execution workflows and providing structured ratings and evidence-based feedback.

What the day-to-day looks like

• Review three AI coding agents that were each given the same real developer request.
• Read each agent's full run: the commands it ran, the files it changed, what it checked, and its final message.
• Judge which run did the job best.
• For every task, rate each run on 10 quality areas, such as following instructions, whether the code works, efficiency, safety, honesty and communication.
• Give each run an overall score from 1 to 5 and rank the three.
• Write a short rationale explaining your ranking, with evidence from the runs.

Requirements

• Minimum 3+ years of overall software engineering experience.
• Strong experience with Python.
• Proficiency with shell commands, Git, Docker, and basic software pipeline setup.
• Comfortable running, modifying, and testing real-world projects locally.

Pay and schedule

• $25 per approved task.
• At least 8 hours per day and 40 hours per week, with 4 hours of overlap with PST.
• Contractor assignment for 1 month; expected start date is next week.
• No medical or paid leave.

Location

Fully remote. Eligible locations: India, Pakistan, Nigeria, Kenya, Egypt, Ghana, Bangladesh, Turkey, Mexico, LA.

Assessment and selection

• Complete three practice evaluation tasks as part of the selection process; each requires quality ratings, overall scores, comparative rankings, and written rationales.
• A reviewer assesses each task for technical accuracy, consistency, and quality of reasoning.
• Onboarding happens only after all three practice tasks are approved.
• The evaluation process takes approximately 75 minutes and is assessment-based.

Strict no-AI policy: Do not use ChatGPT or any other AI tool to write your evaluation rationales, explanations, or reasons. All reasoning must be written independently by the candidate. Compliance will be checked, and any use of AI-generated rationales or reasons results in automatic rejection.

The role offers a fully remote environment and the opportunity to work on AI projects with leading LLM companies.