AI Model Evaluator (JavaScript) at Turing

Model Evaluator (JavaScript) | $25/task | Remote (Asia, Africa, LATAM)

Turing, an AI company, is hiring tech-lead-level JavaScript engineers to evaluate how AI coding agents handle real code. The role blends hands-on software engineering with AI evaluation work.

What you'll do

• Review three AI coding agents that were each given the same real developer request. Read each agent's full run — the commands it ran, the files it changed, what it checked and its final message — then judge which run did the job best.
• Rate each run on 10 quality areas, such as following instructions, whether the code works, efficiency, safety, honesty and communication.
• Give each run an overall score from 1 to 5, rank the three, and write a short rationale explaining your ranking with evidence from the runs.
• Analyze and triage GitHub issues across trending open-source libraries.
• Set up and configure code repositories, including Dockerization and environment setup.
• Evaluate unit test coverage and quality.
• Modify and run codebases locally to assess LLM performance in bug-fixing scenarios.
• Collaborate with researchers to identify repositories and issues that are challenging for LLMs, with opportunities to lead a team of junior engineers.

Requirements

• 3+ years of hands-on software development experience.
• Strong experience in JavaScript/TypeScript; Python, Java, Go, PHP or similar also accepted.
• Proficiency with Git, Docker, shell commands, build tools and test output.
• Ability to understand and navigate complex codebases, and to run, modify and test real-world projects locally.
• Experience contributing to or evaluating open-source projects is a plus.
• Nice to have: previous participation in LLM research or evaluation projects, or experience building or testing developer tools or automation agents.

Selection process

• Complete three practice evaluation tasks as part of selection (assessment-based; the evaluation process takes about 75 minutes), each requiring quality ratings, overall scores, comparative rankings and written rationales.
• A reviewer assesses each task for technical accuracy, consistency and quality of reasoning; you are onboarded only after all three practice tasks are approved.
• Strict no-AI policy: all rationales and reasoning must be written independently by you. Compliance is checked, and any use of AI-generated rationales or reasons results in automatic rejection.

Pay, schedule and location

• $25 per approved task. Contractor assignment — no medical or paid leave.
• At least 8 hours per day and 40 hours per week, with 4 hours of overlap with PST.
• 1-month contract; expected start date is next week.
• Fully remote. Open to applicants based in India, Pakistan, Nigeria, Kenya, Egypt, Ghana, Bangladesh, Turkey, Mexico and Latin America.