AI Benchmark Engineer — SWE at Turing

AI Benchmark Engineer (SWE) | Contractor | Remote – Select Countries

Turing is seeking experienced SwarmBench Task Engineers specializing in software engineering to design and build high-quality multi-agent benchmark tasks grounded in real-world coding workflows. This is a short-term contractor engagement with one of the world's fastest-growing AI companies.

About Turing

Turing accelerates the advancement of powerful AI systems by partnering with leading AI labs to push frontier model capabilities in reasoning, coding, agentic behavior, and more. The work you contribute here directly shapes how the next generation of AI agents perform in real engineering environments.

Role Overview

In this role, you will create benchmark tasks based on real open-source code changes — including bug fixes, migrations, and refactors — used to evaluate how effectively AI agents navigate large codebases, apply precise modifications, and produce correct, testable outputs. You'll work within the Harbor evaluation framework and design verification logic across multi-agent decomposition strategies.

Day-to-Day Responsibilities

• Build multi-agent benchmark tasks from real-world open-source code changes (bug fixes, migrations, refactors)
• Run and validate tasks inside Docker environments using the Harbor evaluation framework
• Write clear, precise task instructions covering file paths, function signatures, expected behavior, and constraints
• Design and implement Python-based verification scripts to validate agent-generated code changes
• Create decomposition strategies that distribute complex code changes across independent sub-agents
• Debug and refine tasks within containerized environments to ensure reproducibility and determinism
• Analyze task performance signals and iteratively improve task quality, clarity, and difficulty

Requirements

• 5+ years of experience in Python and JavaScript development
• Familiarity with AI coding benchmarks such as SWE-bench or Terminal-Bench
• Strong ability to read and navigate large open-source codebases (e.g., Django, Flask, FastAPI, Node.js)
• Solid understanding of Git workflows: pull requests, diffs, cherry-picking, and commit-level navigation
• Comfortable writing Dockerfiles, building images, and debugging containerized environments
• Experience writing test scripts using pytest, unittest, or custom assertion-based frameworks
• Ability to produce clear, unambiguous technical specifications

Engagement Details

• Commitment: 8 hours/day with a 4-hour overlap with PST
• Type: Contractor (no medical/paid leave benefits)
• Duration: 4 weeks, starting next week
• Eligible Locations: Bangladesh, Brazil, Colombia, Egypt, Ghana, India, Indonesia, Kenya, Nigeria, Turkey, Vietnam

Why Work With Turing?

• Contribute to cutting-edge LLM evaluation at the frontier of AI research
• Collaborate with global teams on high-impact, technically challenging work
• Flexible, fully remote engagement