AI Benchmark Task Author at Turing

AI Benchmark Task Author | Contractor | Remote | Rate varies | Up to 40 hrs/week with 4 hrs PST overlap

Turing, based in San Francisco, works with frontier AI labs and enterprises on AI research, data, and training. It is hiring an analytical researcher and red-teaming specialist to build evidence-based web-research benchmark tasks that expose weaknesses in state-of-the-art AI models (e.g., Claude Opus 5 Max). You will discover hard-to-surface facts, build precise research tasks around them, test them directly against leading LLMs, and diagnose exactly how the model's search and reasoning paths fail.

What the Work Involves

• Research niche topics to find factual ground truths that cannot be surfaced with simple, direct search queries.
• Verify facts against primary or highly credible sources, cross-check for contradictory evidence, and document exact source URLs.
• Draft clear, answerable benchmark tasks designed to expose structural weaknesses in model research strategies.
• Run tasks against designated SOTA models to confirm they reliably trigger the intended model failures.
• Trace model run logs, search trajectories, and intermediate outputs to pinpoint where and why the model failed, distinguishing genuine model limitations from ambiguous prompts.
• Apply validated "model-breaking" research patterns across new domains; strong performers also review tasks authored by peer trainers.

Requirements

• 1+ years in investigative research, fact-checking, AI benchmark task authoring, red-teaming, or technical QA review.
• Experience authoring, testing, or reviewing tasks for AI benchmarks such as GAIA, BrowseComp, SimpleQA, SWE-bench, Terminal-Bench, or Humanity's Last Exam (HLE).
• Intermediate-to-advanced web search and OSINT skills; ability to navigate academic archives, local records, or obscure databases.
• Ability to inspect model reasoning chains, search logs, and source selection to diagnose root causes of failure.
• Precision writing: clear, unambiguous benchmark prompts with bulletproof, primary-sourced answer keys.
• Bachelor's degree or higher in a relevant engineering domain (described as a must in the listing).

Preferred Qualifications

• Experience designing prompt-injection or reasoning-trap tasks that break SOTA models.
• Advanced OSINT capabilities or published investigative/academic research demonstrating deep source verification.
• Niche domain expertise in areas rich with hard-to-find records (e.g., local government filings, regional sports history, legal archives, historical patents).
• Experience as a lead reviewer, benchmark auditor, or quality owner on AI evaluation projects.

Schedule and Engagement

• Up to 40 hours per week, with 4 hours of overlap with PST.
• Contractor engagement, up to 4 weeks long.
• Hourly rate is not stated in the listing.

Hiring Process

Candidates submit a job interest form and an assessment. Those who do well on the assessment go through a review and proceed accordingly.