ML Engineer — Model Evaluation at Mercor

ML Engineer – Model Evaluation & Experimentation | $60–90/hr | Remote (US)

Join a leading AI lab's GenAI team and play a central role in shaping the next generation of agentic evaluation benchmarks for frontier models. As a ground-truth expert, you'll design and execute complex, multi-step ML tasks that expose exactly where state-of-the-art models fall short — working in close collaboration with top AI researchers.

This is a full-time W-2 position with Cincinnatus LLC, fully remote within the United States at approximately 35 hours per week.

Key Responsibilities

• Design Tasks: Translate real ML research ideas — such as modifying RL reward computations — into well-defined, multi-step evaluation tasks.
• Run Experiments: Implement changes, execute training experiments end-to-end, and rigorously analyze results to establish ground-truth solutions.
• Explore RL Concepts: Build tasks around reinforcement learning fundamentals including reward functions and training behavior.
• Evaluate Frontier Models: Assess how leading AI models handle your tasks and document where and why they fall short.
• Collaborate: Work closely with researchers and fellow ML experts to ensure tasks remain consistent, rigorous, and fair.

Core Qualifications

• MSc or PhD in machine learning, computer science, or a related STEM field — or equivalent research-heavy practical experience.
• 1+ years in a research or research-engineering role.
• Hands-on experience training and evaluating ML models end-to-end (setup, execution, and analysis).
• Strong familiarity with large language models: capabilities, limitations, and evaluation techniques.
• Proficiency in Python and Git across both scripting and notebook environments.
• Basic understanding of reinforcement learning (reward functions, policy training) preferred.
• Prior experience in AI training, model evaluation, or benchmark authoring is a plus.
• High attention to detail, strong written communication, and the ability to work independently through ambiguous problems.

About the Employer

Cincinnatus LLC is an enterprise staffing company that partners with leading technology firms to place highly skilled professionals in structured, role-based engagements. Cincinnatus serves as the employer of record, providing W-2 employment, payroll, benefits, and compliance while embedding employees directly within client teams.

Cincinnatus is an Equal Employment Opportunity employer and is committed to providing reasonable accommodations for qualified individuals with disabilities throughout the hiring process.