Applied Engineer – AI Evaluation & Data Systems
Hourly Contract | Remote (U.S. & Canada) | $100–$150/hr | Long-term
Are you a full-stack engineering generalist ready for a high-impact, long-term project in AI? Mercor is partnering with Cognition to build the data collection flywheel powering Devin—the next-generation AI Software Engineer. In this role, you’ll help define how we test and evaluate advanced AI coding agents, shaping the future of developer productivity.
About the Role
This isn’t a traditional software engineering or research position. Instead, you’ll design and oversee robust evaluation systems that test how the AI agent writes code. Your work will involve:
Building and Managing Coding Evaluations: Create and supervise test suites for diverse coding tasks (e.g., migrating codebases from JavaScript to TypeScript).
Data Analysis: Analyze how Devin performs on real user cases and investigate customer usage data to spot improvement opportunities.
System Testing: Extensively test AI agent behavior across scenarios, surfacing strengths and weaknesses.
Communication: Present your findings and actionable insights directly to Cognition’s research team.
What We’re Looking For
Several years of professional full-stack engineering experience.
Detail-oriented, hard-working, and passionate about technology.
Actively seeking a long-term, impactful opportunity.
Based in the United States or Canada.
Nice to Have
Experience leading teams or managing technical projects.
Familiarity with automated testing, developer tools, or AI systems.
Role Highlights
Work directly with Cognition’s world-class research team.
Fully remote and flexible hours (minimum 15 hours/week, part-time start).
Long-term project (12 months+), with the opportunity for a full-time Applied Engineer position at Cognition for top performers.
Collaborate as both an individual contributor and a team lead.
Weekly payments via Stripe Connect.
About Mercor
Mercor connects leading technical experts with top AI research labs. Our investors include Benchmark, General Catalyst, Peter Thiel, Adam D’Angelo, Larry Summers, and Jack Dorsey.
Equal Opportunity
We welcome all qualified applicants regardless of legally protected characteristics and provide reasonable accommodations as needed.
Ready to help shape the future of AI-driven software engineering? Apply today and make your impact with Mercor and Cognition!
