Software Engineering & Systems Design Expert at Mercor

Software Engineering & Systems Design Expert | $45–80/hr | Worldwide Remote

Mercor is seeking experienced software engineers to evaluate and improve AI systems used by developers globally. This flexible, remote role allows you to apply your technical expertise to shape how conversational AI reasons about code, generates solutions, and explains complex technical concepts.

Why This Role Exists

Mercor partners with leading AI teams to enhance the quality and reliability of conversational AI systems. In software engineering contexts, these systems must demonstrate correct reasoning, strong problem-solving ability, and adherence to real-world best practices. You'll play a critical role in evaluating and improving how AI models handle coding tasks across various complexity levels.

What You'll Do

• Evaluate LLM-generated responses to coding queries for accuracy, reasoning, clarity, and completeness
• Conduct fact-checking using trusted public sources and authoritative references
• Execute code and validate outputs using appropriate testing tools
• Annotate model responses by identifying strengths, improvement areas, and inaccuracies
• Assess code quality, readability, algorithmic soundness, and explanation quality
• Ensure model responses align with expected conversational behavior and system guidelines
• Apply consistent evaluation standards following clear taxonomies and detailed guidelines

Who You Are

• Hold a BS, MS, or PhD in Computer Science or closely related field
• Have significant real-world software engineering experience
• Expert in at least one programming language (Python, Java, C++, JavaScript, Go, Rust)
• Able to solve HackerRank or LeetCode Medium and Hard-level problems independently
• Have contributed to well-known open-source projects with merged pull requests
• Possess significant experience using LLMs while coding and understand their strengths and limitations
• Demonstrate strong attention to detail and comfort evaluating complex technical reasoning
• Fluent in English

Nice-to-Have Specialties

• Prior experience with RLHF, model evaluation, or data annotation
• Track record in competitive programming
• Experience reviewing code in production environments
• Familiarity with multiple programming paradigms or ecosystems
• Experience explaining complex technical concepts to non-expert audiences

What Success Looks Like

• Identify incorrect logic, inefficiencies, edge cases, or misleading explanations in AI-generated code
• Deliver feedback that improves correctness, robustness, and clarity of AI coding outputs
• Produce reproducible evaluation artifacts that strengthen model performance
• Help build AI systems that developers can trust for real-world coding assistance

Why Join Mercor

This flexible, remote role offers experienced software engineers the opportunity to directly impact how AI systems reason about and generate code. Your technical expertise will contribute to high-impact AI development work, improving systems used by developers worldwide.