Ruby Software Engineering Expert at Mercor

Ruby Software Engineering Expert | $60–100/hr | Worldwide Remote

Mercor is seeking experienced Ruby engineers to evaluate and improve AI-generated code responses for leading AI development teams. This contract role is ideal for senior engineers who want to apply their expertise to high-impact AI training and evaluation work — fully remote, flexible hours.

About the Role

Mercor partners with top AI organizations to enhance the quality, accuracy, and reliability of conversational AI systems. In this role, you'll assess how AI models reason about code, generate solutions, and explain technical concepts — helping shape the next generation of AI coding assistants used by developers worldwide.

What You'll Do

• Evaluate LLM-generated responses to coding and software engineering queries for accuracy, reasoning, clarity, and completeness
• Conduct fact-checking using trusted public sources and authoritative references
• Execute code and validate outputs using appropriate tools to verify correctness
• Annotate model responses by identifying strengths, weaknesses, and factual or conceptual inaccuracies
• Assess code quality, readability, algorithmic soundness, and explanation quality
• Ensure model responses align with expected conversational behavior and system guidelines
• Apply consistent evaluation standards using clear taxonomies, benchmarks, and detailed evaluation guidelines

Who You Are

• Holds a BS, MS, or PhD in Computer Science or a closely related field
• 5+ years of real-world software engineering experience in technical roles
• Expert-level proficiency in Ruby
• Able to independently solve HackerRank or LeetCode Medium and Hard-level problems
• Has contributed to well-known open-source projects with merged pull requests
• Experienced using LLMs in coding workflows with a strong understanding of their strengths and failure modes
• Strong attention to detail with the ability to identify subtle bugs and logical flaws in complex technical reasoning
• Fluent in English

Nice-to-Have

• Prior experience with RLHF, model evaluation, or data annotation
• Competitive programming background
• Experience reviewing code in production environments
• Familiarity with multiple programming paradigms or ecosystems
• Ability to explain complex technical concepts to non-expert audiences

Why Join Mercor

This role offers experienced engineers a unique opportunity to directly influence how AI systems reason about and generate code. Work remotely on your own schedule while contributing to AI tools trusted by developers around the world.