Generalist – English & Russian at Mercor

AI Evaluation Specialist (English & Russian) | $25–40/hr | Worldwide Remote

Mercor is seeking a bilingual English & Russian AI Evaluation Specialist to assess and improve AI-generated responses. In this contract role, you will analyze Russian-language AI outputs and provide structured, English-language feedback to help shape the next generation of AI models. Please note: A Bilingual Competency interview in Russian is required to be considered for this role.

What You'll Do

• Fact-check AI-generated responses using trusted public sources and external tools
• Generate high-quality human evaluation data by identifying response strengths, areas for improvement, and factual inaccuracies
• Assess reasoning quality, clarity, tone, and completeness of model responses
• Ensure AI outputs align with expected conversational behavior and system guidelines

Who You Are

• Hold a Bachelor's degree
• Native Russian speaker with strong English proficiency
• Experienced with large language models (LLMs) and familiar with how they are used
• Possess excellent English writing skills and can articulate nuanced, detailed feedback
• Strong attention to detail with a background in structured analytical thinking (e.g., research, policy, analytics, linguistics, or engineering)

Nice-to-Have

• Prior experience with RLHF, model evaluation, or data annotation
• Experience writing or editing high-quality content
• Ability to make fine-grained qualitative judgments across multiple outputs

What Success Looks Like

• You consistently identify factual inaccuracies, reasoning errors, and communication gaps
• You produce clear, reproducible evaluation artifacts
• Your feedback drives measurable improvements in AI response quality and user experience