Generalist - English & German at Mercor

AI Evaluation Generalist | $39.78/hr | Remote (Europe & US)

Mercor is seeking bilingual English and German speakers to evaluate and improve conversational AI systems. This flexible contract role involves assessing LLM responses, fact-checking content, and providing high-quality human feedback to enhance AI reliability and user experience.

What You'll Do

• Evaluate LLM-generated responses for accuracy, clarity, and helpfulness across diverse topics
• Conduct fact-checking using trusted public sources and external tools
• Generate high-quality evaluation data by annotating response strengths and identifying areas for improvement
• Assess reasoning quality, tone, completeness, and alignment with conversational guidelines
• Apply consistent annotations following detailed evaluation taxonomies and benchmarks
• Ensure model responses meet expected standards and system requirements

Who You Are

• Hold a Bachelor's degree
• Native speaker or C2-level fluency in German
• Significant experience using large language models (LLMs) and understanding their practical applications
• Excellent writing skills with ability to articulate nuanced, detailed feedback
• Strong attention to detail and ability to identify subtle issues
• Adaptable across topics, domains, and varying project requirements
• Background in structured analytical thinking (research, policy, analytics, linguistics, engineering)
• Excellent college-level mathematics skills

Nice-to-Have Qualifications

• Prior experience with RLHF, model evaluation, or data annotation
• Experience writing or editing high-quality content
• Familiarity with evaluation rubrics, benchmarks, or quality scoring systems
• Experience making fine-grained qualitative judgments between multiple outputs

Why Join Mercor

Work at the frontier of human-in-the-loop AI development and directly shape how advanced language models behave in real-world applications. This flexible, remote contract role offers competitive rates and the opportunity to contribute meaningfully to AI systems used by millions globally.

Contract Type: Full-time or Part-time available
Geography: Restricted to Europe and USA