Posted over 30 days ago. Roles like this often stay open for a while, so it's still worth applying.
Generalist - English & German at Mercor
posted 9 months agoAI Evaluation Generalist | $39.78/hr | Remote (Europe & US)
Mercor is seeking bilingual English and German speakers to evaluate and improve conversational AI systems. This flexible contract role involves assessing LLM responses, fact-checking content, and providing high-quality human feedback to enhance AI reliability and user experience.
What You'll Do
- Evaluate LLM-generated responses for accuracy, clarity, and helpfulness across diverse topics
- Conduct fact-checking using trusted public sources and external tools
- Generate high-quality evaluation data by annotating response strengths and identifying areas for improvement
- Assess reasoning quality, tone, completeness, and alignment with conversational guidelines
- Apply consistent annotations following detailed evaluation taxonomies and benchmarks
- Ensure model responses meet expected standards and system requirements
Who You Are
- Hold a Bachelor's degree
- Native speaker or C2-level fluency in German
- Significant experience using large language models (LLMs) and understanding their practical applications
- Excellent writing skills with ability to articulate nuanced, detailed feedback
- Strong attention to detail and ability to identify subtle issues
- Adaptable across topics, domains, and varying project requirements
- Background in structured analytical thinking (research, policy, analytics, linguistics, engineering)
- Excellent college-level mathematics skills
Nice-to-Have Qualifications
- Prior experience with RLHF, model evaluation, or data annotation
- Experience writing or editing high-quality content
- Familiarity with evaluation rubrics, benchmarks, or quality scoring systems
- Experience making fine-grained qualitative judgments between multiple outputs
Why Join Mercor
Work at the frontier of human-in-the-loop AI development and directly shape how advanced language models behave in real-world applications. This flexible, remote contract role offers competitive rates and the opportunity to contribute meaningfully to AI systems used by millions globally.
Contract Type: Full-time or Part-time available
Geography: Restricted to Europe and USA
How to apply for this role
- Upload your resume — keep it up-to-date and in English. Mercor will auto-fill your profile from it.
- Complete the AI interview — a 15-minute conversation about your experience. Be ready to discuss specific projects and challenges you've solved.
- Submit your application — only about 20% of applicants finish all the steps, so completing yours puts you well ahead.