Bilingual AI Evaluator - English & Mandarin at Mercor

Bilingual AI Evaluator | $36.16/hour | Remote (Taiwan, Malaysia, USA)

Join Mercor in shaping the future of conversational AI systems. We're seeking bilingual professionals fluent in English and Mandarin Chinese to evaluate and improve large language model responses, ensuring they meet the highest standards of accuracy, clarity, and usefulness.

About This Role

As a Generalist AI Evaluator, you'll work with leading AI teams to assess and enhance general-purpose conversational AI systems. Your evaluations will directly impact how millions of users interact with AI technology, ensuring responses are accurate, well-reasoned, and aligned with human expectations.

Key Responsibilities

• Evaluate LLM-generated responses for accuracy, clarity, and effectiveness across diverse topics
• Conduct thorough fact-checking using trusted public sources and external verification tools
• Provide detailed annotations on response quality, identifying strengths and areas for improvement
• Assess reasoning quality, tone, completeness, and conversational appropriateness
• Apply consistent evaluation standards following established taxonomies and guidelines
• Generate high-quality human feedback data to improve AI system performance

Required Qualifications

• Bachelor's degree required
• Native speaker or ILR 5/C2 fluency in Chinese (Mandarin)
• Significant hands-on experience using large language models
• Excellent writing skills with ability to articulate nuanced feedback clearly
• Strong attention to detail and ability to identify subtle issues
• Adaptable across various topics, domains, and project requirements
• Background in structured analytical thinking (research, policy, analytics, linguistics, engineering)
• Excellent college-level mathematics skills

Preferred Experience

• Prior work with RLHF, model evaluation, or data annotation
• Experience writing or editing high-quality content
• Background in comparative analysis and qualitative judgment
• Familiarity with evaluation rubrics, benchmarks, or quality scoring systems

What Success Looks Like

• Consistently identify factual inaccuracies, reasoning errors, and communication gaps
• Produce clear, consistent, and reproducible evaluation artifacts
• Contribute feedback that leads to measurable improvements in AI response quality
• Help ensure AI systems meet quality standards before public release

Why Mercor

Work at the frontier of human-in-the-loop AI development with flexible, remote contract opportunities. Your contributions will directly shape how advanced language models behave in real-world applications used by millions. Competitive contract rates aligned with expertise and scope of work.

Contract Type: Full-time or Part-time available