AI Evaluator – English & Urdu at Mercor

AI Evaluator – English & Urdu | $15–20/hr | Worldwide Remote

Mercor is seeking a bilingual AI Evaluator fluent in Urdu (native) and English (strong proficiency) to assess and improve AI-generated responses. This is a part-time contract role open to candidates globally. Please note: You must complete the Bilingual Competency interview in Urdu to be considered.

What You'll Do

• Evaluate Urdu AI-generated responses, identifying strengths, weaknesses, and factual inaccuracies
• Conduct fact-checking using trusted public sources and external tools
• Assess reasoning quality, clarity, tone, and completeness of model responses
• Generate high-quality human evaluation data to help build the ideal AI response
• Ensure model outputs align with expected conversational behavior and system guidelines

Who You Are

• Holds a Bachelor's degree
• Native Urdu speaker with excellent written English skills
• Experienced with large language models (LLMs) and understands how they are used
• Detail-oriented with a background in structured analytical thinking (e.g., research, policy, linguistics, engineering, or analytics)

Nice-to-Have

• Prior experience with RLHF, model evaluation, or data annotation
• Experience writing or editing high-quality content
• Ability to make fine-grained qualitative judgments across multiple outputs

What Success Looks Like

• You consistently identify factual errors, reasoning gaps, and communication issues in AI responses
• Your evaluations are clear, reproducible, and actionable
• Your feedback drives measurable improvements in AI response quality