This job post has expired on June 12, 2026. It is likely that the position has already been filled.
AI Evaluator – English & Urdu at Mercor
posted 5 months agoAI Evaluator – English & Urdu | $15–20/hr | Worldwide Remote
Mercor is seeking a bilingual AI Evaluator fluent in Urdu (native) and English (strong proficiency) to assess and improve AI-generated responses. This is a part-time contract role open to candidates globally. Please note: You must complete the Bilingual Competency interview in Urdu to be considered.
What You'll Do
- Evaluate Urdu AI-generated responses, identifying strengths, weaknesses, and factual inaccuracies
- Conduct fact-checking using trusted public sources and external tools
- Assess reasoning quality, clarity, tone, and completeness of model responses
- Generate high-quality human evaluation data to help build the ideal AI response
- Ensure model outputs align with expected conversational behavior and system guidelines
Who You Are
- Holds a Bachelor's degree
- Native Urdu speaker with excellent written English skills
- Experienced with large language models (LLMs) and understands how they are used
- Detail-oriented with a background in structured analytical thinking (e.g., research, policy, linguistics, engineering, or analytics)
Nice-to-Have
- Prior experience with RLHF, model evaluation, or data annotation
- Experience writing or editing high-quality content
- Ability to make fine-grained qualitative judgments across multiple outputs
What Success Looks Like
- You consistently identify factual errors, reasoning gaps, and communication issues in AI responses
- Your evaluations are clear, reproducible, and actionable
- Your feedback drives measurable improvements in AI response quality
How to apply for this role
- Upload your resume — keep it up-to-date and in English. Mercor will auto-fill your profile from it.
- Complete the AI interview — a 15-minute conversation about your experience. Be ready to discuss specific projects and challenges you've solved.
- Submit your application — only about 20% of applicants finish all the steps, so completing yours puts you well ahead.