Audio Generalist Evaluator Expert at Mercor

Audio Generalist Evaluator Expert | $50/hr | Remote

Mercor is seeking a skilled English (US) Audio Generalist Evaluator Expert to support a high-impact AI audio research project with a leading research lab. This short-term, structured engagement involves transcription, annotation, and model evaluation tasks that directly contribute to training and benchmarking advanced language models. Ideal for candidates with strong analytical or academic backgrounds who are fluent in American English and passionate about AI research.

Key Responsibilities

• Transcribe & Analyze Audio/Video: Listen to, analyze, and transcribe English (US) audio and video content with strict adherence to formatting, stylistic, and accuracy guidelines. Capture nuances such as tone, register, regional expressions, and spoken American English variations.
• Define Evaluation Standards: Develop detailed rubrics and grading guidelines for general consumer audio contexts. Document standards to ensure consistency across reviewers and identify linguistic nuances, colloquialisms, dialectal variations, and edge cases specific to American English.
• Model Testing & Grading: Run prompts through language models, assess generated outputs against predefined criteria (accuracy, fluency, completeness), and provide structured feedback to improve model performance.
• Benchmarking & Quality Assurance: Participate in QA and review cycles, maintain consistency across datasets, and collaborate with project leads to resolve ambiguities and refine task design.

Minimum Qualifications

• Native or near-native fluency in English (US), spoken and written
• Strong writing, editing, and critical thinking skills
• Familiarity with American English accents, regional vocabulary, and contemporary usage
• Ability to accurately transcribe and analyze English audio across general consumer contexts
• Self-directed with strong time management skills; able to commit 10–20 hours per week

Preferred Qualifications

• College students or recent graduates welcome
• Background in linguistics, humanities, journalism, social sciences, or translation/localization
• Prior experience with transcription, annotation, localization, or research workflows
• Familiarity with regional American English dialects
• Interest in AI, language models, or applied research environments

Application Process

Complete a short AI-led interview (approximately 15 minutes). Selected candidates will be onboarded and invited to begin project work promptly.