ML Challenge Task Auditor at Mercor

ML Challenge Task Auditor | $70–90/hr | Remote (US)

Join a frontier AI lab's quality assurance effort by evaluating the rigor, correctness, and methodology of applied machine learning tasks used to train and benchmark cutting-edge AI models. This is a high-impact contractor role for experienced ML practitioners who can think critically about experiment design and evaluation standards.

What You'll Do

• Assess the quality and methodological soundness of applied ML tasks and experiments
• Evaluate model-selection reasoning, hyperparameter tuning decisions, and evaluation methodology
• Identify issues such as data leakage, metric gaming, and train/test/CV hygiene violations
• Provide clear, rubric-based written feedback to guide task improvement
• Critique ML claims against supporting evidence and assess reproducibility

Required Qualifications

• 3+ years of hands-on applied or experimental ML experience (experiment design, model selection, evaluation methodology)
• Strong understanding of data-quality rigor, including leakage detection and cross-validation best practices
• Proficiency with standard ML frameworks such as PyTorch, TensorFlow, scikit-learn, or XGBoost
• Demonstrated ability to critically evaluate ML results and reproduce experimental findings

Preferred Qualifications

• Experience in ML competitions or benchmarks (e.g., Kaggle)
• Graduate-level research background or publication record in applied ML
• Prior experience with task grading, peer review, or structured evaluation

Please note: This role focuses exclusively on applied and experimental ML rigor — it is not an LLM application development or MLOps position.