Benture logo
Mercor logo

AI Evaluation Rubric Auditor at Mercor

posted 35 minutes ago
mercor.com Contractor remote $60-85/hr 29 views

AI Evaluation Rubric Auditor | $60-85/hr | Remote

The work

You will audit tasks completed by Project Reviewers who are testing AI agents on realistic shopping and delivery tasks. For each complete task (prompt and grading rubric), you will determine whether the rubric's marked failures were fair by checking:

  • Was the information available?
  • Does the rubric accurately grade what the prompt asked for?
  • Would a valid answer have been marked wrong?
  • Is the rubric structure correct per the guidelines?

You will also give feedback to reviewers on prompt and rubric issues you find.

Requirements

  • Substantial experience writing or reviewing rubrics for LLM evals — list the projects in your application
  • Experience providing feedback to writers or reviewers
  • Consistent availability
  • Can read a long agent transcript and find where it went wrong
  • Precise writing, and you check the data instead of trusting the reference answer

Pay and commitment

Hourly contract role paying $60-85/hr. The work is fully remote.

How to apply for this role
  • Upload your resume — keep it up-to-date and in English. Mercor will auto-fill your profile from it.
  • Complete the AI interview — a 15-minute conversation about your experience. Be ready to discuss specific projects and challenges you've solved.
  • Submit your application — only about 20% of applicants finish all the steps, so completing yours puts you well ahead.
Benture is an independent job board and is not affiliated with Mercor.
Benture logo
See All Jobs
Apply Back