Posted over 30 days ago. Roles like this often stay open for a while, so it's still worth applying.
Construction Estimating AI Evaluator at Mercor
posted 3 months agoConstruction Estimating AI Evaluator | $50/hr | Remote (US & Canada)
Join a cutting-edge benchmark dataset project focused on evaluating AI models for visual document understanding and instruction-following within the Construction Estimating domain. This is an exciting opportunity for industry experts to directly shape the future of AI capabilities in a specialized field.
About the Role
As a subject matter expert, you will author complex, grounded tasks designed to test AI model performance. Each task must include a clear ground-truth output and an objective evaluation rubric, ensuring rigorous and meaningful benchmarks.
Key Responsibilities
- Design and author challenging visual document understanding tasks rooted in construction estimating workflows
- Develop clear ground-truth outputs and objective scoring rubrics for each task
- Evaluate AI model responses for accuracy, instruction-following, and domain relevance
- Provide structured feedback to improve model performance in the construction domain
Requirements
- Demonstrated expertise in construction estimating (e.g., quantity takeoffs, bid preparation, cost analysis)
- Strong attention to detail and ability to define objective evaluation criteria
- Comfortable working with visual documents such as blueprints, specs, and cost sheets
- Based in the United States or Canada
Engagement Details
- Rate: $50/hr
- Hours: ~15–20 hrs/week
- Location: Fully remote
- Type: Part-time contractor engagement
This is a private shortlist opportunity and is not publicly listed on the general job board.
How to apply for this role
- Upload your resume — keep it up-to-date and in English. Mercor will auto-fill your profile from it.
- Complete the AI interview — a 15-minute conversation about your experience. Be ready to discuss specific projects and challenges you've solved.
- Submit your application — only about 20% of applicants finish all the steps, so completing yours puts you well ahead.