Skip to main content
Mercor (via AI Gig Scout) logo

AI Evaluation Rubric Auditor (LLM Evals) (Remote, Contract, $60–$85/hr) — Mercor via AI Gig Scout

Mercor (via AI Gig Scout)
2 days ago
Contract
Remote
Worldwide
$60 - $85 USD hourly

AI Evaluation Rubric Auditor — Mercor via AI Gig Scout

Pay: $60–$85/hr · Location: Remote (Canada, United States, Argentina, Bolivia, Brazil, Chile, Colombia, Ecuador, Guyana, Paraguay, Peru, Uruguay, United Kingdom) · Type: Contract

Full listing and apply link: https://aigigscout.com/jobs/mercor/ai-evaluation-rubric-auditor-list-aaaboeazz5app9cxnf9dvrub?utm_source=kubernetesjobs&utm_medium=jobboard&utm_campaign=aigigscout_growth


The work

We are testing AI agents on realistic shopping and delivery tasks. Your role as a Project Auditor is to audit the work done by our Project Reviewers. You will audit complete tasks (prompt and grading rubric) and determine if their rubric failures are fair:

- Was the information available?

- Does the rubric accurately grade what the prompt asked for?

- Would a valid answer have been marked wrong?

- Is the rubric structure correct per the guidelines?

This role includes providing feedback to reviewers regarding prompt and rubric issues.

Must-haves

- Substantial experience writing or reviewing rubrics for LLM evals. Please list the projects in your application.

- Experience providing feedback to writers/reviewers

- Consistent availability

- Can read a long agent transcript and find where it went wrong.

- Precise writing, and you check the data instead of trusting the reference answer.

Source: AI Gig Scout job page. Apply there for the original platform listing. Disclosure: posted by AI Gig Scout, an independent listing site (not Mercor); the apply link may be a referral link.