Skip to main content
Mercor (via AI Gig Scout) logo

AI Evaluation Rubric Reviewer, ~15 hrs/week (Remote, Contract, $40–$75/hr) — Mercor via AI Gig Scout

Mercor (via AI Gig Scout)
20 hours ago
Contract
Remote
Worldwide
$40 - $75 USD hourly

AI Evaluation Rubric Reviewer — Mercor via AI Gig Scout

Pay: $40–$75/hr · Location: Remote · Type: Contract

Full listing and apply link: https://aigigscout.com/jobs/mercor/ai-evaluation-rubric-reviewer-list-aaaboeiksoinqks-uxoz7g1?utm_source=kubernetesjobs&utm_medium=jobboard&utm_campaign=aigigscout_growth


The work

We are testing AI agents on realistic shopping and delivery tasks. You'll review AI-drafted tasks (prompt, grading rubric) along with model attempts, and decide whether each failure is fair:

Was the information available?

Does the rubric accurately grade what the prompt asked for?

Would a valid answer have been marked wrong?

Is the rubric structure correct per the guidelines?

Fix the criteria that aren't fair, flag the tasks that can't be saved, and write a short note on each call.

Must-haves

Substantial experience writing or reviewing rubrics for LLM evals. Please list the projects in your application.

Can read a long agent transcript and find where it went wrong.

Precise writing, and you check the data instead of trusting the reference answer.

Source: AI Gig Scout job page. Apply there for the original platform listing. Disclosure: posted by AI Gig Scout, an independent listing site (not Mercor); the apply link may be a referral link.