Job
Expert Talent Pool, Evals and Benchmarks
Clera Clera offers roles in legal evaluation of AI systems and ML internship opportunities
Senior evaluation role designing benchmarks and leading grading for AI evaluation projects. Fully remote worldwide.
At a glance
- Deadline
- Location
- Remote
- Salary
- 75 – 175 USD / hour
- Paid
- Yes
- Employment type
- Full Time
- Minimum years of experience
- 5
- Listed
About this opportunity
Join Clera’s network of senior evaluation experts supporting AI research projects. You will design benchmarks, write rubrics, develop gold-standard answers, and lead grading operations to ensure rigorous assessments of AI models.
This fully remote, senior-level position offers project timelines ranging from one week to several months, with compensation of $75 to $175 per hour depending on project and experience.
You’ll calibrate graders, analyze model failures, run model comparison studies, and validate benchmarks for contamination and ambiguity. Strong English communication and Python proficiency are essential.
Projects span worldwide, with flexible scheduling across many time zones. Expect to contribute to multiple evaluations and potentially long-term collaborations.
What you get
- Compensation of $75 to $175 per hour
- Fully remote work
- Project-based engagements from one week to several months
- Exposure to cutting-edge AI evaluation projects
Responsibilities
- Design benchmarks: scope, question types, difficulty, and scoring methods
- Write and review rubrics for consistent grading
- Develop gold-standard answers and test sets
- Lead grading rounds: calibrate graders, measure agreement, resolve disputes
- Analyze model failures and recommend improvements
- Design and conduct model comparison studies
- Validate benchmarks for contamination, ambiguity, and saturation
- Design user studies on how people rate and interact with model outputs
Requirements
- At least 5 years of experience in evaluation, testing, or assessment
- Experience designing a benchmark or assessment
- Experience writing or reviewing rubrics and gold-standard answers
- Proficiency in Python and evaluation tools (Inspect, lm-eval-harness, HELM, promptfoo)
- Experience analyzing model/test failures and recommending fixes
- Background in ML evaluation, QA, psychometrics, or related fields
- Clear written English communication
How to apply
- Apply through the Himalayas platform listing linked in the post
Ready to apply?
WorkInRwanda never asks for money to apply. If anyone does, report it. Report this listing
About Clera
Clera is involved in legal and artificial intelligence work, offering roles that blend EU law expertise with AI evaluation. The listings indicate remote opportunities available worldwide and focus on senior legal professionals with EU knowledge as well as entry-level ML work.
See everything from Clera