AI Code Evaluation Specialist
Budget / SalaryHourly project
TypeFreelance project
LocationRemote
Posted1 hour ago
AI Trainer - Reasoning & Code Evaluation
Contract | Fully Remote | $40 - $55 per hour
Grade, stress-test, and improve coding assistants used by 200,000+ developers.
ABOUT THE ROLE
We're a small, remote agency of engineers and technical writers. Our clients
build coding assistants, and we do the evaluation and training work they cannot
scale internally. That means judgment: someone has to decide, at scale and with
a straight face, whether a model's answer is actually correct, merely
plausible, or quietly broken.
You will be that someone. This role is equal parts code review, technical
writing, and adversarial thinking. You will spend your days reading model
output that looks right, proving where it is wrong, and turning those findings
into rubrics and datasets the research team can train against.
It is repetitive work done at a high level of attention, and it directly
changes what the model learns. If you have ever been the person who finds the
off-by-one error in a code review that everyone else approved, this job is
built for you.
WHAT YOU'LL DO
- Write and maintain grading rubrics for code generation, debugging, and
multi-step reasoning tasks.
- Score model outputs against those rubrics and write the rationale that lets
a second person reach the same score without asking you.
- Build adversarial prompt sets that expose real failure modes: hallucinated
APIs, silent off-by-one errors, confident but wrong explanations, tests that
pass without testing anything.
- Run pairwise preference comparisons and help convert messy human judgment
into a reward signal that is stable enough to train on.
- Design targeted fine-tuning datasets from your own failure analysis rather
than from generic scraping.
- Review other trainers' work and keep inter-annotator agreement above 0.8 on
rubric-graded samples.
- Write short, specific failure reports that research can act on the same
week. "Model is bad at recursion" is not a report; a reproducer is.
WHAT WE'RE LOOKING FOR
- Bachelor's degree in computer science, mathematics, or a related technical
field, or equivalent demonstrated experience. We care about the work, not
the parchment.
- 3+ years writing production code in Python, TypeScript, Go, or similar. You
can read a diff and spot the bug before running it.
- Comfort with ambiguity. A rubric says "the answer is correct" - you can
argue precisely where the boundary sits and defend it in writing.
- Strong written English. You can explain why an answer is wrong in three
sentences without hedging.
- Tolerance for repetitive, high-attention work. Throughput matters;
consistency matters more.
- 20+ hours per week for at least three months.
NICE TO HAVE
- Prior AI training, annotation, or RLHF work (any provider).
- A math or competitive programming background.
- Experience with evaluation harnesses: lm-eval-harness, custom pytest-style
suites, or an internal eval platform.
- A second language, for multilingual evaluation work.
HOW WE WORK
Tools: our internal annotation platform, GitHub, Linear, Slack.
- Async-first. Written feedback beats meetings; decisions get recorded.
- Weekly calibration sessions: the whole team grades the same 10 samples and
reconciles every disagreement. This is the most useful hour of the week.
- No screenshot surveillance or keystroke tracking. We measure the quality and
consistency of what you produce, not how long your cursor moved.
RATE AND TERMS
- $40 - $55 per hour on a 6-month, extendable contract, set by domain depth and
calibration performance. Invoiced twice monthly, paid within 10 business days.
- Onboarding and ramp-up are paid at the full rate. The first two weeks are
structured: rubric training, shadow grading, then supervised grading.
- No exclusivity. You may hold other contracts as long as quality holds.
- Contractor and referral candidates are both welcome.
HOW WE'LL KNOW IT'S WORKING
- Weeks 1-2: calibrated on the rubric set, with agreement against senior
graders at or above 0.75.
- Months 1-2: you own one evaluation suite end to end, from rubric to
published results.
- Month 3 onward: your rubric proposals ship with light edits, and your
failure analyses start appearing in release notes.
We hire on demonstrated ability. We do not discriminate on the basis of race,
religion, gender, age, disability, or national origin.
Contract | Fully Remote | $40 - $55 per hour
Grade, stress-test, and improve coding assistants used by 200,000+ developers.
ABOUT THE ROLE
We're a small, remote agency of engineers and technical writers. Our clients
build coding assistants, and we do the evaluation and training work they cannot
scale internally. That means judgment: someone has to decide, at scale and with
a straight face, whether a model's answer is actually correct, merely
plausible, or quietly broken.
You will be that someone. This role is equal parts code review, technical
writing, and adversarial thinking. You will spend your days reading model
output that looks right, proving where it is wrong, and turning those findings
into rubrics and datasets the research team can train against.
It is repetitive work done at a high level of attention, and it directly
changes what the model learns. If you have ever been the person who finds the
off-by-one error in a code review that everyone else approved, this job is
built for you.
WHAT YOU'LL DO
- Write and maintain grading rubrics for code generation, debugging, and
multi-step reasoning tasks.
- Score model outputs against those rubrics and write the rationale that lets
a second person reach the same score without asking you.
- Build adversarial prompt sets that expose real failure modes: hallucinated
APIs, silent off-by-one errors, confident but wrong explanations, tests that
pass without testing anything.
- Run pairwise preference comparisons and help convert messy human judgment
into a reward signal that is stable enough to train on.
- Design targeted fine-tuning datasets from your own failure analysis rather
than from generic scraping.
- Review other trainers' work and keep inter-annotator agreement above 0.8 on
rubric-graded samples.
- Write short, specific failure reports that research can act on the same
week. "Model is bad at recursion" is not a report; a reproducer is.
WHAT WE'RE LOOKING FOR
- Bachelor's degree in computer science, mathematics, or a related technical
field, or equivalent demonstrated experience. We care about the work, not
the parchment.
- 3+ years writing production code in Python, TypeScript, Go, or similar. You
can read a diff and spot the bug before running it.
- Comfort with ambiguity. A rubric says "the answer is correct" - you can
argue precisely where the boundary sits and defend it in writing.
- Strong written English. You can explain why an answer is wrong in three
sentences without hedging.
- Tolerance for repetitive, high-attention work. Throughput matters;
consistency matters more.
- 20+ hours per week for at least three months.
NICE TO HAVE
- Prior AI training, annotation, or RLHF work (any provider).
- A math or competitive programming background.
- Experience with evaluation harnesses: lm-eval-harness, custom pytest-style
suites, or an internal eval platform.
- A second language, for multilingual evaluation work.
HOW WE WORK
Tools: our internal annotation platform, GitHub, Linear, Slack.
- Async-first. Written feedback beats meetings; decisions get recorded.
- Weekly calibration sessions: the whole team grades the same 10 samples and
reconciles every disagreement. This is the most useful hour of the week.
- No screenshot surveillance or keystroke tracking. We measure the quality and
consistency of what you produce, not how long your cursor moved.
RATE AND TERMS
- $40 - $55 per hour on a 6-month, extendable contract, set by domain depth and
calibration performance. Invoiced twice monthly, paid within 10 business days.
- Onboarding and ramp-up are paid at the full rate. The first two weeks are
structured: rubric training, shadow grading, then supervised grading.
- No exclusivity. You may hold other contracts as long as quality holds.
- Contractor and referral candidates are both welcome.
HOW WE'LL KNOW IT'S WORKING
- Weeks 1-2: calibrated on the rubric set, with agreement against senior
graders at or above 0.75.
- Months 1-2: you own one evaluation suite end to end, from rubric to
published results.
- Month 3 onward: your rubric proposals ship with light edits, and your
failure analyses start appearing in release notes.
We hire on demonstrated ability. We do not discriminate on the basis of race,
religion, gender, age, disability, or national origin.
Apply on Freelancer →
Project sourced from Freelancer.com. Applications happen directly on the original platform — we never collect your data.