Managed human evaluation · code only
Eval data from people who can read the diff.
DataSynchrs runs named teams of vetted software engineers who grade AI-generated code: pairwise comparisons, RLHF preference data, agent trajectories. Every delivery ships with inter-rater agreement scores. No anonymous crowds.
{
"client": "[redacted]",
"task_type": "pairwise_comparison",
"domain": "agentic_coding",
"items_delivered": 1840,
"evaluation_team": ["ENG-04", "ENG-11", "ENG-17"], // same engineers since batch_001
"inter_rater_agreement": {
"metric": "cohens_kappa",
"score": 0.81, // measured, not assumed
"double_rated_share": 0.20,
"disagreements_adjudicated": 112
},
"status": "shipped"
}what a delivery actually looks like
// what we grade
01 / the-problem
Generic labeling platforms break on code.
Not for lack of QA. The failure is structural: code evaluation is engineering work, staffed and priced like it isn't.
failure_mode[0]
Generalist raters can't read code
A crowdworker can tell you which paragraph reads better. They can't tell you which of two diffs introduces a race condition, or whether an agent's fix addressed the root cause or papered over it. Code evaluation is engineering work.
failure_mode[1]
Unmeasured labels are just noise
Most platforms hand you labels with no reliability metrics attached. If you don't know whether two raters would agree on the same item, you don't know whether you're training on signal or on coin flips.
failure_mode[2]
Anonymous raters, rotating weekly
When the people grading your outputs change constantly, calibration never compounds. Rubric interpretation drifts batch to batch, quality is unauditable, and there's no one to ask when a label looks wrong.
failure_mode[3]
Piecework pricing rewards speed
A careful agent-trajectory review takes thirty to forty minutes. Per-task marketplaces price it like a captcha, so a captcha's worth of attention is exactly what it gets.
02 / how-it-works
From eval spec to measured delivery.
Scope the rubric
We start from your eval spec, or help you write one. Task types, grading rubric, edge cases, and a gold set we calibrate against before anything scales.
Assemble your team
A named team of engineers matched to your stack, vetted through coding screens and paid calibration rounds on the gold set. You see who passed, and who's grading.
Run the evaluations
Evaluators read the diffs, run the code, and grade against the rubric. A fixed share of items is double-rated; disagreements are adjudicated by a senior reviewer.
Deliver with receipts
Every batch ships with Cohen's Kappa per task type, disagreement logs, and per-evaluator stats. If agreement drops, we recalibrate before the next batch, not after.
03 / why-us
The parts that are hard to fake.
vetting = coding_screen + paid_calibration
Engineering-grade evaluators
Every evaluator passes the kind of coding screen you'd run for a hire, then paid calibration rounds before touching production tasks. They read diffs, run code, and review trajectories the way they'd review a teammate's PR.
delivery.kappa = 0.81 // every batch
Agreement scores on every delivery
Cohen's Kappa ships with every batch, computed on double-rated overlap. You see the reliability of the labels before you train on them. We're accountable to a number, not a vibe.
team = ["ENG-04", "ENG-11", "ENG-17"]
Named, consistent teams
The same engineers grade your work week over week, identified in every delivery. Calibration compounds, drift gets caught early, and when you question a label, there's a person behind it.
conflicts_of_interest = []
Coding-only. Lab-independent.
We don't train models, we aren't owned by a lab, and we don't do general-purpose labeling. Your rubrics, prompts, and preference data stay yours, and we have no stake in any benchmark.
04 / the-numbers
Measured, batch over batch.
κ 0.00
avg Cohen's Kappa across recent batches
0%
of items double-rated, every single batch
0%
of deliveries shipped with an IRA report
0h
from scope call to pilot proposal
05 / the-platform
Bring your own evaluators. Keep the measurement.
The platform is the stack our own evaluation teams work in every day, opening up as a product. If you run your own evaluators, an in-house team or a labeling company, you can run them on it and get the same measured output.
Try it yourself
you're the evaluator. two tasks are assigned to you in batch_047 ↓
queue / task_2481
1 of 2 assigned
// Bug report: retry storm under load. Grade the model's patch.
src/retry.ts
@@ -13,5 +13,6 @@
for (let attempt = 1; attempt <= max; attempt++) {
const base = opts.baseMs;
- const delay = base * attempt;
+ const delay = base * 2 ** attempt;
+ const jitter = rand() * base;
await sleep(delay + jitter);
if (await tryOnce()) return;
}
// your label
correctness
code_quality
root_cause_fix
tests_adequate
regression_risk
flags
confidence
justification0/20
diff_native_review
A review surface built for code
Side-by-side diffs with repo context and agent-trajectory playback. Not a text box with syntax highlighting bolted on.
rubrics_as_versioned_config
Rubrics with a version history
Rubrics live as versioned config behind calibration gates. Every label records exactly which rubric version graded it, so drift is visible instead of silent.
agreement_math_built_in
IRA computed, not compiled
Double-rating, Cohen's Kappa, and disagreement queues run automatically on every batch. The reliability report your clients ask for writes itself.
// early partners onboard directly with our team
06 / next-step
Skeptical is the right starting point.
Book twenty minutes. We'll walk through a redacted real delivery: agreement scores, disagreement logs, evaluator profiles. You can judge the signal yourself. No deck, no drip campaign.