Scoring with Rubrics
A rubric is the contract between an interviewer and the candidate’s score. It says “here is what we are evaluating, here is what each level looks like, and here is how the parts combine into a number.” Done well, rubrics let two interviewers grading the same candidate arrive at similar conclusions.
This page explains how Rubric’s rubrics are structured, how to score in practice, and how the math works.
What a rubric contains
Every interview ships with a rubric defined in its interview.yaml file. A rubric has:
- Three to five categories — distinct evaluation dimensions. Typical examples: “Problem Analysis”, “Solution Design”, “Trade-off Reasoning”, “Communication”, “Operational Awareness”.
- A weight per category — a number that says how much that category contributes to the final score. Weights normalize, so
[2, 1, 1, 1]and[20, 10, 10, 10]produce the same final score. - Six descriptors per category — one for each score from 0 through 5. The descriptors are short, behavior-anchored sentences. They are the most important part of the rubric.
A descriptor reads like this:
3 — Solid. Identifies the core problem and proposes a working design. Discusses at least one major trade-off (e.g., consistency vs. availability) without prompting.
The 0–5 scale isn’t a percentile. It’s a calibration ladder.
The 0–5 scale
| Score | Label | What it usually means |
|---|---|---|
| 0 | None | No signal. Candidate didn’t engage with this dimension. |
| 1 | Weak | Attempted but missed fundamentals. |
| 2 | Below average | Got the basics but had material gaps. |
| 3 | Solid | Met the bar. Working design, reasonable trade-offs, clear communication. |
| 4 | Above average | Clearly above the bar. Anticipated edge cases, defended choices well. |
| 5 | Strong | Standout. Brought up considerations the interviewer didn’t think to ask. |
Most candidates score in the 2–4 range across most categories. A 5 is rare; a 0 means the round didn’t actually probe that dimension and the score doesn’t count toward calibration.
Weights and the final score
Each category has a weight. The final score is a weighted average of the individual category scores, on the same 0–5 scale.
final = sum(category_score × category_weight) / sum(category_weight)Example: a rubric with categories scored 3 / 4 / 3 and weights 2 / 1 / 1 produces:
(3 × 2) + (4 × 1) + (3 × 1) 13
─────────────────────────── = ── = 3.25
2 + 1 + 1 4A final of 3.25 says “solid, leaning above average”. The hiring decision (Hire / No Hire / Follow Up) takes the final score plus the per-category breakdown into account — see Interview Reports.
Real-time vs end-of-session scoring
Both work. Pick what suits the round.
- Real-time — score each category as soon as you see strong signal. Useful when the round explores categories sequentially (e.g., the first 10 minutes are problem analysis, the next 15 are solution design). Easy to adjust later.
- End-of-session — leave categories blank and score everything at once after End Session. Useful when the round bounces between topics and you need the full arc to calibrate fairly.
A common hybrid: drop in tentative scores during the round, then take two minutes after End Session to re-read the descriptors and lock in the final numbers before the report drafts.
Conventions for descriptors
If you generate your own interviews (see Generating Interviews), the rubric is part of the package. The descriptors are where most of the leverage is — they’re the part that decides whether two interviewers grading the same candidate land on similar scores. A few conventions that help if you hand-edit a generated rubric:
- Behavior, not adjective. “Mentions consistent hashing without a hint” beats “Strong distributed-systems instincts.” The first is observable, the second is inference.
- Each step up is a real step. A 4 should require something a 3 doesn’t. If 3 and 4 read interchangeably, the rubric will compress to “either solid or weak” and lose its discriminating power.
- Ground the 0 and 5. Edge cases are the calibration anchors. What does it look like when the candidate brings nothing to this dimension? What does world-class look like?
- Short. Four lines per descriptor max. The interviewer is going to read these mid-session.
When a rubric isn’t working
If you’re running an interview and consistently can’t tell which score applies, the rubric is the problem, not you. Two fixes:
- Edit the descriptors of the generated interview to sharpen the boundaries between adjacent scores.
- Drop a category that doesn’t actually surface in 30 minutes. Three sharp categories beat five fuzzy ones.
Edits to a generated interview’s rubric take effect on the next session you start.