shortlist-ai

Evidence-backed, blind resume shortlisting with LLMs, plus the evaluations and bias tests to check it.

Give it a job description and a folder of resumes. It returns a ranked shortlist in which every judgment cites a quote from the candidate's resume, the model never sees names or other identity signals, and the score is computed in plain code you can audit.

It also works the other way round: give it your resume and a folder of job postings, and it ranks the jobs by fit and lists the must-haves you don't show yet. See For job seekers.

This is decision support, not a decision-maker. Automated hiring tools are regulated in several places (e.g. NYC Local Law 144, the EU AI Act's high-risk category). shortlist-ai is designed to help a person review candidates faster and more consistently. Don't use it to reject anyone automatically. See Responsible use.

Why another resume ranker?

Most "AI resume screeners" are one prompt: "here's a JD and a resume, give me a score out of 10." That produces a number you can't check, from a model that saw the candidate's name, age signals and address. shortlist-ai is built around four ideas:

How it works

Job description ──► requirements (must-have / nice-to-have) ──► you can review & edit them
                                                                        │
Resumes ──► extraction (PDF/DOCX/TXT) ──► blind profile ──► keyword prefilter (large pools)
                                                                        │
                                    LLM judges each requirement, quoting evidence
                                                                        │
                                    code verifies quotes, computes weighted score
                                                                        │
                                    ranked report with evidence + review flags

• Two backends. claude (Anthropic API; reads PDFs as pages, so scanned and multi-column resumes work) or local (an MLX model on Apple Silicon: nothing leaves your machine).
• Structured outputs everywhere. Every model call returns a schema-validated object (API structured outputs, or constrained decoding locally via Outlines).
• Scoring: score = 100 × Σ(weight × credit) / Σ(weight), with must-have = 3, nice-to-have = 1, met = 1, partial = 0.5, not met = 0.
• Years of experience are added up in code. For a requirement that states a year count ("3+ years of data engineering"), a separate call shows the model each role's title and work, without durations, and asks whether the role's main work is in that field. Code adds up the durations of those roles, counting overlaps once: at least N years is met, fewer is partial, none is not met. The evidence is the role lines themselves. See Years of experience, computed in code.
• Blind profile: removes name, email, phone, links, home and job locations, school names and graduation years; replaces dates with durations; neutralizes pronouns and honorifics. Job titles, companies, accomplishments, skills, certifications and degrees stay.
• Injection defenses: resumes are untrusted input. PDF text a reader can't see is stripped before any model reads the file, and the candidate is flagged. "Can't see" is judged where each string is drawn: the same colour as what's behind it (white on white, but not white on a dark sidebar or gradient banner), transparent, under 4pt, off the page, or in invisible mode on a page with no scanned image. Extracted skills and certifications that don't appear in the visible text are removed and flagged, which also catches injections that get past the first check. The eval set includes a resume with a planted injection.
• Caching: extractions are cached per file, so re-ranking against a new job costs only the scoring calls.

Quick start

git clone https://github.com/vinayvobbili/shortlist-ai && cd shortlist-ai
python -m venv .venv && source .venv/bin/activate
pip install -e .                 # Claude backend
pip install -e ".[local]"        # + local MLX backend (Apple Silicon, Python ≤ 3.13)

export ANTHROPIC_API_KEY=...     # for --backend claude

# 1. Turn the job description into requirements, then review/edit them
shortlist requirements eval/jobs/senior_data_engineer_gcp.md --out reqs.json

# 2. Rank a folder of resumes
shortlist rank --requirements reqs.json eval/resumes/ --top 5 --out shortlist.md

# Or skip the review step and fully local:
shortlist rank --backend local --jd eval/jobs/senior_data_engineer_gcp.md eval/resumes/

# 3. For a borderline candidate: how much does the score depend on wording that shouldn't matter?
shortlist stability --requirements reqs.json eval/resumes/c15_carlos_diaz.md

For job seekers

shortlist jobs my_resume.pdf saved_jobs/            # a folder of postings (.md/.txt/.pdf/.docx)
shortlist jobs my_resume.pdf a.md b.pdf --top 2     # or individual files

It extracts your resume once, turns each posting into requirements, and scores your resume against each one with the same evidence-checked scorer. The report ranks the jobs and lists the gaps for each: must-haves the resume doesn't clearly show. A gap often means the experience is missing from the resume, not from you, so it doubles as a to-do list for tailoring the resume to each job.

• Scores are comparable across jobs because each is the weighted share of that job's requirements met. Ties go to the job where you meet a larger share of the must-haves.
• Postings are cached, and you can review them like in the recruiter flow: save shortlist requirements posting.md --out posting.json, edit it, and pass the .json instead.
• --backend local keeps your resume on your machine.

Evaluation

eval/ holds 18 synthetic candidates in mixed formats (DOCX, text PDF, two-column PDF, scanned PDF, plain text, Markdown, and one PDF with hidden prompt-injection text) and six job descriptions. Every candidate has a relevance grade (0–3) for every job, written before any model was run and explained in eval/README.md. The set includes deliberate near-misses, such as a data engineer on AWS applying to a GCP role and the reverse.

Each (job, resume) pair is scored once, and the grid is read both ways: per job for candidate ranking (shortlist rank) and per resume for job ranking (shortlist jobs). --repeats N adds runs with the requirements shuffled and reports how much the results move, which is the noise floor for comparing two versions (see the held-out section below).

shortlist eval --backend local --repeats 3 --out results/eval_local.md

Results with the local backend (Qwen3.5-9B, 4-bit, MLX, on an M4 Mac mini), from results/eval_local.md:

¹ At the ceiling: only one (frontend) or two (cloud security) candidates are graded 2 or higher.
² Wei, a backend engineer graded 1 for four jobs and 0 for two, gets Incident Response (graded 0, score 22) third, above the GCP data engineering job (18). Wei's backend years no longer count as data engineering, which is right; all of Wei's matches are weak.

The tables are for the requirements in the order written. Across three orderings (two of them shuffled), results moved a little: mean NDCG@3 0.96–0.99, top-1 correct 5–6 of 6 jobs and 13 of 13 resumes in every ordering, and 819 of 850 verdicts (96%) identical in every ordering. See the held-out section for why orderings are the right measure of noise here.

Read this as "no regressions", not a benchmark. The set is small and synthetic, and the same people wrote it and the tool. What it does show:

• Near-misses land where they should in both directions. Oliver (AWS) ranks the AWS role first; Rahul and Amara (GCP) rank the GCP role first. For the GCP job, Oliver ranks below both GCP engineers and is flagged for the missing BigQuery and Dataflow must-haves.
• Non-requirements were left out. Three postings include lines that aren't job qualifications ("young team", "graduates of top universities", lifting 25 lbs for a desk job). None became a requirement.
• The injected resume scores 0 for every job. Its hidden text is excluded, and none of the planted skills reach the scorer.
• The scanned PDF fails with the local backend (no text layer), which the metrics don't show because that candidate is graded 0 for every job. Use --backend claude or OCR scans first.
• The BM25 prefilter is rough. In the example above, a cut to 6 kept a grade-0 candidate and dropped a grade-1 one. The eval scores every candidate, so this doesn't affect the metrics above.
• "N+ years of X" requirements were the least stable when the model judged them. For "3+ years of professional data engineering experience", it rejected a nurse's 10 years when that requirement came first. Later in the list, it often marked the requirement met: "The profile explicitly states 10 years of professional experience, which exceeds the 3+ years requirement." In the shuffled orderings, 9 of the 17 candidates for the AWS job flipped to met this way, raising the scores of weak candidates. Years are now added up in code, and none of those verdicts change with ordering. See the two sections after the held-out one.
• The Claude backend has unit tests for its request shape but hasn't been evaluated yet. Results welcome: shortlist eval --out results/eval_claude.md.

A scorer fix, checked on held-out data

The first version of the scorer missed one call on this set: it gave Priya 100 for Cloud Security Engineer (graded 2) and ranked Priya above Nadia, who does that job. The evidence showed why: hands-on AWS security experience was marked met because of an AWS certification plus "Terraform" in the skills list. The same score tied with Incident Response, Priya's best fit, so the job ranking was wrong too.

The fix is a rule in the scoring prompt. If a requirement asks for hands-on, production or professional experience, a certification or skills-list entry alone is partial. If it only names a tool, a skills-list entry is enough. The results above are with the fix: both misses are gone and nothing else moved. But the fix was designed around this exact case, so that is expected, not evidence.

The evidence is eval/heldout/: seven new resumes and three new jobs, committed with expected per-requirement verdicts before the fix and before any model saw them. Each job has a candidate whose certifications and skills list match but whose work history doesn't, plus cases where being too strict would be wrong.

Each version was run three times: once with the requirements in the order written and twice with them shuffled (shortlist eval --repeats 3). The local model decodes greedily, so an identical rerun repeats itself exactly. Shuffling the requirement order, which scoring ignores, shows how much the model's answers depend on details that shouldn't matter. Ranges are min–max over the three orderings.

Honest reading: the fix moves errors from lenient to strict, and helps accuracy a little at most. Too-lenient verdicts fell (mean 2.3 to 1.0), which is what the fix was meant to do, and agreement no longer dips below 41. But strict errors took their place. In two of three orderings, a Terraform nice-to-have was marked partial for a candidate who lists Terraform, which breaks the fix's own "only names a tool" rule. The ranges overlap, and three orderings of 44 verdicts are too few to call a one-verdict gain.

What the orderings show about the tool itself:

• Ordering alone changes a few verdicts. About 1 in 15 verdicts flipped just from shuffling the requirements, so a one- or two-verdict gap between single runs isn't evidence. Before this measurement, the before/after comparison was one run each, and one ordering for the old prompt would have scored 38/44 rather than 41.
• The flips are on borderline evidence, and they cluster on weak candidates. Most moved one step (partial ↔ met or partial ↔ not_met). Daniel (sysadmin with Kubernetes certificates) had 5 of the 11 flips with the fixed prompt, and Daniel's SRE score moved by up to 27 points. In one ordering the model also missed Daniel's CKA certificate, a plain error.
• Rankings stayed correct. In every ordering of both prompts, NDCG@3 was 1.00 and every top pick was right, for candidates and for jobs. Scores move, but not across the gaps between strong, adequate and weak fits. Treat borderline scores as a range, not a point.

The held-out set has now been looked at, so the next change needs fresh cases. Before/after reports: results/heldout_local_before.md, results/heldout_local.md. The second one has since been rerun with years computed in code, which raised Daniel's SRE score from 37 to 47 (see below), and again with the quote-check fix (47 to 57, "Checking one score" below). Verdict agreement is now 41–42 of 44, and rankings stayed correct.

A fix that didn't work: hiding the total years

The obvious suspect for the "N+ years" problem was the profile itself. It opens the experience section with Experience (total 10 yrs), and the nurse's met quoted that line. So the change removed the total and kept only per-role durations. It was tested the same way as the scorer fix: eval/heldout_years/ was committed first (nine candidates, three jobs with "N+ years of X" must-haves). Then the change was made, and every set was run with three orderings.

It was reverted. Without the total, the model adds up unrelated roles itself. For a mechanical engineer against the AWS data engineering job, it quoted three roles and said: "The candidate has over 16 years of professional experience, which exceeds the 3+ years requirement." The total wasn't the cause. When "3+ years of professional data engineering experience" isn't near the top of the list, the model drops the qualifier and counts every year. Nothing else moved outside ordering noise.

The years set also showed that the problem is narrower than it looked:

• Long careers in unrelated fields (teacher, warehouse supervisor) were never marked met, with or without the total. Those jobs list 4–5 requirements; the dev-set AWS job lists 10.
• Career-changers are the weak spot. Two years in a SOC after 7½ years of IT help desk was marked met for "3+ years in a SOC or security operations role" in most orderings. That tied the candidate with a 7-year incident responder and cost the SOC job its correct top pick.
• Adding up relevant roles works. Two backend roles of about 3 years each were counted as meeting "5+ years" in every ordering.

Reports: results/heldout_years_local_before.md (with the total, as shipped at the time) and results/experiments/no_total_line/ (reverted change).

Years of experience, computed in code

The next attempt took the arithmetic away from the model. For each "N+ years of X" requirement, a separate call lists the candidate's roles (title, company and highlights, but no durations) and asks for two things: the field the requirement names, and whether each role is in it. Code adds up the durations of the roles that count. The main scoring call is unchanged; its verdict on the years requirement is replaced.

The first version asked yes/no per role. It was tested on eval/heldout_years2/, committed first: 35 of 35 in every ordering. But the original scorer also got 35 of 35 there, so that set couldn't tell the two apart. Reading the per-role answers on the first held-out set showed a new problem: roles that only touch a field were counted. A platform engineer counted as security engineering because the work "involves hardening", and an ML engineer counted as infrastructure because the models ran on Kubernetes. So a third set, eval/heldout_years3/, was written around that case and committed before the fix. Each job has candidates whose main work is in the field and candidates whose work touches it. The fix asks for three answers per role, main / touches / no, and counts only main.

The other sets, checked for regressions (three orderings each):

Honest reading: it works on the case it was built for, and nothing got worse.

• Only set 3 is a clean test of the shipped version, and it's small: 29 verdicts from ten synthetic candidates, written by the same person who wrote the fix, around the failure the fix targets. It shows the fix does what it was meant to on new cases, not that it generalizes.
• The dev set and years set 1 had been seen, so their gains agree with set 3 but aren't independent evidence. Set 2 is a null result.
• It's stricter by design. A role that touches the field gets no credit, however long. Nadia, a cloud security engineer, gets none of Nadia's years counted toward "security operations or incident response"; the Incident Response score went from 53 to 44 (still ranked third, as graded). The reasoning names the roles that touch the field, so a reviewer can see what was left out.
• Some calls are judgment. A systems administrator's years count as infrastructure engineering, which raised Daniel's SRE score on the first held-out set from 37 to 47. That set leaves the verdict out on purpose as ambiguous.
• It costs one extra model call per years requirement per candidate, and any requirement that states a number of years goes this way, even if the number isn't the point of it.

Reports: results/heldout_years3_local.md and _before, the same pair for years set 2 and years set 1, and results/experiments/yes_no_roles/ for the yes/no version.

Checking one score: shortlist stability

shortlist stability scores one resume against one job with the requirements in ten orders and shows which verdicts move, with the evidence and reasoning behind each version. It's meant for a borderline candidate: a score that moves between orders is a range, not a point.

Run on Carlos (analytics engineer, graded 3) for Data Analyst, it found the score moving from 58 to 88, and the cause was mostly in the code, not the model. The model sometimes quoted the skills line shortened to the item it needed ("Skills: Python" from "Skills: SQL, dbt, Snowflake, Looker, Python, Excel"). That isn't a substring, so the quote check rejected it and downgraded a correct verdict; in one order, three of them. The check now also accepts a labeled list with items left out, when one profile line has that label and every quoted item as whole words. Made-up items still fail, and "Java" doesn't match "JavaScript".

What's left is the model's own judgment on two borderline requirements: whether "light Python scripting for automation" meets "Python or R for data analysis", and whether "Excel" in a skills list is partial or not met for "advanced Excel". Reasonable reviewers could split on both.

On the eval sets (three orderings each), the same fix changed two verdicts, both toward the expected answer: Terraform for Daniel on the first held-out set (not met → partial, as labeled; agreement 41 → 41–42 of 44) and Python scripting for Ben on the dev set (partial → met). No ranking moved. The shortened quote is rarer than Carlos's case suggests, but when it happens it costs a whole verdict. Reports: results/stability_carlos_data_analyst_before.md, results/stability_carlos_data_analyst.md.

Evidence that isn't a quote

Scoring a real resume against a real 13-requirement detection-engineering posting with shortlist jobs gave 35/100 and 1 of 7 must-haves. Checking the quotes one by one showed that the local model often padded evidence with entries that aren't quotes at all: the requirement's own wording ("Practical experience using LLM APIs") or a note about what's missing ("Splunk is not mentioned"). Eleven of the 13 assessments had one. Since one unfound quote downgrades the verdict, four correct "met" verdicts (Python, LLM APIs, incident response, open source), each backed by real quotes, dropped to partial.

The quote check now sets these entries aside before verifying, and flags each case: an entry that isn't in the profile and is either contained in the requirement's text or says something "is not mentioned" (or similar) is not counted as evidence. The prompt also says so directly. A made-up quote still downgrades, and a "met" left with no real quote still counts as having no evidence.

The "after" run also uses the new prompt, so part of the change comes from the model following it. On a second posting, the one remaining flag was a real paraphrase ("cut" for "cutting"), which still downgrades as intended. The eval sets have not been re-run with this change.

Fairness testing

shortlist fairness eval/resumes/                               # free leak check
shortlist fairness eval/resumes/ --jd <job> --measure --candidates <id> <id>   # score sensitivity (costs calls)

1. Leak check (no model calls): each resume is rewritten under ten names associated with different genders and ethnicities (following audit-study methodology), with matching emails and pronouns. The blind profiles must come out byte-for-byte identical. If they are, the scorer receives the same input regardless of apparent identity. Any difference is reported as a leak.
2. Sensitivity measurement: scores the same variants with the name shown (a Name: line above the blind profile, so the name is the only difference) and compares the spread across the ten names to the spread across ten requirement orders for one name (model noise; a plain re-score would show none, because local decoding is deterministic). It also reports the mean gap between the names paired with "she" and "he", and each name's score against the candidate's mean. This measures what blinding protects against for a given model.

Results with the local backend (results/fairness_local.md):

• Leak check: 17/17 blind profiles identical across all ten name/pronoun variants (the scanned PDF was skipped because it has no text layer locally).
• Sensitivity: showing the name changed nothing. Six candidates, chosen to include strong and borderline fits for two jobs (Incident Response: Priya, Ben, Nadia; Data Analyst: Carlos, Sam, Jordan), each scored under all ten names: every candidate got the same score under every name, so the she − he gap is 0 too. Requirement order moved the same candidates by up to 29 points (Carlos: 58–88). In a spot check (Carlos under two names), the name was in the prompt, and the model's summary and reasoning were word for word the same.

Honest reading: for this model and prompt, on this sample, the name isn't what moves scores; wording and order are. Blinding wasn't needed to keep these six scores fair. It still guards against models that do react to names, and against other signals the name test doesn't cover: the name-shown profile still has pronouns, locations, schools and dates removed. Six candidates, two jobs and one model is a small sample, and one greedy decode per name can't show a small average bias the way many sampled runs could. The scoring prompt tells the model the profile has been anonymized and not to speculate about identity, which may be part of why it ignores the name.

Known limitations

• Hidden-text detection is geometric, not visual. It knows the colour of flat shapes but not of images, gradients or embedded form objects, so text drawn over those is assumed visible (skills grounding still applies). Text inside embedded form objects isn't checked. Across 390 PDFs that ship with macOS and installed apps, the only flags were text under 4pt in small icons.
• Grounding is literal. If the model renames a skill ("JS" → "JavaScript"), the renamed skill is removed and flagged. On the eval set this happened zero times, but review the flags.
• Scans have no text layer to check extraction against; grounding is skipped for them, and the local backend can't read them (OCR first, or use --backend claude).
• Years of experience depend on one judgment per role. The model decides whether each role's main work is in the field; roles that only touch it get no credit, and titles like "systems administrator" vs. "infrastructure engineer" are a judgment call. Durations come from the dates as extracted, so a wrong date gives a wrong total. Check the evidence for those requirements.
• Small eval set. 18 synthetic candidates and six jobs (plus four held-out sets of seven to ten resumes and three jobs each) is enough to catch regressions, not to certify accuracy. Add graded data from your own (consented, anonymized) hiring before relying on it.

Responsible use

• Keep a person in the loop. Use the ranking to decide who to read first, not who to reject. Candidates below the prefilter cut are listed as not assessed, not rejected.
• Review the requirements. The job description is turned into an editable list first (shortlist requirements). Remove anything that isn't a real job requirement, since the tool enforces whatever you give it.
• Check the flags. Unverifiable evidence, missing assessments and missing must-haves are surfaced for review, not hidden.
• Blinding reduces bias; it doesn't remove it. Company names, writing style and activities can still carry demographic signal, and the requirements themselves can encode bias (e.g. years-of-experience thresholds). Audit outcomes on your own data.
• Know your obligations. Depending on where you hire, automated screening can require bias audits, candidate notice, or human review (NYC LL 144, EU AI Act, Illinois AI Video Interview Act, GDPR Article 22, and others). This project is not legal advice.
• Protect candidate data. Don't commit real resumes (private/ is git-ignored). With --backend claude, resume content is sent to Anthropic's API; use --backend local if data must stay on your machine.

Development

pip install -e ".[dev]"
pytest            # no API key or model needed: tests use a deterministic fake backend
ruff check .      # lint (CI runs both)

Project layout: shortlist_ai/ (documents → extract → blind → prefilter → score → pipeline → report; plus fairness, evaluate, cli), tests/, eval/.

Ideas welcome, especially embedding-based prefiltering, more evaluation data, and more counterfactual dimensions (age signals, disability disclosures, career gaps).

License

MIT