Speaker Attribution Accuracy: 100-Hearing Public Test

The result

• By words is the share of spoken words carrying the right speaker, so a long misattributed statement counts more than a short one. By speakers is the share of distinct voices in each hearing that were named correctly.
• Precision equals recall. A voice the transcript left unnamed counts as wrong, never as skipped.
• 82 of the 100 hearings had every speaker right. 21 speakers (9,979 words) were wrong across the other 18; each one is listed below.
• 95% interval, from resampling the 100 hearings (10,000 bootstrap draws, seed 20261027, computed after scoring): 98.7% to 99.9% by words, 98.2% to 99.3% by speakers.
• Before the reference corrections described below, the same run scored 99.1% by words and 98.5% by speakers.

Members and witnesses. This breakdown was made after scoring; the pre-registration didn't call for it.

Each voice is classed by who the print says it is. A sitting Member of Congress counts as a member, also when testifying on a witness panel. Witnesses are the people on the hearing's official witness list, plus former Senator Roy Blunt introducing a nominee. Where a name could be either, we read the print. The role of every voice is in roles_post_hoc.jsonl.

Check it yourself

Recompute the headline from one small file. per_hearing.csv has one row per hearing and a TOTAL row last:

• by words = sum(word_correct) / (sum(word_correct) + sum(word_wrong)) = 1,639,116 / 1,649,095 = 99.39%
• by speakers = sum(cluster_correct) / (sum(cluster_correct) + sum(cluster_wrong)) = 1,669 / 1,690 = 98.76%

The same figures are in summary.json, and every hearing's row is in the table at the bottom of this page.

Rerun the scoring. The scorer, the frozen references, our transcripts as scored (speaker labels and text, no timestamps) and the roster files are all published. With Python 3.12, jiwer 4.0 and beautifulsoup4 4.14, for each hearing:

cd code/scorer
python score.py --hearing-id <hearing_id> --chrg <CHRG id> \
  --print <the GPO print for that CHRG id> \
  --segments ../../transcripts/<hearing_id>.segments.jsonl \
  --reference ../../references/<hearing_id>/reference_final.json \
  --roster-cache ../../roster_cache --out out/<hearing_id>
python pool.py "out/*/*-per-cluster.jsonl" --out pooled.json

Download the files from the GitHub repository. For the six hearings with a corrected reference, use references_corrected/<hearing_id>/reference_final.json instead (see "Corrections after scoring"). Each GPO print is on GovInfo (links in the table below). The scorer refuses a print whose text doesn't match the hash recorded in the reference. GovInfo's public mirror rewrites email addresses in three prints, so those three are included in prints/. On 2026-10-27 we reran all 100 hearings from these published files alone: pooled.json and all 1,698 per-voice rows came out identical to the published results.

Check the files. SHA256SUMS lists the SHA-256 of every published file. Run shasum -a 256 -c SHA256SUMS in a copy of the repository.

Which hearings, and how they were chosen

• Population: 143 hearings. These are House and Senate committee hearings of the 119th Congress with a GPO printed record that HillGenius had never processed. Hearings we had already processed were left out, so the system was tested only on hearings it had never seen. The population and its filters are in population_report.md; population.json has SHA-256 c86a8d286e82a9214ecf5d83524c3b30ba4c592d02da1e4937480b005ed58bb9.
• Checked before the draw. Every record was checked by hand: the video plays and is the right hearing (Senate recordings confirmed by their caption text), and the witness list matches the print. Ten of the 153 hearings first frozen were dropped, leaving 143, for the reasons in readiness_exclusions_2026-10-26.jsonl: five had already been processed, four had a print describing a different hearing, and one was an organizational meeting with no witnesses.
• The draw. The seed is the output value of a public NIST Randomness Beacon pulse, 2026-10-27T03:30:00Z (index 2001207, chain 2), a value nobody can know before it is published. sampler.py with that seed shuffles the population and takes the first 100: 79 House and 21 Senate hearings, held between 2025-01-15 and 2026-07-14. Anyone can rerun the draw: the seed is in sample.json.
• No replacements. Replacements were allowed only for a hearing that couldn't be measured at all, such as one with no public video. None was needed, so all 100 drawn hearings were processed and scored.
• Processing. All 100 ran fresh on Tiro 1 in production, frozen at commit 9f3d03d0, with nothing reused from earlier runs. It is the same system customers use.

The reference: the official printed record

Each hearing's right answer comes from its GPO printed record on GovInfo, the official account of who said what. The reference was built and frozen before any HillGenius transcript of these hearings was compared to it.

1. Two independent readings of each print. A rule-based parser reads the print's speaker headers ("Mr. Smith."). A language model (Gemini 2.5 Pro) reads the print text alone and names the speaker of each turn. Member names resolve to Bioguide ids through the same roster lookup on both sides.
2. Where the two readings agree, that is the reference.
3. Where they disagree, a reviewer decides from the print excerpt alone. That meant 2,517 disagreement spans across the 100 hearings. The reviewer was an AI agent (Claude), and every decision is logged with its reasoning. All 2,517 were settled from the print without needing the video. Decisions: parser 1,839, language model 642, split between the two 26, someone else 10. In total 47,141 of 2,156,515 reference words changed speaker.
4. Frozen and hashed before scoring. Each hearing's reference (references/<hearing_id>/reference_final.json) and its SHA-256 were committed before scoring began.

Audit of the agreeing turns. Review covered only the spans where the two readings disagreed, so the turns where they agreed were audited too. We drew 200 agreeing turns at random, weighted by length and seeded from the same beacon value, and checked each against the print. One was wrong (0.5%): a "Chairman Green" resolved to the wrong member, the same error fixed by correction 1 below. See audit_review.md.

Every reviewed span is published with its print excerpt, both readings and the decision: references/<hearing_id>/ holds disagreements.json, review_sheet.md and review_log.jsonl.

How the score is computed

1. Our transcript's words are matched to the print's words in order. A word takes the print's speaker only when it sits in a run of at least 3 matching words and isn't right at a change of speaker.
2. The pipeline splits each hearing into voices and names each voice. A voice's right answer is the person the print says spoke most of its matched words.
3. Our name for the voice is compared with that answer: by Bioguide id when the print's speaker is on the hearing's member roster and our label carries a member id, and by surname otherwise (witnesses, staff, members not on the roster).
4. By speakers counts voices right or wrong. By words counts each voice's words by the majority speaker, right or wrong together.

Member ids, cross-checked. A surname match could credit a different member with the same name, so a separate check compares every scored voice's member id with the ids in the reference the score used (check_member_ids.py, output member_id_check.jsonl). It changes no score. Of the 1,669 voices scored right, 1,189 carry the reference's member id and none carries a different one. 2 name the member in full with party and state but carry no id (two members testifying as witnesses in 64314). The other 478 have no member id in the reference, mostly witnesses. For 12 of those, another member of the chamber shares the surname; we read each print, and each names the member we labeled (member_id_review.md). No voice scored wrong carries the reference's member id.

What this leaves out, stated plainly:

• Words that don't match the print. Our 100 transcripts have 1,807,969 words. 1,649,095 of them (91.2%) are counted in the headline. Most of the rest don't line up with the print, because members edit their remarks before printing, statements are inserted for the record, and transcription differs, or they sit right at a speaker change. Two smaller groups follow.
• Words inside a voice that belong to someone else. When the pipeline folds a few words from another person into a voice, such as a witness's "Yes, sir" inside the chairman's voice, the scorer leaves them out instead of counting them wrong. That is 3,784 words (0.23%). Counted as errors, the by-words figure is 99.2%. Some of these are slips in the reference rather than ours; they were not checked one by one.
• Voices with no clear answer. 8 voices (10,680 words) are discarded as ambiguous: three or more people each have at least 20 matched words in the voice, or two do and the larger holds less than 90% of its matched words. 45 voices whose leading speaker has fewer than 3 matched words are not scored.

The scoring rules were written and the scorer's code hashed before the draw. The full text is the pre-registration, PREREGISTRATION.md. Per-voice rows for all 100 hearings are in per_cluster.jsonl.

What we got wrong: 21 speakers, 9,979 words

Every error, with the speaker the print names and the label our transcript carried:

Errors in the official record

Tiro 1 identified errors in the official record: five passages in GPO hearing prints that credit the words to the wrong person, where Tiro 1 named the person who spoke. Items 1–4 come from HillGenius's September 2026 backlog run, on hearings outside this test. Item 5 comes from this test. Items 2–5 can be checked from the print text alone; item 1 needs the video.

Every item is in official-record-errors.json: the print's SHA-256, the wrong header lines, a line-numbered excerpt, the quotes below, the video timestamps, and the clip and print image. Line numbers count lines of the print's text as govinfo's API serves it (https://api.govinfo.gov/packages/<package>/htm); they are the same in the public govinfo HTML copy linked for each item.

1. "Chairman Wicker." for Sen. Ted Budd · CHRG-119shrg62893 · print on govinfo, lines 2240, 2255, 2288, 2313, 2319, 2323, 2328

Evidence strength: video needed. The print text alone does not settle this one. It is consistent with Budd but never names him as the speaker; the video does. Budd is on camera at the MR. BUDD nameplate speaking this passage. Tiro 1 labels the same voice Budd from 8083.1 s to 8373.4 s, which covers all seven 'Chairman Wicker.' headers from line 2240 to line 2328 ('Thank you again. Senator Slotkin.'). The video check covers line 2240 only.

• Lines 721–722: "Chairman Wicker. Thank you for that commitment to the speed of urgency. Senator Reed." Wicker had already taken his own questioning turn early in the hearing and handed off to Reed. It is consistent with someone else holding the gavel at line 2240, but it does not prove it: a chair can take a second turn.
• Line 117: "Mullin, Budd, Schmitt," Budd is listed as present.
• Line 6946: "Questions Submitted by Senator Ted Budd" Budd submitted questions for the record, yet no spoken turn in the print is headed 'Senator Budd.'. This fits a turn that was printed under another name, but it does not say which turn.
• Line 2513: "Senator Sheehy. [Presiding.]" The print shows the gavel changing hands during this hearing, so senators other than Wicker presided at times. The print never marks Budd as presiding.

Video: Senate video, at 2:14:43 (the Senate player has no start-time link); clip, source seconds 8083.4–8097.6.

2. "Senator Kelly." for Sen. Gary Peters · CHRG-119shrg63037 · print on govinfo, lines 2799, 2844, 2881

Evidence strength: print text alone settles it. The chair recognizes Peters on the line before the header and thanks Peters when the passage ends. In between, the speaker talks about his home-state base in Michigan, and the roster lists Peters for Michigan and Kelly for Arizona. Kelly's own turn was earlier, and no turn in the print is headed 'Senator Peters.' All three 'Senator Kelly.' headers between Wicker recognizing Peters (line 2798) and thanking him (line 2884) belong to Peters. Tiro 1 labels Peters for 8367.1-8484.4, 8546.0-8596.8 and 8666.0-8671.7 s.

• Lines 2797–2798: "Chairman Wicker. Thank you, Senator Sullivan. Senator Peters." The chair recognizes Peters on the line just before the header.
• Line 2804: "Selfridge Air National Guard Base in Michigan" The speaker talks about a base in his home state, Michigan.
• Line 2808: "place those KC-46s in Michigan" Michigan again.
• Line 62: "GARY C. PETERS, Michigan" In the print's committee roster, Peters represents Michigan.
• Line 65: "MARK KELLY, Arizona" Kelly represents Arizona.
• Lines 2605–2606: "Thank you very much, Senator Schmitt. Senator Kelly." Kelly had already been recognized for his own turn earlier.
• Line 2693: "Senator Kelly. Thank you. Thank you, Mr. Chairman." Kelly's own turn ends here, about 100 lines before line 2799.
• Line 2884: "Chairman Wicker. Thank you, Senator Peters." The chair thanks Peters right after the passage ends.

Video: Senate video, at 2:19:23 (the Senate player has no start-time link); clip, source seconds 8363.85–8382.8.

3. "Mr. Fitzpatrick." for Rep. Scott Fitzgerald · CHRG-119hhrg61179 · print on govinfo, line 4210

Evidence strength: print text alone settles it. The chair recognizes Fitzgerald, and the section heading and the prepared statement that follows are both Fitzgerald's. 'Fitzpatrick' appears nowhere in the print except this one header. One header: Fitzgerald's whole oral statement runs under 'Mr. Fitzpatrick.' (lines 4210-4234).

• Lines 4203–4204: "The Chair now recognizes Representative Fitzgerald." The chair recognizes Fitzgerald.
• Lines 4207–4208: "TESTIMONY OF HON. SCOTT FITZGERALD, A REPRESENTATIVE IN CONGRESS FROM THE STATE OF WISCONSIN" The print's own section heading names Fitzgerald.
• Line 4235: "[Mr. Fitzgerald's prepared statement follows:]" Right after the oral statement, the print introduces Fitzgerald's prepared statement.
• Lines 4241–4242: "testify today about bicyclist and pedestrian safety." The prepared statement opens on the same topic, bicyclist and pedestrian safety, as the oral statement at lines 4211-4212.
• Line 175: "Hon. Scott Fitzgerald, a Representative in Congress from the" The table of contents lists Fitzgerald's oral statement.

Video: committee video at 2:17:04; clip, source seconds 8224.1–8239.5.

4. "Mr. McCormick." for Rep. Tom McClintock · CHRG-119hhrg58952 · print on govinfo, lines 2578, 2595, 2620, 2631, 2636

Evidence strength: print text alone settles it. The chair recognizes 'the gentleman from California, Mr. McClintock' and closes with 'The gentleman from California yields'. McClintock (CA) is present and on the roster. 'McCormick' appears only as these five turn headers, and he is not on the attendance list. McClintock's whole questioning turn, from the chair's recognition (lines 2576-2577) to 'The gentleman from California yields' (line 2652), is headed 'Mr. McCormick.' five times. Tiro 1 labels McClintock for 2949.0-2990.3, 3013.5-3088.2, 3130.6-3150.9 and 3202.4-3261.4 s.

• Lines 2576–2577: "we will start out with the gentleman from California, Mr. McClintock." The chair recognizes McClintock on the lines just before the header.
• Line 656: "Present: Representatives Tiffany, McClintock, Fulcher," McClintock is listed as present. No McCormick appears in the attendance list.
• Line 65: "Tom McClintock, CA" The subcommittee roster lists McClintock as a California member.
• Line 2599: "sawmills operating in California" The speaker talks about his own state, California.
• Line 2652: "Mr. Tiffany. The gentleman from California yields" The chair closes the turn by naming 'the gentleman from California'.

Video: committee video at 49:02; clip, source seconds 2942.2–2986.4.

5. "Mr. Baird." for Judge Paul Black · CHRG-119hhrg61126 · print on govinfo, lines 755, 1261

Evidence strength: print text alone settles it. The chair calls on Judge Black, and the section heading names Black. The speaker says "I'm Paul Black," his prepared statement follows, and the chair thanks Judge Black. Baird introduced himself as Douglas Baird in his own earlier statement. Two passages. Black's opening statement (lines 755-851) and his answer to the chair's question at line 1254 (line 1261) are both headed 'Mr. Baird.'. Tiro 1 labels Black for 2782.241-3077.826 s and 4534.0-4598.402 s.

• Lines 750–751: "Mr. Fitzgerald. Thank you, Professor Baird. Judge Black, you may begin." The chair thanks Baird for his finished statement and calls on Judge Black.
• Line 753: "STATEMENT OF THE HON. PAUL M. BLACK" The print's own section heading names Black.
• Lines 756–757: "I'm Paul Black, United States Bankruptcy Judge for the Western District of Virginia." The speaker introduces himself as Paul Black.
• Line 673: "I'm Douglas Baird" Baird's own opening statement, earlier in the print, has him introduce himself as Douglas Baird.
• Line 852: "[The prepared statement of Mr. Black follows:]" Black's prepared statement follows this passage.
• Line 855: "Mr. Fitzgerald. Thank you, Judge Black. Judge Harner." The chair thanks Judge Black when the statement ends.
• Line 1254: "Mr. Fitzgerald. Thank you. Judge Black, there have been" Second instance: the chair puts a question to Judge Black...
• Line 1261: "Mr. Baird. No, I really haven't in my court." ...and the answer is headed 'Mr. Baird.'. Baird is a law professor and has no court.
• Line 1265: "Western District of Virginia" The answer refers to Black's own district.

Video: committee video at 46:16, second passage at 1:15:33; clip, source seconds 2776.4–2791.9.

Tiro 1's labels for items 1–4 are from the production transcripts Tiro 1 produced in September 2026; those hearings have not been reprocessed since. Item 5's label is from this test's transcript (hearing page). Review settled the "Mr. Baird." header from the print's own text before any transcript was scored, so this test's reference credits Judge Black (review_log.jsonl, span d002). Where a test print misspelled a speaker's name, review settled it the same way, and the reference used the right name.

Nobody searched the prints for errors. These turned up while checking Tiro 1's transcripts against them, so there may be more.

Corrections after scoring

The pre-registration said the score would be computed once, with no corrected version. We departed from that, and both numbers are published.

Before corrections: 99.1% by words (1,620,934 of 1,635,050) and 98.5% by speakers (1,663 of 1,688). Files: results/locked_run/.

After that run we checked every member id in the references against each print's own list of members (bioguide_check.txt). In several cases our transcript matched the official print but was scored wrong because the reference had an error. Three corrections were applied by rule to all 100 hearings, not case by case:

1. Member identity. Our member database lacked the surname-matching field for Mark E. Green (TN), Sylvester Turner (TX) and Marjorie Taylor Greene (GA), so the reference resolved "Green" and "Turner" to Al Green and Michael Turner. Both prints' committee rosters list Mark E. Green (Chairman) and Sylvester Turner, and the two references that carried the wrong member ids (60649 and 60692) were corrected to them.
2. One speaker labeled two ways in the reference (for example "RUBIO" and "Rubio"), which split a voice's vote.
3. A print error. The 59601 print heads the witness Nicholas McMurray's turns "Mr. Murray." The pre-registration said print errors noticed after scoring would stand against us. This one was noticed after scoring, and correcting it changed no result: that voice stays ambiguous. Print errors that review noticed before scoring were settled from the print's own text (see "Errors in the official record"). None of the 21 wrong speakers traces to a print error.

Six voices changed, and none went against HillGenius:

Corrections can only be found by looking, and a search of references that turns up only errors in our favor could be one-sided. So the blind audit above matters: in 200 randomly drawn turns, checked without seeing our transcripts, it found one error, the same Green error. The frozen reference files are unchanged. The six corrected references are published in references_corrected/, the rules in corrections.json, and the script in rescore_corrected.py.

One other departure. The pre-registration promised a calibration of the parser against the AI reading on the 207-hearing development pool. Only a 5-hearing pilot was run. The two agreed on 95.9% of words at first, and on 98.8% to 100% after parser fixes on the model used, and the parser was right in all 9 disagreements checked by hand. This does not affect the score.

Limitations

• Scope. Committee hearings with a GPO print, held from early 2025 to mid 2026. GPO prints appear months after a hearing, so this doesn't measure live transcripts, floor proceedings or the most recent hearings.
• Witness lists. For 34 hearings our stored witness list was incomplete and was completed from the print's witness list before processing. Nominee introducers were not added, which is why former Senator Roy Blunt is counted wrong.
• Videos. The check before the draw confirmed each video was the right hearing, not that it started at the gavel. Two started mid-testimony, and both produced errors listed above.
• The reference can be wrong where both readings agree. The audit found 1 error in 200 agreeing turns (0.5%), and it was the Green error already corrected. Other errors of that kind would remain, in either direction.
• Sample size. 100 hearings; see the interval above.
• We tested ourselves. HillGenius chose the method, built the reference and ran the scoring. The reviewer of disputed spans was an AI agent working for us, not an independent party.
• The pre-registration was not public in advance. The rules, population, sampler and scorer hash were committed to HillGenius's private repository before the beacon pulse, not posted publicly, so you can't check those timestamps yourself. You can check the draw: the population and the public beacon value reproduce the sample exactly.

What isn't published

Word and segment timestamps, the time boundaries of each voice, the code that aligns the print to the audio, the rows of our development hearings, and the event census the population was filtered from (our list of committee events, with which ones we had processed) together with the scripts that filter it. population_report.md gives the count at every filter step. The time alignment has training value to a competitor, and nothing on this page depends on it: the score above was rerun without it.

Some published files have internal details cut, and a few are held back. None of it is needed to check the score:

• Held back from code/scorer/: the tools that build the reference (the language-model reading and the scripts that compare and freeze the two readings), the scorer's internal notes and its tests. The score is recomputed from the frozen references, which are published.
• code/scorer/roster.py: its database lookups are cut. That code runs only when a hearing has no roster file, and every hearing's roster is in roster_cache/. So in code/scorer/, shasum -a 256 -c --ignore-missing SHA256SUMS passes for every file except roster.py.
• roster_cache/: the database queries that built each roster are removed; the rosters are complete.
• PREREGISTRATION.md: internal details cut, each marked [redacted: ...]. The SHA-256 of the unredacted original is at the top of the file.

Auditors can see the full kit, including timestamps and commit history, under an agreement. Ask through our contact page.

Files

Every file, on GitHub: https://github.com/hillgenius/speaker-attribution-test-2026-10 holds every published file, including each hearing's transcript, reference, review files and roster, with a file guide in its README.

Every hearing

Each title opens our transcript of that hearing as scored, next to its score and a link to the official print.

Version, accountability and corrections

Version 1.0, published 2026-10-28. Hearings processed by Tiro 1 and scored on 2026-10-27 at pipeline commit 9f3d03d0; test kit commit 5826159a. Run by HillGenius. Accountable: Zack Dareshori, founder. If you find an error in this page, the data or the scoring, tell us through our contact page. A correction will be dated here, with the earlier version kept.