Skip to main content
Fit to CareCare into Action

The Record

Did the Prime Minister answer? On the same 27 answers, two AI readers agreed 93% of the time; pairs of people agreed 30% to 63%

Anthony Lawton, Fit to Care

We are testing whether AI can help people judge political answers more consistently, and so reward answering over winning. Two AI readers agreed with each other on 82% of 9,480 answers from Prime Minister's Questions. Pairs of human markers, checking a sample, agreed on between 30% and 63%. When one marker read the machine's reasons, he changed his mind on 7 of the 11 differences he looked at again.

What we found

  • The machines agree with each other. Two AI readers, working separately to one written rulebook, agreed on 82% of 9,480 answers from ten years of Prime Minister's Questions. Allowing for chance, that is "substantial" agreement on the usual scale (kappa 0.74).
  • People find it much harder. On the 27 answers all three of our human markers have marked, all three agreed on only 6 (22%). Pairs of us agreed on 63%, 37% and 30%; allowing for chance, two of the three pairs were no better than chance and the third was only fair. On those same 27 answers, the two AI readers agreed on 25 (93%).
  • Reading the machine's reasons changed minds. Marking blind, my first choice matched the AI on 16 of 29 answers (55%). After reading its reasons, I agreed on 23 (79%). Of the 11 differences I looked at again, I accepted the AI's reading in 7.
  • Non-answers follow patterns. Where an answer fell short, the most common patterns were changing the subject (2,175 answers), attacking the questioner (1,356) and a narrow reassurance in place of the thing asked (773).

How far do readers agree, beyond chance?

The weak spots are real and are set out in full below: the human sample is small, the markers were not chosen at arm's length, we marked from a four-line summary of the rule (two of us untrained) while the AI readers worked from the full rulebook, and two AI readers agreeing shows they are consistent, not that they are right.

Why we are asking

Socrates thought understanding grew from plain questions and honest answers. The Sophists taught people how to win. Public argument today often rewards the second skill: an exchange is scored by who came out on top, not by whether the question was answered.

Fit to Care is running a small public experiment on that problem. Can AI help people see more clearly, and more consistently, when a question has been answered? And if the patterns were visible to everyone, would questioners and answerers behave a little more like Socrates and a little less like the Sophists?

Before any of that, the measure has to be trusted. So we tested the machines against people, and the people against each other.

One answer, three verdicts

In December 2018 a Labour MP asked the Prime Minister a direct question. The MP reminded the Prime Minister of two earlier statements that had not held: no general election, then one; no pulling of the meaningful vote, then it was pulled. Would she now rule out a general election and a people's vote?

The reply came in two parts. A general election "at this point in time, in the middle of our negotiations, would not be in the national interest". And "we should respect the result of the referendum that took place in 2016". (Hansard, House of Commons, 12 December 2018.)

I marked it partial: "would not be in the national interest" is not "I rule it out". The second marker judged it no answer at all. The third judged it a full answer, and so did both AI readers. When I then read the AI's reasoning, that the Prime Minister had stated her position on both options, I changed my mind.

The record and the check

Our public record of Prime Minister's Questions covers 2016 to 2026. Two AI readers, both Google Gemini models, read every answer, 9,480 of them, against one written rulebook. Most answers are coded as a full answer, a partial answer or no answer. The record is free to read. Nobody commissioned or paid for it. Fit to Care sells analysis to the public sector, some of it AI assisted; the record is not for sale.

To check the machines, three of us are marking the same 50 answers, drawn at random: 25 to questions from the Prime Minister's own side and 25 from opposition MPs. Each of us chose before seeing the AI's verdict or anyone else's marks. The markers are me (I built the record), a consultant who is bidding for work jointly with Fit to Care (no work or payment has changed hands yet), and a PR professional who knows me personally but has no business connection to Fit to Care. The other two did this as a favour.

The second marker has now marked all 50, matching the AI readers on 26 (52%); the third marker, on 27 so far, has matched them on 13 (48%).

When the two AI readers disagree

The two readers never see each other's work. When they disagree, a fixed rule decides, and nobody at Fit to Care settles a disputed answer by hand:

  • One says full, the other partial: the record shows partial.
  • One says partial, the other no answer: the record shows "partial or no answer". Both agree the question was not answered in full, so the count of answers not answered in full is the same either way. There are 696 of these.
  • One says full, the other no answer: the record shows "could not be judged". There are 212 of these in the 9,480 answers. From October 2026, new cases go to a third reader, which lists each thing asked, checks whether each was answered, and counts how many were.

How to read the agreement figures

Kappa is the usual way to measure agreement beyond chance: 0 is chance, 1 is perfect, and below 0 is worse than chance. On a widely used scale (Landis and Koch, 1977), the two AI readers, at 0.74 across all 9,480 answers, reach "substantial" agreement. Two of our three human pairs are no better than chance.

That gap is not a verdict on the people. In Peter Bull's long-running research on how politicians reply to questions, two raters coding two broadcast interviews reached a kappa of 0.82 (Bull, 1994, reported in Bull and Waddle, 2019). Non-expert annotators sorting US presidential interview answers into clear reply, ambivalent or clear non-reply reached 0.64 across three coders (Thomas and others, 2024). Two of our markers were volunteers with no training, and all three of us marked from a four-line summary of the rule, while the AI readers worked from the full rulebook. Our guess, untested here, is that training and the full rule would narrow the human gap. A machine can apply one written rule the same way at scale; whether it is the right rule is a separate question.

What changed when I could read the machine's reasoning

For the last 30 answers I marked, the page showed me the AI's reasons after I had made my choice, and I could note whether they changed my mind. The other two markers were also shown the reasons after making their choice; whether the reasons changed their minds is not counted here.

Matching the AI where I could read its reasons

Blind, my first choice matched the AI on 55% of these answers (16 of 29; one I left as "can't tell"). After reading its reasons, I agreed on 79% (23). Of the 13 differences, I looked again at 11. In seven I accepted the AI's reading, often because it kept closer to what was literally asked; in four I did not. Two I did not revisit. Agreeing after reading a fluent reason is not proof the reason was right.

Four times I still disagree. The clearest is from March 2020. An MP asked for an assurance that GP surgeries would get protective equipment that worked, after, the MP said, at least one GP surgery had received masks with 2016 expiry dates. The reply began "To the best of my knowledge" and said the equipment was of the correct standard (Hansard, House of Commons, 18 March 2020). Both AI readers called that a full answer. I do not: an assurance given only "to the best of my knowledge" is not, to me, a full answer. Across all 49 answers I judged, the AI was the stricter reader in 13 of our 20 first differences. Put simply, the machine was stricter than me in most of my differences; the masks answer shows one way I can be the stricter one.

The bigger question

Our AI readers also note how an answer falls short. Of the 4,752 answers the record shows as partial (2,126) or no answer (2,626), not counting the 696 shown as "partial or no answer", 3,695 carry at least one noted pattern. The most common: the subject was changed (2,175), the questioner was attacked (1,356), and a narrow reassurance was given in place of the thing asked (773). These describe answers, not people, and the same rules are applied to everyone.

Here is what that looks like for one session. We chose it by rule rather than picking: the most recent session in the record, 9 September 2026, and its first ten questions. Each answer takes the colour of its published code, and where the readers noted a pattern, the chart says which.

One session, ten questions

The first ten questions include all six from the Leader of the Opposition, so they are not typical: across all 24 questions that day, 8 answers were full, 8 partial, 5 no answer and 3 could not be judged.

These colours judge the answers only. A planned development is to give each question a colour too, judging the question and never the person asking it, against published criteria of its own: was it a clear question that could fairly be answered? That is not yet part of the record, and it will be tested before it is added.

Here is the hypothesis we want to test, and cannot yet prove. If those patterns were made visible, consistently and without fear or favour, would behaviour shift on both sides of the exchange? Questioners might ask cleaner questions, and answerers might change the subject less, because the change would be seen. That would be a small direction of travel towards the Socratic habit, and towards public trust earned by answering.

None of that matters unless the measure itself is trusted. Hence this check, and its candour.

Weak spots and limits

  • The sample is small. The 27 answers all three of us have marked are the first 27 the third marker finished, not a fresh random draw, and 23 remain, so the figures will move.
  • The markers were not chosen at arm's length.
  • We marked from a four-line summary of the rule, and two of us had no training; the AI readers worked from the full rulebook.
  • Consistency is not correctness. The AI readers apply the same rulebook to every speaker, every party and every year. Two readers agreeing shows they apply it consistently. It does not show the rulebook is right or even-handed, and because both readers are Gemini models, some of their agreement may come from a shared make. This check tests none of that.
  • Nothing here tests whether anything said was true.

Try it yourself

We have put ten of the answers, with the rule and a key, into a short exercise that is free to use. Write to anthony.lawton@ffmi.co.uk and we will send it. Mark them before you look at the key, then see who you agreed with. If you teach, it makes a good half hour.

The harder question is one I cannot answer alone. What would it take for a measure like this to be trusted? How many markers, how trained, chosen by whom, and what level of agreement? If you know someone who should mark the 50 with no link to us, I would like to hear from them at the same address.

Who was told first

The chart above shows one identifiable session. Before this piece went up, we sent the findings for that session to the offices of the Prime Minister, the Leader of the Opposition and the Liberal Democrat leader, at least 48 hours ahead. Apart from an automatic acknowledgement, no reply had been received when this piece was published. Any reply we receive will be added here.

Fit to Care, a public sector consultancy, publishes the record as a public service. No party, person or client commissioned or paid for it.

References: Landis, J.R. and Koch, G.G. (1977) Biometrics 33(1), 159 to 174. Bull, P. (1994) On identifying questions, replies, and non-replies in political interviews. Journal of Language and Social Psychology 13(2), 115 to 131. Bull, P. and Waddle, M. (2019) Journal of Language Aggression and Conflict 7(1), 56 to 78. Thomas and others (2024) "I Never Said That": A dataset, taxonomy and baselines on response clarity classification. Findings of EMNLP 2024, ACL Anthology 2024.findings-emnlp.300.

Contains Parliamentary information licensed under the Open Parliament Licence v3.0.

Sources

  • Hansard, House of Commons Official Report, Prime Minister's Questions, January 2016 to September 2026, read through the Fit to Care public record of Prime Minister's Questions (method, rulebook and corrections log at /resources/pmqs-record).
  • Fit to Care human check of the record: 50 answers drawn at random, marked by three people, figures as at 14 October 2026; the three-way figures are the 27 answers all three had marked by 7 October 2026, unchanged since.

The data behind this piece: Did they answer? A public record of Prime Minister's Questions

No record is perfect. If you find an error, write to anthony.lawton@ffmi.co.uk with the subject "The Record correction" and it is corrected within 48 hours, in public, on the piece itself.