The worst questionnaire error: quoting your own SOC 2 exception as if it were the control

The passage in your audit report that most exactly matches a control question is often the auditor's finding that the control failed. Text search cannot tell the difference. It is actively drawn to it.

A failure mode with no natural floor

Most questionnaire mistakes are boring: a stale date, a vague sentence, an inconsistency between two rows. They cost time. This one is different in kind, because the answer it produces is not weak — it is confidently inverted. You assert a control is in place by quoting the paragraph proving it was not.

It is also the mistake most likely to be made by a tool rather than a person, which is precisely why it deserves attention now.

Why search engines are attracted to the wrong paragraph

Consider the question “Is multi-factor authentication enforced for all administrative access?” and two candidate passages from your report. One is the control description. The other is an exception: “Multi-factor authentication was not enforced for all administrative access during the review period.”

Nearly every relevance-scoring method will rank the exception higher. It shares more vocabulary with the question. Common retrieval pipelines discard words like not as noise before scoring, so the single word carrying the entire meaning is the one word thrown away. The more precisely an exception contradicts the question, the better it scores.

We found this in our own engine by testing it, not by reasoning about it: a seeded exception scored a perfect match and drafted an answer asserting the opposite of the source. The lesson generalises past our implementation — if any tool drafts from your SOC 2 by similarity, assume this failure exists until shown otherwise.

Two signals that catch it, and why you need both

Polarity. Compare the negation of the question against the negation of the passage. If one negates and the other does not, something is wrong. This is cheap and catches the blunt cases.

Provenance. Audit reports rarely write “not”. They write deficiency, exception noted, management response, did not operate effectively. Word lists never finish; we measured our own first attempt catching one realistic exception wording in eight. The durable signal is structural: which section did the passage come from? A paragraph under Exceptions Noted describes what failed regardless of how the sentence is phrased.

Neither signal is complete, which is the point. Both route to a human rather than rewriting anything, because a heuristic that silently “fixes” evidence is a worse problem than the one it solves.

What to do by hand

When you copy a sentence out of your SOC 2 into a questionnaire, look at the heading above it before you look at the sentence. If that heading is about findings, exceptions, deficiencies or management responses, the paragraph is describing a gap — and the honest answer is a gap answer with your remediation, not a quotation.

That answer is not a lost deal. An acknowledged exception with a dated remediation plan reads as competence. The same exception discovered by the reviewer inside your own attached report reads as something much worse.