Method · From the series on measurement and adequacy
A score that drops is not, by itself, good news.
Two measurements of the same organization can differ for many reasons, and only one of them matters. A program that can tell that reason from the other four demonstrates results. One that cannot produces charts.
The five conditions
Scope: both measurements cover the same set of domains and services. Mode: both public, or both complete, never one of each. Coverage: collection modules reached equivalent proportions in both. Engine version: the same. Calibration: the same. If all five hold, the difference between results is attributable to what changed in the environment. If any one fails, it is not.
How coverage misleads
This is the most common failure mode and the hardest to notice. When a collection module does not complete, whether from a filtered port, a timeout, or a service that was down at that moment, the model applies a fallback value. A fallback is not neutral. It is an assumption, and an assumption can be more or less favorable than the reality it replaced.
The practical consequence is uncomfortable. An organization that exposed more services publicly between two collections may have more observed signals in the second run, and a different result without anything in the environment having improved or worsened. The reverse holds too: a service temporarily unavailable during the second collection lowers coverage and moves the result without any technical decision having been made.
Why public and complete never compare
In public mode, data shelf life enters through a sector prior, an estimate derived from what is typical for that sector. In complete mode, it enters through the organization's own declaration, data category by data category. The two readings start from different kinds of evidence and belong to different rulers.
This means adding a declaration does not refine the public result. It swaps the instrument. The number may rise, fall, or stay the same, and none of those three movements says anything about the environment having improved. Presenting that transition as progress is the costliest error on this list, because it looks like a good result.
The objection
My score dropped after the program. Doesn't that prove it worked?
It proves it if, and only if, the five conditions hold between the two measurements and the drop is concentrated in the contributors the program actually addressed. A drop spread across signals nobody touched points to a change in coverage or method. The useful question is not how much it dropped. It is where it dropped, and whether the scope matches what was changed.
What to require of any comparison
- Scope declared at both ends, and identical between them.
- Mode declared: public or complete, never mixed.
- Per-module coverage in both measurements, with the fallbacks applied.
- Engine version and calibration explicit in both results.
- The contributors that changed, to check they match what was altered.
Outside the scope of this text
- Even with all five conditions satisfied, attribution holds for the measured scope, not for the whole organization.
- The sector ruler is a static reference base with a cutoff date. It does not update between one run and the next.
From the series on measurement and adequacy
Why impact belongs to the scope that changed
Remediating one asset does not reduce organizational exposure proportionally. Why aggregating impact requires a rule that does not yet exist, and what to report until it does.
Projection is not execution
The governance roadmap projects the effect of declared actions. What that projection demonstrates, what it does not, and why confusing the two is the easiest mistake to make.
Discuss an assessment
If this text described a problem you have, the conversation starts from the scope you need to measure, with the limits written down before any proposal.