Part of a punched card printed with rows of digits, with rectangular holes punched through many of its columns.
Detail of an eighty-column punched card, a format long used to record data for tabulating machines and early computers. Wikimedia Commons.

Methods notes

Three short notes on problems that recur in work with health data. Each sets out the problem in general terms and how the firm handles it.

Written November 2026

Case definitions

Before a condition can be counted, someone has to decide what counts as having it. That decision is the case definition, and there is seldom only one reasonable choice. Diabetes shows the problem well. A health survey can identify it in at least four ways.

  • The respondent says a doctor has diagnosed it.
  • The respondent takes insulin or another glucose-lowering medicine.
  • A blood test finds glycated hemoglobin (HbA1c) at or above 6.5 percent.
  • A fasting blood test finds plasma glucose at or above 126 milligrams per deciliter.

Each of the four picks out a somewhat different group. Some adults with a diagnosis have laboratory readings under both thresholds because their treatment is working. Others meet a threshold and have never been told. The two laboratory tests also disagree with each other, so adding or dropping one changes the count.

A definition based on diagnosis alone counts those who know they have the condition, and one that accepts any of the four counts the most. The two answer different questions. Estimates built on different definitions cannot be compared as though they measured one thing. A trend can mislead in the same way when the definition changed partway through.

The choice is also limited by what was measured. In a national survey, fasting glucose is measured only in the respondents asked to fast, and that subsample has its own weights and wider margins of uncertainty.

The firm states the definition before the analysis begins. The main estimate is then reported under each reasonable alternative, with a note of how far it moves.

Withheld counts

Agencies that release health data often withhold low counts so that no patient can be picked out. The Centers for Medicare & Medicaid Services, for one, does not show a count from 1 to 10 in the files it releases for general use. A row based on so few claims is dropped, or the cell is left blank.

The rule protects privacy, but once rows are missing, the rows that remain no longer add up to the true total.

Prescribing data listed by prescriber and by drug show the effect. A prescriber with a handful of claims for each of several drugs can have most of those rows withheld. The total for that prescriber is given in a separate summary file and still includes them. The sum of the drug rows then falls short of the stated total.

One prescriber, five drugs

DrugClaims shown
Drug A212
Drug B57
Drug Cwithheld
Drug Dwithheld
Drug E18
Sum of the rows shown287
Total in the summary file301
Not shown14
Illustration. The numbers are made up to show the arithmetic.

The shortfall is not spread evenly. It is larger for drugs that are seldom prescribed, for prescribers with few patients, and for places with few residents, which are often the cases of most interest.

Wherever the agency gives its own total, the firm compares it with the sum of the detailed rows. The size of the gap is then known and can be reported. Missing rows are not counted as zeros.

Adverse event reports

When a patient has a bad reaction while taking a medicine, a clinician, the patient, or the manufacturer can report it to the Food and Drug Administration. The reports are gathered in the FDA Adverse Event Reporting System, which holds millions of them. It is one of the main ways a rare harm is first noticed once a medicine is in wide use.

A count of reports cannot be read as a rate. The system does not record how many patients took the medicine, so there is no denominator. Many reactions are never reported, and reporting rises when a medicine is new or in the news. Some events are reported more than once. A report records a suspicion and does not establish that the medicine caused the event.

The reports can show disproportion, an event that makes up a larger share of the reports for one medicine than of the reports for others. The reporting odds ratio measures it from a table of four counts.

The four counts behind a reporting odds ratio

Reports of the eventReports of other events
The medicineab
The comparisoncd

Reporting odds ratio = (a ÷ b) ÷ (c ÷ d)

A ratio above 1 means the event makes up a larger share of the reports for the medicine than of the reports for the comparison.

A ratio above 1 is a reason to look closer. The number also rests on choices that are easy to leave unstated.

  • Which reported terms count as the event.
  • Whether the medicine must be the primary suspect or may appear in any role.
  • Which other reports form the comparison.
  • Whether duplicate reports were removed, and how.

The firm fixes these choices in the plan. The ratio is then calculated again with each choice changed in turn, and the results are set side by side.

These notes describe methods in general terms. They report no results, and nothing in them is medical advice.