A quitting study reports many tests: read the testing plan before picking one result

List the claims that were tested together

A study may measure use, satisfaction, engagement and several follow-up points. Not every number is a separate confirmatory test, and descriptive reporting should not be treated as though it necessarily is. Locate the analysis plan’s distinction between the principal claim, other planned claims and exploratory analyses before counting attractive results.

FDA’s October 2022 methods guidance explains the false-conclusion concern when multiple endpoints are tested and describes planned ways to handle it. The guidance concerns drug trials; we borrow only the statistical reading concept. It supplies no vaping result, approval status or universal rule for correcting every collection of numbers.

US Food and Drug Administration: Multiple Endpoints in Clinical Trials (October 2022)

Make an original claim-family map

Draw a central box for the claim being promoted. Around it, list the outcomes and time points the report used to support that claim. Beside each, record whether the analysis plan labels it primary, secondary, exploratory or otherwise. Then add the actual stated strategy for making several tests interpretable together, keeping an unclear strategy marked unclear.

In a fictional report, three outcomes were all planned, but a post highlights only the favorable test. Checking that the highlighted outcome appears in the plan does not answer how testing three claims was handled. The missing question concerns their joint interpretation. The invented report has no treatment finding and does not show that any real authors selected results improperly.

Distinguish ordering from selective omission

A prespecified sequence may make later confirmatory conclusions conditional on earlier tests. Other approaches can distribute testing allowances across a defined family. Readers do not need to choose a statistical method to ask which strategy was used. Preserve the authors’ terminology and seek methodological clarification if the implications are beyond the report’s explanation.

This differs from simply finding whether results were reported. Complete reporting is valuable, but showing every test does not by itself control the chance of at least one erroneous conclusion across them. Conversely, several reported outcomes do not automatically invalidate a study. The intended inference and analysis plan determine what question needs to be assessed.

Repeat the result with its status intact

If a result is described as exploratory or outside a confirmatory testing sequence, keep that description when sharing it. Do not promote it to the study’s definitive overall conclusion. If the public summary omits the testing plan, ask for the relevant methods rather than inventing an adjusted p-value or claiming a particular conclusion must be false.

The useful output is a map linking the promoted claim, related tests and documented multiplicity strategy. It is not a formal statistical appraisal. Health choices require appropriate professional interpretation of the complete evidence. You can still organize ordinary support and routines while leaving a technical finding unresolved rather than borrowing it as a personal success forecast.