An AI quitting tool claims fairness: look for group-specific performance
Read the population, metric and uncertainty behind a fairness claim rather than treating one overall accuracy score as evidence for every group.
No Vape · Updated

Name the task and the fairness claim
A system might classify a recorded event or estimate an outcome. Before reading group comparisons, locate what it actually does and how the report defines the relevant groups. A general promise that an AI works for everyone does not identify an evaluation. Do not assign yourself a result simply because a broad demographic label appears in a table.
NIST's AI Risk Management Framework playbook recommends context-specific fairness metrics and examination across, within and intersecting groups. It includes error-rate questions among possible evaluations. This is general AI-evaluation guidance, not NIST validation of a vaping product or a rule that one numerical comparison settles every fairness question.
Build a group-by-metric reading card
Write the test task, metric definition, group definition, number of relevant observations and stated uncertainty. Keep unavailable cells marked unavailable. An overall accuracy value cannot fill in a missing group result. Nor does a single group average explain every person within that group or every combination of characteristics.
For a fictional example, a classifier report supplies an overall error summary but no group-specific errors. A promotion calls it equally reliable for all users. The reader can verify the overall result while leaving the equal-performance claim unresolved. The scenario supplies no real model outcome and does not establish that a particular group was harmed.
Keep the comparisons on the same footing
Check whether groups were evaluated on the same task, outcome definition and operating conditions. A rate based on few relevant cases carries a different uncertainty question from a broad total. Avoid ranking groups by rounded percentages without the report's supporting information. A missing comparison is a limit on the claim, not permission to guess its direction.
Different fairness metrics ask different questions; a model can satisfy one comparison without establishing another. You do not need to select a universal fairness formula or audit the system yourself. Record the actual reported criterion and ask how it relates to the provider's stated use. Keep technical interpretation with qualified expertise when the details exceed the public explanation.
Separate evaluation from your support decision
Even a well-described group comparison would not show that using a prediction improves quitting outcomes or safely selects treatment. A treatment-effect subgroup analysis asks another question about an intervention, not merely how a prediction system makes errors. Preserve both distinctions when a product advertisement blends them into one personalized promise.
You can decline to provide private histories while a fairness claim remains unclear. A concise question to the provider can name the task and ask which tested populations and group metrics support the exact claim. This card does not assess a named tool, provide an individual forecast or assign a clinical category. It identifies the evidence needed to repeat the fairness statement accurately.
Sources and scope
Sources inform the guidance. The worksheets and examples are No Vape editorial suggestions, not tested treatments.
Written with AI assistance and checked against the cited sources. No clinical review is claimed. This guide covers practical planning, not individual medical treatment. How we research and write