Inter-rater reliability in vaping research: who made the ratings?
Identify different observers rating the same occasion before treating a reliability claim as a validated outcome or a quitting benefit.
No Vape · Updated

Find the people behind the number
A paper might describe observers watching behavior, coders classifying interview answers or people rating the same material. Locate the actual task. An observer identifying an action is not the same as a participant reporting an internal feeling. A broad label such as reliable measurement can hide whose judgments were compared.
COSMIN’s measurement taxonomy distinguishes ratings by different people on the same occasion from repeated measurement over time. We use that limited terminology here, not a formal assessment of any study. This guide sets no coefficient threshold and does not train readers to classify someone else’s vaping or assess dependence.
COSMIN Initiative: COSMIN Manual Version2.0 — reliability conditions
Draw a rater-by-material map
For each reported comparison, note which raters assessed which material, what instructions they had and whether the report says their judgments were independent. Keep the target occasion visible. If different people rated different events, simply noticing different scores does not isolate a difference between raters.
Consider an invented exercise in which two coders receive the same anonymized descriptions of everyday breaks. They classify each description using a stated rule. Their consistency concerns that coding task. It does not reveal whether a person felt an urge, whether an unmentioned event occurred or whether the written descriptions captured the whole day.
Keep an adjudicated answer separate from initial ratings
Some reports describe a later discussion that resolves disagreements. Preserve the distinction between the ratings used for the reported reliability assessment and a final agreed classification. A consensus table is not automatically a record of what each person independently judged. Ask which stage the number actually describes rather than accusing the authors of concealment.
Also check whether the comparison concerns trained research raters under particular instructions. Evidence from that arrangement does not establish that untrained friends, a new coding guide or an automated app will produce the same results. Keep unavailable details marked unknown; do not reconstruct them from a confident abstract.
Write a bounded reading conclusion
A useful note names the task, raters, shared material or occasion, reported assessment and population. It can end with an unanswered question about the coding stage. That is enough to clarify a claim without calculating a replacement statistic or asking for identifiable participant material.
Repeatability, validity and quitting benefit remain separate questions. Comparing two instruments also needs the actual measurement-comparison evidence, rather than borrowing a rater label. Your own ordinary notes can describe what you experienced without acquiring research credentials. Any consequential clinical interpretation belongs with an appropriately qualified professional and the real instrument context.
Sources and scope
Sources inform the guidance. The worksheets and examples are No Vape editorial suggestions, not tested treatments.
Written with AI assistance and checked against the cited sources. No clinical review is claimed. This guide covers practical planning, not individual medical treatment. How we research and write