Medical Device Vigilance Manual

Signal Detection

Chapter 4 — Quantitative and Qualitative Frameworks

4.1 The 2×2 Contingency Table

Disproportionality analysis begins by arranging spontaneous-report data into a 2×2 contingency table. Each cell counts the number of device–failure-mode pairs observed in the reporting database.

Failure Mode of Interest All Other Failure Modes
Device of Interest A Device + Failure B Device + Other Failures
All Other Devices C Other Devices + Failure D Other Devices + Other Failures

4.2 Proportional Reporting Ratio (PRR)

PRR =  A / (A + B)  ÷  C / (C + D)
Variables are colour-coded to the contingency table above.

SDR (Signal of Disproportionate Reporting) Criteria

All three thresholds must be met simultaneously to flag a signal:

PRR ≥ 2
Proportional Reporting Ratio
χ² ≥ 4
Chi-squared statistic
N ≥ 3
Minimum case count (cell A)
All three conditions must be satisfied concurrently.
Practical Example — Nebulizer A Assembly Errors: Compare the proportion of assembly-error reports for Nebulizer A against the proportion of assembly-error reports for all other nebulizers in the database. A PRR ≥ 2 would indicate that Nebulizer A's assembly errors are reported at least twice as frequently as expected relative to its peers.

4.3 Key Limitations of Disproportionality Analysis

⚠

Underreporting

Spontaneous reporting captures only a fraction of real-world events. For non-serious incidents, an estimated 90–99% go unreported, severely limiting the denominator and distorting rate calculations.

↻

Weber Effect

Reporting peaks shortly after product launch as users encounter the device user interface for the first time, then declines over time — not because failures decrease, but because novelty-driven reporting wanes.

★

Notoriety Bias

High-profile recalls or media coverage of a device can stimulate a surge of reports for that specific product, inflating disproportionality scores independently of any true change in risk.

●

Small Cell Instability

When cell counts (especially cell A) are very small, the PRR becomes highly volatile. A single additional report can dramatically shift the ratio, producing unreliable signals.

4.4 Bayesian Approaches to Signal Detection

Bayesian methods introduce prior distributions that “shrink” extreme estimates toward the overall database rate, producing more stable results when cell counts are small.

WHO-UMC Method

BCPNN

The Bayesian Confidence Propagation Neural Network uses shrinkage algorithms to compute an Information Component (IC). The IC measures the degree to which a device–failure pair is reported more often than expected. Confidence intervals (IC025) exceeding zero indicate a potential signal.

FDA Method

MGPS / EBGM

The Multi-item Gamma Poisson Shrinker produces an Empirical Bayesian Geometric Mean (EBGM). By pooling information across all device–failure combinations, MGPS prevents a single rare report from triggering a false alarm — the prior pulls extreme estimates back toward the database average.

Both methods stabilise disproportionality scores when cell sizes are small, making them more suitable than the PRR for rare device–failure combinations.

4.5 Signal Management Lifecycle

Hover over each stage to view its description.

Systematic screening of spontaneous reports, literature, and databases using quantitative algorithms to identify potential safety concerns.
Signal
Detection
1
▶
Clinical and epidemiological review to determine whether the detected statistical signal represents a genuine safety concern or a false positive.
Signal
Validation
2
▶
Ranking validated signals by severity, frequency, clinical impact, and public-health relevance to allocate resources effectively.
Signal
Prioritization
3
▶
In-depth assessment including root-cause analysis, benefit–risk evaluation, and review of all available evidence for the prioritised signal.
Signal
Evaluation
4
▶
Formulating risk-minimisation measures such as labelling changes, design modifications, field safety corrective actions, or further studies.
Recom­mendation
5
▶
Implementing approved corrective actions, monitoring their effectiveness over time, and closing the signal once risk has been adequately mitigated.
Action &
Tracking
6
⚡
Agentic AI in Signal Detection: Modern device signal detection is augmented by Agentic AI. Algorithms like k-Nearest Neighbors (kNN) identify historically similar incidents across millions of reports, allowing manufacturers to anticipate risks in new device iterations before adverse trends become statistically visible in traditional disproportionality analyses.