Analytics · experiment statistics

analytics-significance

Testing many metrics manufactures attractive p-values unless the full family is corrected and non-significance is kept separate from no effect.

Public · synthetic demo

The problem

A raw p-value answers the wrong question after repeated testing.

Cherry-picking significant rows inflates false discoveries. Calling a non-significant result “no effect” ignores what the sample was too small to detect.

Input

A synthetic family of proportion comparisons.

Family size: 32
Alpha: 0.05
Target power for MDE: 0.8

Headline arms:
control n=4000
real-effect variant n=4000

The money shot

Correction cuts five naive discoveries to two.

Comparisons

Raw p ≤ 0.05

Bonferroni significant

BH significant

The headline effect is 0.0265 with interval 0.0126–0.0404. Its raw p-value is 0.0002 and both corrected values are 0.0061.

How it's verified

Every verdict carries effect, interval, correction, and MDE.

Comparison	Effect	Raw p	BH p	Verdict
headline real effect	0.0265	0.0002	0.0061	significant after correction
headline null variant	0.0010	0.8796	0.8796	not detected; MDE 0.0185
metric_01	0.0800	0.0170	0.1356	raw only

Honest limitations

Correction cannot see tests that were hidden from the family.

The data is synthetic and does not validate a real product effect.
The tests assume independent observations and do not check sample-ratio mismatch or repeat users.
A non-significant row still permits effects below its reported MDE.
The rolling confidence band is a trailing-window interval, not a causal or forecasting model.

The verdict stays literal: not detected is never rewritten as no effect. jigonyoo.com · Back to hub