Many comparisons and combined evidence ​
A report that compares many criteria, or runs many suites, makes many tests at once. The chance that at least one of them looks significant by luck grows with their number. This page defines the error-rate formulas that control that chance and the Simes combination of p-values. The conventions of the shared page apply to every function here.
Words used in the sections:
- m. The number of tests.
Error rates for many tests ​
The three functions of this section live in packages/stats/src/rates.ts. They take the number of tests and a level. All three use the stable forms given, so that a small level keeps its relative accuracy.
familywiseErrorRate ​
ts
familywiseErrorRate(m: number, alpha: number): number- Definition. The chance of at least one false rejection in
mindependent tests of true nulls, each at levelalpha:1 - (1 - alpha)^m. Compute it as-expm1(m log1p(-alpha)).mis a non-negative integer andalphais in [0, 1]. The result is 0 whenmis 0, whateveralphais. It is 1 whenalphais 1 andmis at least 1. - Returns. A number in [0, 1].
- Throws. A
RangeErrorfor anmthat is not a non-negative integer, and for analphathat is NaN, infinite or outside [0, 1]. - Example.
familywiseErrorRate(10, 0.05)is 0.4012630607616211.familywiseErrorRate(1, 0.05)is 0.05.familywiseErrorRate(0, 0.05)is 0.familywiseErrorRate(5, 1)is 1.
bonferroniThreshold ​
ts
bonferroniThreshold(m: number, alpha: number): number- Definition. The level for each of
mtests that keeps the family-wise rate at mostalpha:alpha / m.mis a positive integer andalphais in [0, 1]. - Returns. A number in [0, 1].
- Throws. A
RangeErrorfor anmthat is not a positive integer (0 included), and for analphathat is NaN, infinite or outside [0, 1]. - Example.
bonferroniThreshold(10, 0.05)is 0.005.bonferroniThreshold(4, 1)is 0.25.bonferroniThreshold(7, 0)is 0.
sidakThreshold ​
ts
sidakThreshold(m: number, alpha: number): number- Definition. The level for each of
mindependent tests that makes the family-wise rate exactlyalpha:1 - (1 - alpha)^(1/m). Compute it as-expm1(log1p(-alpha) / m).mis a positive integer andalphais in [0, 1]. The result is 1 whenalphais 1, and 0 whenalphais 0. It is never belowbonferroniThreshold(m, alpha). - Returns. A number in [0, 1].
- Throws. A
RangeErrorfor anmthat is not a positive integer (0 included), and for analphathat is NaN, infinite or outside [0, 1]. - Example.
sidakThreshold(10, 0.05)is 0.005116196891823702.sidakThreshold(1, 0.05)is 0.05.sidakThreshold(1000, 0.05)is 5.129197890901781e-05.
Combining p-values ​
The function of this section lives in packages/stats/src/combine.ts. It turns several p-values for one shared null hypothesis into one number.
simesPValue ​
ts
simesPValue(pvalues: number[]): number- Definition. Sort the p-values ascending. The result is the minimum over ranks
ifrom 1 to m ofm p(i) / i. It never exceedsp(m), so it is at most 1. It is the p-value of the global null hypothesis that every individual null is true, valid under independence and under positive dependence of the tests. - Returns. A number in [0, 1].
- Throws. A
RangeErrorfor an empty array, and for a p-value that is NaN, infinite or outside [0, 1]. - Example.
simesPValue([0.01, 0.04, 0.03, 0.2])is 0.04, from the sorted values 0.01, 0.03, 0.04, 0.2 and the products 0.04, 0.06, 0.0533, 0.2.simesPValue([0.04, 0.04, 0.04, 0.9])is 0.05333333333333334.simesPValue([0, 0.5])is 0.