Classification and agreement ​
This page defines the scores of a classifier or a judge and the agreement between raters. Each section gives the formula, the result and the inputs it refuses. The conventions of statistics capabilities hold for every function: the RangeError rules and the accuracy.
Status: partly built. balancedAccuracy, fBeta, gwetAc1 and weightedKappa exist in @agent-nexus/driftbench-stats; the other functions are planned.
Two terms recur.
- Confusion table. A plain object
{ tp, fp, fn, tn }: the counts of true positives, false positives, false negatives and true negatives. The four fields are required. Each is a non-negative integer. Other fields are ignored. - Category label. An integer that names the category a rater chose. Two labels are the same category when they are equal numbers.
Confusion table ​
These functions belong in packages/stats/src/classification.ts.
balancedAccuracy ​
ts
balancedAccuracy(table: { tp: number; fp: number; fn: number; tn: number }): number- Definition. The mean of the true positive rate and the true negative rate:
(tp / (tp + fn) + tn / (tn + fp)) / 2. - Returns. A number in [0, 1].
- Throws. A
RangeErrorwhen a field is missing or is not a non-negative integer, and whentp + fnis 0 ortn + fpis 0 (a class with no true case has no rate). - Example.
balancedAccuracy({ tp: 40, fp: 10, fn: 5, tn: 45 })is 0.8535353535353535.balancedAccuracy({ tp: 1, fp: 0, fn: 9, tn: 90 })is 0.55.balancedAccuracy({ tp: 0, fp: 5, fn: 5, tn: 0 })is 0.
fBeta ​
ts
fBeta(table: { tp: number; fp: number; fn: number; tn: number }, beta: number): number- Definition. The weighted harmonic mean of precision and recall, in the form
(1 + beta^2) * tp / ((1 + beta^2) * tp + beta^2 * fn + fp). Abetaabove 1 favours recall; abetabelow 1 favours precision. When the denominator is 0 the result is 0. The fieldtnis validated but does not enter the formula. - Returns. A number in [0, 1].
- Throws. A
RangeErrorwhen a field is missing or is not a non-negative integer, and whenbetais not greater than 0. - Example. For the table
{ tp: 40, fp: 10, fn: 5, tn: 45 },fBeta(table, 1)is 0.8421052631578947,fBeta(table, 2)is 0.8695652173913043 andfBeta(table, 0.5)is 0.8163265306122449.fBeta({ tp: 0, fp: 0, fn: 0, tn: 9 }, 1)is 0.
Agreement ​
These functions belong in packages/stats/src/agreement.ts.
weightedKappa ​
ts
weightedKappa(table: number[][], weights: 'linear' | 'quadratic'): number- Definition. The table has one row and one column per ordered category. Cell
table[i][j]is the number of subjects that rater A put in categoryiand rater B put in categoryj. Letnbe the sum of all cells,r_ithe sum of rowiandc_jthe sum of columnj. The disagreement weight of a pair of categories isw_ij = |i - j|for'linear'andw_ij = (i - j)^2for'quadratic'. The weights are not rescaled. The observed disagreement isD_o = sum(w_ij * table[i][j]) / n. The expected disagreement isD_e = sum(w_ij * r_i * c_j) / n^2. The result is1 - D_o / D_e. WhenD_eis 0 the result is 1. This happens when both raters use one and the same single category for every subject, and for a table with one cell. - Returns. A number, at most 1.
- Throws. A
RangeErrorwhenweightsis not the string'linear'or the string'quadratic', when the table has no row, when it is not square (a row whose length differs from the number of rows), and whennis 0. - Example.
weightedKappa([[20, 5], [10, 15]], 'linear')is 0.4, the same as with'quadratic'(on two categories both weightings give the unweighted kappa). For[[11, 3, 1], [2, 9, 4], [0, 3, 12]],'linear'gives 0.65 and'quadratic'gives 0.7333333333333333.weightedKappa([[0, 5], [5, 0]], 'linear')is -1.weightedKappa([[7]], 'linear')is 1.
gwetAc1 ​
ts
gwetAc1(a: number[], b: number[]): number- Definition. Two raters label the same
nsubjects;a[i]andb[i]are the labels of subjecti. Letpobe the share of subjects witha[i] === b[i]. For each labelcthat occurs inaor inb, letpi_cbe the number of timescoccurs in the two arrays together, divided by2n. Letqbe the number of distinct labels that occur inaor inb. Whenqis 1 the result is 1. Otherwise the chance agreement ispe = sum(pi_c * (1 - pi_c)) / (q - 1), which equals(1 - sum(pi_c^2)) / (q - 1). Forq= 2 this is2 * pi * (1 - pi)withpithe pooled share of either label. The result is(po - pe) / (1 - pe). Forqof 2 or more,peis at most1 / q, so the denominator is positive. - Returns. A number, at most 1.
- Throws. A
RangeErrorwhen a label is not an integer. - Example.
gwetAc1([0, 0, 1, 1], [1, 1, 0, 0])is -1.gwetAc1([1, 1, 2, 2, 3, 3, 1, 2], [1, 2, 2, 2, 3, 1, 1, 2])is 0.6342857142857142. With 18 subjects both raters label 1, one subject labelled 2 by A only, and one labelled 2 by B only, the result is 0.8895027624309392.