Data collected in a medical study should, from a methodological point of view, be considered as a sample taken from a larger population. The purpose of the statistical analysis is to check whether the differences in the experimental results observed in different subgroups are related to chance or not. The risks of error must be known to assess the validity of the conclusions.

The first order risk, also called the alpha risk, is the risk of announcing a wrongly positive conclusion, that is to conclude that there is a significant difference that in reality does not exist. By convention, an alpha risk of 5 p. 100 is generally accepted. This means that it is acceptable to announce a statistically positive test when no difference exists in at most 5 p. 100 of the cases. After recording and processing the data, the statistical analysis produces a value called p that is the exact value of the first order risk in the given situation. If p is less than or equal to the alpha risk accepted before the study, it can be concluded that the observed difference is statistically significant at the chosen alpha level and that the p value represents the risk of first order risk in the given situation. If p is greater than the initially accepted alpha, the observed difference is not considered to be significant at the alpha level.

But the assertion that two samples are equivalent, also involves a second order risk, also called the beta risk, that must be known. The beta risk is the risk of announcing wrongly negative results, that is to conclude that two samples are equivalent while in reality they are different. The number of elements in each sample necessary to demonstrate a difference becomes greater as the size of the difference becomes smaller. The beta risk increases as the alpha risk decreases, the number of cases becomes smaller, and the difference to detect becomes smaller. If a difference is not statistically significant at the chosen alpha level, the beta risk of an erroneous conclusion of equivalence is generally less than or equal to 20 p. 100.

In most cases, the beta risk is not determined before the study but after, being calculated from the alpha risk, the sample size, and the non-significant difference observed. If the beta risk is found to be greater than 20 p. 100, no conclusion can be drawn and the study data are useless. It is therefore preferable to define both the alpha and beta risk and the smallest clinically pertinent difference, and to calculate the necessary sample size, before initiating the study.

Let us take a numerical example where two different treatments, A and B, are given to two groups of 100 patients each. Treatment A produced success in 70 cases and treatment B in 80 cases. The chi-squared test yields a p value of 0.10. The observed difference is thus not statistically significant at an alpha level of 5 p. 100. In this case, the calculated beta risk is 54 p. 100. With 200 patients and a beta risk of 20 p. 100, a difference of 20 p. 100 in the success rates between the two groups cannot be detected. If it is accepted that a difference of 10 p. 100 between the success rates is clinically pertinent, to have an acceptable beta risk of 20 p. 100 and detect the difference, the study would have to include 500 patients instead of 200.

In conclusion, when a comparative study concludes that there is no significant difference between two groups, one cannot deduct that these two groups are identical unless the beta risk is less than 20 p. 100. If the beta risk is greater than 20 p. 100, or if it is not mentioned, one cannot conclude that the two groups are equivalent.

Les statistiques ont pour but de vérifier si les résultats observés expérimentalement sont liés ou non au hasard. Les tests statistiques répondent aux lois des probabilités, et sont donc entachés de risques d'erreur inhérents à leur formulation mathématique. Ces risques doivent être connus pour apprécier la validité des conclusions. Le risque de première espèce est bien connu : c'est le risque d'énoncer à tort une conclusion positive, ou de considérer à tort une différence comme significative. Le risque de deuxième espèce est souvent méconnu : c'est le risque de ne pas énoncer à tort une conclusion positive, ou de ne pas détecter une différence significative qui existe en réalité. Ce risque, encore appelé puissance, évalue la plus petite différence détectable avec les données disponibles. En cas d'absence de différence significative issue du test utilisé, la connaissance de ce risque est indispensable pour savoir si les deux échantillons sont effectivement identiques, ou si la différence existant effectivement est trop faible pour être mise en évidence. Sans mentionner ce risque, il est impossible de conclure à l'équivalence entre deux échantillons.