Considerations on the region of interest in the ROC space

Vocal Range Variations Elicited With a Standard Mandarin Reading Passage in Individuals With Voice Disorders

PURPOSE

This study examined whether the "Three Bears Passage" (TB), a standard Mandarin reading passage, could elicit significant vocal range variations in individuals with voice disorders. Relative sensitivity of TB versus another existing standard reading passage, "Passage in Mandarin" (PM), for differentiating between individuals with and without voice disorders was also evaluated.

METHOD

Forty-two individuals with normal voice and 30 individuals with voice disorders participated in the study. Maximum fundamental frequency (f0), minimum f0, mean f0, f0 range, maximum vocal intensity, minimum intensity, mean intensity, and intensity range of all participants reading aloud the two passages were measured with Praat to construct speech range profiles (SRPs).

RESULTS

Significantly larger vocal range was found for TB than for PM in individuals with voice disorders, including significantly higher maximum f0, mean f0, maximum intensity, mean intensity, and significantly larger f0 range and intensity range. Significantly more limited vocal range was observed in individuals with voice disorders than those without, with more obviously restricted SRPs while reading aloud TB compared to PM. Receiver operating characteristic analysis suggested that TB was more sensitive than PM in distinguishing between individuals with and without voice disorders.

CONCLUSIONS

Our findings supported the potential of TB as a standard clinical assessment tool for evaluating pathological changes in vocal range. Future studies should explore if therapeutic approaches based on the passage or variations of it could be developed for overcoming functional limitations and restrictions in vocal range for specific voice disorders.

The Matthews correlation coefficient (MCC) should replace the ROC AUC as the standard metric for assessing binary classification

Binary classification is a common task for which machine learning and computational statistics are used, and the area under the receiver operating characteristic curve (ROC AUC) has become the common standard metric to evaluate binary classifications in most scientific fields. The ROC curve has true positive rate (also called sensitivity or recall) on the y axis and false positive rate on the x axis, and the ROC AUC can range from 0 (worst result) to 1 (perfect result). The ROC AUC, however, has several flaws and drawbacks. This score is generated including predictions that obtained insufficient sensitivity and specificity, and moreover it does not say anything about positive predictive value (also known as precision) nor negative predictive value (NPV) obtained by the classifier, therefore potentially generating inflated overoptimistic results. Since it is common to include ROC AUC alone without precision and negative predictive value, a researcher might erroneously conclude that their classification was successful. Furthermore, a given point in the ROC space does not identify a single confusion matrix nor a group of matrices sharing the same MCC value. Indeed, a given (sensitivity, specificity) pair can cover a broad MCC range, which casts doubts on the reliability of ROC AUC as a performance measure. In contrast, the Matthews correlation coefficient (MCC) generates a high score in its [Formula: see text] interval only if the classifier scored a high value for all the four basic rates of the confusion matrix: sensitivity, specificity, precision, and negative predictive value. A high MCC (for example, MCC [Formula: see text] 0.9), moreover, always corresponds to a high ROC AUC, and not vice versa. In this short study, we explain why the Matthews correlation coefficient should replace the ROC AUC as standard statistic in all the scientific studies involving a binary classification, in all scientific fields.