Random Forests for Hearing Disorder Diagnosis

A new simulation study directly compares two powerful methods for turning questionnaire answers into a diagnosis. The research, led by Catherine Bain and colleagues, finds that a common machine learning algorithm may outperform a standard psychometric technique when the assessment itself contains hidden biases that affect different groups of people differently. This has direct implications for the accurate diagnosis of conditions like tinnitus, misophonia, and hyperacusis, where self-report questionnaires are central to understanding patient experience.

Two Roads to a Diagnosis: IRT vs. Random Forest

The study compared two classification approaches. The first, based on Item Response Theory (IRT), is the bedrock of modern psychological testing. IRT estimates a person’s latent trait level—such as the severity of sound intolerance—from their pattern of answers. This score is then compared to a diagnostic cut-point. The second approach used a machine learning algorithm called random forest (RF). Instead of estimating an intermediate trait score, RF analyzes complex patterns in the raw item responses to predict diagnostic class membership directly.

Researchers ran thousands of computer simulations, creating virtual populations and varying key factors: sample size, test length, the correlation between items, and, most importantly, the presence and severity of Differential Item Functioning (DIF). DIF occurs when a test question has different statistical properties for different groups (e.g., by age, gender, or culture) even when those groups have the same underlying level of the trait being measured. It is a form of measurement bias. The team used a single-group IRT model as a baseline, representing common practice that assumes no DIF exists.

DIF is the Deciding Factor in Model Performance

The results were clear. Under ideal conditions with no DIF, both IRT and random forest produced essentially equivalent classification accuracy, sensitivity, and specificity. The methods were interchangeable.

This parity broke down as DIF was introduced and its severity increased. The classification performance of the single-group IRT model steadily declined. Because it assumes all items function the same for everyone, its estimates—and the subsequent diagnoses—became less accurate when that assumption was violated. In contrast, the random forest algorithm’s performance remained robust and stable across all levels of DIF severity. It was largely unaffected by the introduced bias.

“These findings suggest that RF may maintain more stable classification performance than IRT-based classification when DIF is present but not explicitly accounted for in the model,” the authors write. This makes RF a strong candidate for diagnostic classification when clinicians or researchers suspect hidden test bias but cannot identify its source or structure.

Practical Implications for Hearing and Sound Disorder Assessment

For clinicians and researchers working in hearing health, this study highlights a critical consideration in tool selection and development. Questionnaires for conditions like misophonia and hyperacusis are often developed and validated on specific populations. DIF can creep in if a question about “annoyance” is interpreted differently across cultures, or if sensitivity to “chewing sounds” varies by age. This study shows that if unaddressed, such bias can compromise diagnostic accuracy if using traditional methods.

The random forest approach offers a potential safeguard. Its ability to handle complex, non-linear relationships in data allows it to navigate around DIF without needing to model it explicitly. This could lead to more equitable and stable diagnostic tools across diverse patient groups. It also aligns with a broader trend of using data-driven computational approaches to address complex health challenges.

The Trade-off: Robustness vs. Interpretability

The authors are careful to note that each method has strengths and limitations, centering on a key trade-off. IRT models are highly interpretable. A clinician can see exactly how each item contributes to the trait score and understand why a particular diagnostic threshold was chosen. This transparency is valuable for patient counseling and clinical decision-making.

Random forest, like many machine learning models, operates more as a “black box.” While its predictions can be highly accurate, it is often difficult to extract a simple, intuitive explanation for why it classified a specific individual in a certain way. In an applied setting, this lack of interpretability can be a significant drawback, even if the classification is robust.

The work by Bain, Manapat, and Manapat provides a data-driven framework for making this choice. When measurement invariance is certain or can be statistically confirmed, IRT remains a powerful and interpretable standard. In contexts where diverse populations are assessed and hidden biases are a concern, the robustness of random forest may make it the preferable option, provided its limitations are understood.

Source: Bain, C., Manapat, P. D., & Manapat, D. (2024). A Comparison of Item Response Theory and Random Forest for Diagnostic Classification in the Presence of Differential Item Functioning. Journal of Behavioral Data Science. DOI: 10.35566/jbds/bainmmbg.