Random Forest Models for Hearing Disorder Diagnosis

When Psychological Tests Are Biased, Machine Learning Holds Steady

Psychological questionnaires are central to diagnosing conditions like tinnitus, misophonia, and hyperacusis. Clinicians rely on them to separate clinical from non-clinical cases. But what happens when the questions on these tests are biased? A new simulation study by Catherine Bain, Patrick D. Manapat, and Danielle Manapat examined how two common statistical methods handle this bias, with direct implications for hearing and sound tolerance disorders. They compared traditional psychometric modeling against a machine learning algorithm, testing their resilience to a problem called differential item functioning (DIF).

Testing Two Diagnostic Engines: IRT vs. Random Forest

The researchers used Monte Carlo simulations, a method that generates thousands of virtual datasets with known properties, to create a controlled testing ground. They simulated individuals responding to a diagnostic questionnaire. Key variables they manipulated included the presence and severity of DIF, the sample size, and the length of the test.

DIF occurs when a question on a test has a different meaning or difficulty for different groups, even if those groups have the same overall level of the trait being measured. For example, a question about “distress in restaurants” might function differently for someone with social anxiety versus someone with pure sound sensitivity, even if their overall misophonia severity is identical. This bias can distort scores and lead to misclassification.

The team pitted two classification approaches against each other. The first was a standard psychometric method: single-group Item Response Theory (IRT). IRT estimates a person’s latent trait (e.g., misophonia severity) from their item responses and then uses a cut-point on that score to assign a diagnosis. This method is common but assumes all test items are completely fair (invariant) across all subgroups—an assumption often violated in practice.

The second approach was a machine learning algorithm called Random Forest (RF). Unlike IRT, RF does not first estimate a latent trait. Instead, it uses the pattern of direct item responses to predict diagnostic class membership. The study’s goal was to see which method maintained better classification accuracy—correctly identifying “cases” and “non-cases”—as DIF was introduced and worsened.

Random Forest’s Robust Performance Under Bias

The results were clear. When DIF was absent or very mild, both IRT and Random Forest produced essentially equivalent classification metrics. They were both accurate tools for diagnosis under ideal conditions.

The situation changed dramatically as the researchers increased the severity of DIF in the simulations. The classification performance of the standard IRT model declined. Because it incorrectly assumed all items were unbiased, its trait estimates became distorted, leading to more diagnostic errors.

In contrast, the Random Forest algorithm’s performance remained stable. Its classification accuracy, precision, and recall did not fall away even under conditions of severe DIF. The algorithm’s ability to find complex, non-linear patterns in the data directly from the items allowed it to adapt to the biased items without a significant loss in diagnostic power.

This finding is detailed in the source paper (DOI: 10.35566/jbds/bainmmbg).

Implications for Diagnosing Hearing and Sound Tolerance Disorders

For clinicians and researchers working with tinnitus, hyperacusis, and misophonia, this study highlights a practical decision point. The choice of diagnostic method may affect who gets a diagnosis, especially in diverse populations where cultural, age, or comorbid factors could introduce DIF into common questionnaires.

The traditional IRT approach offers high interpretability. Clinicians can examine item parameters to understand which questions are most informative for the latent trait, which aids in test development and refinement. However, this study shows its vulnerability when unseen biases exist in the data.

Random Forest, while often acting as more of a “black box,” offers a significant advantage in robustness. It can be a viable alternative for diagnostic classification when DIF is suspected but its specific source—whether it’s related to age, a comorbid condition like hyperacusis mechanisms, or another factor—is unknown, unmeasured, or complex. This makes it particularly useful for real-world clinical data, which is often messy and multifaceted.

This trade-off between interpretability and robustness is central. In research settings where understanding the construct is key, IRT remains vital. But in applied screening or diagnostic contexts where the primary goal is accurate and stable classification, methods like Random Forest warrant serious consideration. This aligns with growing interest in applying random forests for hearing disorder diagnosis to handle complex patient data.

A Tool in the Box, Not a Replacement

Bain and colleagues are not suggesting machine learning should replace psychometric theory. Instead, their work shows that RF can be a powerful supplemental tool, especially during the early stages of investigating a new patient population or when using an existing questionnaire in a different group.

For instance, a questionnaire validated on adults with tinnitus might exhibit DIF when used for children with misophonia. While researchers work to identify and resolve the specific biased items, using an RF model for initial classification could help prevent systematic diagnostic errors in the interim.

The study underscores a fundamental principle: the statistical model should match the complexity of the data. For clean, well-understood data with invariant items, IRT is excellent. For complex, real-world data where biases may lurk, a method like Random Forest that makes fewer assumptions can provide more reliable diagnostic decisions, ensuring patients receive accurate and fair assessments.