Can a Small Sample Size Capture a Big Truth? A Hypothetical Simulation Study on Diagnostic Accuracy of AI vs Radiologist in Tuberculosis Detection through X-rays

Abstract

Background: Artificial intelligence (AI) tools are increasingly being adopted in diagnostic radiology. However, early-phase evaluations often rely on small sample sizes due to logistical constraints. It remains unclear whether such small studies can reliably detect real differences in diagnostic accuracy between AI models and human experts. Objectives: To assess whether a small sample size (n = 30) can reliably detect statistically significant differences in diagnostic accuracy between an AI model and a radiologist for TB detection from chest X-rays under varying performance scenarios. Methods: A Monte Carlo simulation was conducted in R (v4.5.0) using 1000 iterations per scenario. Three diagnostic accuracy scenarios tested were: scenario 1. AI (98%) vs Radiologist (70%), scenario 2. AI (90%) vs Radiologist (75%) and scenario 3. AI (92%) vs Radiologist (80%). Predictions were simulated using defined sensitivity and specificity values, and accuracy was computed. A one-sided paired t-test assessed whether AI outperformed the radiologist. Empirical power was calculated as the proportion of simulations yielding p-values < 0.05. Results: Scenario 1 with a large accuracy gap (28.2%), achieved 96% empirical power. Scenarios 2 and 3, with moderate and small differences (~14%), achieved only 46.8% and 37.5% power, respectively. Conclusion: Small samples can detect large diagnostic performance differences but are unreliable for moderate or small differences.