Marie-Luise Kromrey, Laura Steiner, Felix Schön, Julie Gamain, Christian Roller, Carolin Malsch

• Background The integration of artificial intelligence (AI) into radiology aims to improve diagnostic accuracy and efficiency, particularly in settings with limited access to expert radiologists and in times of personnel shortage. However, challenges such as insufficient validation in actual real-world settings or automation bias should be addressed before implementing AI software in clinical routine. Methods This cross-sectional study in a maximum care hospital assesses the concordance between diagnoses made by a commercial AI-based software and conventional radiological methods augmented by AI for four major thoracic pathologies in chest X-ray: fracture, pleural effusion, pulmonary nodule and pneumonia. Chest radiographs of 1506 patients (median age 66 years, 56.5% men) consecutively obtained between January and August 2023 were re-evaluated by the AI software InferRead DR Chest ® . Results Overall, AI software detected thoracic pathologies more often than radiologists (18.5% vs. 11.1%). In detail, it detected fractures, pneumonia, and nodules more frequently than radiologists, while radiologists identified pleural effusions more often. Reliability was highest for pleural effusions (0.63, 95%-CI 0.58–0.69), indicating good agreement, and lowest for fractures (0.39, 95%-CI 0.32–0.45), indicating moderate agreement. Conclusions The tested software shows a high detection rate, particularly for fractures, pneumonia, and nodules, but hereby produces a nonnegligible number of false positives. Thus, AI-based software shows promise in enhancing diagnostic accuracy; however, cautious interpretation and human oversight remain crucial.