Evaluation of Carebot AI MMG Medical Device for Breast Lesion Detection and Density Assessment

Completed

• Conditions: Breast Tumor Benign; Breast Cancer

• Registration Number: NCT06428799
• Lead Sponsor: Carebot s.r.o.

Brief Summary

Comparison of accuracy of clinician and DLAD image evaluation (Carebot AI MMG v2.2)

1. Comparison of the Accuracy of Density Assessment by Clinician and DLAD (DENS)

2. Comparison of Accuracy of Lesion Assessment by Clinician and DLAD (MASS, CLASS)

Detailed Description

The mammography studies were acquired from three independent sites: Site 1 (EUC Mamocentrum Brno) and Site 2 (Hospital Šumperk, a.s.) specialise in routine screening mammography, and Site 3 (Masaryk Memorial Cancer Institute) is a comprehensive oncology facility primarily dedicated to diagnostic mammography, i.e. performing additional examinations in case of a suspicious finding (recall).

The ground truth was obtained by consensus of two board-certified radiologists with expertise in radiology and diagnostic methods, and 13 and 27 years of experience with mammography image interpretation, respectively.

For comparative analysis, a team of five independent radiologists with clinical experience in interpreting mammography images was established. Three of the clinicians were junior (2, 2, and 4 years of experience, respectively) without board-certification; two physicians were senior (7 and 8 years of experience, respectively), board-certified.

Study Timeline

• Study Start: 2022-01-14
• Primary Completion: 2024-04-24
• Study Completion: 2024-04-24
• Status Verified: 2024-05
• Study First Submitted: 2024-05-07
• Study First Submitted QC: 2024-05-20
• Last Update Submitted: 2024-05-20
• Study First Posted: 2024-05-24
• Last Update Posted: 2024-05-24

Recruitment & Eligibility

• Status: COMPLETED
• Sex: Female
• Target Recruitment: 122

Inclusion Criteria

• The medical device is intended for use in women over 18 years of age who are indicated for screening mammography using digital mammography.

Exclusion Criteria

• The medical device cannot be used in patients with breast implants.
• The medical device cannot be used in male breast examination.
• The medical device cannot be used in patients under 18 years of age.

Study & Design

• Study Type: OBSERVATIONAL
• Study Design: Not specified

Primary Outcome Measures

Name	Time	Method
Performance Test	2024	A multicenter, multi-reader, retrospective study was designed to validate the clinical efficacy of the proposed Carebot AI MMG (also referred to as "DLAD"). Using a non-certified medical device, a test set of retrospectively collected mammography studies in standard projections (CC and MLO). The performance of the DLAD was evaluated against the ground truth for individual indications (breast density evaluation, breast lesion detection) using Accuracy.

Secondary Outcome Measures

Name	Time	Method
Comparison of Accuracy of Clinician and DLAD Image Evaluation (Carebot AI MMG v2.2) in Breast Density Assessment	2024	Carebot AI MMG v2.2 classified mammography studies according to the ACR BI-RADS 5th edition, i.e. breast tissue density assessment into A/B/C/D classes. The performance of the Carebot AI MMG device was assessed relative to the ground truth and then compared with the performance of five independent radiologists with varying levels of experience (RAD 1-RAD 5). A rigorous statistical analysis was used in the BI-RADS breast density classification to evaluate the performance of each method - the proposed Carebot AI MMG v2.2 medical device and the compared radiologists in the multi-reader study. The analysis focused on key metrics including Accuracy, F1 Score (Macro-Averaged), Precision (Macro-Averaged), Recall (Macro-Averaged) and Cohen's Kappa (κ) to assess the strength of agreement. Given that all scans were evaluated by all radiologists in the comparison, a bootstrapping method that involves resampling the test data 1000 times.
Comparison of Accuracy of Clinician and DLAD Image Evaluation (Carebot AI MMG v2.2) in Breast Lesion Detection	2024	The medical device (DLAD, Carebot AI MMG v2.2) analyzed mammography studies in standard projections (CC and MLO) and classified the presence of lesions ("present" x "absent") at the mammography study level. DLAD performance was assessed relative to the ground truth and then compared to the performance of five independent radiologists with varying levels of experience (RAD 1-RAD 5). The investigators quantified the performance of diagnostic tests based on Sensitivity, Specificity and Balanced Accuracy. The investigators further assessed the statistical significance of differences between DLAD and individual radiologists using appropriate statistical tests. To determine the reliability of the metrics examined, the investigators calculated 95% confidence intervals using Wilson scores. To evaluate the statistical significance of differences in Sensitivity and Specificity between DLAD and individual radiologists, the investigators applied McNemar's test with a continuity correction.

Trial Locations

Locations (3)

EUC Mamocentrum Brno; 🇨🇿 Brno, Czechia

Masaryk Memorial Cancer Institute; 🇨🇿 Brno, Czechia

Hospital Šumperk; 🇨🇿 Šumperk, Czechia