Large Language Model-Assisted Imaging cTNM Staging Annotation and Uncertainty Recognition for Prostate Cancer Based on Chinese PSMA PET/CT Reports
试验速览
- 阶段
- 不适用
- 状态
- 招募中
- 入组人数
- 4,600
- 试验地点
- 1
- 主要终点
- Accuracy of LLM-Assisted Imaging cTNM Staging Annotation
研究概览
简要总结
This observational study will develop and validate a large language model-assisted workflow for imaging cTNM staging annotation and uncertainty recognition in prostate cancer using Chinese PSMA PET/CT report texts generated during routine clinical care. The study will use de-identified report texts and necessary baseline clinical information only. No additional imaging examination, blood test, treatment, or follow-up visit will be assigned for this study.
The main objective is to evaluate whether a locally or institutionally controlled large language model can identify report-derived imaging cT, cN, and cM categories, extract supporting evidence from the original report, and recognize uncertainty expressions. Model performance will be assessed using an internal independent validation set, external validation reports from two collaborating hospitals, and a prospective validation set of 100 consecutive routine PSMA PET/CT reports. A human-AI comparison will also be performed using physicians from urology and imaging-related specialties with different seniority levels.
详细描述
This is a multicenter observational diagnostic accuracy validation study based on Chinese PSMA PET/CT report texts from patients with prostate cancer or suspected prostate cancer. The study is not designed to evaluate a drug, device, surgical procedure, or imaging intervention. PSMA PET/CT examinations will be performed as part of routine clinical care, and the study will only analyze de-identified report texts and necessary baseline information after the reports have been finalized.
The study consists of retrospective and prospective components. Retrospectively, approximately 4,000 PSMA PET/CT reports from the First Affiliated Hospital of Wenzhou Medical University will be systematically annotated to construct a research database. An internal independent validation set of 300 reports, not used for model development or prompt optimization, will be used to evaluate the performance of the large language model. The reference standard for this 300-report validation set will be established by two experienced urologists through joint annotation, with adjudication by a nuclear medicine expert when needed. External validation will be performed using 110 de-identified reports from the First Affiliated Hospital of Ningbo University and 102 de-identified reports from Liuzhou People's Hospital. In addition, after ethics approval, 100 consecutive routine PSMA PET/CT reports from the First Affiliated Hospital of Wenzhou Medical University will be prospectively included to evaluate the accuracy and operational stability of the frozen model and prompt versions.
The large language model workflow will be deployed locally or in an institutionally controlled environment. The model will be instructed to generate structured JSON outputs, including cT_report, cN_report, cM_report, cT_uncertain, cN_uncertain, cM_uncertain, evidence_T, evidence_N, and evidence_M. The target task is report-derived imaging cTNM staging annotation, not pathological TNM staging or overall AJCC stage grouping. The model output will be used only for research evaluation and methodological analysis and will not be used for clinical diagnosis, treatment decision-making, or patient notification.
A human-AI comparison will be conducted on the 300-report internal validation set. Eight human evaluators from urology and imaging-related specialties, including trainees, residents, attending physicians, and associate chief physicians, will independently annotate the reports before and after learning the annotation manual. Annotation time will be recorded for each round. The performance of human evaluators and the large language model will be compared against the expert consensus reference standard.
The main outcome will be the accuracy of the large language model in identifying cT, cN, and cM categories from Chinese PSMA PET/CT reports. Secondary outcomes will include precision, recall, F1-score, macro-F1, micro-F1, complete cTNM triplet matching rate, uncertainty recognition performance, evidence extraction quality, human-AI comparison results, annotation time, external validation performance, prospective validation performance, and error type distribution. Error analysis will focus on local tumor extent, regional versus non-regional lymph node boundaries, bone and visceral metastasis recognition, equivocal wording, treatment-related context, benign or inflammatory alternatives, and lesions not attributable to prostate cancer.
研究设计
- 研究类型
- Observational
- 观察模型
- Cohort
- 时间视角
- Other
入排标准
- 年龄范围
- 18 Years 至 —（Adult, Older Adult）
- 性别
- Male
- 接受健康志愿者
- 否
入选标准
- •Male patients aged 18 years or older.
- •Patients with clinically diagnosed, pathologically diagnosed, or clinically suspected prostate cancer.
- •Patients who underwent PSMA PET/CT for initial staging, recurrence assessment, treatment response evaluation, metastatic assessment, or other clinical purposes during routine care.
- •Complete or basically complete Chinese PSMA PET/CT report text is available, including imaging findings and/or diagnostic impression.
- •The report text contains information that can be used to evaluate at least one target field, such as local prostate lesion, regional lymph nodes, non-regional lymph nodes, bone metastasis, visceral metastasis, or uncertainty expressions.
- •The research data can be de-identified and replaced by a study identification number before analysis.
排除标准
- •PSMA PET/CT reports unrelated to prostate cancer, or reports clearly irrelevant to the research task.
- •Reports with severely missing, unreadable, or unavailable main text, imaging findings, or diagnostic impression.
- •Reports that cannot be adequately de-identified or contain residual direct personal identifiers that cannot be safely removed.
- •Duplicate records, repeated exports of the same examination, or records for which the unique report version cannot be confirmed.
- •Reports judged by the research team to be of insufficient quality for manual annotation, model evaluation, or statistical analysis.
研究组 & 干预措施
PSMA PET/CT Report Text Validation Cohort
Patients with prostate cancer or suspected prostate cancer who underwent PSMA PET/CT as part of routine clinical care. De-identified Chinese PSMA PET/CT report texts and necessary baseline information will be used for manual annotation, large language model-assisted imaging cTNM staging annotation, uncertainty recognition, internal validation, external validation, prospective validation, and human-AI comparison. No additional examination, treatment, or follow-up will be assigned for this study.
干预措施: Large Language Model-Assisted Report Annotation (Other)
结局指标
主要结局
Accuracy of LLM-Assisted Imaging cTNM Staging Annotation
时间窗: After freezing the model and prompt versions, through completion of internal, external, and prospective validation, up to 18 months
The primary outcome is the accuracy of the large language model in identifying report-derived imaging cT, cN, and cM categories from de-identified Chinese PSMA PET/CT report texts. The LLM-generated cT\_report, cN\_report, and cM\_report will be compared with the expert consensus reference standard. Accuracy, precision, recall, F1-score, macro-F1, micro-F1, complete cTNM triplet matching rate, and confusion matrices will be calculated in the internal 300-report validation set, external validation sets, and prospective 100-report validation set.
次要结局
- Component-Level Accuracy of LLM-Based Uncertainty Recognition(After freezing the model and prompt versions, through completion of all validation analyses, up to 18 months.)
- Complete cTNM Triplet Matching Rate for Human Evaluators and the LLM(During pre-training and post-training human annotation rounds and LLM batch inference, up to 18 months.)
- Annotation Time per Report for Human Evaluators and the LLM(During pre-training and post-training human annotation rounds and LLM batch inference, up to 18 months.)
研究者
Qi Lin, MD
Principal Investigator
First Affiliated Hospital of Wenzhou Medical University