Evaluation of a Vaccine Chatbot on HPV Vaccine Confidence and Hesitancy: a Randomized Controlled Trial and Implementation Science Study

A randomized controlled trial with 2 arms will be conducted to evaluate the effectiveness of an LLM-powered chatbot on improving HPV vaccination among females aged 15 to 45, followed by a pragmatic implementation science study to assess the public acceptance of chatbot and to identify the facilitators and barriers to its implementation in a real-world setting.

The sample size is calculated based on the primary outcome of the HPV vaccination uptake among females aged 15 to 45 in China. This uptake is around 10% according to the China CDC estimation, and we assume that it can increase to 15% following the intervention. With a significance level (α) of 0.05 and 80% power (1-β=0.8), a sample size of at least 683 per group is needed to detect a 5% absolute difference in vaccination rate (from an assumed baseline rate of 10% to 15% post-intervention). The sample size is inflated to 900 per group to account for potential loss to follow-up and to ensure adequate power for a multi-center design.

Participant recruitment will be conducted using a multistage sampling approach across eastern, central, and western China. In each of three provinces, one urban area and one or two rural area will be selected. Participants meeting the inclusion criteria will be invited to participate and provide informed consent. For those aged 15 to 17, both parental or legal guardian consent and adolescent assent will be obtained. A randomization method will be employed to ensure balanced allocation across two groups, and the randomization sequence will be generated by an independent statistician to ensure a 1:1 allocation ratio. Participants and on-site intervention staff cannot be blinded; personnel responsible for blinded outcome assessment and statistical analysis will remain unaware of allocation.

Participants in the intervention group will be invited to use the HPV vaccine chatbot as well as a routine HPV educational pamphlet . The chatbot powered by a large language model (LLM), offers a scalable, accessible, and interactive platform around the HPV vaccine information. At the recruitment site, participants will access the chatbot through WeChat or a web browser and will be asked to use it for around five minutes. A WeChat group will be established, and the chatbot access link will be sent at one week, half a month, one month, one and half month, two months following the enrollment. The control group will not get access to the vaccine chatbot until the end of the trial and will receive standard care by a routine educational pamphlet about HPV infection and vaccination. Following the intervention, all participants will complete a questionnaire survey on their confidence, literacy, and hesitancy on the HPV vaccine. And their vaccination status will be obtained from official vaccination records at the end of trial.

The intention-to-treat (ITT) analysis will be used to evaluate the effectiveness of the intervention. Between-group comparisons will use t-tests for continuous variables and chi-square tests for categorical variables. HPV vaccine uptake as the primary outcome will be analyzed using log-binomial regression with study center included as a fixed effect and prespecified covariates included for adjustment. Risk ratios and adjusted absolute risk differences with 95% confidence intervals will be reported. If the log-binomial model does not converge, modified Poisson regression with robust sandwich variance estimation will be used. Timing of vaccination will be evaluated using Kaplan-Meier survival curves and log-rank tests, with Cox proportional hazards models to estimate intervention effects on time-to-vaccination. Binary secondary outcomes such as vaccine hesitancy and perceived disease risk will be analyzed using log-binomial or modified Poisson regression, consistent with the primary-outcome analysis; and continuous secondary outcomes such as vaccine confidence and literacy will be analyzed using linear regression models that account for study-center effects and adjust for prespecified baseline covariates. Prespecified subgroup analyses will assess heterogeneity of intervention effects across populations stratified by baseline characteristics. In addition, a per-protocol analysis will be conducted on the primary outcome, and within the intervention group, the effect of intervention intensity measured by chatbot engagement levels on primary outcome will also be evaluated.