Perception of Health Promotion Messages

Not Applicable

Completed

• Conditions: Perception of Vaping

• Registration Number: NCT07042789
• Lead Sponsor: The University of Queensland

Brief Summary

The specific aims of this study are:

1. To compare the perceived message effectiveness (PME) of AI-generated vaping prevention materials with existing traditionally generated (typically by a health authority) prevention materials in a sample of young people. This aim addresses the gap in understanding whether AI can match or exceed the effectiveness of traditional prevention materials in changing youth perceptions and intentions of vaping.

2. To examine the impact of ad labelling (AI-generated, AI-generated with health authority endorsement, and unlabelled) on the PME of vaping prevention messages. Given the novelty of AI in this context, understanding the influence of transparency about message origin and endorsements on message reception is critical.

Detailed Description

Study hypotheses and design were pre-registered on OSF (https://osf.io/nzdt2/) and ethics approval for the study was obtained from The University of Queensland (2024/HE000318).

Six hundred and fourteen participants aged 16-25 years old, living in Australia were recruited through the Nielsen Consumer LLC and marketing databases provided through NielsenIQ Datasets. A quota was applied to ensure that there were equal distributions across age and gender. Informed consent was obtained from all participants prior to the start of the survey.

AI-generated, co-designed ads The detailed methodology of the co-design development process is available to as an OSF Preprint. There were 2 phases for ads generation. In Phase 1, the investigators conducted two focus group (N = 10 total participants, mean age 16.3 years) to evaluate an initial set of 100 AI-generated ads created using a basic approach of zero-shot prompting (model is given a task without any examples of existing vaping prevention), single-prompt usage (one prompt is used to interact with the model) and automatic text-image integration (automatic overlaying of AI-generated text onto corresponding AI-generated images).

Based on the feedback from Phase 1, the investigators refined the approach for Phase 2 to a few-shot prompting approach (model is given a task along with a few examples in the prompt to guide its response) with multi-prompt usage (multiple sequential prompts are used to refine or build upon previous interactions with the model until ads meet quality criteria for accuracy, relevance, and persuasion attempt) and manual text-image integration (text and images) by TS to ensure strong text-image cohesion. Twenty-five AI-generated ads were created using Claude 3 for text and Midjourney v6 for images. These refined ads were evaluated through nine semi-structured interviews (N = 9, mean age 16.3 years).

The feedback from Phase 2 interviews was used to further refine the ads in Phase 3, resulting in a final set of 25 ads, based on five themes: addiction, financial impact, health consequences, industry manipulation and social norms.

Existing ads from official health agencies The investigators systematically identified vaping prevention ads from official health agencies using a two-phase search. First, official health agencies from the U.S. with active anti-vaping campaigns, such as the World Health Organization (WHO), U.S. Food and Drug Administration (FDA), Centers for Disease Control and Prevention (CDC), and National Health Service (NHS, UK) were searched and identified. Second, a comprehensive searches of the health agencies' official websites and social media accounts (Facebook, Instagram, Twitter) using keywords such as "vaping prevention," "e-cigarette awareness," and "youth vaping" were conducted.

To ensure consistency and quality, strict inclusion criteria for the ads were established. Ads were included if they: (1) were created by an official health agency, (2) related to one of our five pre-determined themes, (3) were youth-oriented, (4) were in English, and (5) were available as picture files that were in square format or could be cropped to square format.

Twenty-five ads, prioritising those from WHO and FDA due to their global reach and extensive youth-focused campaigns, were selected. The logos and other descriptors of the health agency that created them were carefully removed. All 50 ads are publicly available through our GitHub repository: https://github.com/gckc123/AIvaping.

Procedure This study employed a 2 (material source: AI-generated vs. existing health agency) by 4 (source labelling conditions: control condition, AI condition, health agency condition and combined condition) mixed experimental design, where ad source was manipulated within-subjects and source labelling was between subjects. Each participant viewed and evaluated 50 ads in random order, of which 25 were generated by AI in the co-design process, and 25 were existing ads from official health agencies, giving a total of 30,700 observations.

Participants were randomly assigned to one of four source labelling conditions to examine how labelling influenced ad evaluation: (1) Control condition where ads were presented without any source labelling, (2) AI condition where ads were labelled with the text "Made with AI", (3) Health agency condition where ads were labelled with the text "Made by the World Health Organization (WHO)", and (4) Combined condition where ads were labelled with the text "Made with AI, by the World Health Organization (WHO)".

Measures Perceived Message Effectiveness (PME) Each ad were rated with five items adapted from the validated University of North Carolina (UNC) Perceived Message Effectiveness (PME) Scale for Youth. Two items assessed effects perceptions, i.e. the ads' potential to change attitudes and behaviours, and three items assessed ad perceptions, i.e., judgements about ad characteristics.

The effects perceptions included the following two survey items: "This ad makes me think vaping is _____" (1 = a very bad idea, 4 = neither a good nor a bad idea, 7 = a very good idea) and "This ad _____ from vaping" (from 1 = strongly discourages me to 7 = Strongly encourages me). Lower score indicates stronger effect perceptions. The ad perceptions included the following three survey items: "This ad grabbed my attention.", "This ad provided useful information." and "This ad was convincing." Participants rated these items on a 7-point Likert scale from 1 (Strongly disagree) to 7 (Strongly agree). Higher score indicates stronger ad perception.

2.4.4 Manipulation Check At the end of the experiment, participants were asked "Throughout this study, you viewed several health ads about vaping. How were these ads labelled?" Participants then selected the response option that best matched what they remembered seeing including "The ads were labelled as "made with AI", "The ads were labelled as "made with AI, by the World Health Organization (WHO)", "The ads were labelled as "made by the World Health Organization (WHO)", "The ads did not have any specific label".

Data analysis All analyses were conducted in R version 4.3.3 with the lmer and lmerTest packages. Linear mixed models were used to test the difference in rating between the AI-generated ads and existing ads from official health agencies. For each of the five PME measures, two models were fitted, one with the labelling conditions and one without. Likelihood ratio tests were used to test the overall effect of labelling. Two random intercepts were specified, one was used to account for the repeated measure from participants and one was used to account for the repeated presentation of ads across participants. The non-inferiority margins were set at 0.5 points in the 7-point scale.

For the hypothesis regarding labelling effects (H2), main analysis using the full sample (N = 614) and sensitivity analyses using only the 462 participants (75%) who correctly selected their assigned labelling condition in the manipulation check were conducted.

Study Timeline

• Study Start: 2024-08-24
• Primary Completion: 2024-11-20
• Study Completion: 2024-11-20
• Status Verified: 2025-06
• Study First Submitted: 2025-06-04
• Study First Submitted QC: 2025-06-19
• Last Update Submitted: 2025-06-19
• Study First Posted: 2025-06-29
• Last Update Posted: 2025-06-29

Recruitment & Eligibility

• Status: COMPLETED
• Sex: All
• Target Recruitment: 763

Inclusion Criteria

• Living in Australia
• Aged 16-25 years
• Able to read English
• Committed to reviewing each advertisement and providing honest, personal responses

Exclusion Criteria

• Unwilling to commit to reviewing each advertisement honestly
• Unable to promise to provide genuine responses to survey questions

Study & Design

• Study Type: INTERVENTIONAL
• Study Design: FACTORIAL

Primary Outcome Measures

Name	Time	Method
Perceived Message Effectiveness (PME)	Within 30mins of exposure	Each ad was rated with five items adapted from the validated University of North Carolina PME Scale for Youth. Two items assessed effects perceptions, i.e. the ads' potential to change attitudes and behaviours, and three items assessed ad perceptions, i.e., judgements about ad characteristics.

Secondary Outcome Measures

Name	Time	Method
Vaping and Smoking Behaviour	Within 30mins of exposure	Questions regarding participants' experience with cigarettes/other tobacco products, and e-cigarettes/vapes. These measures allow for the classification of participants according to their smoking and vaping status and are adapted from the National Drug Strategy Household Survey (NDSHS) 2019.
Vaping susceptibility	Within 30mins of exposure	For participants who have never vaped, susceptibility to vaping will be assessed using a 3-item scale adapted from past studies (Noar et al., 2020). The items assess how likely it is that the respondent will: (1) use an e-cigarette or vape soon, (2) use an e-cigarette or vape in the next year and (3) if one of your best friends were to offer you an e-cigarette or vape, would you use it?
Intention to quit	Within 30mins of exposure	For participants who are currently vaping, the question "Are you currently thinking about quitting vaping?" will be asked.
Health behaviour	Within 30mins of exposure	Participants will be asked 13 questions regarding their current health behaviours modified from Hampson et al. (2019) version, with too similar questions removed and three questions modified to better suit the focus of this study. These questions measure what current actions are being done to protect ones own health.
Trust in health agencies	Within 30mins of exposure	Participants will be asked 14 questions regarding their trust in public health authorities adapted from Holroyd et al. (2021), to investigate the possible relationship between trust in authorities and trust in health adverts.
Demographics	Within 30mins of exposure	Information on age, gender, indigeneity, subjective financial situation and the primary language spoken at home will be collected. These measures are designed to detail the profile of the study participants and were adapted from the NDSHS 2019, the Williams Institute (2014) and Williams et al., (2017).
General attitudes towards artificial intelligence scale (GAAIS)	Within 30mins of exposure	Participants will answer 20 items about their attitudes towards AI which is adapted from Schepman and Rodway (2023) to investigate the relationship between labelling and negative attitudes towards AI.

Trial Locations

Locations (1)

National Centre for Youth Substance Use Research; 🇦🇺 Brisbane, Queensland, Australia