K-ORCA: Testing a Decision Support Tool and Group Process for Selecting Interventions

Not Applicable

Not yet recruiting

• Conditions: Social Facilitation; Implementation Science; Policy; Child Welfare; Consensus; Mental Health; Decision Making; Decision Support Technique; Child Abuse; Decision Making, Shared
• Interventions: Behavioral: Optimizing Responses with Collaborative Assessments-Live (ORCA-L); Behavioral: Optimizing Responses with Collaborative Assessments-Automated (ORCA-A)

• Registration Number: NCT06375551
• Lead Sponsor: Chestnut Health Systems

Brief Summary

This proposal responds to NIMH Objective 4.2.c to develop "decision-support tools and technologies that increase the effectiveness and continuous improvement of mental health interventions" by leveraging the Family First Prevention Services Act (FFPSA) policy opportunity. First, a web-based platform to host (a) a decision-support tool and (b) automated facilitation for group decisions with the tool will be developed with state partners' feedback. Next, decision makers leading their states' FFPSA quality improvement efforts will be engaged to pilot a decision-support intervention comprised of the tool and live or automated facilitation, and to evaluate the implementation quality of evidence-based programs adopted with the decision-support intervention.

Detailed Description

Over 268,000 children entered foster care in the United States during fiscal year 2019 despite the existence of Maltevidence-based programs (EBPs) that can successfully prevent child maltreatment. Evidence-informed decision-making (EIDM) can lead to the adoption of EBPs that are likely to be successfully implemented and result in the intended impacts. Despite the existence of EBPs to prevent child maltreatment, EBPs are underutilized in child welfare. Proposed study activities in this career development award respond to NIMH Objective 4.2.c by developing, refining, and piloting an implementation strategy to increase EIDM when decision makers are adopting EBPs to improve mental health and child welfare services. Activities leverage an unprecedented federal policy opportunity to prevent child maltreatment and test implementation decision support strategies in the real world: The Family First Prevention Services Act (FFPSA). FFPSA aims to prevent child maltreatment and foster care entries by providing federal funding to states for implementing EBPs that support child and parent mental health, reduce parental substance misuse, and improve parenting skills. This project engages decision makers from four states implementing FFPSA to extend the candidate's prior work developing an innovative EIDM tool to support EBP adoption: Optimizing Responses through Collaborative Assessments (ORCA). ORCA is based in multi-criteria decision analysis, which provides a structured approach to reach quality, evidence-informed group decisions. Given that group decisions with tools such as ORCA benefit from facilitation, this study will develop and test two facilitation modalities to accompany ORCA: automated (ORCA-A) and live (ORCA-L). In addition, a web-based platform to host the ORCA tool and the ORCA-A facilitation strategy will be developed with decision makers' input (Aim 1). The impact of ORCA-A and ORCA-L on quality of decision experiences, processes, and outcomes then will be tested (Aim 2). Implementation process quality for each EBP adopted with ORCA-A and ORCA-L will be monitored (Aim 3). To enhance her strong foundation in the implementation, decision, and systems sciences, the candidate, Dr. Gracelyn Cruden, will leverage these research activities and training activities to expand her skillset to include: leading implementation studies in service systems, employing a continuum of methods for engaging community members, leveraging interdisciplinary approaches for eliciting members' preferences, conducting advanced simulation model testing, and using advanced statistics for multilevel, longitudinal studies. Cruden will be mentored by an outstanding team led by Primary Mentor Dr. Lisa Saldana, who is accompanied by Mentors Dr. Jason Chapman, Dr. Lindsey Zimmerman, Dr. R. Christopher Sheldrick, Dr. Jonathan Purtle, and Consultants Dr. David Vanness, and Ms. Clare Anderson. Upon completion of these activities, Cruden will be equipped to lead an interdisciplinary research team that will support quality mental health and child welfare services by increasing decision makers' use of EIDM.

Study Timeline

• Study First Submitted: 2023-11-20
• Study First Submitted QC: 2024-04-16
• Study First Posted: 2024-04-19
• Status Verified: 2024-07
• Last Update Submitted: 2024-07-11
• Last Update Posted: 2024-07-15
• Study Start: 2024-07-31
• Primary Completion: 2027-07-31
• Study Completion: 2027-07-31

Recruitment & Eligibility

• Status: NOT_YET_RECRUITING
• Sex: All
• Target Recruitment: 80

Inclusion Criteria

• Non-institutionalized
• Adults (18 years or older).
• English-Speaking

Aim 1:

• Involved in initial state decision-making related to Family First Prevention Services Act.
• Willing to participate in two data collection occasions.

Aim 2:

• Non-institutionalized
• Adults (18 years or older). English-Speaking.
• Involved in ongoing state decision-making related to Family First Prevention Services Act.
• Willing to participate in three measurement occasions.

Aim 3:

-Willing to participate in multiple (bi-weekly up to two years) measurement occasions.

Read More

Exclusion Criteria

-Not involved in or potentially influencing child welfare intervention decisions

Read More

Study & Design

• Study Type: INTERVENTIONAL
• Study Design: PARALLEL

Arm && Interventions

Group	Intervention	Description
ORCA-L	Optimizing Responses with Collaborative Assessments-Live (ORCA-L)	Decision Makers receiving live facilitation to accompany ORCA
ORCA-A	Optimizing Responses with Collaborative Assessments-Automated (ORCA-A)	Decision Makers receiving automated facilitation to accompany ORCA

Primary Outcome Measures

Name	Time	Method
Mean changes in perceived potential community health impact pre-post intervention	Baseline; Immediately Post-intervention; 12-month follow-up	This will be measured across a validated, quantitative sub-scale from a measure for community partnered research: Community Health Improvement. This scale has a range of 1 (Strongly Disagree) to 5 (Strongly Agree). This is part of Aim 2b: Decision quality.
Descriptive changes in ordinal intervention rankings assisted (with ORCA) compared to unassisted (no ORCA)	Baseline; Immediately Post-intervention	Decision makers will rank interventions in an ordinal manner at baseline using a study-specific survey. Rankings will be calculated post-intervention by ORCA as a function of completing the ORCA tool. The number of ordinal rankings will depend on the number of interventions considered, to be determined by participants. A rank of 1 = highest, followed by 2 ...n. This is part of Aim 2c: Decision quality.
Descriptive differences in implementation process fidelity (timing) for interventions adopted with ORCA	Immediately Post-intervention; bi-weekly through 12-month post-intervention	A validated measure of implementation activities across 8 stages and 3 phases will be used (Universal Stages of Implementation Completion). Dates are recorded for each activity to be reported or as missing the date, but still being completed. One of the results for the calculated scores is the duration (time elapsed). The ideal duration depends on the proportion score.
Descriptive, mean group-level changes in perceived appropriateness of intervention for implementation	Baseline; Immediately Post-intervention; 12-month post-intervention	The validated, quantitative Intervention Appropriateness Measure (IAM) will be used to assess perceived acceptability for each intervention considered with ORCA. Scale ranges from 1 (Completely Disagree) to 5 (Completely Agree). This outcome is part of Aim 2b regarding decision quality operationalized as intervention fit to local context.
Descriptive differences in implementation process fidelity (activities completed) for interventions adopted with ORCA	Immediately Post-intervention; bi-weekly through 12-month post-intervention	A validated measure of implementation activities across 8 stages and 3 phases will be used (Universal Stages of Implementation Completion). Dates are recorded for each activity to be reported or as missing the date, but still being completed. One of the results for the calculated scores is the proportion of activities completed. A higher proportion of activities is generally positive.
Mean changes in decision commitment pre-post intervention	Baseline; Immediately Post-intervention	A two-item, Likert-scale measure regarding the decision maker's confidence in and commitment to implementing the selected intervention(s) will be gathered using a two-item measure. This uses a 5-point Likert scale ranging from 1 (Strongly disagree) to 5 (Strongly Agree). These scales relate to Aim 2a: decision experience quality.
Mean changes in decision assumptions, decision goals Pre and Immediately Post-intervention	Baseline; Immediately Post-intervention	A two-item, quantitative measure will inquire about assumptions and goals in the decision to adopt an intervention. This uses a 5-point Likert scale ranging from 1 (Strongly disagree) to 5 (Strongly Agree). This outcome is part of Aim 2b: Decision quality based on intervention fit to local context.
Mean changes in decision experience quality pre-post intervention	Baseline; Immediately Post-intervention	This will be measured across five validated, quantitative sub-scales from a measure of community partnered research: Partner Values, Synergy, Conflict and Cooperation, Participation, Participatory decision-making. Each scale has a range of 1 (Strongly Disagree) to 5 (Strongly Agree).
Qualitatively described rationale for program adoption from pre-post intervention	Baseline; Immediately Post-intervention	This is a study-specific, short, open-ended survey to understand decision makers' rationale for adopting programs during initial Family First Prevention Services Act decisions. There is no scale for the items, given the qualitative, open-ended nature. These are one survey of similar, but related constructs of intervention feasibility, acceptability, and appropriateness. This is part of Aim 2c: Decision quality.
Descriptive, mean group-level changes in perceived acceptability of intervention for implementation	Baseline; Immediately Post-intervention; 12-month post-intervention	The validated, quantitative Acceptability of Intervention Measure (AIM) will be used to assess perceived implementation feasibility for each intervention considered with ORCA. Scale ranges from 1 (Completely Disagree) to 5 (Completely Agree). This outcome is part of Aim 2b regarding decision quality operationalized as intervention fit to local context.
Descriptive, mean group-level changes in perceived feasibility of intervention Pre-post intervention	Baseline; Immediately Post-intervention; 12-month post-intervention	The validated, quantitative Feasibility of Intervention Measure (FIM) will be used to assess perceived implementation feasibility for each intervention considered with ORCA. Scale ranges from 1 (Completely Disagree) to 5 (Completely Agree). This outcome is part of Aim 2b regarding decision quality operationalized as intervention fit to local context.

Secondary Outcome Measures

Name	Time	Method
Time to adoption or re-adoption of interventions with ORCA	Baseline to adoption of an intervention for implementation, assessed for up to 24 months	Days elapsed from completing ORCA initial time to official decision regarding adopting or re-adopting an intervention. This outcome relates to Aim 2a: decision process quality.
Qualitative changes in decision reasoning pre- post-intervention	Baseline; Immediately Post-intervention	Group discussion transcripts will be analyzed using directed content analysis to identify themes. At baseline and post-intervention, a single-item, study-specific questionnaire will ask participants to express their rationale for adopting the selected intervention(s).
Qualitative changes in decision goals pre- post-intervention	Baseline; Immediately Post-intervention	Group discussion transcripts will be analyzed using directed content analysis to identify themes. At baseline and post-intervention, a single-item, study-specific questionnaire (the same as in Outcome 12) will ask participants to express their rationale for adopting the selected intervention(s).
Qualitative changes in decision consensus pre- post-intervention	Baseline; Immediately Post-intervention	Group discussion transcripts will be analyzed using directed content analysis to identify themes. At baseline and post-intervention, a single-item, study-specific questionnaire (the same as in Outcome 12) will ask participants to express their rationale for adopting the selected intervention(s).