Evaluation of enhanced mental and behavioral health training for family medicine residents: a research protocol

Abstract

Background

The treatment gap for mental and behavioral health (MBH) in the United States (US) remains a major public health concern. Given the growing need for a robust MBH workforce, particularly for underserved populations, calls for integrated MBH in primary care have been mounting. Family medicine providers, who know and can treat all members of a family within the same setting, are uniquely positioned to manage MBH conditions.

Objectives

With HRSA funding, the University of Utah Family Medicine Residency (UUFMR) seeks to address gaps in mental health services by enhancing or developing MBH training and partnerships. This protocol describes the project’s evaluation. The evaluation aims to identify areas to improve training content, describe training capacity, and assess intermediate outcomes of improved trainings.

Methods

The evaluation consists of three components: analyzing current curriculum and best practices, developing or enhancing trainings with partners, and assessing residents’ and graduates’ confidence in providing MBH care.

Results

The results from this protocol fill gaps in the current literature regarding evaluation methods for provider- and organizational-level outcomes of increased quality and capacity of residency training in MBH. Further, the results provide practical guidance for other residencies seeking to integrate MBH training into their curriculum.

Conclusion

Considering the resources committed to the ongoing enhancement of resident education, it is crucial to evaluate the implementation and outcomes of improvements to ensure that limited resources are well-utilized. Assessing the training capacity developed through collaboration supports progress toward creating a high-quality, accessible, and integrated mental and behavioral healthcare system in primary care.

Introduction

The treatment gap for mental and behavioral health (MBH) in the United States (US) remains a major public health concern. Over 1 in 5 Americans experience mental illness each year, and more than 1 in 5 children currently or at some point in their life have experienced a debilitating mental illness [1]. Rural, LBGTQ+, and other medically underserved populations often have less access to MBH services and resources, even though they are at similar or greater risk of mental illness [2, 3]. To address the treatment gap and strengthen the MBH workforce, calls have been made for the integration of MBH services into primary care settings. Integration involves a collaborative approach to MBH care, and leverages clinical and community partners to reach underserved populations and improve patient outcomes [3, 4].

Primary care has played an increasingly important role in filling the mental health treatment gap in the US over the past decades; this role includes providing an increased volume of mental health services, prescribing more and a wider range of medications for mental health disorders, and treating a wider range and severity of mental health disorders [5]. Given that individuals with MBH diagnoses, including substance use disorders, access both primary care and hospital-based services at greater rates than individuals without these diagnoses [6], primary care residencies must strengthen their capacity to train physicians who are prepared to treat MBH conditions across patient populations [7,8,9,10,11]. In particular, family medicine providers, who know and can treat all members of a family within the same setting, are uniquely positioned to manage MBH conditions [12, 13].

Training family medicine residents to collaborate with other clinic- or community-based mental health professionals prepares them to practice MBH care in their current and future practice. Residencies dedicate substantial resources to enhance the quality of training. However, there is limited research on the implementation and outcomes of improvements to MBH training in family medicine residencies, making it challenging to determine whether progress toward integrated MBH care has been made. This article describes the evaluation of a project aiming to increase the number of family medicine physicians prepared to provide MBH care through increased training capacity and quality.

Methods

Project

This project takes place at the University of Utah Health, Salt Lake City, Utah, and is supported by the Health Services and Resources Administration (HRSA) of the US Department of Health and Human Services as part of an award totaling $2.5 million. The contents of this protocol are the authors’ and do not necessarily represent the views of nor an endorsement by HRSA, or the US government. The project focuses on three interrelated goals:

• Develop or enhance MBH training and preparation,

• Strengthen partnerships for MBH across populations, and

• Increase the number of primary care physicians who are trained to manage MBH conditions.

Evaluation design

The evaluation aims to identify areas to improve training content, describe training capacity, and assess intermediate outcomes of improved trainings. To achieve these aims, the evaluation utilizes various methods and samples including residents, faculty, and partners.

Figure 1 shows the components of the evaluation and how they inform or influence project goals. Component 1 is an analysis of the current curriculum and a comparison with best practices from professional bodies that serves as the rationale for improving trainings. Component 2 represents the bulk of project work, where new trainings are developed, or where current trainings are enhanced in conjunction with community and clinical partners in MBH. Component 3 assesses the intermediate outcomes (e.g., perceived confidence of current and former residents in providing mental and behavioral healthcare) of applied MBH training.

Fig. 1

Project evaluation framework

Curricular components of family medicine residency

Figure 2 shows how the components of the residency curriculum are structured. Family medicine residency at the University of Utah lasts for three years, and the curriculum includes didactics and rotations covering a full spectrum of family medicine topics. Didactics are structured lessons, usually delivered by faculty or other partners, and are heavily focused in the first year of residency to build foundational knowledge. There are four categories of didactics: intern-specific training, training for residents in years 2 and 3, Journal Club, and Grand Rounds. Within each didactic, there are categories of topics, such as behavioral science or pediatrics. Topics consist of individual activities that correlate with a specific competency (e.g., depression management in children). Curricular enhancements occur at the activity level for didactics.

Fig. 2

Curriculum components of FM residency training

Rotations are hands-on training experiences encompassing the full scope of family medicine practice. During rotations residents visit and work with different clinical or community-based sites for approximately one month. There are required rotations for all residents, and elective opportunities. Rotations correspond with different family medicine required competencies, and curricular enhancements also occur at the activity level.

Residents apply the knowledge and skills obtained during didactics and rotations in their longitudinal continuity clinic. Throughout residency, they provide full-spectrum family medicine care to their own panel of patients, supervised by an attending family medicine physician with progressive independence. Enhancements to continuity clinic often involve modifications in interactions and teaching between attending physicians and residents (Fig. 2).

Component 1: comparative needs assessment

The first component is a comparative needs assessment and consists of a survey-based analysis of the MBH content in the current curriculum. The goal of the analysis is to identify didactic trainings and rotations that are related to MBH for comparison with professional standards, competencies, and guidelines regarding family medicine residency training in MBH [13]. The findings from this analysis support the identification of activities for targeted enhancement and promote the integration of MBH concepts across the curriculum.

Participants

Faculty members on the residency’s curriculum committee complete the surveys. To facilitate survey completion among faculty with busy schedules, the project director schedules one-on-one meetings with participants well in advance to reserve participants’ time and be available for questions.

Survey development

To catalog the existing curriculum and develop a reference for survey participants, the evaluation team collects and compiles the objectives for each activity into tables organized by topic. Each table includes the name of the activity and the learning objectives of the activity.

An electronic survey is developed in Qualtrics based on this catalog. The survey asks participants to rank the relationship between activities’ objectives and MBH. Relatedness to MBH is measured in three categories: not related (1), moderately related (2), and highly related (3).

Data collection

Surveys are administered in two waves to reduce the time burden on participants. Surveys on didactics are administered first. Each participant receives an email containing survey information and survey instructions, the survey link, and the curriculum catalog for reference. Participants complete a total of 4 surveys, one for each didactic sub-component. Each survey takes about 3–5 min to complete, or 20 min in total.

A few weeks later, the surveys on rotations are administered in the second wave. Participants receive another email containing survey information and instructions, the survey link, and the curriculum catalog for reference. Participants complete a total of two surveys, one for each rotation sub-component. Each survey takes about 3–5 min to complete, or 10 min total.

Data analysis

Responses are quantified into a consensus score - the proportion of respondents who agree with one another on the relationship between an activity and MBH. A consensus score is calculated for each activity and for each overall topic. For each activity, the consensus score is the average of each respondent’s relatedness score. The average of all the consensus scores for activities in a topic informs the topic consensus score.

Component 2: organizational network analysis

A significant portion of residency training occurs in clinical and community-based settings. Therefore, the achievement of project goals is largely tied to improving training through the development of partnerships. The second component documents and analyzes the role of partnerships in providing didactics and rotations and consists of a survey-based organizational network analysis. The use of network analysis to evaluate partnerships between community-based organizations and academic institutions like medical residencies is underutilized; partnerships are often assumed, and not systematically evaluated [14]. The goal of the analysis is to demonstrate that partnerships have been formed and strengthened to enhance MBH training and preparation.

Participants

The network is defined by clinical and community partners who provide letters of support during the grant application process, or who join the project later. These organizations include integrated behavioral health and primary care sites and community-based organizations that are interested in providing MBH training to residents. The point of contact for communication and collaboration about grant activities is invited to complete the survey.

Survey development

Two articles exploring the value of network analysis for addressing complex healthcare needs in the community inform survey content and structure [14, 15]. The survey measures relationships and collaboration between organizations at two time points, before the project award kickoff meeting and currently. The survey results will be analyzed to determine: (1) network density (i.e., the percentage of all possible relationships that are formed), (2) the type and frequency of collaboration, and (3) bidirectional satisfaction with the partnership. The survey components are summarized in Table 1.

Table 1 Network survey components

Data collection

Surveys are administered via email using Qualtrics. Survey completion is tracked, and respondents who have not completed the survey are reminded every two weeks. Follow-ups are sent until no new responses are collected, or until all possible respondents have completed the survey. The survey takes about 10 min to complete.

Data analysis

Data analysis is conducted in RStudio using the network analysis tool. Network density is measured from 0 to 100%, with 100% representing a perfectly connected network, and 0% representing an unconnected network where members do not know each other. An increase in network density indicates that relationships have formed among partners. Regarding level and type of collaboration, an increase in the frequency or type of collaboration demonstrates stronger relationships. We also assess the correlation between different results.

Component 3: MBH learning outcomes surveys

Residents who receive enhanced MBH training (e.g., attending new rotations with community partners) during their residency ideally graduate feeling confident in their ability to manage MBH conditions across patient populations and life stages. The third component assesses this outcome through two surveys, one for residents and one for residency graduates. The goal of these surveys is to demonstrate an increase in the number of physicians who are confident providing in MBH care.

Participants - graduate survey

The participants for the graduate survey are any graduates of the residency program before 2023. Graduates in the classes of 2023, 2024, 2025, and 2026, are included in the following year’s iteration of the survey. The maximum number of participants is about 200 graduates across the project duration.

Participants - resident survey

The participants for the resident survey include all residents entering the family medicine residency program from 2023 to 2027. In total, the survey will be completed up to 120 times by 60 residents, with residents completing the survey up to three times.

Survey development

Both surveys are developed, piloted, reviewed, and revised by content experts, including family medicine physicians and educators and experienced evaluation staff. In the development process, reducing survey length survey is a major priority to protect residents’ and graduates’ time.

Survey development - graduate survey

The graduate survey consists of four sections. The first section collects information on demographics and graduation year. The second section asks about current practice location and general characteristics. Section three is the bulk of the survey and includes questions about the participants’ confidence when managing MBH conditions, providing services for mental and behavioral healthcare, and ensuring culturally and linguistically appropriate MBH care. Participants also indicate what learning experiences contributed the most to areas they feel confident in (e.g., residency training, on-the-job training), and asks if confidence could have been improved through more training during residency. Section four uses validated questions to assess the participants’ wellbeing and perceived level of burnout in the context of providing mental and behavioral healthcare [16].

Survey development - resident survey

The resident survey consists of 5 sections. The first section asks general questions about residency training (e.g., year, number and type of rotations completed). Section two includes questions about residents’ intentions for a future practice location. Section three mirrors the graduate survey and asks residents about their confidence managing and treating MBH conditions. Residents describe which learning experiences contributed the most to their confidence (e.g., rotations, self-study) and how their confidence could be improved through specific training methods during residency. Section four asks about other intentions for future practice (e.g., providing education, seeking loan forgiveness, practicing internationally). The final section collects demographic and other background information.

Data collection

Both surveys are administered via Qualtrics email to residents or graduates. Survey completion is tracked, and follow-up emails are sent every two weeks to residents or graduates who have not completed the survey. Follow-ups are sent until no new responses are collected or enough responses are collected for analysis.

Data analysis

Analyses for both surveys will include descriptive statistics such as frequencies, means, and standard deviations to summarize demographic or other survey responses, and to compare descriptive statistics across the years of implementation to monitor changes. Additionally, given sufficient sample sizes, some item responses will be combined into scales for statistical tests including t-tests, ANOVA, or chi-square tests. Regression models may also be employed to test for relationships between variables.

Results

The results from this protocol fill gaps in the current literature regarding evaluation methods for provider-level (e.g., confidence), and organizational-level (e.g., partnerships) outcomes of increased quality and capacity of residency training in MBH. Further, results provide practical guidance for other residencies seeking to further integrate MBH training into their curriculum.

While calls for increased MBH training in primary care residencies have been growing for many years [17,18,19,20,21,22,23], research on the outcomes and implementation of enhanced training in MBH, particularly in family medicine residency, is scarce [24,25,26]. The results from this evaluation will help to fill these gaps.

Conclusion

This protocol describes the evaluation of a project aimed at improving mental and behavioral health training for family medicine residents. Considering the resources committed to the ongoing enhancement of resident education, it is crucial to evaluate the implementation and outcomes of improvements to ensure that these limited resources are well-utilized. Moreover, assessing the training capacity developed through collaboration supports progress toward creating a high-quality, accessible, and integrated mental and behavioral healthcare system in primary care. More research is needed, especially in the family medicine setting, to implement effective training practices for residents and describe the outcomes.