Impact of a continuing medical education program on primary care providers - an analysis of a child psychiatry education program in Saskatchewan

Abstract

Background

Treating children and youth with mental health problems in the primary care setting is challenging for many reasons. One barrier to care is a lack of physician knowledge of child psychiatry. Saskatchewan has implemented a 6-month course developed by the non-profit REACH Institute and adapted to Canada (CanREACH), focused on teaching and implementing evidence-based child psychiatric diagnostic and treatment methods into primary care. Our study focused on determining this program’s impact on primary care providers (PCPs) and patients.

Methods

To determine the impact of CanREACH, we assessed the knowledge and skills gained by PCPs and determined whether these were retained over time. To evaluate systemic impact, we examined if the pre-training and post-training referrals to an outpatient child psychiatry clinic were impacted for PCPs who had taken the course.

Results

PCPs developed significant skills and comfort in assessing, diagnosing and treating various mental illnesses and retained this knowledge over time. PCPs who had taken the course had a significant reduction in referrals (0.9 per PCP) made to child psychiatry, as compared to those who had not taken the course (1.3 per PCP) (p < 0.05). This long-term retention of skills provides reassurance about the effectiveness of the CanREACH program.

Conclusions

Education improved the capacity of primary care providers to manage child and adolescent psychiatry cases, reducing the need to refer children to psychiatry subspecialists. This ultimately improved access to outpatient child psychiatry care for Saskatchewan residents.

Introduction

Psychiatric disorders are common among children and adolescents, with estimates that 1 in 5 have a diagnosable mental illness [1, 2], and 70% of all mental health problems have their onset in childhood and adolescence [3]. Access to mental health care is severely limited in Canada [4], with a significant shortage of specialists [5]. While an improved stepped-care model would allow patients to receive care with the least intensive resource [5,6,7], PCPs identify skills and knowledge deficits as barriers to care provision and an increased rate of referral to subspecialists [8, 9]. With limited access to care, emergency room visits for childhood mental health concerns are rising in Canada [10], which is costly and limited in scope.

Continuing medical education (CME) to enhance mental health care competencies in PCPs can be an effective strategy to fill the gaps in knowledge of psychiatric care to children and adolescents and promote collaborative and stepped care [11, 12]. Nevertheless, not all CME programs studied have shown a sufficient and/or sustained impact [13,14,15]. The REACH Institute (www.TheReachInstitute.org) has developed the Patient-centered Mental Health in Pediatric Primary Care (PPP) program: an intense interactive 16-hour course followed by six months of bi-weekly case-based virtual peer consultation based on adult education and behaviour change science [16]. This program has improved evidence-based Attention-Deficit/Hyperactivity Disorder (ADHD) diagnosis and management by PCPs [17]. The REACH Institute’s PPP training program has been adapted to the Canadian context (CanREACH) and delivered in Alberta [18] and Ontario before being introduced in Saskatchewan in 2018. In Alberta, research has shown advanced knowledge and comfort in assessing and diagnosing mental health (MH) problems, a trend to decreased referrals to the Emergency Department, a decreased length of hospital admission days, and an associated cost-saving averaging $20,000 per trained PCP [18, 19]. The impact of CanREACH on referrals to Child and Adolescent Psychiatry subspecialists has not been evaluated to date.

Our research seeks to build on prior knowledge by studying the impact of CanREACH on referrals to child and adolescent psychiatry subspecialists in the outpatient setting. We also assess the impact of CanREACH on knowledge and competence in assessing and diagnosing nine MH presentations. We explore self-perceived intended and reported changes in clinical practice and summarize the study findings concerning training experience, learning outcomes, and behavioural and systems changes utilizing Kirkpatrick’s Model of Learning (KML) framework [20].

Methods

Program description

The CanREACH program is an interactive educational program reviewing evidence- based guidelines for the assessment and treatment of common pediatric mental health presentations offered through the Division of Continuing Medical Education at the University of Saskatchewan to all primary care providers working in the province. (Appendix 1)

Data collection

Program participants completed self-report surveys, developed by REACH for program evaluation, at three time points (i.e., pre-training (T1), completion of the 3-day training (T2) and the end of 6-month training (T3)), (November 2019 – July 2022). These surveys were used to evaluate participants’ knowledge, comfort, and current approaches to assessing, diagnosing, treating, and managing each of the nine MH conditions (Appendix 1). Additional survey items addressed competence in prescribing psychotropic medications, rating scales, screening tools and learning experiences. Open-end questions addressed intended (T2) and reported changes (T3) in PCPs’ practices.

The number of referrals made by trained PCP one year before and one year after the 3-day training was quantified from the Electronic Medical Records (EMR) of the Division of Child Psychiatry in Saskatoon. The participant demographic data is reported under Results and Table 1.

Table 1 Demographics

Data analysis

Quantitative data analysis (two-sided) was performed using Statistical Package for the Social Sciences (SPSS v28) at the 5% significance level.

Self-reported surveys

Descriptive statistics characterized study participants and outcomes. The impact of the CanREACH on educational outcomes was examined using the Wilcoxon Signed Rank test (individual scores) and a mixed model for repeated measures (overall scores, i.e. scores from all MH conditions). For dichotomous variables, comparisons between pre- and post-training were conducted using McNemar’s test.

Referral data

The number of referrals made by trained PCPs one year before and after training was compared using the Wilcoxon Signed Ranks test. Data on average # referrals/PCP was not normally distributed, therefore non-parametric tests were used to compare median between studied groups.

Kirkpatrick’s ‘model of learning’ framework

The KML framework (Appendix 2) was applied to evaluate learners’ educational experience holistically, changes in knowledge, and the benefits of clinical care on primary beneficiaries, i.e., patients [20].

Qualitative data

Open-ended questions were asked about changes the PCP would integrate into their practice after the completion of the program. PCPs’ answers were independently reviewed by four coders. Upon reaching consensus, an inductive coding scheme was developed. Content was grouped into common themes/ subthemes were further clustered into overarching categories and were further quantified as to prevalence of themes and subthemes (%).

Results

Demographics

One hundred twenty-five participants were invited to complete self-reported surveys. Most were family physicians, psychiatrists, pediatricians, nurse practitioners, and medical trainees (Table 1).

Knowledge in assessing, diagnosing, treating, and managing MH conditions

Statistically significant and sustainable improvements in scores of knowledge of assessing and diagnosing MH conditions were observed immediately and at six months compared to the baseline score (p < 0.001) (Fig. 1A). Comparable improvements in knowledge of treating and managing MH conditions were noted (p < 0.001) (Fig. 1B). Total scores at T2 and T3 were not different (Assessing and Diagnosing: p > 0.05 (Appendix 3); Treating and Managing: p > 0.05 (Appendix 4; Fig. 1E).

Fig. 1

Knowledge and comfort in assessing, diagnosing, treatment, and management of mental health conditions. Figures A, B, C, and D show average knowledge and comfort scores in assessing, diagnosing, treating, and managing nine mental health conditions across the study period. Day-3 post-training scores are consistently higher than pre-training for all disorder categories in all knowledge and comfort measures. Changes from day 3 to 6 months post-training are smaller, with average scores relatively stable. Similar results can be seen in the cumulative total scores (E and F). Thicker lines (Attention-Deficit/Hyperactivity Disorder (ADHD), Anxiety, Depression) indicate disorders that were the primary focus of the training. Dashed lines indicate disorders that were the least focus of the training

The degree of knowledge improvement varied across MH conditions (Fig. 1A). Although knowledge scores regarding assessing and diagnosing increased rapidly and were steady at T3 for all conditions, continuing improvement over time (i.e. T3 vs. T2) was reported only for ADHD (Z = -2.83; p < 0.01), Anxiety Disorders (Z = -2.98; p < 0.003), Major Depressive Disorder (MDD) (Z = -2.86; p < 0.004) and Bipolar Disorder (BD) (Z = -2.15; p < 0.03) (Appendix 3). The knowledge scores in treating and managing continuously improved for anxiety disorder (Z = -2.24; p < 0.003) and BD (Z = -2.15; p < 0.03) only (Appendix 4).

Comfort in assessing, diagnosing, treating, and managing MH conditions

There was a statistically significant improvement in self-reported comfort in assessing and diagnosing MH conditions immediately after training compared to the baseline (p < 0.001) (Fig. 1C F). The improvement was sustained at the 6-month follow-up assessment (p < 0.001). However, no further change was observed compared to T2 (p > 0.05). Improvement of comfort score in treating and managing MH conditions was also rapid (p < 0.001) and sustained (p < 0.001) (Fig. 1D). No difference was observed between T3 and T2 (p > 0.05). An immediate and sustained increase in comfort scores in the assessment and diagnosis of MH conditions were comparable across conditions (p < 0.001) (Fig. 1C; Appendix 5). Comfort scores further improved for ADHD (Z = -2.68; p < 0.01), anxiety (Z = -2.71; p < 0.01), and BDs (Z = -2.33; p < 0.05). Positive sustainable changes in comfort of treatment and management of MH conditions were also comparable across conditions (Fig. 1D), however comfort scores further improved (i.e. T3 vs. T2) for ADHD (Z = -3.27; p < 0.001), anxiety disorder (Z = -2.50; p < 0.05), MDD (Z = -2.51; p < 0.05), and suicide (Z = -2.15; p < 0.05) only (Appendix 6).

The current approach to assessment, diagnosis, treatment, and management of MH conditions

Participants reported improving their approach to assessing and diagnosing MH conditions (Figs. 2 A and 2 C). Instead of referring to most suspected cases, trained PCPs felt capable of assessing and diagnosing less complex (the majority of) cases. This rapid improvement (p < 0.001) advanced over time (p < 0.05) and was sustainable at the six-month point (p < 0.001). This positive change was observed across all mental health conditions (Appendix 7). However, continuous improvement was identified for anxiety disorder (Z = -2.92; p < 0.004) and MDD (Z = -2.92; p < 0.004) only. More PCPs reported that they managed most cases with only occasional consultations after training rather than referring most cases and/or assessing and managing only less complicated cases pre-training (p < 0.001)) (Fig. 2B C, Appendix 7 and 8). No difference was noted between T3 and T2 (p > 0.05). The scores for current clinical practices, in favour of managing most cases independently, increased over time across all MH conditions (p < 0.001) (Appendix 8). This positive change was further advanced at T3 for anxiety disorder only (Z = -2.76; p < 0.01).

Fig. 2

Current approach in assessment and diagnosis of mental health conditions. Figures A and B show current referral practices (treating, managing, assessing, and diagnosing) for nine mental health conditions at the three time points of the study. Lower average scores indicate a tendency of the trained physician to refer patients with the disorder, while higher scores indicate a tendency to assess, diagnose, treat, and manage themselves. Average scores for all disorders increase from pre-training to 3-day post-training but stabilize at the 6-month pos- training timepoint. The total score for assessment and diagnosis (C) ranked from 9 to 45 (since the individual scale is from 1 to 5. The total score for treatment and management ranked from 9 to 54 (since the individual scale is from 1 to 6). Thicker lines (ADHD, Anxiety, Depression) indicate disorders that were the primary focus of the training. Dashed lines indicate disorders that were the least focus of the training

Comfort with prescribing psychotropic medications

Most PCPs felt comfortable prescribing selective serotonin reuptake inhibitors (SSRIs) and stimulants at baseline (Table 2). In contrast, only half of the PCPs felt comfortable prescribing atypical antipsychotics, and the majority were uncomfortable with prescribing mood stabilizers. The number of PCPs with comfort in prescribing the studied psychotropic medications significantly increased at T2 vs. T1 (p < 0.05). No additional improvements were observed at T3 (p > 0.05); however, this positive shift was sustained for prescribing stimulants, atypical antipsychotics and mood stabilizers (p < 0.05).

Table 2 Comfort with prescribing psychotropic medications

Use of rating scales and screening tools

Utilization of rating scales and screening tools variably increased for both diagnosis and assessment and treatment and monitoring of patient’s psychosocial status and the studied MH conditions after training (Appendix 9). Positive changes were immediate and sustainable (p < 0.05). Before training, some scales and screening tools were used more often (∼ 60%) for ADHD, Anxiety Disorders and Depression compared to other conditions (∼ 25%). This positive change was not evident at T2 for some MH conditions with relatively better-established pretraining practice (p > 0.05) (ADHD, Anxiety Disorders and Depression).

Self-perceived anticipated and reported changes in clinical practice

The dominant theme in PCPs’ answers regarding CanREACH’s impact on clinical practice is related to Quality of Service (QoS) (Appendix 2 and Appendix 10). The majority discussed the positive impact of CanREACH on educational awareness, comfort, and competence. Professional Satisfaction and Program Delivery were other themes addressed by the PCPs (Appendix 11).

Referral patterns

Ninety-five of the 125 trained PCPs were identified in the EMR. The total number of referrals made by PCPs after training (87 referrals) was lower than pre-training (105 referrals). There was a significant reduction in the number of referrals made per PCP after the training (0.9 referrals post-training vs. 1.3 referrals pre-training; p = 0.04).

Interpretation

This study documents the positive impact of Can-REACH on PCPs’ clinical practice, demonstrating sustainable improvements in knowledge and confidence in diagnosing and treating nine pediatric mental health conditions. Many continuing medical educational programs have been shown to produce self-reported changes in knowledge and self-confidence, but few have produced objective changes in actual practice behavior. We observed a significant reduction in referrals made by PCPs to child and adolescent psychiatrists after completing Can-REACH. Our finding indicates how a well-designed and tailored CME program delivered by psychiatrists and primary care providers can contribute to the adoption of stepped-care principles, enhancing timely access to mental health care for children and adolescents as close to home as possible. Although no change in referral rates for outpatient mental health services before and after training was reported when the comparable training program was evaluated for the Child and Adolescent Addiction Mental Health and Psychiatry Program in Calgary, Alberta (Canada), the referral frequency was reduced for emergency services for trained PCPs [19]. This likely reflects that our study looked specifically at referrals to subspecialty child and adolescent psychiatrists and not to child and youth mental health care teams. The finding from these studies underscores the importance of access to allied mental health services to provide care in conjunction with PCPs, reserving subspecialty child and adolescent services for those with the most complex needs [21].

The reduction in the number of referrals per PCP in our study likely results from PCPs becoming more knowledgeable, confident and competent in assessing, diagnosing, treating and managing a wide range of MH conditions after CanREACH. Our findings from self-reported surveys strongly support this and align with previous observations [12, 15, 22]. Moreover, our findings enhance our understanding of how CanREACH has a positive but variable impact on several MH conditions. Although sustainable improvements were observed across all MH conditions covered in the course, continued improvements were observed for anxiety disorders, MDD, ADHD, BD and suicide risk to various degrees. Variability regarding disorder-specific improvements likely stems from enhanced course content regarding these conditions, given their increased prevalence and amenability to management in a primary care setting. Interestingly, we noted CanREACH training-related improvements in knowledge, comfort, and competence, even concerning MH conditions that were not intensively covered during teaching sessions. A positive “spillover” effect might be a reason for the favourable outcomes leading to a more positive impact on learners’ professional advancement.

Trained PCPs also report significantly increased utilization of rating scales and screening tools for diagnosing, assessing, and monitoring pediatric mental health conditions, likely contributing to the reduction in the number of unnecessary referrals. Utilization of screening tools has been shown promise in improving the quality of referrals and reducing the burden on ambulatory health care services [12, 21,22,23]. Additionally, PCPs reported enhanced comfort with prescribing studied psychotropic medications after CanREACH, likely indicating PCPs’ improved competence to manage MH conditions with confidence and independence. This aligns with comparable findings from previous reports, emphasizing the enhanced comfort of PCPs in prescribing a broad range of medications, moving from an initial comfort with antidepressants to a more active approach in managing and adjusting medications, even for psychostimulants and mood stabilizers [12]. Notably, trained PCPs reported improving their approach to assessing and diagnosing MH conditions. Instead of referring to the most suspected cases, they could independently assess and diagnose the mildest to moderate complexity cases [24, 25]. This should improve access to Child Psychiatry outpatient services by focusing on subspecialty services for those in greatest need [26].

We summarize the multifaceted positive impact of the CanREACH program on pediatric mental healthcare in Saskatchewan through the lens of Kirkpatrick’s Framework. As with most studies on medical educational programs, there is evidence of positive learner reactions. Learners rated the program as a valuable learning experience and expressed satisfaction and endorsement. The learners reported heightened comfort and competence in all aspects of clinical care for MH disorders (i.e. QoS) and a change in their clinical practice. Trained PCPs were willing to recommend CanREACH to colleagues. Assessment of changes in behaviour is planned for a future paper that analyzes the specifics of psychiatric care provided by PCPs. However, most importantly, there is evidence of change at the top of Kirkpatrick’s Model (Results), with fewer referrals for subspecialty Child Psychiatry care. This benefitted patients who received timely care close to their residence with a familiar care provider. This observable change in clinical practice bears testimony to the program’s tangible impact on PCPs’ behaviour. It aligns harmoniously with the overarching goal of CanREACH to equip PCPs with the competence required to provide quality care for pediatric mental health conditions.

Limitations

We did not have access to referral data for the Regina health region, though we captured most of the provincial data. Increasing the number of trained PCPs could facilitate demonstrating enhanced positive changes over the years. Survey data was based on self-report which carries inherent bias.

Conclusion

The CanREACH educational program resulted in fewer referrals to Child Psychiatry subspecialty services. It also significantly improved Primary Care Providers’ knowledge and skills in treating common mental health presentations in children and adolescents. This should lead to a reduced waitlist and improved access to subspecialty services for those with the most complex needs.