Use of patient-reported outcome measures (PROMs) in primary care-based mental health programming: an environmental scan of Alberta, Canada

Abstract

AbstractSection Background

Many Primary Care Networks (PCNs) in Alberta collect Patient Reported Outcome Measures (PROMs) to support patient-centered care. However, there is limited knowledge on what tools are currently being administered across PCNs and how the data is used. For this study, we focused on PROMs for mental health programming (MHP). Our objectives are to identify what PROMs are currently being administered in PCNs and what domains they measure for MHP; understand PCNs’ capacity to implement and use PROMs data effectively for their PCN MHP; describe how PROMs are currently being reported in PCNs for MHP; and understand the feasibility of having standardized and consistent measurement of PROMs in general across PCNs.

AbstractSection Methods

This environmental scan employs a survey for PCN evaluators (those responsible for managing PROMs data for their PCN), tailored to examine PROMs in PCN MHP across all populations. Evaluators from all 39 Alberta PCNs were invited to complete the survey on behalf of their PCN. It included closed and open-ended questions. Survey results were aggregated and reported by objective.

AbstractSection Results

Evaluators from 20 PCNs (51%) completed the survey, with a mix of rural/urban and across all five health zones. Nine out of 20 reported 11 tools currently being collected and seven out of nine reported using more than one tool for MHP. The most used tools were the EQ-5D-5 L (7/9) and PHQ-9 (6/9). Seven respondents indicated the EQ-5D-5Lwas useful or sometimes useful; five reported the PHQ-9 was useful or sometimes useful. While the use of each PROM varied, most PROMs are used for clinical care decisions and internal reporting. Most respondents indicated standardizing PROMs across PCNs would be challenging, however having alignment of PROMs and sharing best practices for PCNs would be beneficial.

AbstractSection Conclusions

These results provide a better understanding of the current use of PROMs in PCNs, specific to MHP, which will be further examined through future narrative conversations. Overall, this study informs primary care leadership on the current use of PROMs and supports the advancement of PROMs use in Alberta.

Background

Primary Care is the first point of contact within the Canadian health system offering a spectrum of services including mental health services [1, 2]. The province of Alberta largely relies on Primary Care Networks (PCNs) for team-based primary health care delivery [3]. PCNs are geographically located in five health zones and each PCN operates as an independent entity to support the health needs of their communities, with their own organizational structure and staffing roles. About 84% of primary care physicians are registered with one the province’s 39 Primary Care Networks (PCNs). Together, these PCNs represent more than 3,800 doctors and 1,000 health care providers which offer a variety of services and programs to nearly 3.6 million Albertans [4]. The size and capacity of PCNs are based on their catchment population, with services and programming tailored to local needs. Given the ongoing mental health concerns of the general population [5], most PCNs offer mental health programming (MHP) to their communities. A total of 29 PCNs directly conduct MHP services, which vary in delivery models; short- or long-term support, group or individual, in-person or virtual, referral-based or not, and are provided by a number of different providers (behavioural health consultants, social workers, psychologists, etc.). Although differences exist in the delivery of such programs, they are all designed to address issues such as anxiety, depression, stress and other mental health symptoms throughout the life span. Incorporating patient-reported outcome measures (PROMs) into these programs helps evaluate their effectiveness [6], ensuring interventions are tailored to patient needs and contribute to improved mental health outcomes.

We developed this environmental scan to determine what PROMs are currently being used and how the data is used for MHP in PCNs. PROMs are validated, standardized instruments for people to appraise and self-report health status and outcomes [7], making them well-suited for MHP. They play a critical role in understanding individual and population-level health trends [8, 9], evaluating health programs and services and the effectiveness in healthcare delivery [10, 11], and informing policy [7, 12]. PROMs are being used routinely in many PCNs for various purposes [6] including in MHP. Due to the emotional, behavioural and symptom-based nature of MHP, PROMs provide additional data for PCNs to use at the individual/clinical (micro), group/PCN (meso) or broad population/provincial (macro) level [13], but how is it collected? Data shows that the use of patient’s electronic medical records (EMRs) enables systematic measurement for many indicators. By integrating PROM tools in EMRs, clinicians, patients, administrators and analysts will have access to readily available PROMs data to inform individual’s and population health [6]. Likewise, consistent and standardized measurement (PROMs and otherwise) would contribute to provincial level PCN evaluations and support overall health system quality improvement [6].

Many PCNs have made significant efforts to address patient-centered care and the patients’ medical home [14], including through the introduction of PROMs. However, there is limited knowledge on what tools are currently being administered in PCNs and how the data are being used. Furthermore, not all PCNs have equitable staffing (including a dedicated evaluator), access to EMRs/software and analytical capacity due to numerous reasons (i.e., high costs, lack of analysts and training) and data collection, storage and management practices vary [6]. This environmental scan aims to: (1) identify what PROMs are being administered and what domains they measure for MHP, (2) understand the PCNs’ capacity to implement and use PROMs data effectively for MHP, (3) describe how PROMs are currently being reported in PCNs and, (4) understand the feasibility of standardization and consistency of all PROMs within PCNs in Alberta.

Methods

Study design

This environmental scan was directed by the Alberta PROMs and EQ-5D Research and Support Unit (APERSU) PCN Working Group, comprised of five PCN staff evaluators. Each member is employed within an Alberta Health zone (regional zones in Alberta are: Calgary, Edmonton, Central, North and South). This group helps APERSU identify gaps in primary care and research, and support the development of our research projects, including this study. They identified the need to understand what PROMs other PCNs were collecting, and how the data was being used. Together, we determined the best approach was to focus on MHP, as many PCNs are tasked with evaluating their mental health services and include PROMs. To assess the meso-level application [13], a survey of current practices and perspectives was co-developed by this group and administered.

The survey’s development was informed by a broad search through published and grey literature. Search terms included “patient reported outcomes”, “patient reported outcome measurements”, “evaluation in primary care”, “patient reported outcomes in mental health”, “patient reported outcomes in primary care”, “patient reported outcomes tools in Alberta”, “evaluation frameworks patient reported outcomes”, and “patient reported outcomes measures Alberta in primary care”. Additionally, we analyzed our internal database to identify any supplementary information. Findings were presented to the APERSU PCN Working Group, and survey questions were developed, refined and finalized by consensus.

Survey data collection

Data was collected from September 2023– December 2023. The findings of our literature searches and discussions with the PCN Working Group informed our survey development and activities. The survey was co-developed over several meetings and supports our four objectives. For the first objective, we asked what PROM tools are being administered and identified what domains they measure. Further, we collected information for each PROM tool and reported its purpose, frequency of data collection, how it was selected, how the data is collected, to get a better understanding of the PROMs used. Objective two examines PCN evaluators capacity to implement and use PROMs data effectively by determining the enablers and barriers to implement PROMs in MHP. We asked about confidence in interpreting the data and usefulness. Additionally, we asked about data collection barriers PCN evaluators encountered when administering PROMs and social barriers observed when collecting PROMs from patients, and outcomes of PROMs (individual-patient level and program level) at their PCN. The third objective explored how PROMs data are currently being reported and to whom. The fourth objective aimed to understand standardization and consistency of PROMs overall, beyond MHP. Qualtrics [15] was used to administer the survey. The survey is available upon request.

Recruitment

We used convenience sampling to target and invite PCN evaluators (individuals responsible for evaluating their programming at their PCN) to participate in the survey. Not all PCNs have a dedicated evaluator on staff. In some PCNs, program evaluation is completed by a clinical nurse manager, or other leadership staff, and smaller PCNs may share evaluation roles with another PCN. For this study, we will refer to all participants as ‘PCN evaluators’. PCN evaluator’s roles varies across PCNs, including tasks such as implementing data collection, managing, analyzing, interpreting, and reporting data to inform PCN programming. While we recognized not all PCNs conduct MHP; all 39 were invited to complete the survey and two reminders were sent via email.

To administer the survey, it was important to use multiple engagement strategies for our convenience sampling to ensure we contact the appropriate persons for all PCNs. Along with our PCN Working Group, we held several conversations with key stakeholders from provincial and national health organizations, to develop a robust recruitment strategy. This collaboration was critical in connecting with hard-to-reach smaller and rural PCNs, as identifying PCN evaluators proved challenging due to their limited availability and heavy clinical workloads. We also attended three meetings with a PCN evaluator community of practice, to present our objectives and garner support. During these sessions, we highlighted the study’s value and encouraged participation. Following those meetings, 48 people were invited to complete the survey from 39 PCNs. We sent out 26 personalized emails to PCNs within our network to present our study and share the survey link. Through these individualized survey invitations, we ensured that all survey participants were directly involved in evaluating programming at their PCN. As some PCNs may have more than one person evaluating MHP, one person was designated to respond on behalf of their PCN and was encouraged to seek additional input from other staff members.

Data analysis

We exported the survey results via Excel, and cleaned and organized the quantitative data by question, then by objective. We analyzed responses from closed ended questions with descriptive statistics and used thematic analysis for open-ended questions. We coded responses to identify recurring themes and patterns. By combining both quantitative and qualitative methods by objective, it provided deeper understanding and insights to explain our findings. This environmental scan used principles of engagement and served as a professional engagement activity; for this reason, ethics approval was not required. However, the requirements outlined in the Canadian Tri-Council Policy Statement: Ethical Conduct of Research Involving Humans [16] was followed.

Results

Twenty of the 39 (response rate 51%) Alberta PCNs participated in our survey, with a mix of rural and urban settings, and across five health zones. All respondents were PCN staff members responsible for evaluating programming at their PCN. From our survey and literature search, MHP was directly offered at 29 PCNs at the time, however, only 13 reported administering PROMs as part of their MHP.

Objective 1: Identify what PROMs are currently being administered in PCNs for MHP

Four PCNs did not indicate which tool(s) they administer for their MHP. Nine out of 20 respondents reported 11 tools currently being collected for MHP at their PCN and we captured the domains they measure shown in Table 1. Seven of those nine reported using more than one tool; including 5/7 use the PHQ-9 alongside the EQ-5D-5L. The most used tools were the EQ-5D-5L (7/9) and PHQ-9 (6/9). None of these PCNs reported involving patients in planning for PROMs administration. Since we collected information for each PROM tool, the survey results were aggregated to highlight the use of PROMs for MHP overall. The most common methods of administration or data collection are pen and paper or online (e.g., REDCap, EMR, other survey platforms). The frequency of PROMs collected varies across PCNs; at the initial visit, end-of-care visit, every visit, or dependent on patient care. Respondents reported ‘clinical care’ as the main reason for collecting a specified PROM (84%), ‘evaluation’ (52%) and to ‘describe population health status’ (16%) were other reasons (n = 25). The most common factor considered when choosing a specified PROM is the ‘evidence-base supporting the use of that PROM in primary care settings’ (77%; n = 22). This was the case for the EQ-5D-5L (respondents also included costs and domains) and the PHQ-9 (respondents also shared the outcome suited their measurement needs and easy interpretation). At the time of the survey, four PCNs indicated PROMs are integrated in their EMR. The EMRs reported included Wolf, Med Access, Accuro, Ava, and Telus CHR. Three PCN evaluators reported they do not use EMRs to collect PROMs at their PCN, they use a different approach. Seven did not respond to this question.

Table 1 PROMs administered for MHP across nine PCNs

“[The EQ-5D-5L] provides a simple understanding of change in physical and mental health status over the course of the workshop [PCN MHP]. It also allows us to compare the scores to other programs and services where it is used, such as with our vulnerable populations.” (PCN 5).

Objective 2: Understand the PCNs’ capacity to implement and use PROMs data effectively

To understand PCNs’ capacity to implement and use PROMs data effectively in MHP, we asked open-ended questions about enablers (Table 2) and barriers (Table 3). The most common enablers to implement PROMs were EMR integration, employee buy-in, and the tool being easy to complete. Implementation barriers were divided into two different categories based on our responses: data collection and clinic barriers. Common data collection barriers included time constraints, length of PROMs and low response rates from patients. Barriers at the clinic included lack of standards at clinics, suboptimal workflow and usability of the tool for staff and/or clinicians. Additionally, when given a checklist of data collection and social barriers to collect PROMs, respondents checked loss to follow up, burdensome workload and staff capacity most frequently and differences in language or culture, functional limitation or disability and limited literacy levels were reported as common social barriers (Table 4).

Table 2 Open-ended enablers to implement PROMs in MHP

Table 3 Open-ended barriers to implement PROMs in MHP

Table 4 Checklist of barriers experienced when collecting PROMs

Nine respondents reported how PROMs were used at their PCN. All respondents strongly agreed or agreed PROMs helped identify treatment plans or improved the design of patient visits to be focused on symptoms and factors important to them. Most also strongly agreed or agreed PROMs are useful at the PCN level to evaluate programs to inform MHP (e.g. resource allocation and content). However, one respondent disagreed. When asked whether PROMs have improved services within their PCN’s MHP, four respondents strongly agreed or agreed, four neither agreed nor disagreed, and one was unsure. Most respondents strongly agreed or agreed PROMs outcomes helped justify funding support for MHP changes while two indicated they neither agreed nor disagreed, and one was unsure. When asked if the tool was useful or not useful, all nine PCNs reported the tools to be useful or sometimes useful. Further, seven respondents indicated the EQ-5D-5L was useful or sometimes useful; five reported the PHQ-9 was useful or sometimes useful.

Objective 3: Describe how PROMs are currently reported in PCNs

PCNs report PROMs to different audiences, internally or externally, based on the tool they use. Internally, eight respondents stated they report PROMs data to staff, which includes management, PCN clinicians, member physicians and PCN leadership. Externally, five include PROMs results in their annual reports to Alberta Health (Ministry of Health). Four respondents report PROMs data and results to their PCN board. Only one reported sharing their results with other PCNs (EQ-5D-5L). Two respondents stated not reporting their PROMs data for certain tools (the PHQ-9; and one does not report their GAD-7 data).

Objective 4: Understand the feasibility of standardization and consistency of all PROMs across PCNs

Understanding PCNs perspectives on standardization and consistent measurement is key to help inform the future direction of PROMs in primary care in Alberta. When asked to describe the possibilities, we received mixed responses from six respondents. PCN evaluators expressed the value in standardization could include aligning PROMs for program evaluation, providing feedback to counsellors or Behavioural Health Consultants (BHCs) and determine their counselling impact, and having one organization or system identify appropriate assessments. However, others expressed concerns and challenges. One respondent shared it’s not a necessary outcome measure due to the differences in PCNs and sharing best practices between PCNs instead would be helpful. Other challenges included determining the frequency of administration and supporting PCNs in evaluating PROMs, the diversity of programming, differences in intervention approaches and target populations across PCNs, and the lack of authority to mandate the use of tools in family physician practices/medical homes. When asked how standardization of PROMs looks in your setting, one PCN evaluator responded:

“Guidelines on how to capture our PROMs, uniform tools and metrics and provider training. Guidelines on how frequently PROMs should be captured… Alberta wide reporting to see how we compare to others in terms of capturing data and follow up if help is needed to capture data. Follow up for quality control as well.” (PCN 4).

Discussion

Our study demonstrates a province-wide environmental scan of the current collection practices, use and underlying perspectives about PROMs in Alberta’s PCNs for MHP. The results highlight specific tools, predominantly the EQ-5D-5L and the PHQ-9, which are openly available free of charge. The top three factors for choosing the EQ-5D-5L were: (1) evidence base supporting the use of PROMs in primary care settings, (2) no cost and (3) the domains gathered. While the EQ-5D-5L is designed to assess general aspects of health to understand a patient’s quality of life that are not specific to a particular disease [17], it can be beneficial to evaluate outcomes of mental health services and programming as well [18]. The EQ-5D-5L is useful for evaluating health-related quality of life outcomes of treatment for patients with mental health illness such as depression and anxiety [19, 20]. However, Brazier (2018) suggested this tool may not be as effective for more severe mental health conditions such as schizophrenia [21].

Condition specific and generic PROMs (such as PHQ-9 and EQ-5D-5L) can be used at the same time for specific patient populations or conditions, which was the case for most PCNs in this study. From our survey, six out of seven PCNs that used the EQ-5D-5L used at least one other condition-specific tool (five PCNs used the PHQ-9 along with the EQ-5D-5L). The respondents stated the top three reasons for choosing the PHQ-9 were: (1) the emerging evidence base supporting the use of PROMs in primary care settings, (2) the outcome suited their measurement needs and (3) the tool was easy to interpret. The PHQ-9 is a widely used tool to screen depression in primary care settings in high- and low-income countries [22]. For example, in Spain, mental disorders are mostly diagnosed in primary care centers. Major depressive disorder is highly prevalent in Spain, yet it’s commonly underdiagnosed and most do not receive appropriate treatment. PHQ-9 was found highly satisfactory for diagnosing this disorder [23]. It is also an effective clinical marker for clinical complexity among patients and supports patient management [24].

The PROMs selection steps outlined in Al Sayah, Jin & Johnson [17], specify it is essential to consider the purpose of collection prior to choosing a tool and integrating it within a workflow. This is because the relevance and appropriateness for the patient population and the specific health conditions are important considerations, although challenging in primary care, hence why this environmental scan focuses only on PROMs used in MHP. Additionally, this corroborates with the most common factor considered when PCN evaluators chose a PROM for MHP at their PCN; evidence-base supporting the use of that PROM in primary care settings. The reliability and validity of the tool(s) are critical to ensure accurate and consistent data collection; many of the tools reported are well studied and referenced in Table 1. Additionally, the ease of administration and interpretation, both for patients and healthcare providers, should be considered and will facilitate integration into routine practice. Finally, the tools’ ability to provide actionable insights that can directly inform and improve patient care decisions is a key factor in its selection and sustainability in practice.

When selecting PROMs for use in primary care-based MHP, the steps above, which can be applied to other health programming in primary care, may seem daunting to an overworked, under-resourced primary care practice, although the value of each step is known to influence overall implementation. Collecting PROMs routinely in MHP also promotes the supporting care of individual patients and populations while simultaneously monitoring the quality of services [25]. This highlights the importance of partnership across government, PCNs, clinics and research. APERSU (www.apersu.ca) is an example of embedded support services within Alberta’s health system to facilitate information gathering (like this environmental scan), research activities and implementation strategies [26]. Despite the challenges, PROMs provide direct insights into patients’ perspectives on their mental health status and treatment outcomes [8].

Standardizing the collection of PROMs ensures consistent, reliable data, enabling healthcare providers to track progress over time and make informed clinical decisions [12, 27], in addition to program-level decisions [6]. This approach facilitates the comparison of outcomes across different populations and settings, enhancing the ability to identify best practices and improve care [27, 28]. Like our survey respondents expressed, there is value to standardize but also hesitation when considering the logistics across jurisdictions. However, with more awareness of our findings and PROMS overall, it will become better known that PROMs empower patients by actively involving them in their care, promoting patient-centered approaches and improving overall satisfaction and engagement with treatment [29].

This study is not without limitations. There is the potential of bias in the survey responses obtained, as those who chose to respond may not be representative of others. As well, the system is constantly changing, so the data may be outdated, which can affect the accuracy and reliability of this environmental scan. Conducting this environmental scan required substantial resources, including time, funding, and personnel. Staffing changes within PCNs may have limited the scope and depth of the scan, potentially impacting the comprehensiveness and usefulness of the findings. However, as Snowdon et al. [30] stated, this understanding of PROMs use in primary care provides valuable insights on the purpose, scope and practical considerations, to inform future implementation. This will inform future work on gathering additional insights from stakeholders to further leverage the conversation towards standardization of PROMs across Alberta.

Conclusion

We have compiled an assessment of the context and use of PROMs in PCNs for MHP. These results provide a better understanding of the current use of PROMs in PCNs, specific to MHP, which will be further examined through future narrative conversations. Overall, this study informs leadership on the current use of PROMs and supports the advancement of PROMs use in Alberta.