March 2024Regular Article: Shifting Mindsets: The Impact of a Patient Portal on Functioning and Recovery

Shifting Mindsets: The Impact of a Patient Portal on Functioning and Recovery in a Mental Health SettingChanger les mentalités : l’impact d’un portail patient sur le fonctionnement et le rétablissement dans un contexte de santé mentaleBrian Lo, MHI^1,2,3,4

, Hwayeon Danielle Shin, RN, MScN^3,4, Jessica Kemp, MHI^1,3, Mikayla Munnery, MSc^3,4, Sheng Chen, PhD³, Clement Ma, PhD^3,5, Damian Jankowicz, PhD¹, Rohan Mehta, MBA¹, Alexandra Harris, MN/MHSc, PhD^6,7, Moshe Sakal¹, Ryan Pundit, MBA¹, Kevin Chung, MHI⁸, Craig Kuziemsky, PhD⁹, Sarah Rossetti, RN, PhD, FACMI, FAMIA¹⁰, and Gillian Strudwick, RN, PhD, FAMIA, FCAN^1,3,4img1

The Canadian Journal of Psychiatry / La Revue Canadienne de Psychiatrie
2024, Vol. 69(3) 217‐227
© The Author(s) 2023
Article reuse guidelines:
sagepub.com/journals-permissions
DOI: 10.1177/07067437231197060
TheCJP.ca | LaRCP.caimg3

Abstract
Objective:This study aims to understand whether higher use of a patient portal can have an impact on mental health functioning and recovery.Method: A mixed methods approach was used for this study. In 2019–2021, patients with mental health diagnoses at outpatient clinics in an academic centre were invited to complete World Health Organization Disability Assessment Scale 12 (WHODAS-12) and Mental Health Recovery Measure surveys at baseline, 3 months, and 6 months after signing up for the portal. At the 3-month time point, patients were invited to a semistructured interview with a member of the team to contextualize the findings obtained from the surveys. Analytics data was also collected from the platform to understand usage patterns on the portal.Results: Overall, 113 participants were included in the analysis. There was no significant change in mental health functioning and recovery scores over the 6-month period. However, suboptimal usage was observed as 46% of participants did not complete any tasks within the portal. Thirty-five participants had low use of the portal (1–9 interactions) and 18 participants had high usage (10+ interactions). There were also no differences in mental health functioning and recovery scores between low and high users of the portal. Qualitative interviews highlighted many opportunities where the portal can support overall functioning and mental health recovery.Conclusions: Collectively, this study suggests that higher use of a portal had no impact, either positive or negative, on mental health outcomes. While it may offer convenience and improved patient satisfaction, adequate support is needed to fully enable these opportunities for patient care. As the type of interaction with the portal was not specifically addressed, future work should focus on looking at ways to support patient engagement and portal usage throughout their care journey.Objectif: La présente étude vise à comprendre si l’utilisation accrue d’un portail patient peut avoir un effet sur le fonctionnement et le rétablissement de la santé mentale.Méthode: Une approche de méthodes mixtes a été utilisée dans cette étude. En 2019–2021, des patients ayant des diagnostics de santé mentale à des cliniques ambulatoires d’un centre universitaire ont été invités à remplir des enquêtes WHODAS- 12 et MHRM à la base, et à 3 et 6 mois après s’être inscrits au portail. Au bout de 3 mois, les patients étaient invités à une entrevue semi-structurée avec un membre de l’équipe pour contextualiser les résultats obtenus des enquêtes. Les données analytiques ont aussi été recueillies de la plateforme pour comprendre les modèles d’usage sur le portail.Résultats: Globalement, 113 participants ont été inclus dans l’analyse. Il n’y a eu aucun changement significatif dans le fonctionnement de la santé mentale et les scores de rétablissement dans la période de 6 mois. Cependant, un usage sous-optimal a été observé comme 46% des participants n’ont accompli aucune tâche dans le portail. Trente-cinq participants avaient un faible usage du portail (1–9 interactions) et 18 participants avaient un usage élevé (10 + interactions). Il n’y avait pas non plus de différences du fonctionnement de la santé mentale et des scores de rétablissement entre les usagers légers et élevés. Les entrevues qualitatives ont fait ressortir de nombreuses possibilités où le portail peut soutenir le fonctionnement global et le rétablissement de la santé mentale.Conclusions: Collectivement, cette étude suggère qu’un usage plus élevé d’un portail n’avait pas d’effet, soit positif ou négatif, sur les résultats de santé mentale. Bien qu’elle puisse offrir commodité et une meilleure satisfaction des patients, il faut un soutien adéquat pour permettre pleinement ces opportunités pour les soins aux patients. Comme le type d’interactions avec le portail n’a pas été spécifiquement abordé, les travaux futurs devraient se concentrer sur la recherche de façons de soutenir l’engagement de patients et l’usage du portail durant leur période de soins.Keywordspatient portals, clinical informatics, nursing informatics, evaluation, mental healthIntroductionSince 2010, there has been a proliferation of health information shared with patients through a patient portal.^1,2 Many platforms allow patients and families to perform various tasks including scheduling appointments, viewing lab results and reviewing clinical documentation.^3,4 Since the COVID-19 pandemic, patient portals have become an instrumental tool for communicating polymerase chain reaction (PCR) test results to confirm diagnoses for patients and families.^5-7 This has led to exponential adoption and use of portals in primary and acute care settings.⁸ Particularly, in the US, the 21st Century Cures Act has mandated the sharing of clinical documentation with patients and families.^9-12While many studies have yielded important lessons for successful portal implementations in primary and acute care settings (e.g., the importance of engaging patients and families and clinical workflows),^8,13-15 in the mental health domain, it is still relatively uncommon to share health information with patients and families due to stigma and impact on therapy.¹⁶ Despite growth in this area,^17-20 there has been significant concern and apprehension around sharing notes with patients. In particular, there are perceptions among healthcare professionals that information sharing may ruin therapy and negatively impact recovery.^{12,16,19,21-23} While Canada does not have a national policy such as the “Cures Act” in the United States to proactively share clinical notes with patients, the clinician concerns noted above may further impact the adoption of patient portals.^23-25Despite ongoing efforts to evaluate the impact of patient portals on mental health care, the current literature lacks evidence on whether the patient portal is detrimental to therapy and/or has a negative impact on mental healthcare.¹⁷ More importantly, little is known about the impact of the portal on patients’ mental health recovery.^18,26 A recent literature review on the impact and benefits of patient portals has found insufficient evidence to understand their impact on clinical care.²⁷ Providing quantitative and qualitative data from real-world environments can help validate some of these concerns and inform future efforts to support the sustainable scale-up of this tool. Thus, the present work focuses on examining the impact the use of a patient portal has on mental healthcare and the drawbacks that clinicians fear. The specific aims are as follows:Aim 1: Determine if higher use of mental health patient portals is associated with changes in mental health functioning and recovery over a 6-month period.Aim 2: Determine whether the higher usage of the mental health patient portal affected the overall trajectory of mental health functioning and recovery.Aim 3: Understand the perceptions and experiences of patients on the use of a mental health patient portal for supporting mental health functioning and recovery.MethodsA convergent, concurrent mixed methods approach (QUAN + qual)²⁸ was used to examine the relationship between a patient portal and mental health functioning and recovery. Further details can be found in the protocol paper published in BMJ Open.²⁹ This work is informed by the Canadian Institute of Health Research Strategy for Patient-Oriented Research Framework,³⁰ which outlines key principles for engaging patients to improve care delivery through research. The study was approved by the Research Ethics Board at the Centre for Addiction and Mental Health (REB 044/2018) and the University of Toronto (REB No. 40342). A patient and family advisory committee was engaged throughout the study.SettingThis study was conducted at a large academic and teaching mental health hospital in Toronto, Ontario. The organization implemented a comprehensive electronic health record in 2014 and achieved Stage 7 on the HIMSS Electronic Medical Record Adoption Model.³¹ In 2018, the hospital launched its mental health patient portal to several outpatient clinics. The mental health patient portal allows patients to (a) view parts of their health record (e.g., discharge notes) and lab tests; (b) complete self-assessments; (c) attend virtual care appointments; and (d) message their care provider. Patients are offered portal access by their clinician or administrative staff and are free to choose whether or not they want to use the patient portal for their care.ParticipantsPatients receiving care at 1 or more outpatient clinics offering the patient portal were invited to the study from June 2019 to April 2021. When patients signed up for the portal, they received a pamphlet about the portal including a flyer to the study. Patients could also sign-up for the study on the registration page and patient portal home page. To be eligible for the study, the patient must have self-reported signing up for the portal within the last 2 weeks of enrolling in the study. There were no exclusion criteria based on diagnosis or demographics. Based on the sample size calculation outlined in the protocol, about 100 patients (accounting for ∼30% attrition) were required for the study.²⁹ A sample size of N = 68 completers achieved 80% power (alpha = 5%) for detecting a 30% drop in World Health Organization Disability Assessment Scale 2.0 (WHODAS 2.0) from baseline to T2 (equivalent to a small to medium effect size of Cohen’s d = 0.27). All participants provided informed consent in written form as outlined in the REDCap eConsent recommendations.^32,33Data CollectionAfter enrolling in the study, the quantitative strand of the study began. The quantitative strand involved completing survey instruments online using REDCap at baseline, 3-month and 6-month mark. After consent, they were asked to complete a baseline package asking them about their demographics, which include sex, primary diagnosis, and ethnicity, among others. After completing the demographics section, participants were asked to complete 2 survey instruments at each of the 3 time points. The WHODAS 2.0—Short Version³⁴ and the Mental Health Recovery Measure (MHRM)³⁵ were used to assess mental health functioning and recovery respectively. Both of these instruments are scales frequently used in both clinical care and mental health research.^34,36-38 Details about the domains and reliability of these instruments can be found in the protocol.²⁹In addition to collecting survey data for the quantitative strand, usage data^39-41 on the mental health patient portal were collected from the back-end platform throughout the 6 months of the study. This includes the duration, frequency and usage of the different functionalities of the portal.^42,43 The usage data was validated internally as part of the implementation process and was matched based on the patient’s name reported in the study. As part of the analysis, participants who have used the portal were stratified into low (1–9 interactions) and high usage (10+ interactions) for the analysis.For the qualitative strand, at the 3-month mark, participants were invited to complete a 1-hour semistructured virtual interview⁴⁴ with a graduate student in health informatics (BL, JK or MM) using a qualitative descriptive approach.⁴⁵ Participants were asked to share their experience using the portal, their perceptions on how the portal has impacted their mental health functioning and recovery journey, and challenges that hindered the value and benefits of the patient portal, among other outcomes. The question guide (Supplemental Appendix A) builds on the existing literature^17,18,46,47 and the needs suggested by our Patient and Family Advisory Council. Participants were invited to the qualitative strand until data saturation⁴⁸ was achieved.Data AnalysisThe quantitative analysis was carried out using SAS Enterprise Guide v7.1 (SAS Institute, Cary, NC, USA). Descriptive statistics (mean, standard deviation, frequency, and proportion) were used to summarize participant characteristics. For each study outcome, linear mixed effect models with random participant effects were performed to model the change in outcome over the 3 time points.⁴⁹ Linear contrasts were conducted to compare across the 3 time points. While the main analysis was not adjusted, an additional exploratory analysis adjusting for sex, age and diagnosis was conducted. Missing data was imputed using SAS PROC MI at the item level. A total of 5 imputed datasets were generated and SAS PROC MIANALYZE was used to aggregate the findings. Two-sided p-values < .05 were considered statistically significant. For the analytics data, a doctoral student (BL) familiar with analytics and user engagement led the data cleaning and analysis. Given that usage data is considered approximate and dirty data (flawed, uncertain, proximate and sparse [FUPS]),⁵⁰ any redundant data entries were removed. In order to segregate the users across low and high users, the total number of interactions related to the main use cases of the portal was summated. This includes the: (a) number of appointments viewed, (b) clinical notes downloaded/viewed, (c) number of lab results viewed, (d) self-assessments completed, and (e) number of messages reviewed and sent. The trajectory in functioning and mental health recovery between low and high users was then compared by stratifying it based on the number of interactions with the portal.The qualitative data was analyzed by JK and HDS using the content analysis protocol outlined by Elo and Kyngäs⁵¹ using a deductive approach. JK and HDS deductively analysed the same constructs asked in the MHRM and developed new codes inductively when appropriate. In order to ensure trustworthiness⁵² of the analysis, the analysis was done in duplicate independently (JK and DS) and an audit trail of the discussion and decision points was maintained.ResultsA total of 113 participants were enrolled and consented to participate in the study. Overall, most participants identified as male sex at birth (68%), between 25 and 64 years old (34%), never married (55%), and Caucasian or European (68%) (Table 1). With regard to the most responsible psychiatric diagnoses, 38% of participants indicated they had a mood disorder (e.g., depression and bipolar disorder); 31% had a diagnosis other than mood disorders, anxiety or psychotic disorders.Quantitative AnalysisMental Health Functioning. In the unadjusted model of WHODAS over time (Table 2 and Figure 1A), there was a decreasing trend in WHODAS score from baseline to 3 months (mean difference = −1.26; 95% confidence interval [CI], −2.70 to 0.19; p-value = .09). There were nonsignificant decreases in WHODAS from baseline to 6 months (mean difference = −1.07; 95% CI, −2.56 to 0.42; p-value = .16). The model adjusted for sex, age and diagnosis showed similar results.Tab1Mental Health Recovery. A total of 110 participants were included in the analysis of MHRM scores (3 participants did not complete the baseline and/or subsequent MHRM surveys). Based on the model with time as an independent variable (Figure 1B), there was no significant difference in MHRM scores between baseline and 3 months (mean difference = 1.20; 95% CI, 1.77 to 1.86; p-value = .23). There were also no differences between baseline and 6-months (mean difference = −0.31; 95% CI, −0.86 to −0.23; p-value = .76). Similar findings were obtained after adjusting for sex, age and diagnosis.Impact of Portal Usage on Functioning and Recovery. Usage data was retrieved for 99 participants (87%). Some participants’ usage data could not be retrieved due to the use of a different name in the patient portal account. In terms of usage, about 46 participants (46%) did not complete a task in the portal (e.g., viewed a note and completion of a selfassessment). Thirty-five participants (35%) had low usage of the portal (1–9 interactions) and 18 participants (18%) had high usage, which is defined as more than 10 interactions. In terms of interactions, the majority comprised selfassessments (311 interactions from 16 users), followed by viewing upcoming appointments (133 interactions from 33 users), messages (58 interactions from 18 users), notes (56 interactions from 22 users) and labs (45 from 5 users). In the model with time adjusted for sex, age and diagnosis, there were no significant differences between participants with low usage and those with high usage on mental health functioning score, F(2,85) = 0.36, p = .70, and mental health recovery score, F(2,83) = 1.37, p = .26.Tab2Fig1Qualitative AnalysisOf the participants who completed the survey, 10 participants also completed the interview portion of the study. Overall, the majority of participants were male, 25–64 years old with a mood disorder and had interacted with the portal (see Supplemental Appendix B). A summary of the themes and representative quotes is listed in Table 3 and in detail below.Overcoming Stuckness Through Engagement With the Portal. Individuals described that the portal frequently served as an adjunct tool to support their understanding and treatment process. As outlined by 1 participant (Quote S1), providing information can enhance the efficiency of patients in understanding their care, and serve as another tool to enhance the recovery process. In fact, with the limited time during face-to-face encounters, the ability to review their notes at any time can provide another opportunity to improve the therapeutic relationship and process (Quote S2). However, when the clinician did not use the tool, interested patients may perceive that their clinicians do not support their involvement and engagement in the care. As mentioned by another participant (Quote S3), the lack of interest from clinicians in the use of the tool led to doubt about the clinician’s compassion towards the patient. Thus, the patient portal may be seen to connect with the clinician outside of their appointment time, serving as a tool to help patients move forward with recovery.Basic Functioning and Self-Empowerment for Care Planning. After participants reviewed their health information, they felt that the portal provided them with the ability to have empowerment over their own health data. In Quote B1, understanding their own health can help encourage them to self-reflect on their progress to date and provide possible rationales as to why they may feel a certain way. For some participants, this can encourage them to be in control of their own self-management and recovery. Moreover, the information about medications and appointments can help them understand their care plan. Participants also appreciated having a copy of the medications they need to take on a regular basis, which affirmed the plan they had discussed with their clinician (Quote B2). Similarly, the portal can help them keep track of their appointments and documents they need to read beforehand (Quote B3/B4). Thus, the portal acts as an aid to enhance their independence in adhering to the treatment plan while in the community.Learning and Self-Redefinition. As part of reading notes, participants outlined the potential of using the portal to learn in terms of who they are and gain knowledge about their mental health. For example, 1 participant (Quote L1) recounted that the ability to review their health information within the portal helped them understand who they are as a person living with a mental illness. Moreover, sometimes, during the visit, it can be hard to interpret all the recommendations and discussions made with the clinician, even with assistance from others (Quote L2). By reviewing the exact language and verbiage that is written in the documentation, it reduces the chance of misinterpreting recommendations from the clinician. Lastly, the ability to review information after the encounter provides patients with the opportunity to reflect on their progress and the space to have disagreements with clinicians about their care. One participant (Quote L3) recounted how being able to review their notes allowed them to identify different perspectives on this issue and devise a plan to ensure that both perspectives can be visible in future conversations. Collectively, viewing notes in the portal has improved the ability for patients to learn and grow about their own mental health journey and care.Impact on Overall Well-Being. While many individuals had positive experiences with the patient portal, some participants were also able to identify situations where the lack of appropriate support can hinder their well-being. For example, 1 participant (Quote W1) recounted that depending on the journey when the note was documented, there could be the risk of “replaying” parts of their life that were challenging. These documents may not be in alignment with their perceptions, and, in some cases, such as Quote W2, have language that may be considered offensive (Quote W2). Thus, reading health information can have an impact on their overall wellness.New Potentials, Advocacy and Enrichment. In addition to reading their own health information for care planning, participants also spoke of how the portal can enhance engagement and greater awareness of their own rights in achieving their care goals. One participant (Quote P1) highlighted that being able to view their notes in a comfortable space at home allowed them to review the milestones and overall progress they have achieved. Moreover, access to their own health information via a portal can offer them the possibility to share their health information more easily and openly with other family members (Quote P2). Collectively, this allows patients to become more empowered and to identify any discrepancies they encounter throughout their care.Integration of Quantitative and Qualitative FindingsThe research team integrated the findings to contextualize the impact of patient portals on mental health functioning and recovery (Table 4). Overall, there was no significant impact of portal use on the mental health functioning and mental health recovery scales. However, there were mixed perceptions identified from patient interviews. A clear promise to enhance mental health functioning and recovery using the portal has been expressed, but patients also highlighted limitations of the portal that hindered their ability to realize the promise and impacted their care. Mixed experiences identified in the qualitative strand partially explain nonsignificant results in the quantitative strand and that the portal does not have a unanimous positive impact.DiscussionTo our knowledge, this is one of the first studies outlining concrete evidence of the impact a patient portal implemented in a Canadian mental health hospital has on the functioning and recovery of patients suffering from mental health challenges. Contrary to the concerns outlined in numerous articles,^{12,16,19,21-23} this work has provided the foundation for us to reconsider the potential concerns around portals in mental health settings and think about how we can best position the delivery of patient portals to enhance mental health clinical care outcomes.⁴⁶Tab3There has been significant interest in the impact of a patient portal on mental health functioning. In our study, we found no significant differences over the period of 6 months in mental health functioning, which suggests that patient portals do not negatively affect therapy and mental health care. This work complements the review by Zhang et al.,²⁷ who found the use of patient portals was associated with increased participation in their care (patient activation). Our findings from the qualitative interview with patients may shed some light on the mixed results. While patient portals can support adherence to appointments and medications, there can be difficulties in fully interpreting their health information.²⁷ Thus, delivery of compassionate care through the portal requires exploring ways to ensure that clinical practices are supportive and conducive to the use of the tool in practice.⁵³ Moreover, as outlined in Quote W2, the findings also raise highlight potential communication issues through the portal. While many assessments are considered standard (e.g., mental health status exam), the approach to which the results are written and summarized by clinicians may be perceived as offensive to patients and families.^54,55 While studies on the concept of sharing notes (OpenNotes) have highlighted some strategies to mitigate this including reviewing with patients and setting expectations, further work is necessary to provide feasible approaches to address these challenges within the mental health context.^56,57 The authors are currently conducting a study that focuses on addressing these challenges within the mental health context.⁵⁸ Nevertheless, this work highlights that the portal may offer conveniences and improve satisfaction while not necessarily impacting the quality of care.Tab4In addition, our quantitative strand found that there were no significant changes in recovery scores over time. These findings are in contrast to the findings from a previous study from another Canadian mental health care centre, where they found a significant increase in recovery scores.⁵⁹ Moreover, many interview participants in our current study highlighted the important role of viewing their health information for learning and self-understanding of themselves and their condition. However, the mixed findings in our current study and the suboptimal usage of portals suggest that many of the opportunities to enhance outcomes could be limited if there is insufficient support to enable the benefits. Implementation frameworks such as the Consolidated Framework for Implementation Research⁶⁰ can be useful to support the identification of key facilitators and challenges in supporting the adoption and use of these innovations.Lastly, it is worth highlighting the suboptimal level of user engagement observed within the portal. While there is no gold standard measuring user engagement,^41,61 the majority of participants had <2 interactions with the portal and only a small fraction of patients used the portal repeatedly. In this context, the low usage by participants may be suggestive of high levels of disengagement with the technologies, thereby leading to a futile impact on change in health behaviour and outcomes.⁶² Unfortunately, disengagement with a digital tool is complex and Perski et al.⁶³ suggest numerous factors related to the tool and environment that can hinder engagement with the tool. Moreover, there has been discussion about the importance of health equity and inclusivity consideration in supporting the use of digital tools.⁶⁴ Thus, future work should uncover factors that influence how individuals remain engaged with portals in mental health settings.There are also broad policy implications from this work. In Ontario, improving patient access to health information has become a provincial priority and Patient Portal Standards have been developed to ensure that these tools have the appropriate functionalities (e.g., single digital identity) for patients to effectively embed it as part of their care.⁶⁵ These guidelines align closely with the findings from the qualitative findings, in that there is a need to look at how clinic workflow and clinician engagement should be involved in supporting patient access to health information for improved efficiency in care planning. It would also be useful to look at how patients and families can be better supported to bridge the digital divide by using the portal. In the current study, due to the early stages of implementation, there was limited support (other than a technical hotline) being provided to support the challenges (e.g., education) related to digital health equity. As digital health equity^66,67 continues to be a growing concern for usability and scale, looking at interventions that focus on empowering patients and families with the right knowledge and competencies is critical.LimitationsFrom a methodological perspective, given that this is an observational study without a comparator group in a single academic Canadian site, it is unclear whether the trajectory of these patients would be different if they had not used the portal. In particular, due to the imbalanced mix of mental health disorders in the sample, limited clinical anchors at baseline and the nature that each patient has a different mental health trajectory, it is likely that we were underpowered to detect any difference among some of the subpopulations. Moreover, as patient portals function within a complex health-care environment, it is unclear how the specific patient portal software and the evolving workflows within the clinical environment contributed to the observed findings. As such, the findings should be considered with studies on clinician perspectives. It is important to note that the study was conducted during the COVID-19 pandemic, which may have influenced how the portal was used by participants. Due to the limitations in sample size and rapidly changing environment, we were unable to add targeted questions specific to the pandemic and compare use before and during the pandemic. For the qualitative interviews, the imbalanced and small sample size also hindered the ability to compare the perceptions and experiences between high and low users and generate useful insights for optimizing the platform.ConclusionOver a 6-month time period, there were no significant differences in mental health functioning and recovery, suggesting that the portal does not have a detrimental effect on the treatment and recovery of patients. However, usage of the portal remains suboptimal. From the qualitative findings, numerous opportunities to enhance patients’ mental health function and recovery through portal use are highlighted, but appropriate support for patients and clinicians is required to achieve the portals’ fullest potential and engagement.AcknowledgementsThe authors of this paper would like to acknowledge the Centre for Addiction and Mental Health for their overall support of this work. In addition, we would like to acknowledge Anna Chyjek, Emily Winters and Uzma Haider for their support in this project.Declaration of Conflicting InterestsThe authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.FundingThe authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This project is supported by a Strategy for Patient-Oriented Research (SPOR) Grant from the Canadian Institutes for Health Research (Funding Reference Number: PEH-157079) as well as a Sigma/ Canadian Nurses Foundation Grant from the Sigma Foundation for Nursing. In-Kind contributions were also provided by the Centre for Addiction and Mental Health (CAMH).ORCID iDBrian Lo

https://orcid.org/0000-0003-1021-884X
Supplemental MaterialSupplemental material for this article is available online.References

Lye CT, Forman HP, Daniel JG, Krumholz HM. The 21st Century Cures Act and electronic health records one year later: will patients see the benefits? J Am Med Inform Assoc. 2018;25(9):1218-1220.

Mehan WA Jr, Brink JA, Hirsch JA. 21st Century Cures Act: patient-facing implications of information blocking. J Am Coll Radiol. 2021;18(7):1012-1016.

Alturkistani A, Greenfield G, Greaves F, Aliabadi S, Jenkins RH, Costelloe C. Patient portal functionalities and uptake: systematic review protocol. JMIR Res Protoc. 2020;9(7):e14975.

Ammenwerth E, Neyer S, Horbst A, Mueller G, Siebert U, Schnell-Inderst P. Adult patient access to electronic health records. Cochrane Database Syst Rev. 2021;2:CD012707.

Turer RW, DesRoches CM, Salmi L, Helmer T, Rosenbloom ST. Patient perceptions of receiving COVID-19 test results via an online patient portal: an open results survey. Appl Clin Inform. 2021;12(4):954-959.

Margolius D, Hennekes M, Yao J, et al. On the front (phone) lines: results of a COVID-19 hotline. J Am Board Fam Med. 2021;34(Suppl):S95-S102.

Imborek KL, Krasowski MD, Natvig P, Merrill AE, Diekema DJ, Ford BA. Experience with pretravel testing for SARS-CoV-2 at an Academic Medical Center. Acad Pathol. 2021;8:23742895211010247.

Lo B, Charow R, Laberge S, Bakas V, Williams L, Wiljer D. Why are patient portals important in the age of COVID-19? Reflecting on patient and team experiences from a Toronto Hospital Network. J Patient Exp. 2022;9:23743735221112216.

Waseem N, Kircher S, Feliciano JL. Information blocking and oncology: implications of the 21st Century Cures Act and open notes. JAMA Oncol. 2021;7(11):1609-1610.

Arvisais-Anhalt S, Lau M, Lehmann CU, et al. The 21st Century Cures Act and multiuser electronic health record access: potential pitfalls of information release. J Med Internet Res. 2022;24(2):e34085.

Genes N, Darrow B, Vijayaraghavan M, Rucker DW. The 21st Century Cures Act and emergency medicine – part 1: digitally sharing notes and results. Ann Emerg Med. 2022;79(1):7-12.

Magid SK, Cohen K, Katzovitz LS. 21(st) Century Cures Act, an information technology-led organizational initiative. HSS J. 2022;18(1):42-47.

Fujioka JK, Bickford J, Gritke J, et al. Implementation strategies to improve engagement with a multi-institutional patient portal: multimethod study. J Med Internet Res. 2021;23(10): e28924.

Skrzypcak K, McFadden V, Bauer SC. Using electronic health record modifications to increase use of adolescent sensitive notes. Hosp Pediatr. 2023;13(4):e81-ee7.

Wang H, Shen C, Barbaro M, et al. A multi-level analysis of individual and neighborhood factors associated with patient portal use among adult emergency department patients with multimorbidity. Int J Environ Res Public Health. 2023;20(2):1231.

Hoffman J. What the therapist thinks about you. New York, NY: New York Times; 2014. https://archive.nytimes.com/well.blogs.nytimes.com/2014/07/07/what-the-therapist-thinksabout-you/.

Strudwick G, Booth RG, McLean D, et al. Identifying indicators of meaningful patient portal use by psychiatric populations. Inform Health Soc Care. 2020;45(4):396-409.

Leung K, Clark C, Sakal M, Friesen M, Strudwick G. Patient and family member readiness, needs, and perceptions of a mental health patient portal: a mixed methods study. Stud Health Technol Inform. 2019;257:266-270.

Strudwick G, Clark C, Sanches M, Strauss J. Predictors of mental health professionals’ perceptions of patient portals. AMIA Annu Symp Proc. 2018;2018:989-997.

Strudwick G, Yeung A, Gratzer D. Easy access, difficult consequences? Providing psychiatric patients with access to their health records electronically. Front Psychiatry. 2019;10:917.

Dobscha SK, Denneson LM, Jacobson LE, Williams HB, Cromer R, Woods S. VA Mental health clinician experiences and attitudes toward OpenNotes. Gen Hosp Psychiatry. 2016;38:89-93.

O’Neill S, Blease C, Delbanco T. Open notes become law: a challenge for mental health practice. Psychiatr Serv. 2021;72(7):750-751.

Blease C, Kharko A, Hagglund M, et al. The benefits and harms of open notes in mental health: a Delphi survey of international experts. PLoS One. 2021;16(10):e0258056.

Blease C, Torous J, Hagglund M. Does patient access to clinical notes change documentation? Front Public Health. 2020;8:577896.

Blease C, Torous J, Kharko A, et al. Preparing patients and clinicians for open notes in mental health: qualitative inquiry of international experts. JMIR Ment Health. 2021;8(4):e27397.

Walker J, Leveille S, Bell S, et al. Opennotes after 7 years: patient experiences with ongoing access to their clinicians’ outpatient visit notes. J Med Internet Res. 2019;21(5):e13876.

Zhang T, Shen N, Booth R, LaChance J, Jackson B, Strudwick G. Supporting the use of patient portals in mental health settings: a scoping review. Inform Health Soc Care. 2022; 47(1):62-79.

Creswell JW, Plano Clark VL. Designing and Conducting Mixed Methods Research. 2nd ed. Los Angeles: Sage Publications; 2013.

Strudwick G, Kuziemsky C, Booth RG, et al. Engaging patients and family members in the evaluation of a mental health patient portal: protocol for a mixed-methods study. BMJ Open. 2018;8(8):e025508.

Holmes BJ, Bryan S, Ho K, McGavin C. Engaging patients as partners in health research: lessons from BC, Canada. Healthc Manage Forum. 2018;31(2):41-44.

Healthcare Information and Management Systems Society. Electronic Medical Record Adoption Model (EMRAM) [Web]. 2022 [cited 2022 Aug 15]. https://www.himss.org/what-we-do-solutions/digital-health-transformation/maturitymodels/electronic-medical-record-adoption-model-emram.

Lawrence CE, Dunkel L, McEver M, et al. A REDCap-based model for electronic consent (eConsent): moving toward a more personalized consent. J Clin Transl Sci. 2020;4(4):345-353.

Chen C, Turner SP, Sholle ET, et al. Evaluation of a REDCap-based workflow for supporting federal guidance for electronic informed consent. AMIA Jt Summits Transl Sci Proc. 2019;2019:163-172.

Üstün TB, Kostanjsek N, Chatterji S, Rehm J. Measuring health and disability: manual for WHO disability assessment schedule WHODAS 2.0. Geneva, Switzerland: World Health Organization; 2010.

Young S, Bullock WA. The Mental Health Recovery Measure. 2003.

Konecky B, Meyer EC, Marx BP, Kimbrel NA, Morissette SB. Using the WHODAS 2.0 to assess functional disability associated with DSM-5 mental disorders. Am J Psychiatry. 2014;171(8):818-820.

Roosenschoon B-J, Kamperman AM, Deen ML, Weeghel Jv, Mulder CL. Determinants of clinical, functional and personal recovery for people with schizophrenia and other severe mental illnesses: a cross-sectional analysis. PLoS One. 2019; 14(9):e0222378.

Bitter N, Roeg D, van Assen M, van Nieuwenhuizen C, van Weeghel J. How effective is the comprehensive approach to rehabilitation (CARe) methodology? A cluster randomized controlled trial. BMC Psychiatry. 2017;17(1):396.

Baumel A, Muench F, Edan S, Kane JM. Objective user engagement with mental health apps: systematic search and panel-based usage analysis. J Med Internet Res. 2019;21(9):e14567.

Matthews P, Topham P, Caleb-Solly P. Interaction and engagement with an anxiety management app: analysis using large-scale behavioral data. JMIR Ment Health. 2018;5(4):e58.

Miller S, Ainsworth B, Yardley L, et al. A framework for analyzing and measuring usage and engagement data (AMUsED) in digital interventions: viewpoint. J Med Internet Res. 2019;21(2):e10966.

Pham Q, Graham G, Carrion C, et al. A library of analytic indicators to evaluate effective engagement with consumer mHealth apps for chronic conditions: scoping review. JMIR Mhealth Uhealth. 2019;7(1):e11941.

Pham Q, Graham G, Lalloo C, et al. An analytics platform to evaluate effective engagement with pediatric mobile health apps: design, development, and formative evaluation. JMIR Mhealth Uhealth. 2018;6(12):e11447.

Patton MQ. Qualitative research & evaluation methods: Integrating theory and practice. Saint Paul, MN: Sage Publications; 2002.

Sandelowski M. Whatever happened to qualitative description? Res Nurs Health. 2000;23(4):334-340.

Kemp J, Zhang T, Inglis F, et al. Delivery of compassionate mental health care in a digital technology-driven age: scoping review. J Med Internet Res. 2020;22(3):e16263.

Shen N. The eHealth trust model: Understanding the patient privacy perspective in a digital health environment. Toronto, ON: University of Toronto; 2019.

Saunders B, Sim J, Kingstone T, et al. Saturation in qualitative research: exploring its conceptualization and operationalization. Qual Quant. 2018;52(4):1893-1907.

Cnaan A, Laird NM, Slasor P. Using the general linear mixed model to analyse unbalanced repeated measures and longitudinal data. Stat Med. 1997;16(20):2349-2380.

Wolpert M, Rutter H. Using flawed, uncertain, proximate and sparse (FUPS) data in the context of complexity: learning from the case of child mental health. BMC Med. 2018;16(1):82.

Elo S, Kyngas H. The qualitative content analysis process. J Adv Nurs. 2008;62(1):107-115.

Lincoln YS, Guba EG. But is it rigorous? Trustworthiness and authenticity in naturalistic evaluation. New Dir Program Eval. 1986;1986(30):73-84.

van Rijt AM, Hulter P, Weggelaar-Jansen AM, Ahaus K, Pluut B. Mental health care professionals’ appraisal of patients’ use of web-based access to their electronic health record: qualitative study. J Med Internet Res. 2021;23(8):e28045.

Walker J, Leveille S, Bell S, et al. Opennotes after 7 years: patient experiences with ongoing access to their clinicians’ outpatient visit notes. J Med Internet Res. 2019;21(5): e13876.

Dobscha SK, Denneson LM, Jacobson LE, Williams HB, Cromer R, Woods S. VA Mental health clinician experiences and attitudes toward OpenNotes. Gen Hosp Psychiatry. 2016; 38:89-93.

Murugan A, Gooding H, Greenbaum J, et al. Lessons learned from OpenNotes learning mode and subsequent implementation across a pediatric health system. Appl Clin Inform. 2022; 13(01):113-122.

Kassam I, Shin HD, Durocher K, et al. “I think it’s something that we should lean in to”: the use of OpenNotes in Canadian psychiatric care contexts by clinicians. Digit Health. 2022;8:20552076221144106.

Kassam I, Lo B, Durocher K, Shen N, Strudwick G. Identifying implementation strategies to enhance the uptake of OpenNotes in mental health settings: a study protocol. Stud Health Technol Inform. 2022;290:1044-1045.

Kipping S, Stuckey MI, Hernandez A, Nguyen T, Riahi S. A web-based patient portal for mental health care: benefits evaluation. J Med Internet Res. 2016;18(11):e294.

Kirk MA, Kelley C, Yankey N, Birken SA, Abadie B, Damschroder L. A systematic review of the use of the consolidated framework for implementation research. Implement Sci. 2015;11(1):1-13.

O’Brien HL, Toms EG. What is user engagement? A conceptual framework for defining user engagement with technology. J Am Soc Inf Sci Technol. 2008;59(6):938-955.

Price M, Lau F. The clinical adoption meta-model: a temporal meta-model describing the clinical adoption of health information systems. BMC Med Inform Decis Mak. 2014;14:43.

Perski O, Blandford A, West R, Michie S. Conceptualising engagement with digital behaviour change interventions: a systematic review using principles from critical interpretive synthesis. Transl Behav Med. 2017;7(2):254-267.

Crawford A, Serhal E. Digital health equity and COVID-19: the innovation curve cannot reinforce the social gradient of health. J Med Internet Res. 2020;22(6):e19361.

Ontario Health. Patient Portal (PP) Provincial Service Standards. 2021.

Richardson S, Lawrence K, Schoenthaler AM, Mann D. A framework for digital health equity. NPJ Digit Med. 2022; 5(1):119.

Crawford A, Serhal E. Digital health equity and COVID-19: the innovation curve cannot reinforce the social gradient of health. J Med Internet Res. 2020;22(6):e19361.

¹Information Management Group, Centre for Addiction and Mental Health, Toronto, ON, Canada²Information Management & Technology, The Hospital for Sick Children (SickKids), Toronto, ON, Canada³Campbell Family Mental Health Research Institute, Centre for Addiction and Mental Health, Toronto, ON, Canada⁴Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, ON, Canada⁵Division of Biostatistics, Dalla Lana School of Public Health, University of Toronto, Toronto, ON, Canada⁶ Interprofessional Practice, Unity Health Toronto, Toronto, ON, Canada⁷ Lawrence S. Bloomberg Faculty of Nursing, University of Toronto, Toronto, ON, Canada⁸ Ontario Health, Toronto, ON, Canada⁹ Office of Research, MacEwan University, Edmonton, AB, Canada¹⁰ Department of Biomedical Informatics, Columbia University, New York, NY, USACorresponding Author:
Gillian Strudwick, Information Management Group, Centre for Addiction and Mental Health, 60 White Squirrel Way, Room 315, Toronto, ON, M6J 1H4, Canada.
Email: Gillian.Strudwick@camh.caJournal CoverOtsuka LundbeckEisai Canada - DayvigoEditorial BoardVancouver Coastal HealthJournal InfoSage JournalsIn Review: Canadian Guidelines for Forensic Psychiatry Assessments and Report WritingSupplément spécial de la Revue canadienne de psychiatrie: Guide canadien sur l’expertiseCommentary: Predatory Marriage: An Emerging Medicolegal IssueRegular Article: The Association Between Self-Reported Anxiety and Retention in Opioid AgonistRegular Article: Influence of CYP2C19, CYP2D6, and ABCB1 Gene Variants and Serum LevelsRegular Article: Chronic Psychotic Disorders and Correctional Involvement: A Population-Regular Article: Obstacles et perspectives pratiques de la consultation en médecine générale duRegular Article: Shifting Mindsets: The Impact of a Patient Portal on Functioning and RecoveryResearch Letter: Exploring the Tensions Between Judicial Transparency and Respect forSage Deep Backfile & PremierSage PremierSage VideoSage Peer ReviewersCPA Annual Conference Save The Date