The Canadian Journal of Psychiatry / La Revue Canadienne de Psychiatrie2023, Vol. 68(12) 916‐924© The Author(s) 2023Article reuse guidelines:sagepub.com/journals-permissionsDOI: 10.1177/07067437231164571TheCJP.ca | LaRCP.ca
Background: Repetitive transcranial magnetic stimulation (rTMS) is recommended in Canadian guidelines as a first-line treatment for major depressive disorder. With the shift towards competency-based medical education, it remains unclear how to determine when a resident is considered competent in applying knowledge of rTMS to patient care. Given inconsistencies between postgraduate training programmes with regards to training requirements, defining competencies will improve the standard of care in rTMS delivery.
Objective: The goal of this study was to develop competencies for rTMS that can be implemented into a competency-based training curriculum in postgraduate training programmes.
Methods: A working group drafted competencies for postgraduate psychiatry trainees. Fourteen rTMS experts from across Canada were invited to participate in the modified Delphi process.
Results: Ten experts participated in all three rounds of the modified Delphi process. A total of 20 items reached a consensus. There was improvement in the Cronbach’s alpha over the rounds of modified Delphi process (Cronbach’s alpha increased from 0.554 to 0.824) suggesting improvement in internal consistency. The intraclass correlation coefficient (ICC) increased from 0.543 to 0.805 suggesting improved interrater agreement.
Conclusions: This modified Delphi process resulted in expert consensus on competencies to be acquired during postgraduate medical education programmes where a learner is training to become competent as a consultant and/or practitioner in rTMS treatment. This is a field that still requires development, and it is expected that as more evidence emerges the competencies will be further refined. These results will help the development of other curricula in interventional psychiatry.
Résumé
Contexte: La stimulation magnétique transcrânienne répétitive (SMTR) est recommandée dans les lignes de conduite canadiennes à titre de traitement de première intention du trouble dépressif majeur. Avec le changement de la formation médicale axée sur les compétences, il demeure incertain de déterminer à quel moment un résident est considéré compétent pour appliquer les connaissances de la SMTR aux soins des patients.
Étant donné les incohérences entre les programmes de formation postdoctorale relativement aux exigences de la formation, définir les compétences améliorera la norme de soins dans la prestation de la SMTR.
Objectif : La présente étude avait pour but de développer les compétences pour la SMTR qui peuvent être mises en œuvre dans un programme d’études de formation basé sur les compétences dans des programmes de formation postdoctorale.
Méthodes : Un groupe de travail a rédigé des compétences pour les stagiaires en psychiatrie postdoctorale. Quatorze experts de la SMTR du Canada ont été invités à participer au processus Delphi modifié.
Résultats : Dix experts ont participé aux trois cycles du processus Delphi modifié. Au total, 20 articles ont atteint un consensus. Il y avait une amélioration de l’α de Cronbach dans les cycles du processus Delphi modifié (l’α de Cronbach a augmenté de 0,554 à 0,824), ce qui suggère une amélioration de la cohérence interne. Le coefficient de corrélation intraclasse s’est accru de 0,543 à 0,805 suggérant ainsi un meilleur accord interjuge.
Conclusions : Ce processus Delphi modifié a donné un consensus d’experts sur les compétences à acquérir au cours des programmes de formation médicale postdoctorale où un apprenant est formé à devenir compétent comme consultant et/ou praticien du traitement de SMTR. C’est un domaine qui nécessite encore un développement, et il est prévu qu’avec l’arrivée de plus de données probantes, les compétences seront encore plus raffinées. Ces résultats contribueront à l’élaboration d’autres programmes d’études en psychiatrie interventionnelle.
Keywordstranscranial magnetic stimulation, psychiatry, competency by design, medical education, Delphi
Transcranial magnetic stimulation (TMS) is a non-invasive neurostimulation procedure delivering magnetic fields to treat psychiatric and neurological disorders. Repetitive transcranial magnetic stimulation (rTMS) is currently recommended as a first-line treatment in the Canadian Network for Mood and Anxiety Treatments (CANMAT) 2016 guidelines for major depressive disorder.1 There is increasing interest and access to TMS internationally including USA, Brazil, Australia, Israel, Germany, Finland, Serbia and the United Kingdom.2 The National Institute for Health and Care Excellence (NICE) guidelines and position statement in 2017 also supports the use of rTMS in the treatment of depression.3 As access to novel neurostimulation strategies is made available, clinicians need to become familiar with treatment options. At the time of this study, access to rTMS is increasing globally and in Canada, 3 out of 10 provinces (Alberta, Quebec and Saskatchewan) and Yukon have publicly funded rTMS interventions. Furthermore, the Health Technology Assessment Committees in British Columbia and Ontario have made formal endorsements for rTMS to be covered under provincial health plans. Once implemented, 90% of the Canadian population will have access to rTMS.4
In the Canadian context, psychiatry resident trainees are required to become competent regarding prescribing and delivering electroconvulsive therapy (ECT) as outlined by the Royal College of Physicians and Surgeons of Canada (RCPSC) training requirements.5 Similar learning objectives for ECT are outlined for the American Board of Psychiatry and Neurology (ABPN).6 As competencies have been developed for ECT, this ensures standardization of training for residents who are receiving teaching despite being at different universities and hospitals. In the new competency-based training curriculum developed by the RCPSC in 2020, there is a requirement to develop knowledge of neurostimulation strategies other than ECT, such as rTMS and deep brain stimulation.7 However, it remains unclear how to determine when a resident would be considered competent in being able to apply knowledge of rTMS to patient care. Currently, although there is exposure to rTMS in residency, inconsistencies exist between programmes with regards to training requirements.8 For those who wish to practice neurostimulation as part of clinical practice, training and competency standards are not yet defined. For example, although rTMS has been approved for use by the NICE guidelines, specific training requirements regarding indication and delivery have not yet been outlined by the Royal College of Psychiatrists in the United Kingdom in the recent 2022 curricula.9
Defining the milestones and competencies for rTMS is important as novel neurostimulation strategies become available and access improves across jurisdictions. Training regarding neurostimulation modalities should be a mandatory component of psychiatric residency. Standardizing training requirements across different training sites using a common list of competencies will be important as competency-based training curricula are implemented across training programmes.10 The list of competencies will need to differentiate between different stages of training, that is, foundations of discipline, core of discipline or transition to practice.11
This study used a modified Delphi methodology to develop a list of competencies for psychiatry residents in training for rTMS. The Delphi methodology is a widely used technique to develop a consensus from a group of experts.12,13 In this modified Delphi methodology, statements are first developed by a working group based on current neurostimulation curricula and available literature. The goal of this study is to develop milestones and competencies for rTMS which will be useful in developing a competency-based training curriculum in Canadian postgraduate psychiatry training programmes. Our work may facilitate similar efforts to develop formal training in interventional psychiatry in other jurisdictions.14
The development of the competencies for rTMS occurred through a modified Delphi process. This study was approved by the Ontario Shores Research Ethics Board (OS REB #21-015-D).
The Interventional Psychiatry Competency Diploma (IPCD) Steering Committee was formed at the University of Toronto in May 2021. A working group was led by Dr Amer M. Burhan (Physician-In-Chief and Research Chair at Ontario Shores Centre for Mental Health Sciences) and Dr Robyn Waxman (Medical Head of Brain Stimulation Clinic, Ontario Shores Centre for Mental Health Sciences). The objective of this working group was to define a set of competencies in the practice of rTMS relevant for residents training in general adult psychiatry, and advanced psychiatry trainees in neurostimulation necessary to become a competent consultant in rTMS.
The working group included current rTMS practitioners (RW, AMB) affiliated with Ontario Shores and a senior psychiatry resident (KSPL) to bring the perspective of both current rTMS practitioners and trainees. Current rTMS training requirements were reviewed using the brain medicine fellow and interventional psychiatry curricula available in the United States15–17 as well as recent work on competencybased education in neurostimulation.11 Competencies were drafted that were relevant for both general adult psychiatry resident trainees and advanced psychiatry trainees in neurostimulation and this list was distributed to rTMS expert panelists for review. None of the members of the working group participated as an expert panelist. The survey items consisted of a questionnaire using a Likert scale ranging from 1 to 5. The survey items were divided into the stages of residency training as outlined by the RCPSC: foundations of training, core of training and transition to practice.18 The survey included opportunities for experts to enter free-text comments relevant to individual items and to enter suggestions for each statement.
rTMS practitioner experts from across Canada were invited to participate in the modified Delphi process. Those who participated were subsequently invited to be co-authors of this manuscript if they agree to provide critical feedback on the drafts of this manuscript. Criteria used to define an “experts” included: practiced rTMS on a regular basis at an academic health sciences centre and have been doing so for at least the past 2 years and has contributed as a clinician educator and/or clinician-researcher in the field. This study included clinicians ranging from the early stages to late stages of their careers. Clinicians were recruited from a variety of sites differing in geographical locations across Canada. Emails were sent to members of the Ontario rTMS network who meet the above criteria to invite them to participate in this study.
Expert panelists communicated findings using Survey Monkey (https://www.surveymonkey.com). Results from the first round of the Delphi process were collected and summarized. For each round of the modified Delphi process, Cronbach’s alpha and the intraclass correlation (ICC) were calculated for internal consistency and interrater reliability respectively. Items that had a level of agreement of less than 80% were modified or removed from the subsequent round of the modified Delphi process. Survey items were refined accordingly based on panelist feedback. The modified survey was approved by the working group and then sent to the expert panelists for a subsequent round of the modified Delphi process. A total of three rounds were pursued and has been suggested to be adequate in achieving consensus for a Delphi study.12 This was set to allow for consensus building over multiple rounds of the survey while minimizing sample fatigue. The final list of survey items consisted of statements that had a rating of at least 80% approval by experts.
Survey items were analysed using IBM SPSS Statistics version 26.19 Means and SDs of survey items were calculated with a summary of relevant expert feedback to expert panelists.
Cronbach’s alpha was used to assess internal consistency and reliability of survey items among experts.20 Cronbach’s alpha assessed the extent to which items within the scale were intercorrelated for each round of the modified Delphi. Results of the Cronbach’s alpha were interpreted as follows: α ≥ 0.9, excellent; 0.9 > α ≥ 0.8, good; 0.8 > α ≥ 0.7, acceptable; 0.7 > α ≥ 0.6, questionable; 0.6 > α ≥ 0.5, poor; and α < 0.5, unacceptable.21
The two-way mixed effects ICC was used to calculate interrater reliability and agreement between experts for each round of the modified Delphi process. A two-way mixed effects model was used given that there are multiple raters making measurements on multiple items in each survey. The two-way ICC is therefore suitable to account for the variability between the multiple raters. ICC values less than 0.5 are indicative of poor reliability, values between 0.5 and 0.75 indicate moderate reliability, values between 0.75 and 0.9 indicate good reliability and values greater than 0.90 indicate excellent reliability.22
Fourteen experts in rTMS were invited to participate in this study. Ten experts consented and participated in all three rounds of the modified Delphi process (Figure 1). The remaining four experts did not reply to the invitation. Table 1 describes the demographic characteristics of the experts. Experts on the panel have been practicing in psychiatry for an average of 23 years and practicing in rTMS for an average of 12 years. Three experts were female. Most experts were practicing in Ontario (N = 7), while others were practicing in Quebec (N = 1), Alberta (N = 1) and British Columbia (N = 1).
There was a total of 21 items in the first formulation of the survey. Two new items were introduced for round 2 of the modified Delphi survey after feedback. These items were related to the mechanism of action of rTMS and identifying contraindications to rTMS. One item related to educating the interdisciplinary team was excluded for concerns about redundancy. Therefore, a total of 22 items were rated for round 2 and round 3. There was a total of 14 items that were reworded over the course of the two rounds. A total of 20 items reached a consensus and 2 items did not reach a consensus. Table 2 shows the statements presented to the expert panelists during round 3 of the modified Delphi survey with mean rating scores and percent agreement.
There was improvement in Cronbach’s alpha over the rounds of modified Delphi process. Cronbach’s α = 0.554 in round 1 and improved up to Cronbach’s α = 0.824 by round 3 suggesting improvement in internal consistency within the survey items. In addition, the ICC improved from ICC = 0.543 in round 1 to ICC = 0.805 by round 3 suggesting improved interrater agreement over the rounds of the modified Delphi process. Items that benefited from rephrasing included items related to managing complications of rTMS and items related to the technical aspects of delivering rTMS. Items, where there were difficulty in achieving consensus, were related to managing maintenance treatment after rTMS and how to deliver academic contributions related to rTMS.
Calculations of means, SDs and ranges of the expert panel’s in response to each statement over the rounds of modified Delphi process show reduction of the SDs and ranges suggesting convergence and stability of agreement among expert panelists (Supplementary Table S1).
This study developed and validated competencies for postgraduate medical education programmes where a learner is training to become competent as a consultant and/or practitioner in rTMS using the modified Delphi methodology. The utilization of the modified Delphi process allowed for quantitative results showing improvement in internal consistency and agreement over the rounds of Delphi as well as the opportunity to incorporate qualitative feedback from comments provided by expert panelists in subsequent rounds of the modified Delphi process.
Although broad learning objectives are previously defined in the fellowship programmes reviewed, this study proposes competencies that can be practically implemented into training programmes for rTMS. The implementation of competency-based medical education is expected to better track resident progression status than time-based training programmes as competencies can track both the amount and volume of training, especially with procedural-based tasks.23 Therefore, a minimum amount of exposure is set to ensure residents are obtaining adequate amount of exposure to a procedure such as rTMS. Implementation of competency-based education also encourages increased direct observation of residents which will improve the quality of feedback provided to learners. Competencies were developed using a competency-by-design framework so that there is a graded level of responsibility in a progressive fashion. Residents in general adult psychiatry training programmes would be expected to successfully complete the competencies in the foundations of discipline stage outlined in Table 2, but advanced psychiatry trainees in neurostimulation would be expected to be competent in the competencies across all stages.
Over multiple rounds of the modified Delphi process, expert panelists expressed concerns about whether the objectives for those competencies can be met based on the limited amount of time allocated for training. For example, for statement 14 “Finding motor threshold for treatment”, experts felt there would not be enough time during training to become competent in the technical skills of rTMS. However, expert panelists felt that this is an important statement to include because motor threshold is the method by which rTMS dose is determined and side effects are managed; therefore, it is an essential competency to achieve that cannot be delegated to others involved in the care delivery such as rTMS technicians.
It is important to clarify that although these are the competencies agreed upon by rTMS experts, these statements may be flexibly interpreted by the learner and the evaluator. For example, statement 4 evaluates the competency to “Inform the patient about potential risks, benefits, and complications of rTMS”. The number of items that could be assessed are quite broad including scalp tenderness, pain, seizures, as well as competency in educating the patient and caregivers.24,25 This is similar for statement 14, “Manage immediate complications (e.g., Seizure) arising from rTMS” where complications may include seizures and tachycardia.25,26 It is expected that even though these may be rare complications arising from rTMS, learners should be able to manage and triage this as these complications can be life-threatening. In addition, with statement 15 “Manage immediate side-effects (e.g., Scalp pain) arising from rTMS”, side-effects of rTMS may include headache, syncope, dizziness, facial twitching and tearing.
These competencies would form the basis for the development of entrustable professional activities (EPAs) as per the RCPSC competence by design model. EPAs are the specific tasks that supervisors can observe and evaluate in trainees. One EPA may integrate multiple items from the list of competencies as demonstrated in Table 3 (“Assessment of Eligibility for rTMS”). The EPAs can therefore incorporate multiple CanMEDS roles such as medical expert, communicator, collaborator and leader. The implementation of these competencies as EPAs will likely require further review with psychiatry programmes before further implementation.
There were two items that did not reach a consensus in the modified Delphi process. Item 18 was “Recommend relapse prevention strategies after an acute course of rTMS treatment (e.g., Medications, maintenance rTMS)”. Understandably, the concern for this statement was that evidence remains limited for how to sustain maintenance after an acute course of rTMS. In addition, training regarding relapse prevention is expected to have been completed through general adult psychiatry training. Item 21, “Provide education and scholarly contributions in rTMS (teaching, audit, or research).”, also did not reach a consensus. It was felt that this is not expected or required for all learners, but it is recommended that these opportunities be made available for those residents who wish to pursue this (e.g., Evaluating efficacy or clinical utility of accelerated rTMS protocols or theta burst stimulation).
This study was limited to clinicians who are practicing at academic centres and may be difficult to generalize to learners who are practicing in community-based settings. In addition, there have also been concerns raised about the implementation of competency-based medical education including increased assessment burden and logistical challenges including increased need for observation of learners.27 Therefore, these factors should be considered when applying these competencies as the use of rTMS generalizes and becomes more readily available.
This study was a first attempt at reaching a consensus for postgraduate medical education programmes for residents in general adult psychiatry and for advanced trainees in neurostimulation and rTMS. This is a field that still requires development and research and it is expected, as more evidence emerges in the field of rTMS, that competencies will be updated to reflect new knowledge in the field (e.g., when there is evidence for rTMS in diagnoses other than unipolar depression or maintenance treatment in rTMS). These competencies will be useful in the development of other curricula such as training requirements for an Interventional Psychiatry Diploma.
Lai KSP: conception and study design, data acquisition, analysis/interpretation of data, drafting manuscript, revision of manuscript and final approval of manuscript; Waxman R: conception and study design, data acquisition, analysis/interpretation of data, drafting manuscript, revision of manuscript and final approval of manuscript; Blumberger DM: data acquisition, interpretation of data, revision of manuscript and final approval of manuscript; Giacobbe P: data acquisition, interpretation of data, revision of manuscript and final approval of manuscript; Hasey G: data acquisition, interpretation of data, revision of manuscript and final approval of manuscript; McMurray L: data acquisition, interpretation of data, revision of manuscript and final approval of manuscript; Milev R: data acquisition, interpretation of data, revision of manuscript and final approval of manuscript; Palaniyappan L: data acquisition, interpretation of data, revision of manuscript and final approval of manuscript; Ramasubbu R: data acquisition, interpretation of data, revision of manuscript and final approval of manuscript; Rybak Y: data acquisition, interpretation of data, revision of manuscript and final approval of manuscript; Sacevich T: data acquisition, interpretation of data, revision of manuscript and final approval of manuscript; Vila-Rodriguez F: data acquisition, interpretation of data, revision of manuscript and final approval of manuscript; and Burhan AM: conception and study design, data acquisition, analysis/interpretation of data, drafting manuscript, revision of manuscript and final approval of manuscript
Additional data can be accessed through Supplementary Table S1.
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Ka Sing Paris Lai https://orcid.org/0000-0001-7650-6356
Daniel M. Blumberger https://orcid.org/0000-0002-8422-5818
Peter Giacobbe https://orcid.org/0000-0001-6642-6221
Supplemental material for this article is available online.
1 Department of Psychiatry, University of Toronto, Toronto, Ontario, Canada
2 Department of Psychiatry, Ontario Shores Centre for Mental Health Sciences, Whitby, Ontario, Canada
3 Department of Psychiatry, Temerty Centre for Therapeutic Brain Intervention, Centre for Addiction and Mental Health, Toronto, Ontario, Canada
4 Department of Psychiatry and Behavioural Neurosciences, McMaster University, Hamilton, Ontario, Canada
5 Department of Psychiatry, University of Ottawa, Ottawa, Ontario, Canada
6 Department of Psychiatry, Queen’s University, Kingston, Ontario, Canada
7 Department of Psychiatry, McGill University, Montreal, Canada
8 Department of Psychiatry, University of Calgary, Calgary, Alberta, Canada
9 Department of Psychiatry, Western University, London, Ontario, Canada
10 Clinical Sciences Division, Northern Ontario School of Medicine, Thunder Bay, Ontario, Canada
11 Department of Psychiatry, University of British Columbia, Vancouver, British Columbia, Canada
Corresponding author:Amer M. Burhan, MD, Msc, FRCPC, Ontario Shores Centre for Mental Health Sciences, 700 Gordon St, Room 5-3007, Whitby, Ontario, Canada L1N 5S9.Email: burhana@ontarioshores.ca
