COVERSTORY
The Novartis Skills & Capabilities Training Team, which provides training to all U.S. field associates, needed to evolve its sales training measurement strategy. While anecdotal feedback indicated the training was perceived as “good,” it was time to develop and collect empirical data to support this claim.
The team realized, “You can’t manage what you can’t measure” and needed clarity about how training and coaching were impacting field team behaviors. They wanted to benchmark results and improvement, and be able to communicate the evidence-based value of training initiatives.
Novartis worked with IC Axon on the measurement strategy and ACTO on the learning dashboards to show the value of commercial learning and development to the organization. The team gathered, centralized and reviewed data to:
Validate the effectiveness of training and make informed decisions about future improvement based on data rather than conjecture.
Measure the impact of sales training on field rep performance.
Track learning time and core capabilities for new hires, which enabled Novartis to understand how much time was invested in training and whether it was leading to improved competency.
To develop a viable learning dashboard, Novartis began by mapping out the elements required to build a strong data story. To do this, Novartis developed the following five-step approach:
Define the problem: Articulate the problem that needs to be solved.
Identify “data logistics”: Outline what types of data are needed to solve this problem.
Consider “people logistics”: Determine who needs to be involved and at which stages.
Push the launch button: Communicate the timeframe in advance of the launch to ensure success.
Measure “how did we do?”: Review, adjust and present data accordingly, and solicit input and discussion.
After outlining the goals and mapping out the data story, Novartis addressed several challenges, which included:
A disparate learning technology ecosystem: Novartis had multiple technology platforms that were used for training, which posed challenges to the data collection process. It was important to the team that we did not add any additional platforms to the already complex ecosystem for the measurement strategy.
Lack of standardization: With different training events having different goals, it was difficult to standardize questions to be answered for all events. However, having standardized survey questions was essential for benchmarking and comparing data across different programs. Novartis worked to incorporate standard questions applicable to all training events. Additionally, a “program-specific” section was introduced, allowing trainers to add custom questions based on the specific training objectives of different training events.
Change management: Initially, each trainer was building their own survey specific to their training, which led to a hodgepodge of data sets. To address the lack of standardization, consistent communication and reinforcement at the leadership level helped to ensure buy-in from the trainers and made the move to the new system more acceptable, efficient and manageable.
This data-driven training initiative has significantly improved and enhanced learning transfer, learner engagement, performance and commitment in the following ways:
Implementing evaluations for new hires: By implementing learner evaluations for Phase 1 (the “readiness” phase involving onboarding and certification), and Phase 2 (the “mastery” phase involving continuous coaching), trainers gather real-time feedback from learners and make necessary adjustments to the training content and delivery to improve the overall program. This allows for continuous improvement and ensures the learning experience was built to meet the needs and preferences of the learners.
Introducing evaluations for managers: Introducing manager evaluations 60 days after new hires completed their training was a novel aspect of this initiative. It provided valuable feedback from managers on how well the new hires performed post-training.This feedback identified gaps in learning transfer and enabled targeted support and development for field teams. The result of this initiative was to strengthen the connection between training (readiness) and real-life performance in the field (mastery).
Incorporating a Net Promoter Score (NPS): Incorporating a NPS for workshops and learning programs was another significant enhancement. By asking learners how likely they were to recommend the course to a colleague, the NPS provided insights into learner satisfaction and engagement. This information allowed trainers to gauge the overall impact and effectiveness of the workshops and learning programs, as well as identify areas for future improvement.
These enhancements contributed to better learning transfer and an increase in learner engagement and commitment. By gathering learner feedback, receiving manager evaluations and utilizing the NPS, trainers have a comprehensive view of the learning experience and can make data-driven decisions to enhance the effectiveness and impact of the training initiatives.
Utilizing one system to measure field effectiveness and learning pre-work, launch, post-launch and pull-through (and pulling in external data) enhanced the efficiency, accuracy and effectiveness of data analytics. This not only allowed Novartis to obtain valuable data for assessing and improving training, but also established a strong, standardized and viable strategic decision-making process.
This initiative was groundbreaking within Novartis; the innovative design and delivery revolutionized how Novartis measures and evaluates learning outcomes.
With the introduction of this strategic learning measurement approach, Novartis developed a standard evaluation process, comprehensive dashboards, a benchmarkable approach and team alignment. In the short term, insight from data has enabled trainers to identify areas of strength, areas that require improvement and opportunities for growth. It also empowered them to make informed decisions and adjustments to enhance the effectiveness of the training programs.
With the implementation of such a measurement strategy, Novartis can now provide key stakeholders with NPS scores for every training event, along with meaningful data on content, learning experience, modality and job impact.
The initiative to improve data measurement, collection, analysis and reporting has led to Novartis measuring training effectiveness and enhancing learning experiences.
Brezden Peter is associate director, learning technology & measurement for Novartis. Email her at brezden.peter@novartis.com or connect through linkedin.com/in/brezden-peter-53838529.
