TRAININGINSIGHTS
By Marcos A. Carreras, Ph.D.
Surveys and other measurement tools are indispensable in today’s data-driven world. Organizations, educators, researchers and various professionals rely on these instruments to gather valuable information, assess performance and make informed decisions.
In this article, we explore the importance of Kirkpatrick level one surveys, and the effectiveness of performance-focused questions. We’ll share a case study on dimensions for assessment of training impact and the significance of standardizing dimensions to generate standard and actionable insights. Finally, we’ll talk about the importance of executive summaries and dashboards to present our data results.
The Kirkpatrick Model is a widely recognized framework for evaluating training and development programs. It consists of four levels, each representing a different dimension of assessment. Level one focuses on reactions, measuring participants’ immediate response to the training experience. It gauges factors such as satisfaction, engagement and relevance.
Level one surveys serve as the foundational step in data collection, particularly in organizational and educational settings. These surveys typically encompass basic information about participants, such as demographics and general feedback. Level one surveys lay the groundwork for deeper analysis by providing context and understanding the characteristics of the surveyed population.
Level one surveys are commonly referred to as smile sheets but can be much more than simple questionnaires seeking participant satisfaction with a program. While smile sheets do measure participants’ immediate reactions and sentiments toward the learning experience, they can be enriched to collect valuable data beyond mere smiles using performance-focused questions.
Traditional Likert type (agree-disagree) questions primarily focus on participants’ satisfaction and engagement, but they fail to address the crucial aspect of performance improvement. Performance-focused questions revolutionize this evaluation process by introducing questions specifically designed to measure participants’ learning outcomes and the application of newly acquired knowledge and skills.
This paradigm shift in approach allows for a more holistic understanding of the training’s impact on individual and organizational performance. According to Will Thalheimer, Ph.D., performance-focused questions have four key components that make more robust level one surveys:
When I join an organization, one of the first steps I always do is to spend time learning about their culture, ways of working and understanding of “pain points.” One of the biggest issues explained to me by stakeholders was that the existing measurement strategy was felt as “stuck.”
Basically, surveys were coming from everywhere, there was no standardization of questions asked and no actual strategy on reports, analytics or presenting data (for example, executive summaries and dashboards), which didn’t allow effective sharing with leadership and other stakeholders. This is when it was decided that a survey and measurement theoretical framework needed to be implemented.
A theoretical framework holds immense importance. It provides a structured and organized approach to understanding, analyzing and interpreting information. It serves as a roadmap that offers a systematic way to define and organize key concepts and ideas. It provide a shared language and understanding among users, reducing ambiguity and enhancing communication.
A robust theoretical framework increases the validity and reliability of data. It helps ensure that the data collection methods, analysis and interpretations align with the established key performance indicators (KPIs) or dimensions, reducing the risk of random or biased results or misinterpretations. It builds on the collective knowledge and with standardized dimensions, allows data comparison and trending.
Our theoretical framework was based on dimensions, which relate to organizational KPIs and objectives. The goal was to have dimensions that were impactful to the business, actionable by the users, easy to understand and quantifiable. Our organization came up with the following five dimensions for our measurement strategy (See Figure 1)
Standardizing these dimensions is essential to ensure consistency and comparability of survey results. By establishing clear definitions and measurement scales for each dimension, survey creators can avoid ambiguity and interpretation variances. Standardization allows for easier analysis of data and facilitates comparisons between different groups or time periods, ultimately leading to more accurate conclusions and actionable insights.
Measurement strategies should also be flexible enough to change when necessary. There can be several triggers for change:
A revision cycle was included as part of the strategy to accomplish this goal. (See Figure 2)
In the ever-evolving landscape of data-driven decision-making, the role of dashboards has become increasingly vital. Dashboards serve as visual representations of survey and measurement data, presenting complex information in a user-friendly and accessible manner.
One of the primary advantages of dashboards is their ability to visualize data effectively. Instead of sifting through spreadsheets and raw numbers, stakeholders can access clear, concise and interactive graphical representations. Charts, graphs and heatmaps are among the many visualization options available in dashboards, enabling users to, at a glance, understand trends, patterns and relationships.
Dashboards offer real-time or near-real-time updates, allowing users to access the most current data available. This dynamic feature is particularly valuable in dynamic environments, where decision-making depends on up-to-date information. Users can monitor dimensions and track progress toward goals in real time, enabling swift action and intervention when necessary.
They can be tailored to cater to specific audiences, providing relevant insights to different stakeholders. Managers, executives or peers may require varying levels of detail and metrics. By customizing dashboards to meet the unique needs of each audience, data-driven discussions become more focused and effective.
Surveys and measurements are potent tools that empower organizations with valuable data to drive informed decision-making. Kirkpatrick level one surveys provide the groundwork for deeper analysis, while performance-focused questions offer a deeper and more targeted insight on the responses. By considering dimensions like confidence, motivation, knowledge stretching, support and training experience, survey creators and analytics professionals can extract comprehensive data to shape effective strategies.
Moreover, standardizing dimensions ensures consistency and comparability, enhancing the reliability and utility of survey results. In harnessing the power of surveys and measurements, individuals and organizations can pave the way for growth, development and success.
As leaders use the data from surveys and measurement tools for decision-making, dashboards emerge as indispensable tools to present data with clarity and impact. By visualizing data, providing real-time updates and customizing for specific audiences, dashboards empower stakeholders to identify trends, anomalies and actionable insights efficiently.
The collaborative nature of dashboards fosters communication among teams, leading to more effective data-driven discussions. Ultimately, the integration of dashboards into the survey and measurement process enhances the utilization of data, propelling organizations and professionals toward success.
Marcos A. Carreras, Ph.D. is an industrial organizational psychologist and senior manager SPC commercial training, for Sanofi. Email Marcos at marcos.carreras-vargas@sanofi.com.
