MEASUREMENT
By James Delaney and Steven Just, Ed.D.
Learning and development (L&D) teams have spent years trying to improve Level One evaluations. Mature organizations have moved beyond simple satisfaction questions (i.e., “Did you like the course?”) and have begun asking more meaningful questions about relevance, confidence and intent to apply learning on the job. That shift was necessary, but it may no longer be enough.
Consistent, standardized learning experiences are beginning to disappear. As learning becomes increasingly personalized, conversational and embedded directly into the flow of work, the idea of evaluating a single shared “course experience” starts to lose relevance. In an AI-driven learning environment, the future of Level One evaluations requires an entirely different approach to measurement.
Traditional Level One evaluations were built on a simple assumption: Learning happens in a defined event with a relatively consistent experience across learners. Everyone attends the same course, consumes the same content and completes the same activities. A post-course survey then attempts to measure reaction to that shared experience. That assumption often no longer holds.
Increasingly, learners rely on AI-powered support that is immediate, contextual and personalized to their specific task or situation. One employee may use an AI assistant to troubleshoot a customer issue. Another may generate a just-in-time job aid before a meeting. Someone else may receive personalized coaching recommendations based on recent performance data. Two people participating in an AI role play may have very different interactions with the same avatar. None of these individuals are necessarily experiencing the same learning interaction, and in many cases, there is no clear “course” at all.
When learning becomes continuous and individualized, traditional course evaluation begins to lose meaning. Asking someone whether they “liked the course” becomes irrelevant when there was no course to begin with. This does not mean Level One evaluations disappear. It means their purpose must evolve.
The problem is no longer just satisfaction surveys. Many organizations have already moved beyond those questions. Even modernized Level One evaluations — with questions about confidence, intent to apply or perceived relevance — still rely on subjective signals. They rely on learners accurately predicting future behavior, which is often unreliable. In AI-enabled learning environments, organizations may no longer need to depend solely on what learners say they will do. Increasingly, they can observe what learners actually do.
AI-driven learning experiences create new opportunities to capture objective indicators of value in real time. Rather than relying exclusively on surveys, organizations can begin measuring learning through interaction patterns and workflow behavior. Examples may include:
Whether an AI interaction resolved the learner’s immediate need.
Frequency of returning to AI-generated performance aids.
Reuse or refinement of generated outputs.
Time-to-completion improvements after support interactions.
Reduction in repeat questions or support escalations.
Patterns of successful task completion following guidance.
Continued use of AI-enabled support tools over time.
These signals are imperfect, but they represent something traditional Level One evaluations rarely could: observable evidence of applied use.
If learning is continuous, evaluation must become continuous as well. In AI-enabled environments, measurement can happen immediately following meaningful learning interactions and directly within the flow of work.
Some of these signals may still involve lightweight prompts embedded into the experience itself:
Did this interaction answer your question?
Were you able to complete the task?
Did this guidance help you move forward?
Would you use this support again?
These questions are more contextual and actionable than traditional post-course surveys because they focus on immediate utility rather than what might happen after a learning event. Increasingly, organizations can supplement subjective responses with behavioral data generated through the learning process itself.
How often did the learner revisit the resource? Did they successfully use the generated performance support? Did the interaction reduce dependency on managers, peers, support desks or retraining? Did it help accelerate completion of real work?
These forms of embedded measurement align far more closely with operational performance than traditional course surveys ever did.
This does not mean learner feedback becomes irrelevant. Subjective perception still matters. Confidence, clarity and perceived usefulness can provide valuable context that behavioral data alone may miss. But surveys should become one signal among many, not the centerpiece of evaluation.
In AI-driven learning environments, organizations can move beyond satisfaction scores and even beyond subjective performance-focused surveys. By combining in-the-moment feedback with observable behavioral signals, learning teams can begin measuring something far more meaningful: Whether support was useful, work improved and learning translated into behavior change.
James Delaney is the founder and principal consultant of Talent Experience Group. Connect through jdelaney@talentexperiencegroup.com or linkedin.com/in/jdlearning.
Steven Just, Ed.D., is CEO and principal consultant at Princeton Metrics. Connect through sjust@princetonmetrics.com or linkedin.com/in/steven-just-081b76.
