The Performance Improvement Blog

New e-Book for Getting Business Results from Training

Getting business results from training, or any learning intervention (for example, coaching) for that matter, often has more to do with organizational issues than it does with design and delivery of the learning event. Learning professionals have become very good at creating powerful learning experiences for participants. Yet, the number of participants who apply this learning to achieving important business results for their organizations is still quite small. Some researchers put that number in the 10% to 20% range. 

Sean Murray, CEO of RealTime Performance, and I want to increase that number. We have identified five factors that contribute to the impact of learning interventions. We believe that when learners and managers of learners attend to these factors, it is much more likely that training will contribute to business success. These factors are: 

1. Alignment: Align learning with strategic goals.
2. Anticipation: Anticipate success.
3. Alliance: Create a learning alliance between learner and boss.
4. Application: Apply learning immediately.
5. Accountability: Hold learner and organization accountable for business results.

These factors make up what we call the "5As Framework". You can read more about this framework in our new e-book, Getting More From Your Investment in Training: The 5As Framework. The e-book also has valuable tips and tools you can use to get more impact from your training programs. You can download the first two chapters free or you can purchase the whole e-book at the RealTime Performance Web site. Let us know what you think.

Learning From Training Evaluation

If you are not doing follow-up evaluation of your training programs, you are not getting the most you can from these performance interventions.  I’ve written about this before but it deserves repeating. Evaluation has many uses. Kirkpatrick uses evaluation to determine learner reactions, increased knowledge, behavior change, and results. Phillips uses evaluation to measure the ROI of training. Brinkerhoff uses evaluation to assess impact of training on the business. I think another just as valid purpose of  evaluation is to ensure learning.

Asking participants questions about their learning makes it more likely that learning will occur. When I read a book, especially fiction, and don’t discuss the book with anyone for several weeks, I forget what I read.  The same phenomenon occurs for most people after attending training events. If they do not think about the material again in a meaningful way within a short period of time, they forget. Follow-up evaluation requires learners to think meaningfully about what they learned.

Simply the process of asking questions of learners has the effect of cementing that knowledge in their brains. Maybe it’s the repetition. Maybe it’s the renewed commitment to the content. Maybe it’s the sense of being held accountable for learning. Whatever the internal process that goes on in learners, the result of asking questions is employees that are more likely to retain new knowledge and apply that learning to their work.

Downturn Opportunities for Performance Improvement

The current downturn in the economy and the much-anticipated-but-slow-in-coming upturn, presents companies with a unique opportunity for performance improvement. John Baldoni writes in his blog post titled, Use the Downturn to Your Advantage,  that this is the time for organizations “…to get rid of anything and everything that does not add value to the bottom line”,  look to employees, customers, and vendors for ways to improve operations, and model for all employees how to overcome obstacles during tough times.

All great advice; but I wouldn’t do any of this without data. Collect and analyze data that describes your processes from both a statistical and human point of view. For example, changing the work schedule to more hours over fewer days might look more efficient on paper, but if it takes an emotional toll on workers, it might not be worth altering. Survey employees, customers, and vendors and use that data as the basis for a conversation with them about improving operations. Ask them what measures would indicate effective operations. Use those measures to assess the change and to make decisions about improvement.

This is the time for radical change…as long as it is supported by evidence. Change simply for the sake of change will cause you to lose the trust of employees, customers, and vendors. Change that is indicated by the facts will gain you the respect of these stakeholders, even if they disagree with your conclusions.

ROI vs. IOB

In his article in the current issue of Chief Learning Officer Magazine, Timothy L. Hill argues for IOB over ROI. That is, when it comes to learning interventions, CLOs should measure “Impact on Business” instead of “Return on Investment”. Hill writes:

ROI typically falls short when it comes to determining how engaged learners were, how much they retained or how job performance skyrocketed after learning. Further, the results of programs have to last long after the classroom or online program concludes. For instance, are employees using their new skills to improve customer service? Are sales on the upswing? Has customer repeat business improved? Have inventories declined? Are the improved results maintained for several quarters, or is it just an immediate spike?

While I agree with his basic supposition, he weakens his argument by using a very narrow definition of ROI and then rejecting the measure as insufficient. Hill defines a training ROI as the ratio of cost savings to the amount spent on training. This is a standard financial definition. The problem is that in the employee training field (See ROI Institute.), ROI has come to mean a comparison of all of the costs of employee learning to all of the impacts and benefits of the intervention, which can include employees retained, sales goals achieved, increased revenue, increased market share, job performance improvement, effective teamwork, customer satisfaction, employee engagement, etc. However, even using this broader definition, ROI is only part of the picture. The problem with a formulaic calculation of ROI is that you don’t know how the learning intervention affected outcomes, you don’t know to what extent the learning intervention contributed to the change, and you don’t know what else could have been done to increase impact on business results.

CEOs want to know more about business results. The important thing is to measure attainment of the learning intervention’s specific goals and produce evidence that is convincing to CEOs and other executives, answering their questions about learning and performance. An ROI calculation might be part of this, but the intended outcomes might require observations by team leaders and co-workers or stories of how employees applied what they learned to achieving business results. For example, in leadership coaching, process and outcomes are often not known at the outset. To evaluate this kind of learning intervention, you need to discover the story about what was done in coaching, what the learner learned, how that learning was applied, and what difference it made for the organization. If this is what Hill means by “IOB”, I’m all for it.

CEOs Want to Know Impact of Training

CEOs want more evidence of impact and less evidence of employee reactions (Kirkpatrick’s Level 1) to training. At least, this is one of the findings from a study by Jack and Patti Phillips recently published in T+D. They surveyed CEOs of large companies selected from both the Fortune 500 list of largest public companies and Hoover’s list of the largest privately-held companies. Ninety-six CEOs responded, which is 21.3 percent of the 451 CEOs who were sent surveys. Only six percent were “very satisfied” with their companies’ current measures of learning and development and 96% indicated that their companies should be doing more to measure the impact of learning and development programs on business results.

I have some concerns about the generalizability of the survey data given the way the sample was constructed, but I commend the effort to try to hear from CEOs about their perceptions of learning interventions in their companies. The findings confirm a common belief among evaluators of learning interventions (training, coaching, etc.) that CEOs are not being given the kind of data they want from the learning and development professionals in their organizations. CEOs want to know the impact on business goals but, instead, they are given attendance figures and told how much employees like the programs. The authors of the report write:

Executives intuitively feel that providing learning opportunities is valuable, and they logically anticipate a payoff in important, bottom-line measures, such as productivity improvements, quality enhancements, cost reductions, time savings, and improved customer service. Yet the frustration comes from the lack of evidence to show that programs really work.

If executives want bottom-line measures, there is much they can do to get what they want. For one, they could ask their learning and development leaders to provide this information. Then, when they receive impact data, they could report these findings widely in their organizations. They could ask their managers about the implications of the findings and what they will do to increase impact in the future. And they could give recognition and rewards to learning and development professionals who provide evidence of results and implement solutions to increase results. CEOs should hold themselves and their entire organizations accountable for the impact of employee learning on business results.

Measure Success or Measure Failure?

Tom Friedman in his July 29 column in The New York Times, titled “59 Is The New 30”, writes about 59 year old pro-golfer Tom Watson almost winning the British open:

This wonderful but cruel game never stops testing or teaching you. “The only comment I can make,” Watson told me after, “is one that the immortal Bobby Jones related: ‘One learns from defeat, not from victory.’ I may never have the chance again to beat the kids, but I took one thing from the last hole: hitting both the tee shot and the approach shots exactly the way I meant to wasn’t good enough. ... I had to finish.”

I have great respect for Tom Friedman and Tom Watson, but this often repeated saying that we learn more from failure than from success, has always bothered me. It’s just not true. Even Tom Watson, if he really thought about it, would probably agree with me. He’s known for decades, that he “had to finish.” Not making the shots he intended to make didn’t teach him anything. I’m a 40 handicap golfer. When I do something good on the course, I try to repeat it. When, as is more often the case, I lose my ball in the rough or five-putt the green, I don’t have a clue why. I don’t learn much from making errors, but I learn a lot from my successes.

When I evaluate the impact of training programs, as well as any kind of learning intervention, I look for examples of success, not to the exclusion of failure to apply learning, but my emphasis is on results that made a positive difference for the organization. Why? Because people learn more from those examples than from failure. I attribute this to three factors: 1) we can extract desirable and repeatable behaviors from successes whereas failures rarely tell us what will work; 2) learners are more willing to be open and honest about their successes; it’s less personally threatening than talking about failure; and 3) talking about success is rewarding whereas talking about failure is punishing - positive reinforcement works better than punishment.

And now there is biological research to support this notion of learning. MIT brain researchers found that monkeys learn more from their successes than their failures due to cell activity. Of course, we don’t know if this mechanism plays out in the same way in humans but it is fascinating to hear about evidence at the neural level.

First, Select the Metrics that Measure Success

Yogi Berra said, “You can see a lot by just looking.” Jacqueline Kosecoff, CEO of Prescription Solutions, a pharmacy benefit management company, makes “looking” a key aspect of her leadership. She believes that effective leadership includes measuring organizational performance and using that information to improve.

In an interview conducted by Adam Bryant for the New York Times, Kosecoff said:

I’m a great believer in measurement. So before I begin the execution phase of any project, I sit down with my team and we ask ourselves: “What are the metrics against which we’re going to measure our success?”

We do two things: We measure where we’re succeeding, and where we succeed we celebrate. And we also measure where we’re not succeeding, and where that happens we ask ourselves, “Can we go back and fix something?” And if so, we do it. And if not, we make sure that we understand where we went wrong, put it into the corporate DNA, so the next project won’t have that flaw.

Another thing I learned was that when you’re involved in a large development project, projects often morph. And when people become advocates of their project, they change some of those metrics so that they can claim success when perhaps it’s not 100 percent legitimate to do so.

So creating the metrics up front, and having a discipline of saying, “O.K., this is where we want to go, and if we don’t achieve it it’s O.K., we’ll try in another way to get there,” is very helpful. Not just for me, but for the whole team.

It sounds like common sense: “create the metrics up front.” However, often this is not what many companies do. They tend to choose a metric out of convenience rather than because it is the best indicator of the success they want. Take customer service for example. Is the goal a positive customer attitude toward the company, customer satisfaction with a buying experience, few customer complaints, a timely response to a question or complaint, a low number of product returns, on-time delivery, customer retention over time, customer referrals, or something else? A hospital can achieve high customer satisfaction scores on a customer survey if nothing bad happened during the patient visit, but is that what you want, merely adequate service? Or, should the service be outstanding? In that case other metrics are needed that require a little more effort, such as measuring the extent to which nurses and doctors treat patients with understanding and compassion (See: If Disney Ran Your Hospital: 9 ½  Things You Would Do Differently)? As Kosecoff suggests, first decide what you mean by “success” and then design a metric that will help you determine when and how that goal is achieved.

 

Employee Training Analytics

Everybody is trying to do more with less. Companies are scrutinizing every expense and trying to eliminate as much waste as possible. This is true for employee training and development programs, as well. The most common way of measuring effectiveness of these programs has been attendance and end-of-session "smile sheets". I and others have been arguing for years that these are poor proxy indicators of learning, let alone performance. Arif Ahmed writes in TrainingZONE that there are better measures of performance:

With today's economic climate, organisations will increasingly find that the drive for efficiency is pushing performance analytics up the agenda. I believe that it is possible to evolve current measurement processes within an organisation by simply adding a series of 'value metrics' such as:

• Has the number of customer queries reduced?
• Has customer satisfaction improved or declined?
• Has quality of output from staff improved or declined?
• Is there an alignment of individual and organisational performance?
• How quickly are technical queries resolved per person?
• Is customer and staff retention getting better or poorer?

I agree that the answers to these questions might be better indicators of training effectiveness than the more common measures, assuming that these are the goals of the training intervention. But even if these are the intended outcomes, how do you know that it was the training that made the difference? How do you find out what else contributed to the change? And how do you find out what are the unanticipated consequences of training? This is where interviews come in. You need to talk to learners and customers to hear their stories that link training to results. When done with care, this approach can be a valid, credible, and practical way to use information to drive waste out of training and development programs.

A Checklist for Evaluating Training

The UK’s Chartered Management Institute offers a seven step “checklist” for evaluating training. Of course, any evaluation checklist is, by definition, an over-simplification of what can be a very complex process. Essentially, the CMI approach consists of clarifying training objectives, comparing trainee knowledge and behavior to objectives, and using this information to improve the training in the future. This is a useful way of thinking about evaluation when the program to be evaluated is highly technical, straightforward, predictable, and knowledge and skills can be learned in a relatively short period of time. But for anything else, such as leadership development, the seven steps fall short. Graham O’Connell, of the UK National School of Government, captures the issues well in his comment about the CMI article:

…I think there are some gaps [in the checklist] which reflect gaps in the wider debate on evaluation. Firstly, let me say unequivocally that evaluation is about securing and demonstrating great value results. But for these to be sustainable results we need to understand why things have worked and, where there have been problems, why those have occurred and what we might do about them. In other words we need to evaluate the process right from concept and the agreement of success criteria, through the design and delivery of the learning interventions, on to the consolidation, application and cascade of learning, and into improved performance and any other intended or unintended outcomes. In a sense evaluation is about learning. Our learning about learners, other stakeholders, organisations, culture, change, business and how learning interacts with all these issues…What we need is an intelligent analysis of the whole process - including satisfaction and results, quality improvement and systemic learning, real business benefits for today and increased capability for the longer term.

I agree with O’Connell; when we think of training evaluation as an opportunity for organizational learning, and not simply a way to determine if the initially intended results were achieved, the approach we must take is very different from CMI’s seven point checklist. We must consider how trainees come to the task of learning, how the training is delivered in the classroom and outside the classroom, what organizational factors (e.g., structure, communication, strategy) are affecting learning, what happens in the workplace over time to facilitate learning or to be a barrier to learning, what are the unintended consequences of the training program, and how are the individual, teams, and the organization affected by what the trainee has learned.

Evaluation Reinforces Learning

In my last post (5/15/08), I talked about using evaluation to understand the impact of training and development programs. Now I want to say a few words about using evaluation to increase the impact of these programs. One aspect of this is using evaluation to set expectations for learning and performance improvement. Jody Hollis wrote about this in The Winning Teams Blog:

By placing emphasis on a formal evaluation process, you are sending signals to staff and managers that you are expecting a certain set of behaviours and actions as a result of the intervention, and that you expect to get the desired results. That is a very powerful message, and is a very different one to asking if they enjoyed the learning experience.

Training evaluation increases performance in another way, too. An evaluation process that asks learners to recall what they have learned and how they have applied that learning helps to sustain learning and application over time. If there is no follow-up, there is unlikely to be sustained learning and application. It’s like learning to play the piano. Teachers don’t require piano recitals because they enjoy listening to the awful sound of beginners; they require recitals because that motivates learners to practice and because the feedback from a recital improves performance. That is, the evaluative comments from the teacher and others help to improve piano playing skills and retain that learning. It’s the same for workplace learning. It doesn’t need to be a “recital” (although requiring a formal presentation to a panel of executives is not a bad idea in some cases), but it is critical that there is some kind of systematic follow-up that requires the learner to recall what was learned and explain how that learning has or has not been applied.

I have interviewed many managers who, several months after their participation in a leadership development program, told me about having intended to apply the new skills, that they have not done so yet, but that my questions reminded them of how they can immediately use that ability in their work. In effect, following up with these managers, reinforces learning and increases the likelihood that they will use that learning in their jobs. Not doing follow-up interviews would have “left money on the table.” I am convinced that evaluation is an essential tool for maximizing the impact of learning interventions.