The Performance Improvement Blog

The Course Catalog Is (Should Be) Dead

Recently, I was asked a very thought-provoking question by the director of HRD for a large organization. I was explaining the “5As Framework” to her and she asked, “What are the implications for what we do here?” I immediately launched into my usual speech about getting managers of learners
involved in actively supporting the learning process. This led to a discussion of all the excuses she hears from managers to explain their lack of involvement in employee learning. It wasn’t until later, after further rumination, that I realized the better answer to the director’s question is, “Throw out the course catalog!” 

Like HRD departments in many large organizations, each year they offer a long list of courses and programs that have all of the current buzz words in their titles. I have no doubt that her department delivers excellent, world-class courses and programs. And I’m sure employees enjoy the courses and even say that the knowledge and skills will be very useful to them in their work. However, I’m also quite sure that very little has changed in the wider organization as a result of the many outstanding events put on by this unit. The problem is the lack of alignment between event-based instruction and important business results.

The competency model, which provides the underlying theory of event-based training, is doing a disservice to the current needs of organizations. Training and development professionals seem to believe that if they can identify the competencies that people need in order to do their jobs and then offer courses to teach those competencies, employees will be more productive and successful. The problem with this way of thinking is that performance has as much (if not more) to do with work culture as it does with job competencies. To be successful, employees need supervisors and co-workers who are supportive of learning and performance improvement, they need a supervisor/boss/mentor who talks to them regularly about learning and results, they need the opportunity to apply new learning and to receive feedback, and they and the organization need to be held accountable for learning and results. If there is this culture of learning, employees will continually improve performance.

According to my colleague, Ralph Jacobson, we have to change the local environment so that people actually “work differently”, not just learn how to work differently. All the courses in the world will not change behavior and improve results unless the local culture supports change. At one time, when jobs were static and organizational change came slowly, a menu of well-designed courses was sufficient. Now, when change has to happen rapidly, when we have to find ways to lower the cost of learning, and when we have better and better online, just-in-time, mobile learning resources, event-based learning is no longer the best answer.

We must provide learning experiences to people when and where they need it and in ways that optimizes application to achieving important business results. We need training and development that is timely, short duration, and project-focused. We need frequent on-the-job feedback from supervisor/boss/mentor to employee. For example, leadership skills are best taught during a challenging, team project while the leader is trying out new behaviors and receiving ongoing feedback from a coach. This way of learning will have much more impact on performance than a course or conference on leadership.

What do you think? Can we throw out the course catalog?

The Learning Alliance and Kirkpatrick's Four Levels of Training Evaluation

In response to my blog post titled, “Kirkpatrick’s Four Levels of Training Evaluation: A Critique”, Wendy Kirkpatrick wrote a comment directing me to a white paper that she co-authored with Jim Kirkpatrick, "The Kirkpatrick Four Levels: A Fresh Look After 50 Years 1959 - 2009." That document describes updated thinking that the Kirkpatrick’s call “The Kirkpatrick Model”. They’ve added some critical elements to the
original “Four Levels”, including: starting evaluation planning by clarifying intended results of training; taking into consideration what happens before and after a training event that facilitates learning; the supervisor/mentor/coach partnership in learning; and "organizational drivers" to results.  I applaud these additions to their model; they are all factors that Rob Brinkerhoff and I described in our 1994 book, The Learning Alliance: Systems Thinking in Human Resource Development.

As we did then and as we do now, Rob and I have great respect for what the Kirkpatricks have done to heighten awareness of the importance of evaluation to the field of workplace learning. However, the problem persists in organizations in the U.S. and around the globe: because people have a label for something, they believe it has value. So, by labeling a measure “Level One”, “Level Two”, “Level Three”, or “Level Four”, training managers believe they are collecting useful information when, in fact, they most likely are not. I wish I had a dollar for every time I've asked training managers about evaluation and they have said to me, "We do Level Ones and, when we can, Level Twos."

I concede that any measurement of training has some reinforcing effects on learning and, by bringing organization attention to the training, may have internal political benefits, as well. But until training and learning specialists measure the impact that a learning intervention has on the business and evaluate the organizational factors that either drive those results or prevent them from occurring, organizations will continue to learn little from evaluating training. 

 

Evaluating Training Through Stories

Participants rated the management training program 4.3 on a five-point scale. Sixty percent said that they have applied the training to their work. What do we know about effectiveness of the program? Not much.

We cannot determine from this data if the management training program made a difference for the organization. If it didn't make a difference, there is little reason to continue to offer the program to managers in the same form. Unless we’re talking about a training company, training is a means to an end, not an end in itself. This is how Maggie Bayless, head of training at Zingerman’s, a “Small Giant” company in Ann Arbor, Michigan, describes this link between training and business results. She is quoted In an interview for a local magazine:

"We call it ‘bottom line training,’ Bayless explains, “because we know that the purpose of Zingerman’s businesses isn’t to train staff. For The University of Michigan, one of their bottom lines is education; that’s why they exist. But Zingerman’s wasn’t launched with the idea of creating an organization where we train staff. Zingerman’s was created to be an organization where people delivered a great experience, with great food and great service. To do that, we need to train staff effectively.”

 “So in order for it to make sense to do training,” Bayless explains, “it has to improve food quality, improve service quality, or improve financial results. Those are our three bottom lines; that’s why we’re here. Otherwise, we shouldn’t be doing it.”

So, if the purpose of training is to achieve bottom-line results, then we should measure how and to what extent the program does that and what the organization needs to do to improve results in the future. A “smile sheet”, or even a follow-up survey, is not going to give you this information. We must tell the employee's story. That is, we must create a narrative that clearly explains the causal links between what people learned in the training program, how they applied that learning, and what the effect was on business results. In addition, the story should describe what the organization did and did not do to facilitate learning and results (e.g., providing an opportunity to apply newly learned leadership skills).

David Zinger describes the power of stories in his blog post, "Employee Engagement Stories: Altering Our Personal GPS." He writes:

Stories are the building blocks of understanding. We understand the world through story. Story is not just something we read in a book or see in a movie. Story is how we make sense of ourselves, situations, and others. Stories function as fundamental building blocks of understanding and also as a social GPS. Stories are the conclusions, judgments, hypothesis and assumptions about what we see and hear. Stories are the way we communicate with ourselves.

It’s like evaluating a summer car vacation by counting the miles driven, gas used, days on the
road, and money spent. All nice to know, but it’s the experiences along the way, anticipated and unanticipated, that make the trip meaningful and memorable.  

 

Kirkpatrick's Four Levels of Training Evaluation: A Critique

At the end of Dan McCarthy’s blog post, “How to Evaluate a Training Program”, in which he explains his pre-post, survey approach to applying the Kirkpatrick four levels of training evaluation, he asks: Has anyone used a system like this, or something better? What do you think, is it worth the bother?  Since McCarthy asked, here is my response.

Yes. I’ve used a similar “system” of evaluation many times in my career as a program evaluator. However, I don’t recommend this approach any longer for most situations. While better than nothing, for many training programs (as well as for coaching, mentoring, simulations, self-directed learning, etc.), this approach does not produce the information needed to continuously improve performance and achieve business results. There are at least six reasons for this.

First of all, there is low correlation among the four levels (reaction, learning, behavior, and results). This takes nothing away from the contribution that the Kirkpatrick model has made to the field over the past 50 years. I hate to think where employee training would be today if we hadn’t been guided by Donald Kirkpatrick’s thinking. However, just because learners liked the training doesn’t mean that they learned anything new, or will apply what they learned, or will contribute to achieving business results. Knowing one level does not predict outcomes at the other levels.

Second, self-report surveys produce unreliable performance data. What people say they did or will do is often not what actually happens (I know this is not shocking news.). This is not to say that we shouldn’t ask; it just means that we have to interpret those findings very cautiously.

Third, it shouldn’t be a decision to evaluate or not to evaluate. Good training programs measure impact; they hold learners (and their organizations) accountable. That’s part of the learning process. And some form of evaluation should always be done because it reinforces learners learning and contributes to performance improvement.

Fourth, the outcomes of leadership training cannot be fully anticipated. To construct a useful survey, we have to be able to anticipate what will be learned, how it will be applied, and what difference it will make so that we know what questions to ask. However, especially with leadership training, that kind of prediction is nearly impossible. The only way to discover all of the outcomes is through direct observation or in-depth interviews.

Fifth, if we apply the Kirkpatrick model strictly, then we’re not asking if the training program is the right thing to be doing in the first place.  Shouldn’t we ask that question first, before we ask about reactions, learning, behavior, and results?

Sixth, performance improvement is never the result of training, alone. Many organizational factors (e.g., manager support) determine whether impact from that learning will occur. If, as McCarthy suggests, we only compare what people knew before training to what they know and do after training, we still will not know what to do to achieve better results.

In most learning interventions, especially for leadership and management training, Robert O. Brinkerhoff’s Success Case Method is a more useful approach to evaluation. I have written about this method in previous blog posts.

Thank you, Dan McCarthy, for asking. 

 

Five Steps to Business Impact From Learning

Five key organizational factors determine if there will be business results from learning: 1) alignment; 2) anticipation; 3) alliance; 4) application; and 5) accountability. Together, these factors make up a framework that contributes to employee success. Sean Murray and I wrote about this in our book, The 5As Framework. Sean will be talking about the framework in a session at the American Society for Training & Development Conference in Orlando, Florida on May 25th. If you go to the ASTD Conference, I encourage you to attend Sean's session.

In his session, Sean will describe the factors and tools participants can use to ensure that learning is a business driver in their organizations. Sean's goal is that participants leave the session understanding each of the 5As and how they impact learning. After attending, participants will be able to conduct a 5As audit to identify the greatest opportunities for increasing business success from training and education in their organizations. Participants will be able to create an action plan for achieving business results. They will hear how other organizations are using the 5As Framework to achieve business success.

Training for No Results

Why do companies continue to deliver corporate-wide employee training in a way that ensures minimal results? Recently, I heard about two examples of national corporate offices requiring that employees in local offices attend their training programs. That is not necessarily a bad thing. However, the way in which each corporation did it is highly wasteful and does nothing to promote the respect and trust of employees, let alone improve performance. Unfortunately, this is typical of centralized training programs. In one case, franchisees were required to send all managers to a franchisor-led program on new customer-management software. In the other case, a local office of a national, land-development company was required to send all new employees to corporate-delivered, project-management software training. 

In each case, employees were told by the home office to attend multi-day training programs in a distant city without any direction to have conversations with their bosses about the importance of this training to the company. Learners and bosses did not talk about how the training fits with individual development goals. They did not discuss mutual expectations for applying the training in the workplace. They did not plan how the training would be applied back home to make a difference in the organization. And they did not identify measures they could use to know if the training had contributed to business results. Without these conversations and goal-setting with their bosses, these employees are highly unlikely to apply what they learned in any significant way. The companies will, once again, have wasted valuable resources. In fact, it probably would have been better not to have offered the training in the first place. 

Multi-location companies seem to believe that if they require employees in local offices to attend company training programs, that will ensure consistency in operations throughout the organization. This is purely wishful thinking. They will be lucky if more than 20% of those participants ever use what they learned back on the job. Many forces conspire to prevent application of learning, not the least of which is a lack of clear expectations for what will be learned and how that learning will be applied. For further explanation of the factors that ensure that training programs contribute to business results, read: The 5As Framework. 

Retroactive Alignment

Nobody would disagree with the notion that alignment of learning interventions with business goals is a good thing. However, too often this connection is rationalized after the fact. That’s too late to make a difference for the organization. David Vance, former president of Caterpillar University, in an interview with ASTD, talks about the importance of aligning learning and business goals at the same time as fiscal-year budgeting is being done. He says:

It’s almost too easy to just say we need to align learning with the goals of the company. Everyone knows that is the correct answer. But in my experience, very few organizations do this strategically and proactively. Many will actually do it retroactively. They will connect the dots between the goals of the organization and the programs they ran last year. To me that is retroactive alignment, and you shouldn’t get credit for that. If you haven’t done alignment before the year began, how do you know if you have budgeted your learning toward the highest priority needs of the company? If you haven’t done it before the year begins, you haven’t given your CEO or governing body the ability to prioritize that spending and to make sure that the learning investment is being (properly) directed to the highest needs of the organization. You can only do that proactively.

I agree with Vance that budgeting and prioritizing are important reasons for aligning learning with company goals. However, those are not the only reasons for alignment. Alignment is also critical to retention and application of learning. Employees need to know that what they will be learning is valued in the organization and will help them contribute to achieving business success. They need to know the path that they will take from learning to application to business outcomes to achievement of strategic goals. Alignment that is apparent to learners is highly motivating and increases the likelihood that what they learn will be applied. If alignment is done retroactively, it cannot motivate learning nor can it ensure the application of that learning.

 

How to Have a Learning Conversation

Employees learn best when they have guidance, support, and performance feedback from someone they trust and respect in the organization. Ideally, this is a boss, supervisor, coach, or mentor who knows why that learning is important and how it aligns with strategic business goals. Managers, at all organizational levels, should have regular learning-focused conversations with the people who report to them. In these conversations, described in our book, The 5As Framework, managers should ask:

1. What do you need to learn in order to help this organization achieve its business goals?
2. Given what you need to learn, what should be your specific learning objectives for the short-term and long-term?
3. How will you and I know if you are making progress toward and achieving those objectives?
4. How can I help you get the most out of your learning opportunities?
5. What resources do you need and how can I help you get them?
6. What should be our schedule for regular conversations about your progress?
7. What do we need to change about this plan?

The key to making this a successful relationship is to keep the focus on how the employee’s learning and performance improvement will contribute to achieving business goals. Companies can no longer (as if they ever could) afford to allow employees to participate in learning activities simply because they are an enjoyable break from the routine, a motivating perk, or conveniently scheduled. Managers must make sure that the time, effort, and money invested in their direct reports is contributing to the organization being successful. 

 

Correlation Without Causation

Organizational leaders often misinterpret social science data. That might be okay on a quiz show like “Whad'Ya Know?”, but not when making important organizational decisions. The problem is that executives, like all human beings, are wired to see causal relationships between behaviors and events that happen at approximately the same time. Christopher Chabris and Daniel Simons talk about this problem in their new book, The Invisible Gorilla: And Other Ways Our Intuitions Deceive Us.

Everyone is susceptible to what these authors call the” illusion of cause.”  Even David Brooks, a highly respected journalist and editorial columnist for the New York Times, demonstrates this bias toward causation in his December 7, 2010 column. He summarizes behavioral research that was reported to him by Kevin Lewis, a social science writer. In these brief summaries Brooks implies causal relationships between behaviors and events. These are the causal relationships Brooks describes in his column:

• Periods of peak fertility in females makes them avoid male relatives
• People who strongly doubt their own beliefs will strongly argue in favor of those beliefs
• Touching among team members improves the team’s performance
• Daylight savings time lowers SAT scores
• Gun shows have no effect on crime
• Low glucose levels result in a loss of self-control and increase the likelihood someone will commit murder
• Extroverted leaders do best when their employees are passive and don’t take the initiative
• Men increase risk-taking in the presence of attractive women
• Men are not attracted to women who dumped their last boyfriends, whereas women are attracted to men who dumped their last girlfriends

Each of these findings from social science research is an excellent example of how even very smart people make the mistake of attributing a causal relationship to two variables that, although correlated statistically, might each be caused by a third variable or the interaction of four, five, six, or more other variables. Maybe when sports teams are winning, players are more likely to touch each other in the excitement of the moment. Maybe when things are going well in a company led by someone who is extroverted, employees don’t see a need to initiate new programs. Many alternative theories can explain why each of those findings summarized by Brooks could be untrue. The problem Brooks, Lewis, and the social scientists have is the "illusion of cause".

Within companies the “illusion of cause” occurs, for example, when executives attribute business success or failure to their leadership, or to a new marketing program, or to a reorganized sales department, or to innovative products, or to a change in the economy. These events might be highly correlated but they are not necessarily the cause of business success or failure. Another area where leaders suffer from the "illusion of cause" is in judgments about the value of employee training and development programs. If there is no significant improvement in the performance of employees who go through training, the training is considered ineffective when, in fact, it could be a host of other behaviors and events that are barriers to learning and to learners applying what they learned. Conversely, if leaders observe an improvement in individual or team performance, they attribute it to the training. That might also not be true for the same reasons. What are examples of the “illusion of cause” that you see played out in your organization?

 

Good Evaluation Facilitates Learning

Evaluation of programs (e.g., training, coaching) and organizations has more to do with facilitating learning than it does with statistics. Typical measures such as ratings of programs, frequency of participation, or fluctuation in revenue are useful only if they become the stimulus for a facilitated discussion of stakeholders that addresses implications for enhancing performance. Good evaluation requires good facilitators.

Kylie Hutchinson writes about this issue in her post on the American Evaluation Association blog, AEA365. She quotes this definition of “facilitator” that appears in the book, Facilitator’s Guide to Participatory Decision-Making:

…an individual who enables groups and organizations to work more effectively; to collaborate and achieve synergy. They are a content neutral party who…can advocate for fair, open, and inclusive procedures to accomplish the group’s work. A facilitator can also be a learning or a dialogue guide to assist a group in thinking deeply about its assumptions, beliefs, and values, and about its systemic processes and context.

As organizations seek to be more accountable and measure the impact of learning interventions, they will need facilitation skills to help people think more deeply about what they believe and what they need to do to improve. Measuring change is part of good evaluation, but unless we can get leaders in an organization to learn from that data and apply that knowledge to improving performance, what’s the point?