The Performance Improvement Blog

Evaluation Is Not Complicated

In his blog post, “Stop Evaluating Training!”, Amit Garg summarizes a presentation that Robert O. Brinkerhoff gave at the Australian Institute of Training and Development conference in April. Garg relates Brinkerhoff’s comments to the challenges of measuring the effectiveness of elearning programs. Garg writes:

So how do you evaluate the success of eLearning that you create?" As a learning solutions vendor, I’ve been asked this question countless times and have also encountered it in many an RFP. Proving the effectiveness of training and showing ROI is no walk in the park and still keeps L&D up at night.

Maybe evaluation of the impact of training is not a “walk in the park”, but it is not “rocket science” either. As Brinkerhoff explains in The Success Case Method, there is a simple logic to assessing impact. First we have to understand the alignment between a learning intervention, whether that’s classroom-based, elearning, coaching, on-the-job structured experiences, or something else, and the intended impact (e.g., customer retention, production, sales, revenue, market share). Then it’s a matter of identifying and interviewing learners who are contributing to that impact and those that are not contributing. The purpose of these interviews is to understand the nature of the impact and why it’s happening or not happening.

Three immutable laws drive this method of evaluation. One is the law of learning as a process not an event. This law holds that learning is a process that starts before an instructional event and continues after the event. To study the event only is to fail to include key aspects of what is facilitating learning and what are barriers to learning.

Another is the law of unintended consequences which holds that when there is change there are always outcomes that weren’t anticipated. A useful evaluation of training will examine these unintended consequences (positive and negative) as well as achievement of objectives of the intervention.   

And the third law holds that significant, lasting change in complex organizations is always the result of many factors, some that are not in the control of trainers and learners. For example, effectiveness of leaders, change in strategic direction, and the economy could have a profound influence on the impact that learners have on their organizations.

It’s not complicated. Understand how the learning intervention is intended to help the organization achieve its goals. Identify employees who are and are not applying what they learned. Interview these employees to find out what they did or did not do, how that affected the organization, what other factors were intervening in that impact, and what can be done to increase learner impact in the future.

The Future of Workplace Training

Paul Matthews, in a blog post titled, “The future of training is not training,” argues that workplace training needs to be about building capability not delivering courses. He writes:

Many people in training seldom stop to think why they are doing the training. The logistics and hassles of keeping the training department running are sufficient to fill up their days and obscure the real purpose.

Matthews goes on to say that the future of training is in building the capability of employees to do their jobs more effectively.

Sean Murray and I present a similar argument in our book, “The 5As Framework,” except we go further to say that training, as well as any learning intervention in an organization, should contribute to achieving the strategic goals of that organization. We wrote:

Companies today can no longer afford to rely on these isolated events to make a difference, whether a one-day skill-building workshop, or a year-long leadership development institute. Maybe there was a time when companies could offer these events to employees without concern for results. Today, resources are too precious. You must make sure that you are maximizing the impact from every performance intervention…This means that rather than a culture of events, you need a culture of learning, one that supports ongoing learning throughout your organization. In a learning culture, the normal behaviors, customs, expectations, and goals are all oriented towards learning and performance improvement. (p.32)

For too long, HR, training departments, and even Learning and Development units, have focused on events. And they have become quite good at doing events. But the future of training is not events, nor is it only capability. It is learning that contributes in a measurable way to the success of the organization.

("The 5As Framework" is now available as an ebook from Learning2BGreat.com)

Related articles

Why Learn?

How to Create a Learning Culture in Organizations

The 0% Solution to Learning

The Unexamined Leadership Program is Not Worth Doing

If you’re not going to evaluate a leadership development program, don’t do the program! It will be a waste of time, money, energy, and trust. End-of-program reactionnaires (aka smile sheets) don’t count as evaluation. I’m talking about a systematic, evidence-based look at why it was done, what was done, how it was done, what happened as a result, how it can be improved, and what the organization learned from the process.

Viv Nunn of UK’s Open University, in an article for TrainingZone, explains some of the reasons for evaluating professional development programs. She writes that evaluation should provide the following:

• Evidence of the extent to which the professional development is contributing to your organisation's success
• Validation that the correct learning solution has been identified and suggestions for programme improvement
• Advice about how to get the most from your L&D budget by considering the workplace learning environment

I would add:

• Reinforcement of learning. That is, the act of evaluating (i.e., observing) a leadership development program reinforces learning. If done well, evaluation heightens awareness of goals, unintended consequences, and the knowledge, skills, and attitudes that were learned.

To achieve these purposes, you have to look beyond the seminar, workshop, internship, mentoring, coaching, online course, or any other method of instruction. Learning, especially about leadership, is affected by the organizational environment and external factors, as well as the program. To fully understand why and how learning did or did not occur and what impact it had, you have to examine organizational factors in addition to program factors.

Nunn provides us with an excellent description of the factors that can affect learning:

If the workplace learning environment isn't right, if learners cannot find opportunities to apply what they have learned at work, if the organisational culture is not one which encourages them to try out their new skills and knowledge, there will be little or no impact to report, regardless of how effective the actual learning solution has been. Isolating learning from its context will not provide useful data alone. Evaluating the context of learning along with the learning solution will give you the greatest opportunity to get the most from your investment.

I hate to think about how many leadership development programs I’ve participated in as either a learner or facilitator that were thought-provoking, information-rich, and emotionally powerful only to find that they resulted in little or no change because there was no immediate opportunity to apply that learning back on the job. Participants might learn about team-building but not have a team to lead in the workplace or be discouraged by management from applying what they learned in their teams.

Evaluation of professional development must examine how workplace culture supports or hinders learning. Knowing whether intended results follow, in time, a professional development program or not, won't help you unless you understand the factors that affected those results. And, as Nunn suggests, it's things like clarity of goals, manager's attitudes toward learning, incentives for professional development, opportunities to apply new learning, and feedback, that have more to say about performance improvement than does the quality of instruction alone. 

Evaluation is not an option; it’s an integral part of the learning process. If you want a leadership development program to be more than entertainment and you want it to achieve learning that results in significant performance improvement, than you must evaluate the program and the organizational evironment of that program.

 

What's Better: eLearning or Classroom?

We need to examine the proliferation of Web-based education before it gets out of hand (It may be
too late.). Online college courses are being promoted as the panacea for producing more college graduates and strengthening the workforce. The Obama administration has been touting the educational value of online courses since coming into office over three years ago, and many colleges, especially community colleges, have adopted that mantra.

Many companies are also convinced that “elearning” is the answer to their employee training and development needs. They are told elearning will be cheaper, faster, more convenient, and more effective than traditional, classroom training. Also, using elearning programs makes managers feel like they are staying on the cutting edge of technology. Nobody wants to be left behind.

Feeding this appetite for online programs is research that seems to support the notion that online is better than face-to-face, classroom instruction. A study by the U.S. Department of Education found a small advantage for online (blended) programs. However, they qualify that finding by writing:

Despite what appears to be strong support for blended learning applications, the studies in this meta-analysis do not demonstrate that online learning is superior as a medium. In many of the studies showing an advantage for blended learning, the online and classroom conditions differed in terms of time spent, curriculum and pedagogy. It was the combination of elements in the treatment conditions (which was likely to have included additional learning time and materials as well as additional opportunities for collaboration) that produced the observed learning advantages.

The problem is that whether online programs are better than face-to-face depends on the purpose of the course, learning goals of the program, skills of the instructor, and expectations of learners, among other things. I published an article about this nearly 10 years ago in Educational Technology. At that time, I wrote (in ilalics):

1. E-learning will not revolutionize training; it is only one instructional method among many, each better at achieving some instructional objectives than others. (I was wrong; e-learning is revolutionizing training. However, the change has been in the accessibility of information, not in the impact of instruction.)
2. Putting a course on the Web or on a CD-ROM does not ensure performance improvement.(Learning and change depend more on expectations of learners and organizational culture than on the mode of delivery.)
3. Employees learn in many different ways. (For some, online instruction fits their style of learning, but for others it is not always the best mode.)
4. E-learning is not a low cost alternative, especially if it is not aligned with the organization's strategic goals. (If the content is not aligned, it will be a waste of time and energy. No matter how cheap it is per learner, it will be too expensive.)
5. Having a vast selection of courses is meaningless. (It’s wonderful, for personal edification, to have access to thousands of courses from Carnegie Mellon, MIT, and other prestigious colleges. But if these courses aren’t related to what learners need to know on the job and can’t be applied immediately, then more is not better.)
6. Work and learning are the same. (The tendency of managers is to think that by having instruction online, employees can use non-work time for learning. But this separation between work and learning is a false dichotomy. Especially in this day and age, employees have to be continually learning; it must be part of their job.)

In the past 10 years, the technology of online instruction has evolved considerably. Interactivity has advanced. Audio and video have continued to improve. Practical mobile tools for just-in-time applications of elearning are now available (e.g., smart phones and tablets). However, the danger is that organizational leaders and technologists will assume that any learning that is computer enabled is better than direct human contact. For some learning, this could be true. For other learning, it is not. We need to align method of content delivery with intended learning outcomes.

photo credit: Mr_Stein via photopin cc

The 0% Solution to Learning

Ideally, organizations should be striving for a culture in which every aspect of the workplace supports continuous individual, team, and whole organization learning. It should be a place where Individuals are learning how to help the organization achieve its strategic goals, where teams are learning how to help the organization achieve its strategic goals, and where the organization as a whole is learning how to achieve its strategic goals.

In this kind of culture, formal classroom training is superfluous. The most important learning occurs in the conversations that employees have with their supervisors and co-workers, in the conversations that teams have about team development and effectiveness, and in whole organization experiences with problem solving and planning. Learning occurs from the feedback and reflection around individual, team, and whole organization successes and failures. Learning occurs because of a performance management system that permeates every aspect of organizational life, not because of isolated instructor-based courses, workshops, and seminars.

Robert O. Brinkerhoff makes this point in his writing about measuring the contribution that training makes to learning and performance improvement. He argues that the isolation score, that indicator of how much training contributes to performance results, should be zero. He writes:

An isolation score of zero, on the other hand, seems to me to be the perfect solution: it means the organization has figured out a way to drive a desired performance improvement and resultant business result solely by manipulating the performance system, so there is no need for any training at all. Isn't this scenario exactly what the performance-oriented practitioners would seek? Training is always an expensive solution. If you can get to the desired business outcome without spending a dime on training - by just adjusting incentives, motivation, performance support, etc. - isn’t this what a thoughtful HRD leader should strive for?

The implication of striving for an “isolation score of zero” is that the whole organization owns learning, not just HR and the CLO, and when the organization needs performance improvement, training is not the first course of action. We have to change the way we think about learning and performance improvement, especially as people need to learn faster and achieve better results.

Unfortunately, it’s a very difficult mental model to change. In a recent post on ASTD’s Learning Circuits Blog, Connie Malamed asks, “What do you think the future role of the training professional will be?” She offers this model of future roles:

Malamed's model is an accurate representation of the current state of things but not where I think the field needs to go in the future. My model would not include “design and create courses.” My model would put “align learning with the strategic goals of the organization” at the center and “create a culture of learning throughout the organization” as the next key role. We have so many learning support options now, including just-in-time elearning (desktop and mobile), coaching, mentoring, simulations, on-demand video, experiential learning, internships, and on and on, that we rarely need to develop courses. Learning and development professionals need to be focused on performance results, not on training. 

 

Evaluate Learning Process

As organizations evaluate their programs and services they need to pay attention to process as well as outcomes. Catherine (Brehm) Rain, of Rain and Brehm Consulting Group, Inc., writes this in aea365:

Process evaluation ensures you answer questions of fidelity… did you do what you set out to with respect to needs, population, setting, intervention and delivery? When these questions are answered, a feedback loop is established so that necessary modifications to the program or the evaluation can be made along the way.

Life is a journey—and so is a long-term evaluation. Stuff happens. However, it is often in the chaotic that we find the nugget of truth, the unknown need, or a new direction to better serve constituents. A well-documented process evaluation assists programs to ‘turn on a dime’, adapt to changing environments and issues, and maximize outcome potential.

The problem with much of the evaluation that goes on in organizations, whether that’s evaluation of training and development programs, communication programs, marketing campaigns, a new sales approach, or strategic planning, is that the process is not examined. You might discover what participants thought of the program from “smile sheets” and you might even know how participants applied what they learned if you are fortunate enough to do a follow-up assessment, but none of this tells you what happened and what could be changed to achieve greater impact in the future. This is what process evaluation does.

If you tell me that you provided coaching, or diversity training, or emotional intelligence training, or relationship selling, that doesn’t tell me what actually happened. These interventions could be very different from organization to organization, from department to department, or from day to day. Even if an organization implements a highly structured program such as The Fish! Philosophy with its video, guidebook, playbook, and accessories, the experience could be very different for participants in different organizations with different facilitators at different times. To assess its value to the organization and improve the program for future audiences, we have to know what happened and how that was experienced by participants.

It’s like a friend telling you that she went to Las Vegas, Nevada for a vacation and had a good time. If that’s all you know, then you don’t know very much about her trip. You couldn’t replicate her experience or improve on it for yourself. Maybe she stayed on the Strip and spent every waking moment gambling in casinos. Or maybe she stayed downtown and spent each day on Hoover Dam, Red Rock, and Grand Canyon sight-seeing trips without ever setting foot in a casino. Both experiences might receive a “five” on the vacation evaluation form but that data would be useless to you until you knew what “Las Vegas Vacation” means in her case.

So too with learning interventions, whether classroom, online, or informal, we need to know the actual process of learning (anticipated and unanticipated) in those situations in order to improve the experience for other learners and maximize impact on organizations. It is insufficient to know outcomes without knowing what happened to achieve those outcomes. 

I’ve evaluated employee training programs that included follow-up coaching. Some participants used the coaching and some did not. Some coaching consisted of several, hour-long sessions and some consisted of 15 to 20 minute sessions. Some coaching was done by phone, some was done online and asynchronous, and some was done face-to-face. Some coaching was learner-centered and some was coach-centered. Without knowing how the actual coaching process was delivered, we shouldn’t be making decisions about the program.

Lately, the trend in the employee training and development industry has been to emphasize measurement of results and ROI. While I applaud this recognition that end-of-program, smile sheets are an inadequate measure of the quality of programs, I think measuring outcomes without describing the factors that contributed to those outcomes is also inadequate. We need both process and outcome evaluation. 

 

Investing in Training in Downturns and Upturns

The U.S. is emerging from a training drought that occurred during the worst economic period since the Great Depression. At least, that’s what’s indicated by organizations that do this kind of
research. ASTD reports an increase in spending on L&D in 2010 (latest year for which data is available); they estimate companies spent about $171.5 billion. Bersin & Associates reports that corporate spending on training increased 9.5% in 2011 – the largest gain in three years.  And Training magazine’s research found a 13% increase in spending on L&D in 2011. Each of these studies uses a different method for calculating spending. However, the trend is undeniable: spending on employee training is increasing.  

With this increase in spending comes a great opportunity to do things right. In our book, The 5As Framework, Sean Murray and I wrote that most training programs significantly impact only 10% to 20% of their participants. This represents a huge waste that no organization can afford any longer. As the worldwide recession comes to an end, organizations must use every dollar very wisely. Garry Marsh, director of the U.K. company THM Business Simulations, gives us some thoughts about best ways to use those dollars. Here are those seven ideas with my spin on each one:

• Get it right first time – Think in terms of what learners need, not the latest training fad (content or delivery). Only provide experiences that will help learners improve performance and contribute to achieving the strategic goals of the organization – no more and no less. Pay attention to the quality of content and quality of the learning intervention.
• Keep employee satisfaction high – Involving employees in opportunities to learn is an effective way to increase engagement. Many employees value learning more than a pay increase. To keep engagement high, make continuous learning part of the culture of the organization.
• Train your managers – Many managers did not reach that position because they were great managers of people. This is something that all managers need to keep learning. And if organizations want to prevent their top talent from leaving as the economy improves, they must give managers the opportunity to learn and grow.
• Focus on customer experience – All organizations should have the improvement of customer experience as one of their top strategic goals. To achieve this goal, they will have to train employees in how to create a better experience. Learning about the customer experience should be on every manager’s agenda.
• Spend less, get more – Organizations should not rely on the usual suspects and usual methods for delivery of training. It’s about learning; it’s not about glitzy programs. It doesn’t have to cost much to develop talent. Maybe a manager will become a more effective team leader with the help of a mentor. Maybe a manager will become more effective at giving performance feedback with the help of a coach. Maybe a manager will become better at financial management by attending the CEO’s “open-book” meetings.
• Be ready for the upturn – As the economy improves and companies start hiring again, the best talent will look elsewhere. They have stayed only because nobody else was growing and hiring. Unless managers continue to see opportunities in their current organizations and they feel respect from their leadership, they will be gone. Organizations should do something about these conditions now.
• Look after yourself – Many resources are available to help any organization be successful while keeping costs low. Look for help internally and externally. Use what the organization already owns when that makes sense and then go outside to peers, consultants, and training organizations when they can help you improve performance and achieve your business goals.

 

A Year of “The Performance Improvement Blog” in Review

It’s that time of year again. It seems like everyone has their top this or top that from 2011. Not to be left out, I thought I would do a little year-end reflecting of my own.

Throughout 2011, I used this blog to shed some light on critical leadership and management issues. The topics ranged from “the vision thing”, to shaping organizational culture, to improving the effectiveness of employee training programs, to building trust, to breaking down communication silos, to doing better presentations, to seeing organizational blind spots, to increasing employee engagement, and more.

Out of all of these blog posts I’ve selected five that seem to have had the most interest for readers. They are:

Hospital Culture Trumps Equipment and Techniques - Culture, with all of its values, assumptions, beliefs, expected behaviors, and norms, is a powerful force within organizations. Culture shapes how things are done. In hospitals, culture has life and death implications. Evidence suggests that health care outcomes are improved when the culture of a hospital encourages trust and honesty among staff and with patients and when people act consistently in accordance with these values.

Kirkpatrick’s Four Levels of Training Evaluation: A Critique – I explain why evaluating training by using a survey that asks trainees questions about each of the “Kirkpatrick Four Levels” is a flawed way to determine if that program is having a meaningful impact on an organization. Although a four-levels survey is a very common way to approach evaluation of training in organizations, it probably will not provide managers with the information they need. I suggest an alternative approach.

Performance Management is Broken - What is intended to pass for performance management in too many organizations today is the annual compulsory performance review. To label this a “performance management system” is to give the process much more credit than it deserves. A perfunctory meeting between a manager and employee once a year to review a standard rating sheet that lists competencies and goals that are probably no longer relevant, is not a performance management system.

How to Use PowerPoint – Millions of presentations are given every day using some form of slideware, often PowerPoint. It is an important communication tool of managers. Yet, management’s messages are not getting across. I frequently hear that employees didn’t know something that had been mentioned in an all-staff presentation, or that they didn’t realize its importance, or that they didn’t understand what it had to do with them. Apparently, PowerPoint slides are, in many cases, failing to communicate key messages. I refer the reader to suggestions for more powerful PowerPoint presentations.

20 Signs of Employee Disengagement - An important question for all managers is, “Do I have an employee engagement problem in my organization?" You don’t need an organization-wide survey to answer this question. Pay attention to what your employees are saying and doing. Look and listen for the clues. This blog post lists twenty tell-tale signs of disengagement.

Wishing you a healthy and prosperous 2012! 

 

Barriers to Learning in Organizations

Continuous acquisition and application of knowledge, skills, and beliefs by individuals, teams, and the  whole enterprise is an essential aspect of high performance organizations. However, barriers to this learning are common in organizations. These barriers must be overcome in order for organizations to have long term success. Twelve of these common barriers are:

1. Program focus – new programs and services are evaluated in isolation rather than as interdependent parts of the whole organization, e.g., a diversity workshop is evaluated by the participants at the end of the workshop, not by everyone in the organization weeks and months after the workshop
2. Limited resources – learning is not given adequate funding and support, e.g., staff are not given resources to experiment with new ideas before risking large scale implementation
3. Work-learning dichotomy – producing and selling things is valued whereas learning is merely tolerated, e.g., little involvement of supervisors in the training of their direct-reports
4. Passive leadership – leaders don’t ask themselves the hard questions and their boards and co-workers don’t question their actions, e.g., a hospital CEO continues to push for hospital expansion while nobody asks, “Is expansion in the best interests of patients, employees, doctors, and the community?”
5. Non-learning culture – organizational values, assumptions, beliefs, behaviors, and norms do not support learning, e.g., recognizing (praise, reward, promotion, etc.) individual success but not recognizing team success
6. Resistance to change – trying new ways of doing things is not encouraged, e.g., individuals are told to be creative and innovative but not allowed to implement their ideas
7. Not discussing the un-discussable – everyone has a shared but un-spoken understanding that certain issues are not to be confronted and resolved, e.g., one employee’s negative attitudes are bringing down morale of the organization but nobody will talk about this problem for fear of retribution
8. Need for control – managers intentionally and unintentionally use organization charts, policy manuals, rules and regulations, performance evaluation, compensation, budget and expense monitoring, security systems, and work-space arrangements to constrain and limit sharing and applying knowledge, e.g., lines of communication are enforced within an organization to the point that a manager in one department may not talk to a manager in another department
9. Focus on short-term, simple solutions – managers taking the most expedient course of action without solving the long term problems within their organizations, e.g., doing multi-person layoffs to reduce costs when the real problem is not offering products and services that customers want
10. Skilled incompetence – the tendency of managers to avoid embarrassing or threatening interactions, place blame on others, and not accept responsibility for problems, e.g., the leader of a team that failed to complete an assignment avoids asking himself, “What is it about what I am doing or saying that contributes to other people behaving in ineffective and destructive ways?”
11. Blame, not gain language – employees tend to use mostly language that is judgmental and punitive rather than language that facilitates learning, e.g., when something goes wrong, the first words out of a manager’s mouth are, “Who is responsible for this problem?”
12. Selective attention – not seeing (literally) and, therefore, not learning from unexpected events, e.g., two employees attend the same presentation by their CEO but understand the new organizational direction in very different ways

What additional barriers to learning have you observed in organizations?

 

The Course Catalog Is (Should Be) Dead

Recently, I was asked a very thought-provoking question by the director of HRD for a large organization. I was explaining the “5As Framework” to her and she asked, “What are the implications for what we do here?” I immediately launched into my usual speech about getting managers of learners
involved in actively supporting the learning process. This led to a discussion of all the excuses she hears from managers to explain their lack of involvement in employee learning. It wasn’t until later, after further rumination, that I realized the better answer to the director’s question is, “Throw out the course catalog!” 

Like HRD departments in many large organizations, each year they offer a long list of courses and programs that have all of the current buzz words in their titles. I have no doubt that her department delivers excellent, world-class courses and programs. And I’m sure employees enjoy the courses and even say that the knowledge and skills will be very useful to them in their work. However, I’m also quite sure that very little has changed in the wider organization as a result of the many outstanding events put on by this unit. The problem is the lack of alignment between event-based instruction and important business results.

The competency model, which provides the underlying theory of event-based training, is doing a disservice to the current needs of organizations. Training and development professionals seem to believe that if they can identify the competencies that people need in order to do their jobs and then offer courses to teach those competencies, employees will be more productive and successful. The problem with this way of thinking is that performance has as much (if not more) to do with work culture as it does with job competencies. To be successful, employees need supervisors and co-workers who are supportive of learning and performance improvement, they need a supervisor/boss/mentor who talks to them regularly about learning and results, they need the opportunity to apply new learning and to receive feedback, and they and the organization need to be held accountable for learning and results. If there is this culture of learning, employees will continually improve performance.

According to my colleague, Ralph Jacobson, we have to change the local environment so that people actually “work differently”, not just learn how to work differently. All the courses in the world will not change behavior and improve results unless the local culture supports change. At one time, when jobs were static and organizational change came slowly, a menu of well-designed courses was sufficient. Now, when change has to happen rapidly, when we have to find ways to lower the cost of learning, and when we have better and better online, just-in-time, mobile learning resources, event-based learning is no longer the best answer.

We must provide learning experiences to people when and where they need it and in ways that optimizes application to achieving important business results. We need training and development that is timely, short duration, and project-focused. We need frequent on-the-job feedback from supervisor/boss/mentor to employee. For example, leadership skills are best taught during a challenging, team project while the leader is trying out new behaviors and receiving ongoing feedback from a coach. This way of learning will have much more impact on performance than a course or conference on leadership.

What do you think? Can we throw out the course catalog?