The Performance Improvement Blog

The Unexamined Leadership Program is Not Worth Doing

If you’re not going to evaluate a leadership development program, don’t do the program! It will be a waste of time, money, energy, and trust. End-of-program reactionnaires (aka smile sheets) don’t count as evaluation. I’m talking about a systematic, evidence-based look at why it was done, what was done, how it was done, what happened as a result, how it can be improved, and what the organization learned from the process.

Viv Nunn of UK’s Open University, in an article for TrainingZone, explains some of the reasons for evaluating professional development programs. She writes that evaluation should provide the following:

• Evidence of the extent to which the professional development is contributing to your organisation's success
• Validation that the correct learning solution has been identified and suggestions for programme improvement
• Advice about how to get the most from your L&D budget by considering the workplace learning environment

I would add:

• Reinforcement of learning. That is, the act of evaluating (i.e., observing) a leadership development program reinforces learning. If done well, evaluation heightens awareness of goals, unintended consequences, and the knowledge, skills, and attitudes that were learned.

To achieve these purposes, you have to look beyond the seminar, workshop, internship, mentoring, coaching, online course, or any other method of instruction. Learning, especially about leadership, is affected by the organizational environment and external factors, as well as the program. To fully understand why and how learning did or did not occur and what impact it had, you have to examine organizational factors in addition to program factors.

Nunn provides us with an excellent description of the factors that can affect learning:

If the workplace learning environment isn't right, if learners cannot find opportunities to apply what they have learned at work, if the organisational culture is not one which encourages them to try out their new skills and knowledge, there will be little or no impact to report, regardless of how effective the actual learning solution has been. Isolating learning from its context will not provide useful data alone. Evaluating the context of learning along with the learning solution will give you the greatest opportunity to get the most from your investment.

I hate to think about how many leadership development programs I’ve participated in as either a learner or facilitator that were thought-provoking, information-rich, and emotionally powerful only to find that they resulted in little or no change because there was no immediate opportunity to apply that learning back on the job. Participants might learn about team-building but not have a team to lead in the workplace or be discouraged by management from applying what they learned in their teams.

Evaluation of professional development must examine how workplace culture supports or hinders learning. Knowing whether intended results follow, in time, a professional development program or not, won't help you unless you understand the factors that affected those results. And, as Nunn suggests, it's things like clarity of goals, manager's attitudes toward learning, incentives for professional development, opportunities to apply new learning, and feedback, that have more to say about performance improvement than does the quality of instruction alone. 

Evaluation is not an option; it’s an integral part of the learning process. If you want a leadership development program to be more than entertainment and you want it to achieve learning that results in significant performance improvement, than you must evaluate the program and the organizational evironment of that program.

 

What's Better: eLearning or Classroom?

We need to examine the proliferation of Web-based education before it gets out of hand (It may be
too late.). Online college courses are being promoted as the panacea for producing more college graduates and strengthening the workforce. The Obama administration has been touting the educational value of online courses since coming into office over three years ago, and many colleges, especially community colleges, have adopted that mantra.

Many companies are also convinced that “elearning” is the answer to their employee training and development needs. They are told elearning will be cheaper, faster, more convenient, and more effective than traditional, classroom training. Also, using elearning programs makes managers feel like they are staying on the cutting edge of technology. Nobody wants to be left behind.

Feeding this appetite for online programs is research that seems to support the notion that online is better than face-to-face, classroom instruction. A study by the U.S. Department of Education found a small advantage for online (blended) programs. However, they qualify that finding by writing:

Despite what appears to be strong support for blended learning applications, the studies in this meta-analysis do not demonstrate that online learning is superior as a medium. In many of the studies showing an advantage for blended learning, the online and classroom conditions differed in terms of time spent, curriculum and pedagogy. It was the combination of elements in the treatment conditions (which was likely to have included additional learning time and materials as well as additional opportunities for collaboration) that produced the observed learning advantages.

The problem is that whether online programs are better than face-to-face depends on the purpose of the course, learning goals of the program, skills of the instructor, and expectations of learners, among other things. I published an article about this nearly 10 years ago in Educational Technology. At that time, I wrote (in ilalics):

1. E-learning will not revolutionize training; it is only one instructional method among many, each better at achieving some instructional objectives than others. (I was wrong; e-learning is revolutionizing training. However, the change has been in the accessibility of information, not in the impact of instruction.)
2. Putting a course on the Web or on a CD-ROM does not ensure performance improvement.(Learning and change depend more on expectations of learners and organizational culture than on the mode of delivery.)
3. Employees learn in many different ways. (For some, online instruction fits their style of learning, but for others it is not always the best mode.)
4. E-learning is not a low cost alternative, especially if it is not aligned with the organization's strategic goals. (If the content is not aligned, it will be a waste of time and energy. No matter how cheap it is per learner, it will be too expensive.)
5. Having a vast selection of courses is meaningless. (It’s wonderful, for personal edification, to have access to thousands of courses from Carnegie Mellon, MIT, and other prestigious colleges. But if these courses aren’t related to what learners need to know on the job and can’t be applied immediately, then more is not better.)
6. Work and learning are the same. (The tendency of managers is to think that by having instruction online, employees can use non-work time for learning. But this separation between work and learning is a false dichotomy. Especially in this day and age, employees have to be continually learning; it must be part of their job.)

In the past 10 years, the technology of online instruction has evolved considerably. Interactivity has advanced. Audio and video have continued to improve. Practical mobile tools for just-in-time applications of elearning are now available (e.g., smart phones and tablets). However, the danger is that organizational leaders and technologists will assume that any learning that is computer enabled is better than direct human contact. For some learning, this could be true. For other learning, it is not. We need to align method of content delivery with intended learning outcomes.

photo credit: Mr_Stein via photopin cc

The 0% Solution to Learning

Ideally, organizations should be striving for a culture in which every aspect of the workplace supports continuous individual, team, and whole organization learning. It should be a place where Individuals are learning how to help the organization achieve its strategic goals, where teams are learning how to help the organization achieve its strategic goals, and where the organization as a whole is learning how to achieve its strategic goals.

In this kind of culture, formal classroom training is superfluous. The most important learning occurs in the conversations that employees have with their supervisors and co-workers, in the conversations that teams have about team development and effectiveness, and in whole organization experiences with problem solving and planning. Learning occurs from the feedback and reflection around individual, team, and whole organization successes and failures. Learning occurs because of a performance management system that permeates every aspect of organizational life, not because of isolated instructor-based courses, workshops, and seminars.

Robert O. Brinkerhoff makes this point in his writing about measuring the contribution that training makes to learning and performance improvement. He argues that the isolation score, that indicator of how much training contributes to performance results, should be zero. He writes:

An isolation score of zero, on the other hand, seems to me to be the perfect solution: it means the organization has figured out a way to drive a desired performance improvement and resultant business result solely by manipulating the performance system, so there is no need for any training at all. Isn't this scenario exactly what the performance-oriented practitioners would seek? Training is always an expensive solution. If you can get to the desired business outcome without spending a dime on training - by just adjusting incentives, motivation, performance support, etc. - isn’t this what a thoughtful HRD leader should strive for?

The implication of striving for an “isolation score of zero” is that the whole organization owns learning, not just HR and the CLO, and when the organization needs performance improvement, training is not the first course of action. We have to change the way we think about learning and performance improvement, especially as people need to learn faster and achieve better results.

Unfortunately, it’s a very difficult mental model to change. In a recent post on ASTD’s Learning Circuits Blog, Connie Malamed asks, “What do you think the future role of the training professional will be?” She offers this model of future roles:

Malamed's model is an accurate representation of the current state of things but not where I think the field needs to go in the future. My model would not include “design and create courses.” My model would put “align learning with the strategic goals of the organization” at the center and “create a culture of learning throughout the organization” as the next key role. We have so many learning support options now, including just-in-time elearning (desktop and mobile), coaching, mentoring, simulations, on-demand video, experiential learning, internships, and on and on, that we rarely need to develop courses. Learning and development professionals need to be focused on performance results, not on training. 

 

Bad News is Good News

When evaluating programs and organizations, bad news can be good news. What I mean by that is any feedback is helpful to an organization if leaders listen to the feedback, look for value in the information, and make improvements accordingly. However, what so often happens is that leaders immediately become defensive and look for ways to discredit or cast doubt on criticism. Almost impulsively, they look for ways to find fault with negative feedback and only accept high ratings and praise. The tendency is to automatically try to identify the people who made critical comments and explain away what was said because of who said it: “They’re always complaining”; “That person is poisoning the well for everyone else”; “We couldn’t give them raises this year so they’re angry”; and on and on.

Some organizations have a culture that is so insular they can’t hear bad news. The National Collegiate Athletic Association (NCAA), “founded more than one hundred years ago as a way to protect student-athletes”, appears to be one of these organizations. In a series of columns written for the New York Times, Joe Nocera has described some examples of the NCAA failing to act in the best interests of students. He writes:

The N.C.A.A. claims — as it always does — that it is acting to protect its athletes “from exploitation by professional and commercial enterprises.” But this is classic N.C.A.A. Orwellian spin. Its true purpose in preventing athletes from engaging with agents while in college is to exacerbate their exploitation. The professional and commercial enterprise doing the exploiting, of course, is college sports itself.

I can’t vouch for the accuracy of the facts that Nocera cites, but it does seem that the NCAA is not making use of this opportunity to look at itself in the mirror and make improvements. Rather than accepting the feedback and explaining what they will do to make the NCAA a better organization for students and its member institutions, organizational leaders have become defensive and have attacked Nocera in the press. Dave Pickle, a long-time employee of the NCAA, wrote a blog post about Nocera titled, “News not fit to print.”

This isn’t a constructive response, especially by an organization that has responsibility not only to its member institutions but also to the wider public.  One of the primary ways in which organizations learn is by periodically examining what they do and what stakeholder’s (customers/clients, employees, suppliers, wider community, etc.) think of what they do. If leaders do not listen to the “bad” as well as the “good”, they cannot learn. Barriers to listening, such as the argumentum ad hominem defense, do not allow for continuous improvement. Like the NCAA, a culture can develop in organizations in which these barriers become ingrained in the way they do business. “Bad news” should provide an opportunity for learning and change. Great organizations turn bad news into news they can use.

 

Evaluate Learning Process

As organizations evaluate their programs and services they need to pay attention to process as well as outcomes. Catherine (Brehm) Rain, of Rain and Brehm Consulting Group, Inc., writes this in aea365:

Process evaluation ensures you answer questions of fidelity… did you do what you set out to with respect to needs, population, setting, intervention and delivery? When these questions are answered, a feedback loop is established so that necessary modifications to the program or the evaluation can be made along the way.

Life is a journey—and so is a long-term evaluation. Stuff happens. However, it is often in the chaotic that we find the nugget of truth, the unknown need, or a new direction to better serve constituents. A well-documented process evaluation assists programs to ‘turn on a dime’, adapt to changing environments and issues, and maximize outcome potential.

The problem with much of the evaluation that goes on in organizations, whether that’s evaluation of training and development programs, communication programs, marketing campaigns, a new sales approach, or strategic planning, is that the process is not examined. You might discover what participants thought of the program from “smile sheets” and you might even know how participants applied what they learned if you are fortunate enough to do a follow-up assessment, but none of this tells you what happened and what could be changed to achieve greater impact in the future. This is what process evaluation does.

If you tell me that you provided coaching, or diversity training, or emotional intelligence training, or relationship selling, that doesn’t tell me what actually happened. These interventions could be very different from organization to organization, from department to department, or from day to day. Even if an organization implements a highly structured program such as The Fish! Philosophy with its video, guidebook, playbook, and accessories, the experience could be very different for participants in different organizations with different facilitators at different times. To assess its value to the organization and improve the program for future audiences, we have to know what happened and how that was experienced by participants.

It’s like a friend telling you that she went to Las Vegas, Nevada for a vacation and had a good time. If that’s all you know, then you don’t know very much about her trip. You couldn’t replicate her experience or improve on it for yourself. Maybe she stayed on the Strip and spent every waking moment gambling in casinos. Or maybe she stayed downtown and spent each day on Hoover Dam, Red Rock, and Grand Canyon sight-seeing trips without ever setting foot in a casino. Both experiences might receive a “five” on the vacation evaluation form but that data would be useless to you until you knew what “Las Vegas Vacation” means in her case.

So too with learning interventions, whether classroom, online, or informal, we need to know the actual process of learning (anticipated and unanticipated) in those situations in order to improve the experience for other learners and maximize impact on organizations. It is insufficient to know outcomes without knowing what happened to achieve those outcomes. 

I’ve evaluated employee training programs that included follow-up coaching. Some participants used the coaching and some did not. Some coaching consisted of several, hour-long sessions and some consisted of 15 to 20 minute sessions. Some coaching was done by phone, some was done online and asynchronous, and some was done face-to-face. Some coaching was learner-centered and some was coach-centered. Without knowing how the actual coaching process was delivered, we shouldn’t be making decisions about the program.

Lately, the trend in the employee training and development industry has been to emphasize measurement of results and ROI. While I applaud this recognition that end-of-program, smile sheets are an inadequate measure of the quality of programs, I think measuring outcomes without describing the factors that contributed to those outcomes is also inadequate. We need both process and outcome evaluation. 

 

There They Go Again: Bashing Brainstorming

What did brainstorming ever do to receive such vitriol? Jonah Lehrer writes in the January 30 issue of The New Yorker:

Typically, participants leave a brainstorming session proud of their contribution. The whiteboard has been filled with free associations. At such moments, brainstorming can seem like an ideal mental technique, a feel-good way to boost productivity. But there is one overwhelming problem with brainstorming. It doesn’t work.

Susan Cain writes in a January 13 opinion piece for the New York Times titled “The Rise of the New Group Think”:

…brainstorming sessions are one of the worst possible ways to stimulate creativity. The brainchild of a charismatic advertising executive named Alex Osborn who believed that groups produced better ideas than individuals, workplace brainstorming sessions came into vogue in the 1950s. …But decades of research show that individuals almost always perform better than groups in both quality and quantity, and group performance gets worse as group size increases… People in groups tend to sit back and let others do the work; they instinctively mimic others’ opinions and lose sight of their own; and, often succumb to peer pressure.

And now Cliff Kuang raises concerns with brainstorming again in an article for Fast Company titled, The Brainstorming Process Is B.S. But Can We Rework It? At least Kuang is asking how to make brainstorming more effective. Let’s brainstorm some possibilities (just kidding).

As I wrote in a previous blog post, “I believe the problem is in the implementation, not the method.” Indicting brainstorming is like saying that elearning is more effective than classroom-based learning. The truth is there is effective elearning and there is ineffective elearning. It all depends on the purpose, design, content, and delivery of the course.  

It’s the same with brainstorming. It all depends on the quality of design and facilitation. The critics are right; if all the facilitator does is allow participants to free associate for an hour and then go home, the process is “B.S.”. When that happens we can predict little creativity and little impact on planning.

However, when it is done well, when it is done in context and with preparation of participants, and when there is follow-up after the initial generation of ideas, then brainstorming can be very effective. A good facilitator:

1. Is clear about the purpose and intended results of the exercise and discusses this with participants
2. Prepares (even trains) participants for doing brainstorming (e.g., gives an example)
3. Establishes norms for how participants are to behave in the session
4. Leads the conversation in such a way that gets everyone involved, gets all of the ideas displayed in front of the group, and does not allow criticism until all ideas are listed
5. Helps the group reach consensus on a subset of ideas on which to do further work
6. Begins action planning to do the follow-up work
7. Helps participants learn (and learn how to learn) from their involvement

There is an art and science to brainstorming. The facilitator must be able to follow the rules of brainstorming but also deviate from and expand on those rules depending on the situation.  

When done well, there are many uses of brainstorming. It could be used to evaluate a program, consider new strategies and methods, identify problems, create a vision for the future of an organization, or solve problems.  Often, brainstorming is used simply to warm up the group for a lengthy strategic-planning or prolbem-solving discussion. The technique creates an atmosphere of trust where everyone is heard and all ideas are respected. Now, what’s wrong with that? 

 

A Year of “The Performance Improvement Blog” in Review

It’s that time of year again. It seems like everyone has their top this or top that from 2011. Not to be left out, I thought I would do a little year-end reflecting of my own.

Throughout 2011, I used this blog to shed some light on critical leadership and management issues. The topics ranged from “the vision thing”, to shaping organizational culture, to improving the effectiveness of employee training programs, to building trust, to breaking down communication silos, to doing better presentations, to seeing organizational blind spots, to increasing employee engagement, and more.

Out of all of these blog posts I’ve selected five that seem to have had the most interest for readers. They are:

Hospital Culture Trumps Equipment and Techniques - Culture, with all of its values, assumptions, beliefs, expected behaviors, and norms, is a powerful force within organizations. Culture shapes how things are done. In hospitals, culture has life and death implications. Evidence suggests that health care outcomes are improved when the culture of a hospital encourages trust and honesty among staff and with patients and when people act consistently in accordance with these values.

Kirkpatrick’s Four Levels of Training Evaluation: A Critique – I explain why evaluating training by using a survey that asks trainees questions about each of the “Kirkpatrick Four Levels” is a flawed way to determine if that program is having a meaningful impact on an organization. Although a four-levels survey is a very common way to approach evaluation of training in organizations, it probably will not provide managers with the information they need. I suggest an alternative approach.

Performance Management is Broken - What is intended to pass for performance management in too many organizations today is the annual compulsory performance review. To label this a “performance management system” is to give the process much more credit than it deserves. A perfunctory meeting between a manager and employee once a year to review a standard rating sheet that lists competencies and goals that are probably no longer relevant, is not a performance management system.

How to Use PowerPoint – Millions of presentations are given every day using some form of slideware, often PowerPoint. It is an important communication tool of managers. Yet, management’s messages are not getting across. I frequently hear that employees didn’t know something that had been mentioned in an all-staff presentation, or that they didn’t realize its importance, or that they didn’t understand what it had to do with them. Apparently, PowerPoint slides are, in many cases, failing to communicate key messages. I refer the reader to suggestions for more powerful PowerPoint presentations.

20 Signs of Employee Disengagement - An important question for all managers is, “Do I have an employee engagement problem in my organization?" You don’t need an organization-wide survey to answer this question. Pay attention to what your employees are saying and doing. Look and listen for the clues. This blog post lists twenty tell-tale signs of disengagement.

Wishing you a healthy and prosperous 2012! 

 

Evidence-Informed Practice

Some nonprofit organizations, in an effort to improve outcomes for their clients and to be more accountable for results, are applying an “evidence-informed practice” approach to their work. They are examining practices that have been proven to be effective and applying this experience to their own delivery of services. They are taking adult learning theory and systems thinking, combining that with evidence from program evaluation studies, and using the information to make their own organizations more effective.

For example, PART (Practice and Research Together) is promoting evidence-informed practice in child welfare agencies across Ontario, Canada. In November, I had the honor and pleasure of speaking at a
retreat in Toronto for members of PART. Formed in 2006, PART is modeled after Research in Practice, a world-renowned partnership of children’s services in the U.K. that is “creating new knowledge, new skills and a new energy to improve outcomes for children, young people and their families.”

At their retreat, I talked with PART members about developing a learning culture in their agencies that would help them in the process of continuous improvement. We discussed making reflection, feedback, and knowledge-sharing the way they function on a day-to-day basis. We identified organizational barriers to learning and how those barriers can be overcome. We explored ideas around building their organizational capacity to apply action-learning through small experiments using Deming’s model of plan-do-check-act. They shared ways in which they are continuously learning from their own experiences and from the knowledge and best practices of others.

Many of these ideas were taken from my book, Developing a Learning Culture in Nonprofit Organizations, and then incorporated into the PART guidebook, Broadening Horizons: Linking Evidence to Child Welfare Supervision, which can be ordered from PART.  I’m very pleased to see a book of mine having practical value and being interpreted for child welfare organizations.

My warning to these child welfare agencies, as well as any organization that’s taking an evidence-based approach, is: guard against ways of thinking that prevent the acceptance of new information. I’ve written previously about “confirmation bias” and the “illusion of cause.” Both of these common ways of thinking are barriers to evidence-informed practice. Human beings have a tendency to attend to data that confirms what they already believe to be true and disregard the rest (e.g., a respected leader would never abuse a child), and to attribute a causal relationship between factors simply because they occur close in time (e.g., a child has poor grades because his father is out of work). We all believe that we are reasonable people and that we base our judgment on facts, but sometimes we won’t accept the truth. 

 

Data is Not Knowledge

I like this quote by the Swedish author Henning Mankell that appeared in a Sunday New York Times op-ed column titled “The Art of Listening”:  “

Many people make the mistake of confusing information with knowledge. They are not the same thing. Knowledge involves the interpretation of information. Knowledge involves listening.

We are awash with data and information. Numbers, facts, and observations are constantly at our fingertips in our Blackberrys, iPhones, and Droids. Online newspapers, magazines, and social media smother us in sense and nonsense. Marketing and survey research companies publish mounds of opinion data daily. The key question we must ask ourselves is, “Are we learning anything? Are we turning that data and information into knowledge that can be applied for the betterment of our organizations?”

TED and the pecha-kucha movement provide excellent examples of how data can become knowledge and move people to action. Those presenters tell a story about their data that engages people emotionally. They realize that just because someone has data doesn’t mean they have knowledge.

This graphic, from Epic.graphic, makes the point: 

 

Leadership and Health Care Reform

The Obama administration’s health care reform legislation will change incentives, which is badly needed, but it will not improve the quality of care unless an integrated, team-based approach also becomes the way health care professionals do business. This kind of approach is what the legislation refers to as ACO (accountable care organization). However, to be successful, such an approach will require health care professionals, especially doctors, to learn new competencies and form new attitudes regarding collaboration and cooperation.

The assumptions behind the new legislation make the need for change abundantly clear:

• the cost of health care in the United States on a per capita basis is too high
• the health of the United States population is too low
• health care in the United States is not sufficiently accessible, particularly for the uninsured
• health care in the United States is not sufficiently coordinated
• the quality of health care services in the United States is too low
• the federal government, as the largest purchaser of health care services in the United States, has the right to demand high quality, reasonably priced, accessible, coordinated health care services
• the federal government should no longer purchase health care services on a fee-for-service (or a la carte) basis because it gives health care providers an incentive to order more tests and perform more procedures in order to make more money
• there needs to be a “sea change” in how health care is delivered and paid for in the United States

Shifting from a fee-for-service model to a high-quality, low-cost model will address part of the problem. Don’t get me wrong - this change is essential, though the challenge with incentives is to ensure that “low-cost” doesn’t become more important than “high quality”. The other part of the problem is managing health care organizations in a fundamentally new way. Rather than the command-and-control style of leadership that is prevalent among doctors, nurses, and other health care professionals, these leaders will need to facilitate teamwork. They will need to respect and listen to the input of other health care providing organizations and integrate diet and life-style experts into the team. They will need to manage these teams so that optimum results are achieved for patients. They will need to keep all of the stakeholders engaged. They will need to draw out the best from everyone. And they will need to do all of this in a cost efficient manner. This is a tall order for people who come to their profession because of their high need for control, for prestige, and, for many, the financial benefits.

Health care professionals can be retrained if motivated. The real transformation of health care in the U.S. will come over time as a different kind of person is admitted to the health care professions and as measures are put in place to evaluate this team-based approach to delivering health care.