The Performance Improvement Blog

Opening the Black-Box to Evaluation

Black-box evaluation is measuring the effects of a program or service without examining the nature of that program or service. It is evaluating what people think and do after the intervention (e.g., training, classroom education, e-learning, coaching, policy change) and then attributing that behavior change or lack of change to what happened in the “black box”. Sometimes a comparison is made post-intervention to what people knew and did before the intervention.  However, nobody opens the black box to see what’s inside. Because of this, we don’t know what aspects of the program had the effect and why they had that effect. So, while we can be accountable for results, black-box evaluation doesn’t help us understand how to replicate those results in the future, we don’t know what should be continued and what should be eliminated, and we don’t know what should be changed about the program to get even better results.

My last two blog posts are about my recent work with Agahozo-Shalom Youth Village, a residential community for orphaned youth in Rwanda. That work provides a good example of the problem with a black-box approach to evaluation. If we only examine the output (e.g., number of graduates, national exam scores) and outcomes (e.g., enhanced self-confidence, enrolling in college or technical school, starting a business), staff would not have the information they need to replicate ASYV, to improve programs, and to possibly figure out how to get the same results at lower cost.  

                   

                              

What ASYV needs is a way to link the contents and process of their programs and activities with outcomes. In addition, they need to know how circumstances of each child before coming to the Village interact with Village activities to affect results. And they need to look for the causes of those pesky unintended consequences that might become a predictable aspect of learning and change.

One of the methods we propose for this evaluation is in-depth interviews of a small sample of students at their time of graduation, then 18 months later, and then six years after that. Much like Brinkerhoff’s Success Case Evaluation Method, the Village staff will use these interviews to tell the stories of the youth. The interviewers will listen for links between what the students experienced at ASYV and how they’ve developed as self-confident, independent adults. They will listen for how the change has occurred from the time the youth came to the Village to how they are today. They will look into the “black box”, which is the Village experience of informal and formal education programs as well as the support of house mothers, counselors, social workers, and other staff, and examine how those experiences contribute to intended and unintended results. 

Continuous Improvement in Agahozo-Shalom Youth Village

In my last blog post I mentioned my evaluation work with Agahozo-Shalom Youth Village in Rwanda. One of the tools we used to help the Village staff think about indicators of success was a model of the residents’ total Village experience. See below.

We asked the staff, “What indicates that you have been successful at each point in this model?” At "baseline", they said that it’s knowing students’ abilities and expectations when they first arrive at the Village. It’s understanding the circumstances they came from and how the Village is different? It’s knowing their hopes and dreams...if at that point they are able to hope and dream.

During the “process” stage, indicators of success include the level of participation of residents in their school and non-school programs. Measures of learning, examples of leadership, examples of volunteering to help others in the Village and outside, and examples of showing initiative were all considered evidence that youth are learning and developing into knowledgeable, self-sufficient, and productive adults.

Some of the indicators of “output” success are the number of students who stay in the Village for four years, the number who graduate from the High School, and the number who pass Rwanda’s national exams. A low incidence of health clinic visits and emotional problems are also indicators of success at this point.

“Outcomes” that staff think are indicators of success include attending the university or a technical institute, starting a business, getting a job, raising a family in a responsible way, and maintaining contact with other graduates from their cohort. Also, taking a leadership position in the community and “giving back” to their communities in other ways are indicators of success.

“Impact” success is apparent in graduates who complete a college education, who are running a successful business, who have an adequate income that is supporting a family, and who are giving back to their communities through teaching, service, and economic development projects.  

What the ASYV staff and I explored together is that, given their theory of change, success is measured at a number of points in the learner’s experience. If they waited, as so often happens in organizations, until after a program (i.e., learning intervention) has ended for learners, it’s too late for continuous improvement. 

When Did You Become A Leader?

I recently returned from a nine-day trip to Rwanda where I teamed with two colleagues to train the leadership team of Agahozo-Shalom Youth Village, a residential community for orphaned youth, in how to use evaluation to continuously improve their educational and personal development programs.

One of the first things we did with the staff was to ask them to share a story about when they discovered that they were leaders. This exercise, which had their enthusiastic participation, set the stage for our week of training and learning. We heard many inspirational stories. We heard stories about:

• Feeling compelled to take action to right an injustice
• Making a difference in other people’s lives and their communities
• Being entrusted with leadership responsibilities by people who are highly respected
• Taking a risk to fill a major leadership role and then succeeding
• Having leadership thrust upon them and rising to the occasion
• Being chosen by peers to be their leader

Interesting to me was that nobody mentioned a college course on leadership, a leadership training program, or leadership coaching. Their newly acquired awareness of their own leadership capabilities came from meeting challenges they faced in everyday life.

When did you first discover that you were a leader? Where were you? What were you doing? What happened that told you that you could lead others and be effective at it?

Executive Coaches Talk About Coaching Effectiveness

I participated in a presentation on coaching at Zingerman’s Roadhouse in Ann Arbor, Michigan on September 21, 2012. My co-presentors were Rob Pasick and Barb Allushuski. We talked about the growth of executive coaching, the range of styles of coaching, the ups and downs of coaching, and how to evaluate coaching. See below:

 

Why Evaluate Executive Coaching

Executive coaching has quickly become a nearly two billion dollar industry. Many companies now rely on coaching for the development of their leaders. Having a coach, once perceived as an admission of failure, is now perceived as a normal part of managing large, complex organizations. And coaching has also become common for entrepreneurs who need help transitioning from leader of a start-up to manager of a rapidly growing company.

With this acceptance of coaching, CEOs and Boards are beginning to ask if the substantial investment they are making in it is paying off.  Unfortunately, we don’t know much about the impact of coaching. Studies of coaching tend to rely on surveys of executives who are being coached. But this method of studying coaching is fraught with problems.

For example, an American Management Association survey of coaching found that respondents from organizations that use coaching more than in the past are also more likely to report two kinds of advantages:

1. They’re more likely to report that their organizations have higher levels of success in the area of coaching.

2. They’re more likely to say that their organizations are performing well in the market, as determined by self-reports in the combined areas of revenue growth, market share, profitability, and customer satisfaction.

The problem with this kind of study of coaching is that it is correlational and correlation does not necessarily mean causation. It is highly likely that high performing organizations choose to use coaching more than low performing organizations. In other words, high performance predicts coaching rather than coaching predicts high performance.

Survey studies of coaching are problematic in other ways, too. One, they lump all coaching into a black box without examining the many variations of coaching. Two, they are based on the assumption that the purpose of coaching is to change the individual when, in many cases, the purpose is to change a team or the organization as a whole. Three, they don’t tell us exactly how the coaching contributed to performance improvement within an environment in which many other factors may have contributed to the change. And four, they rarely look at long term impact, which is what is often the intended purpose of coaching.

To evaluate executive coaching, first we need to define what is meant by "coaching" in the situation being evaluated. Do they mean coaching for greater self-awareness, for developing leadership skills, for making a transition to a new job or role, or for solving a specific problem? Do they mean face-to-face, by phone, or email? Do they mean an hour a day, an hour a week, an hour a month, or as needed. Is the coaching process mostly inquiry or mostly advocacy?  Does the coaching include interviews of supervisors and direct reports? Does it involve observation of work situations and interactions? In other words, what does coaching mean for that executive and that organization?

What we need are studies that tell us what kind of coaching under what circumstances contribute to what kind of results for the organization and over what period of time. Surveys are helpful for knowing the extent of coaching and the general attitude of employees toward coaching. But to understand coaching well enough to be able to replicate effective coaching or improve coaching for an organization, we need to tell the story of the coaching experience. How did the executive get involved in coaching? What were her expectations? How did she work with the coach? What did the coach actually do? How much time did they spend in coaching? What happened? What did the executive learn? What is the executive doing differently? What impact is that having on her team and organization? What evidence is there that this kind of change is occurring and that it is linked to coaching?

When we can answer these questions we will understand what it is about various coaching interventions that have an effect on results.  We will be able to compare coaching to other potential interventions and replicate effective methods. We will know the return on investment. We will know what makes an executive coach qualified to coach. We will know what makes one approach to executive coaching better than another.

Evaluation of coaching is also important to do because it reinforces client learning. When we ask someone questions about what they have learned, it helps to make that learning stick. This is what happens when we ask clients questions about the coaching they received. Their self -awareness is increased, behavior change is strengthened, and long-term follow-through is more likely.

Executive coaching: How do we know it works?

Executive coaching has quickly become a nearly two billion dollar industry. Many companies now rely on coaching for the development of their leaders. Having a coach, once perceived as an admission of failure, is now perceived as a normal part of managing large, complex organizations. And coaching has also become common for entrepreneurs who need help transitioning from a leader of a start-up to a leader of a second and third-stage company.

With this acceptance of coaching, as with any performance improvement intervention, CEOs and Boards are beginning to ask if the substantial investment they are making in it is paying off.  However, we don’t know any more about the impact of coaching then we did in 2008 when I wrote the blog post, “Coaching for Results.” Studies of coaching tend to be surveys of people being coached. Sometimes they include direct reports and supervisors to find out if anyone observes attitude or behavior change in that executive.

The problems with these kinds of studies are fourfold: first, they lump all coaching into a black box without examining the many variations of coaching; second, they assume that the purpose of coaching is to change the individual when, in many cases, the purpose is to change the organization; third, they don’t tell us exactly how the coaching contributed to performance improvement given an environment in which many other factors could have contributed to the change; and fourth, they rarely look at long term impact, which is what is often the intended purpose of coaching.

Researchers Erik de Haan and Christiane Niess write in Training Journal:

Coaching is an organisational intervention. The client is part of the organisation. He brings with him deep traces from the organisation. And if anything changes, he will act differently upon the organisation, where more people will be affected.

So if one is interested in the impact of executive coaching, one would have to measure its effects at the organisational level. This has only been done to a very limited extent. Three hundred and sixty degree assessments by others in the organization about the client have been used in the past to measure effectiveness of executive coaching. However, in those cases, feedback from bosses, peers and direct reports was used essentially to measure the impact of executive coaching on the client, not on others in the organisation. As far as we could determine, organisational feedback on the impact of executive coaching on organisations in a wider sense, has not been investigated or analysed.

Coaching can take many forms so that when someone says they are being coached, that doesn’t tell us very much about their coaching. Do they mean coaching for greater self-awareness, for developing leadership skills, for making a transition to a new job or role, or for solving a specific problem? Do they mean face-to-face, by phone, or email? Do they mean an hour a day, an hour a week, an hour a month, or as needed. Is the coaching process mostly inquiry or mostly advocacy?  Does the coaching include interviews of supervisors and direct reports? Does it involve observation of work situations and interactions? What does coaching mean for that executive and that organization?

What we need are studies that tell us what kind of coaching under what circumstances contribute to what kind of results for the organization and over what period of time. Surveys are helpful for knowing the extent of coaching and the general attitude of employees toward coaching. To understand coaching well enough to be able to replicate successful coaching or improve coaching for an organization, we need to tell the story of the coaching experience. How did the executive get involved in coaching? What were her expectations? How did she work with the coach? What did the coach actually do? How much time did they spend in coaching? What happened? What did the executive learn? What is the executive doing differently? What impact is that having on her team and organization? What evidence is there that this kind of change is occurring and that it is linked to coaching?

When we can answer these questions we will understand what it is about various coaching interventions that has an effect on results.  We will be able to compare coaching to other potential interventions and replicate effective methods.

 

Program Evaluation: A Process for Learning

I was recently asked, “What is your view of program evaluation? How does it differ from research?” At the risk of over-simplifying these two approaches to knowing, I would say that the purpose of evaluation is to improve programs, services, and organizations and to be accountable for results. Research uses many of the same methods as evaluation but has a different purpose. The purpose of research is to advance our knowledge and generalize findings to a wide population, beyond a specific program, service, or organization. For example, if we want to know if a particular medical procedure is effective, we would do research to correlate that procedure with intended outcomes in a range of settings and with a range of people. If we want to know how well that procedure is being implemented in hospital X and what can be done to make the procedure more effective in that hospital, then we would do an evaluation study.  Similar methods of data collection; different purposes.

Even more important is how the data is used.  I believe that the primary purpose of evaluation should be organizational learning. Evaluation of a program, service, or organization should result in stakeholders learning how to achieve success and how to sustain that success over time. Evaluation should help them become smarter about what they are doing for clients, customers, and partners.

Following are the key evaluation principles I’ve concluded from my 30 years of evaluation work[1]:  

Involve staff (internal customers of evaluation) and other key stakeholders in deciding what to measure and how to measure it. Ask them what they want to know and why. Ask them for their thoughts about data collection methods. Ask them to help pilot these methods to determine if the methods will produce the data that is needed.

Choose the method of measurement only after deciding what to measure. The tendency is to use a survey to measure just about everything. But other methods (e.g., interviews, focus groups, observation) can be more useful depending on what you want to know. The appropriateness of each of these methods depends on the kind of data needed, the sources of that data, the circumstances for collecting the data, and how the data will be used. For example, you probably shouldn’t send a questionnaire to users of a literacy program, and you probably shouldn’t convene a focus group of users of a drug treatment program.

Collect data that are credible to stakeholders. Managers might accept the accuracy of staff interviews and focus groups, whereas board members might listen only to output (e.g., number of people served) and financial data (e.g., cost per person served). One group wants to hear numbers; another group wants to hear stories. Know your audience so that you can collect data and report findings that key stakeholders will find credible.

Collect and report data that are useful. The fact that the average rating of the agency is 4.5 on a 5-point scale or that the cost of the adult day care program has gone down 10% in the past year are not particularly useful data for organizational learning. The distribution of ratings across different client groups and finding out why they rated the agency the way they did would be more useful to know. The reasons for the 10% drop in day care costs and the implications for improving the quality and future of the program would be useful to know. Consider what data will help stakeholders make critical decisions.

Report findings in a manner in which stakeholders are receptive to the information. This has to do with the format in which information is reported. You will want all of the various stakeholders to understand your findings and be able to act on the implications. Keep it simple, relate it to the goals that are important to the particular audience, and recommend what should be done about the results. Do not report numbers, only; explain what the numbers mean to the organization.  Tell the story that explains the statistics. If board members are influenced by statistics, then give them charts and graphs. If board members are influenced by client stories, then tell stories. Fit the method to the evaluation questions and learning styles of stakeholders, not the other way around.

Measure the process as well as the outcome. Continuous improvement is achieved by regular assessment of where people are in the process of achieving their goals. This starts by understanding what are key elements that make up the process. Then adjustments to the process can be made, especially as you learn more about staff and client needs and the organization becomes clearer about its goals. Do not wait a year. Give yourself the opportunity for short-term adjustments that will have long-term impact.

Provide just-in-time and just-enough information. Give staff the information they need, when and where they need it. Learning is maximized when people are not overwhelmed with new information, they can relate new information to their work, and they can apply that information to a problem on the job, immediately.

Measure to improve the process, not to blame or punish. Our tendency is to feel threatened by anything that might reveal a lack of personal competency. When we feel threatened, we become less cooperative and less willing to improve performance. Do everything that you can to assure participants that the measures are not being used to make judgments about individuals. Follow through on this promise. Use the data only to help individuals learn, make changes in the organization, and plan for additional activities that will make a difference in everyone’s performance. Do not use the data to chastise or penalize. As Peter Drucker said, “The question isn’t, Do you make mistakes? It’s, Do you learn from them?”

---

[1] Adapted from Developing a Learning Culture in Nonprofit Organizations by Stephen J. Gill, Sage Publications, 2010

 

No-Fear Failure

Fear of failure has a name: atychiphobia. Who knew? While that phobia is a rare, debilitating anxiety with serious mental and physical repercussions, many of us have a reluctance to take risks that might result in what we perceive to be a loss of respect, status, or careers. We also worry about spending time and money on something that doesn’t succeed, but those are only proxies for self-esteem. The problem is that without risk-taking and the failures that inevitably come with that, nothing truly new and innovative is created.

Regina Dugan, former Director of DARPA and now with Google, says in a TED presentation (See below.):

Scientists and engineers defy the impossible and refuse to fear failure…when you remove the fear of failure impossible things suddenly become possible…ask yourself this question, “What would you attempt to do if you knew you could not fail?”…we cannot both fear failure and make amazing new things.

 

Where would we be without airplanes, manned rockets into space, and the internet? Yet these inventions were the result of many prior failures. Dugan argues that risk-taking and failure are simply part of the inventive process.

One of the major barriers to risk-taking in organizations is leadership. Whether real or imagined, we worry that we will be judged harshly by leaders who want to hear only about successes. Many leaders intentionally and unintentionally send the message that failure is not tolerated. An exception to this kind of leader is Kyle Zimmer, head of First Book. In an interview with Adam Bryant of the New York Times, she talks about looking for the experience of failure in the people she hires. She says:

We want people who have tried things, and have failed, and have risen above it. Those indicators that you’re a builder are profoundly important. Because if you’re bright, and you’re a builder, and you’ve overcome the winds that blow against anybody trying to build anything, a lot of other things fall away, like defensiveness.

I’ll say, “Have you ever started anything? From the time you were little, did you invent anything? An organization? Did you start a club?” Then I’ll ask, “What was the hardest part of that? What about failure? Talk to me about failure.” I think that is really important, because if you’re pushing, you’re going to fail. If you’re pushing in whatever you’re doing, you’re going to fail way more than you succeed. It’s that old saying: “You can fail without ever succeeding, but you can’t succeed without ever failing.”

The culture we live in teaches us to fear failure, and I think that’s a huge mistake. When I look back over the history of our organization, the times we’ve been most creative were a result of the pressure of a failure or near failure.

Taking a risk without asking for feedback about the innovation is what should be feared. That truly would be a waste of time and money. Alison Provost, CEO of touchstorm.com, in 30SECONDMBA, talks about the importance of listening to your customers when trying something new. She says:

Ideas don't have to be perfect before you take them to the marketplace. In fact, the true process of innovation is the opposite of that. That essentially, you have your idea and you take it to the marketplace, and you sit with customers and you say, 'This is it.' And they say, 'No, we don't want it quite like that; we want it different than that.' And you adjust it. And you take it back to the office, and back to your folks, who then have to change the product and adjust as you go. And that's the nature of innovating. It's really the fundamental nature of creativity. It's moving in incremental bits to something that is market ready and market worthy. 

It’s not failure that’s the problem. That comes with the territory. The problem is failing to collect information about the innovation or not using data to make improvements and increase the likelihood of success in the future. If we are going to create new businesses and solve the most serious problems that we face as a society, we must continue to try things that have a high likelihood of failure. And we must evaluate what we do and learn from the information. 

 

The Unexamined Leadership Program is Not Worth Doing

If you’re not going to evaluate a leadership development program, don’t do the program! It will be a waste of time, money, energy, and trust. End-of-program reactionnaires (aka smile sheets) don’t count as evaluation. I’m talking about a systematic, evidence-based look at why it was done, what was done, how it was done, what happened as a result, how it can be improved, and what the organization learned from the process.

Viv Nunn of UK’s Open University, in an article for TrainingZone, explains some of the reasons for evaluating professional development programs. She writes that evaluation should provide the following:

• Evidence of the extent to which the professional development is contributing to your organisation's success
• Validation that the correct learning solution has been identified and suggestions for programme improvement
• Advice about how to get the most from your L&D budget by considering the workplace learning environment

I would add:

• Reinforcement of learning. That is, the act of evaluating (i.e., observing) a leadership development program reinforces learning. If done well, evaluation heightens awareness of goals, unintended consequences, and the knowledge, skills, and attitudes that were learned.

To achieve these purposes, you have to look beyond the seminar, workshop, internship, mentoring, coaching, online course, or any other method of instruction. Learning, especially about leadership, is affected by the organizational environment and external factors, as well as the program. To fully understand why and how learning did or did not occur and what impact it had, you have to examine organizational factors in addition to program factors.

Nunn provides us with an excellent description of the factors that can affect learning:

If the workplace learning environment isn't right, if learners cannot find opportunities to apply what they have learned at work, if the organisational culture is not one which encourages them to try out their new skills and knowledge, there will be little or no impact to report, regardless of how effective the actual learning solution has been. Isolating learning from its context will not provide useful data alone. Evaluating the context of learning along with the learning solution will give you the greatest opportunity to get the most from your investment.

I hate to think about how many leadership development programs I’ve participated in as either a learner or facilitator that were thought-provoking, information-rich, and emotionally powerful only to find that they resulted in little or no change because there was no immediate opportunity to apply that learning back on the job. Participants might learn about team-building but not have a team to lead in the workplace or be discouraged by management from applying what they learned in their teams.

Evaluation of professional development must examine how workplace culture supports or hinders learning. Knowing whether intended results follow, in time, a professional development program or not, won't help you unless you understand the factors that affected those results. And, as Nunn suggests, it's things like clarity of goals, manager's attitudes toward learning, incentives for professional development, opportunities to apply new learning, and feedback, that have more to say about performance improvement than does the quality of instruction alone. 

Evaluation is not an option; it’s an integral part of the learning process. If you want a leadership development program to be more than entertainment and you want it to achieve learning that results in significant performance improvement, than you must evaluate the program and the organizational evironment of that program.

 

What's Better: eLearning or Classroom?

We need to examine the proliferation of Web-based education before it gets out of hand (It may be
too late.). Online college courses are being promoted as the panacea for producing more college graduates and strengthening the workforce. The Obama administration has been touting the educational value of online courses since coming into office over three years ago, and many colleges, especially community colleges, have adopted that mantra.

Many companies are also convinced that “elearning” is the answer to their employee training and development needs. They are told elearning will be cheaper, faster, more convenient, and more effective than traditional, classroom training. Also, using elearning programs makes managers feel like they are staying on the cutting edge of technology. Nobody wants to be left behind.

Feeding this appetite for online programs is research that seems to support the notion that online is better than face-to-face, classroom instruction. A study by the U.S. Department of Education found a small advantage for online (blended) programs. However, they qualify that finding by writing:

Despite what appears to be strong support for blended learning applications, the studies in this meta-analysis do not demonstrate that online learning is superior as a medium. In many of the studies showing an advantage for blended learning, the online and classroom conditions differed in terms of time spent, curriculum and pedagogy. It was the combination of elements in the treatment conditions (which was likely to have included additional learning time and materials as well as additional opportunities for collaboration) that produced the observed learning advantages.

The problem is that whether online programs are better than face-to-face depends on the purpose of the course, learning goals of the program, skills of the instructor, and expectations of learners, among other things. I published an article about this nearly 10 years ago in Educational Technology. At that time, I wrote (in ilalics):

1. E-learning will not revolutionize training; it is only one instructional method among many, each better at achieving some instructional objectives than others. (I was wrong; e-learning is revolutionizing training. However, the change has been in the accessibility of information, not in the impact of instruction.)
2. Putting a course on the Web or on a CD-ROM does not ensure performance improvement.(Learning and change depend more on expectations of learners and organizational culture than on the mode of delivery.)
3. Employees learn in many different ways. (For some, online instruction fits their style of learning, but for others it is not always the best mode.)
4. E-learning is not a low cost alternative, especially if it is not aligned with the organization's strategic goals. (If the content is not aligned, it will be a waste of time and energy. No matter how cheap it is per learner, it will be too expensive.)
5. Having a vast selection of courses is meaningless. (It’s wonderful, for personal edification, to have access to thousands of courses from Carnegie Mellon, MIT, and other prestigious colleges. But if these courses aren’t related to what learners need to know on the job and can’t be applied immediately, then more is not better.)
6. Work and learning are the same. (The tendency of managers is to think that by having instruction online, employees can use non-work time for learning. But this separation between work and learning is a false dichotomy. Especially in this day and age, employees have to be continually learning; it must be part of their job.)

In the past 10 years, the technology of online instruction has evolved considerably. Interactivity has advanced. Audio and video have continued to improve. Practical mobile tools for just-in-time applications of elearning are now available (e.g., smart phones and tablets). However, the danger is that organizational leaders and technologists will assume that any learning that is computer enabled is better than direct human contact. For some learning, this could be true. For other learning, it is not. We need to align method of content delivery with intended learning outcomes.

photo credit: Mr_Stein via photopin cc