The Performance Improvement Blog

Evaluation Is Not Complicated

In his blog post, “Stop Evaluating Training!”, Amit Garg summarizes a presentation that Robert O. Brinkerhoff gave at the Australian Institute of Training and Development conference in April. Garg relates Brinkerhoff’s comments to the challenges of measuring the effectiveness of elearning programs. Garg writes:

So how do you evaluate the success of eLearning that you create?" As a learning solutions vendor, I’ve been asked this question countless times and have also encountered it in many an RFP. Proving the effectiveness of training and showing ROI is no walk in the park and still keeps L&D up at night.

Maybe evaluation of the impact of training is not a “walk in the park”, but it is not “rocket science” either. As Brinkerhoff explains in The Success Case Method, there is a simple logic to assessing impact. First we have to understand the alignment between a learning intervention, whether that’s classroom-based, elearning, coaching, on-the-job structured experiences, or something else, and the intended impact (e.g., customer retention, production, sales, revenue, market share). Then it’s a matter of identifying and interviewing learners who are contributing to that impact and those that are not contributing. The purpose of these interviews is to understand the nature of the impact and why it’s happening or not happening.

Three immutable laws drive this method of evaluation. One is the law of learning as a process not an event. This law holds that learning is a process that starts before an instructional event and continues after the event. To study the event only is to fail to include key aspects of what is facilitating learning and what are barriers to learning.

Another is the law of unintended consequences which holds that when there is change there are always outcomes that weren’t anticipated. A useful evaluation of training will examine these unintended consequences (positive and negative) as well as achievement of objectives of the intervention.   

And the third law holds that significant, lasting change in complex organizations is always the result of many factors, some that are not in the control of trainers and learners. For example, effectiveness of leaders, change in strategic direction, and the economy could have a profound influence on the impact that learners have on their organizations.

It’s not complicated. Understand how the learning intervention is intended to help the organization achieve its goals. Identify employees who are and are not applying what they learned. Interview these employees to find out what they did or did not do, how that affected the organization, what other factors were intervening in that impact, and what can be done to increase learner impact in the future.

The Next Big Thing: Big Data

Everybody’s talking about “Big Data” these days. But as with any fast moving bandwagon, it’s good to get off once in a while and take stock of the direction in which you are going. So let’s take a look at the pluses and minuses of Big Data. We now have the technology to collect enormous amounts of information about people and to analyze huge datasets that have implications for the economy, education, health care, and social services, as well as enterprises such as agriculture, manufacturing, retail, and much more. Having data is not the problem; the problem is using it effectively and responsibly.

Wally Bock, referring to Johannes Kepler and his discovery of a third law of planetary motion, writes:

Most Big Data operations are the same kind of brute force computation that Kepler did, only much faster. Big Data is the new panacea. It will do just about everything, we're told. But before we start planning for the era of universal peace and an end to disease and hunger which will surely come, it's worth pondering that the computation is only the middle part of the process.

Kepler was using the Big Data of his day, but he had to know what he was looking for. He had to use good, accurate data. And he had to have some idea of how he could use it when he found it. That hasn't changed simply because we can calculate faster.

Making good use of large datasets takes preparation and follow-through. Take health care, for example.  Even when findings from Big Data have obvious disease prevention implications, people do not do what is suggested by the findings. Michael Hickins writes in CIO Journal that Aetna is using Big Data to predict which of its members is likely to develop cardiovascular disease. But even with this information, there is relatively low compliance with the regimen that will likely prevent heart disease. It’s not sufficient to have data. What's important is what we do with that data to change behavior of individuals, teams, organizations, and communities.

Darin McKeever, Deputy Director at the Bill & Melinda Gates Foundation, in a post titled "Moving From Big Data to Big Wisdom", makes the point that Big Data does not necessarily mean that people have the wisdom to improve organizations and society. That data must be turned into something meaningful. Data must become useful information. Useful information must become new knowledge. And it is that new knowledge that becomes the wisdom to bring needed change in new situations.

Getting to wisdom starts before data mining begins. It starts with key stakeholders coming together to decide what questions they want to answer, how the data will be collected and analyzed, what information can be gleaned from the data, and how individual privacy can be protected. It continues with learning from the analysis (knowledge) followed by generating the insights and wisdom needed to sustain change. Big Data, by itself, is not the solution. It’s what people do with that data and how it is turned into wisdom that makes the difference.

Invisible Gorilla Strikes Again

This time it’s radiologists. In a study reported by NPR, 83% of radiologists failed to see the image of a gorilla on slides that they were reviewing for signs of cancer.  The conclusion by reporter Alix Spiegel is:

… what we're thinking about — what we're focused on — filters the world around us so aggressively that it literally shapes what we see. So, [the researcher] says, we need to think carefully about the instructions we give to professional searchers like radiologists or people looking for terrorist activity, because what we tell them to look for will in part determine what they see and don't see.

This phenomenon of selective attention permeates our society. Drivers don’t see approaching bicycles and motorcycles. Police can’t always determine who’s the victim and who’s the perpetrator in a violent situation. Soldiers responding in a conflict don’t always see the innocent bystanders in their line of sight. Teachers don’t always notice when a student for whom there are low expectations suddenly does something extraordinary.

When I was a teenager, I along with others observed a socio-drama as part of a lesson in tolerance. In this theatrical event, a crime was committed. We were asked to say who committed the crime. To us it was obvious. When the action was repeated it wasn’t so obvious and we learned about how our prejudices and perceptions influence our judgment, especially in quickly evolving situations.

And I wonder about leaders of large, complex organizations.  Do they sometimes display "selective attention"? Do they look at but not see the signs of a failing business model? Do they look at but not see the signs of a recessionary economy? Do they look at but not see managers who are not doing their jobs effectively? Do they look at but not see customers who are dissatisfied and looking elsewhere. Maybe it’s denial, but it also might be an inability to see something that is unexpected. 

What Gets Rewarded Gets Done

To be successful, organizations must align their reward structure with their intended goals. Eduardo Porter, in a New York Times article titled, When Public Outperforms Private in Services, writes:

The success or failure of an organization often depends on whether it can clearly identify its goals and align the interests of managers and employees to serve them. Yet whatever reward structure an organization picks can skew incentives in an undesirable way.

A lack of goal clarity and alignment can have detrimental effects. When oil companies reward profit over both safety and protecting the environment, loss of life and environmental disasters are the result. When pharmaceutical companies reward sales people only for the quantity of drugs sold, patients will be prescribed drugs they don’t need. When states judge schools on the basis of test scores, not learning, students graduate unprepared for college and work. When banks reward mortgage agents for quantity rather than quality, a high percentage of loans fail and the entire economy suffers.

I don’t think it’s a question of private vs. public. Some public institutions and government agencies reward the wrong behavior as do some businesses. I remember consulting with a nonprofit health insurance company that rewarded customer service reps for the number of phone inquiries they handled in an hour. The tradeoff was quantity of transactions for customer satisfaction. Treating a customer with respect and solving the customer’s problem were given lip service but not rewarded in practice.

The problem, as Porter says, is tradeoffs. A company that truly values customers, might have to sacrifice some portion of profit in the short term for a satisfied customer who will continue to come back to that business over the long term. A government or nonprofit that truly values its mission, might have to sacrifice some portion of revenue in order to achieve its service goals. Neither type of organization can afford to run at a deficit but neither is sustainable without alignment of rewards with goals.

A Year of “The Performance Improvement Blog” in Review

It’s time to do a little year-end reflecting on my blog posts from 2012.

Throughout last year I used this blog to illuminate important leadership and management issues. The topics ranged from creating a learning culture to increasing employee engagement to improving organizational communication to evaluating executive coaching, and more.

Out of all of these blog posts I’ve selected five that seem to have had the most interest for readers. Here are the links with a short excerpt from each post:

How to Create a Learning Culture in Organizations - Organizational learning is not about training. Rather, it's about a community of workers sharing in a process of constantly seeking improvement through new knowledge, new skills, and new applications of knowledge and skills to achieving the goals of the organization. They examine what they do, compare that to what needs to be done, reflect on what they have learned, and make the needed change in the organization.

 

Leaders Make or Break Employee Engagement - A leader’s attitude and behavior determine whether an organization has an engaged workforce or not. In other words, what leaders think and feel and say and do has a profound impact on employee satisfaction, enthusiasm for their work, desire to do their best, and commitment to go beyond their job descriptions every day. In an article for The Business Thinker, my colleague Jim Stilwell and I describe 13 ways in which leaders affect employee engagement. 

 

 

What Gets Measured Gets Done...Revisited - Measurement is not enough. An organization has to do something with those metrics in order to “get done” what needs to get done, i.e., organizational learning, performance improvement, and change. The act of measuring things like sales, customer service, product quality, teamwork, employee engagement, or learning, does not necessarily mean that anything will be done differently.

 

 

Why Evaluate Executive Coaching - CEOs and Boards are beginning to ask if the substantial investment they are making in executive coaching is paying off.  Unfortunately, we don’t know much about the impact of coaching. Studies tend to rely on surveys of executives who are being coached. But this method of studying coaching is fraught with problems.

 

 

What Do You Mean? - Dan Pallotta, president of Advertising for Humanity and HBR blogger, in a post titled “I Don't Understand What Anyone Is Saying Anymore”, writes about business jargon and how it affects communication in organizations. The blog post must have hit a nerve because it has broken the record for most comments to a single HBR Blog Network post. 

Wishing you a healthy and prosperous 2013!

Managing to Outcomes in Nonprofit Sector

This is the traditional time to reflect on trends observed during the past year. One major trend I have observed is performance management of nonprofit organizations. Actually, this is an accelerating trend that got its impetus from Peter Drucker decades ago. Nonprofits are becoming more and more focused on what McKinsey & Company calls “managing to outcomes”.  

I know, you thought all organizations managed to outcomes. Not so, particularly in the social sector. Nonprofits have been more like manage-to-what-makes-you-feel-good or manage-to-spend-the-money. It has been less about results and more about the process of doing the work and providing services. Now a shift is occurring towards greater recognition of the importance of measuring outcomes and using that information for learning. That is, learning how to be a more effective organization and how to increase impact of programs and services.

If your organization is involved in philanthropic activity, if you serve on the board of a nonprofit organization (or NGO), or if you work for a nonprofit or a philanthropic foundation, or if you donate to nonprofits, then you should pay special attention to this trend. It will affect what you do and how your money is spent.

This intensified focus on outcomes and learning is a result of a confluence of factors in the growing and increasingly important U.S. nonprofit sector (According to the Foundation Center, “In 2010, nonprofits contributed $804.8 billion to the gross domestic product (GDP); this equates to 5.5 percent of GDP… the nonprofit sector employed 13.7 million people.”). The factors that are contributing to the change include:

• Increasing demand for services for the poor, the disadvantaged, the elderly, and the marginalized
• Less public money for social programs (Mario Morino calls this the “Age of Scarcity”.)
• More people, and a wider spectrum of people, interested in “doing good”
• Greater demand by social investors for evidence of impact
• Greater demand from policy-makers for evidence of impact
• More leaders on nonprofit and foundation boards who want to see results
• Increasing professionalization of the sector (e.g., MBAs leading large nonprofits)

So the need for services is growing while resources are dwindling and, at the same time, funders and boards of directors want more accountability and more impact from their investment. Because of all of these factors, there is pressure on the sector to develop more efficient and effective organizations. These are organizations that embrace a culture of measurement and performance improvement. Mary Winkler of the Urban Institute emphasized this point when she said:

…organizational culture and a predisposition to measurement and managing toward results is perhaps the single most important ingredient to success. A culture of continuous improvement needs to be evidenced at the top. Equally important, however, is the extent to which the culture of continuous improvement is integrated at every level of the organization. 

That means having leaders who are willing to learn from what they do and apply that learning to improving organizations, as well as their programs and services. This is not an easy transition for many of these leaders who have been accountable for only process and output in the past. They will need the support of funders, staff, constituents, consultants, and the public at large, to make this change.

Doing Good Effectively

It’s that giving time of year again. And whether you donate to charities or you are a philanthropist that gives grants to nonprofits, you should care about organizational effectiveness and efficiency.

One commonly used indicator of effective management in organizations that do-good is the percentage of their budgets that goes to administrative costs. The rule of thumb is between 11% and 13%. This is a simple and easily calculated and understandable measure. Unfortunately, it’s not the best indicator. Overhead matters, but effectiveness matters more.

Tina Rosenberg, in a New York Times opinion piece, identifies additional criteria that should be used in judging charitable organizations. She writes that the best organizations…

• Aim to solve the most serious problems (in the normal calculus, this means that providing bed nets to save children from malaria ranks above helping public radio stations or art museums).
• Use interventions that work.
• Employ cost-effective strategies (trachoma surgeries, rather than training guide dogs, to help the blind).
• Are competent and honest. The percentage of donations spent on overhead is one measure of these qualities.
• Can make good use of each additional dollar. This is the hardest point to assess, but it asks whether the group has the program on the ground to use your money well, and whether your donation will make something happen that otherwise wouldn’t.

I would also ask these questions:

• Is the program and the organization sustainable? Assuming the problem being addressed requires a long-term solution, I would want to know if the nonprofit has the leadership, infrastructure, and funding from other sources to stay with the solution as long as is necessary.
• Does the nonprofit have a culture that is consistent with my values? The leaders might have the best of intentions, but if they treat their staff and volunteers poorly, if they don’t pay their vendors, if they relate to low-income and minorities in a disparaging and condescending way, then I want to know this.
• Are they using the best interventions available? Maybe they are using interventions that work but maybe there are other interventions that are more effective and less costly. And maybe there are other interventions that should be tried to test their efficacy.

Toby Ord, in the blog Giving What We Can, argues that we should mainly be looking at cost-effectiveness. He believes that we have a moral imperative, especially when it comes to global health, to support programs that get the most bang for the buck. Ord writes:

…merely moving funding from one intervention to a more cost-effective one can produce almost as much benefit as adding an equal amount of additional funding. This is unintuitive since it isn’t the case when one option is merely 10% or 30% better than another. However, when one option is 10 times or 100 times better, as is often the case in global health, redirecting funding is so important that it is almost as good as adding new funding directly towards the superior intervention. In times of global austerity and shrinking budgets, it is good to know how much more can be done within existing ones.

People give to charities and other nonprofits for all kinds of reasons. Some because of a religious calling; some for the prestige; some because of a personal connection to the agency or the cause; some for a tax deduction; some because of peer pressure (for example, they serve on the board of the agency); some to honor a relative by making a donation in that person’s name; and some to make a difference in society. Regardless of the reason, we should be concerned about effectiveness and what it costs to be effective. No one wants to throw their money away and people receiving the money want to be proud of what they do. 

Community of Practice and the TorranceLearning Download

One of many methods of organizational learning is a “community of practice”. This is a term coined by Etienne Wenger to describe “…groups of people who share a concern or a passion for something they do and learn how to do it better as they interact regularly.” TorranceLearning has created a kind of community of practice that they call the Torrance Download. The Download is a time and place for people to come together to share experiences and ideas, with the intention of everyone learning something new. 

I had the pleasure recently of attending another session of the Torrance Download. Once again the TorranceLearning team created a “…casual, fast-paced and friendly exchange of ideas with our friends, colleagues and clients in learning.” Between noon and 3:00 on a Friday, five concurrent, 30-minute discussions were held for a total of 20 topics. All of the topics were about learning and development and ranged from video production to elearning design to mobile learning to onboarding new employees. The tone, set by Megan Torrance, was one of fun, high energy, and engagement. This is a terrific model for a community of practice, either for people from different organizations or for people internal to one organization.

One type of Download discussion is called “Flash Project”. This is a leaderless conversation for the purpose of creating a list of answers to a specific question. I participated in one of these in which the
question was, “What is learning evaluation?” In 30 minutes, our group came up with an excellent list of ideas and questions that we shared with everyone in attendance at the end of the day.

We said that evaluation of learning is:

• Measuring business outcomes
• Establishing metrics for behavioral change
• Deciding how we want a learner to behave differently
• Thinking about individual, team, whole organization, and community learning
• Asking the learner, “What do you want to be known for?”

We also said that evaluation of learning is answering these questions:

• Did you like it?
• Did you learn it?
• Did you use it?
• Did it make a difference?
• Did you retain it?

And that evaluation of learning is measuring:

• How well organizational communications are aligned with pre, during, and post learning
• What managers are doing to facilitate and reinforce learning
• What resources (people, technology, money) are available to facilitate learning
• Where the gaps are between resources and learning
• Where people are starting from pre-learning
• What we know about learners' situations pre-learning
• What the organizational constraints are to learning and applying learning
• Why learning isn’t happening
• How to provide more opportunities for applying learning on the job
• What learning is still being applied months and years later
• Agile learning - Are learners able to extract useful information in any environment? What can be done to help learners become agile?
• What is being done to facilitate learning before, during, and after training

The striking thing about our Flash Project definition of “learning evaluation” is that we said that evaluation is much more than finding out what learners have in their heads at the end of a training program. We came to the conclusion that evaluation of learning should empasize achievement of business goals and it should measure the learner and learner’s situation before training, during training, and after training. And we agreed that the culture of the organization should also be evaluated because it is a critical factor in learning and application of learning.

Philanthropy, Learning, and Change-Making

The recent U.S. election reminded us on a daily basis of all the serious problems we face as a nation. Political campaigns made the solutions sound simple, as if one person or one policy could put people back to work, improve our educational system, deliver better health care, clean up the environment, or prevent war. It’s just not that simple. As Baldoni writes in his Forbes blog post, “Post-Election: Now What?”:

Now is the time to put the interests of the whole society first. That will require a commitment – indeed a partnership – of community, business, nonprofit and yes government.

Philanthropic foundations, although not specifically called out in Baldoni’s post, are also part of the solution mix. Steve McCormick, President of the Gordon and Betty Moore Foundation, writes that private foundations should be “game-changers”; they should be “…creating solutions to our most critical social problems.” He writes that philanthropists must do some things differently. One of these is measurement. He admonishes philanthropists to:

Use meaningful measures. It’s difficult to chart progress in the not-for-profit sector. Measurement requires creativity, coupled with discipline, to establish the right indicators—qualitative as well as quantitative. But, as the saying goes:  “if you don’t know where you’re going, you’ll never get there.”

Jeff Raikes, CEO of the Bill & Melinda Gates Foundation, in launching MarketsForGood.org (see below), said that foundations should be continually learning from doing. Conversations about learning do not happen as much as they should in foundations and their grantee organizations. They should be sharing information. Raikes says that the answers exist but it’s too hard to find and apply the information. Bigger, faster impact requires finding ways to get better data to social investors and nonprofits that are doing the work on the ground. He said, “Our work relies on the free flow of quality, accessible data and information…that lead to greater impact and a better world.”  

 

I agree with McCormick that private foundations can be “game-changers” and that creative, disciplined measures will help make that happen. And I agree with Raikes that much of the data and information already exists to increase impact on society’s most serious problems; that it is more a problem of access to information than it is one of collecting the data.

However, beyond applying measures and accessing data is the challenge of actually using the information to bring about change. This is where many foundations and nonprofits get stuck. They have the information in reports on their shelves or in computer files, but that’s where it stays. That information needs to be liberated, discussed, and translated into action. If they are going to be change-makers, they need to get their organizations engaged in conversations about the implications of the information, plan the action that must be taken to implement what has been learned, and hold themselves accountable for making change.

What Gets Measured Gets Done...Revisited

One of the most frequently visited posts on this blog is “What Gets Measured Gets Done…or Not.” I suspect that the post is interesting to readers because of the popular axiom, not because of the content of the post. In that post I argue that measurement is not enough. An organization has to do
something with those metrics in order to “get done” what needs to get done, i.e., organizational learning, performance improvement, and change. The act of measuring things like sales, customer service, product quality, teamwork, employee engagement, or learning, does not necessarily mean that anything will be done differently.

A better way to talk about measurement in organizations is the way Terry Starbucker writes about it in his blog post titled, The Truth About “If You Can’t Measure It, You Can’t Manage It”, a variation of the first axiom. He writes:

The thing is, I am a proud member of the “if you can’t measure it, you can’t manage it“ school of business.

It’s not one of those total absolutes, but I’ll say this – it’s pretty darned close.

So when I would hear “proof” that was anecdotal, or based on “feel”, or hearsay, or seven degrees of separation indirectness, I usually don’t buy it.

Terry goes on to describe performance improvement and cost savings he achieved in a former company because he had measures of employee behavior and customer satisfaction and he and others used those numbers to develop solutions for improving performance. He not only had the data but he did something with it.

Of course, many activities are managed without being measured. Many managers organize work, provide direction, and supervise people without measuring anything important. Are they managed well? That’s a different question.

And that leads us to another variation of the first axiom: “You can’t improve what you can’t measure.” I like this wording because it ties measurement to action. I have done many program and organizational evaluation studies for managers who asked me to measure processes, outcomes, and impact… but didn’t want to use the data to change. I suspect the explanation for their behavior is threefold: 1) they were hoping for a glowing report that would make them look good to their bosses and funders; 2) if they didn’t get a glowing report and findings implied the need for significant change in programs and staff, they could always find something about the measurement method that would discredit the findings; and 3) they could be accountable for measuring something without having to be accountable for improving performance.

The organizations that embrace performance improvement are those that make a concerted effort to understand what the metrics mean to their organizations and how they can use the findings to make decisions about what should be sustained and what should be modified. By all means, measure everything that is important to the success of your organization. But only do this if you are going to use the data to guide decisions about improvement.