The Performance Improvement Blog

Good Evaluation Facilitates Learning

Evaluation of programs (e.g., training, coaching) and organizations has more to do with facilitating learning than it does with statistics. Typical measures such as ratings of programs, frequency of participation, or fluctuation in revenue are useful only if they become the stimulus for a facilitated discussion of stakeholders that addresses implications for enhancing performance. Good evaluation requires good facilitators.

Kylie Hutchinson writes about this issue in her post on the American Evaluation Association blog, AEA365. She quotes this definition of “facilitator” that appears in the book, Facilitator’s Guide to Participatory Decision-Making:

…an individual who enables groups and organizations to work more effectively; to collaborate and achieve synergy. They are a content neutral party who…can advocate for fair, open, and inclusive procedures to accomplish the group’s work. A facilitator can also be a learning or a dialogue guide to assist a group in thinking deeply about its assumptions, beliefs, and values, and about its systemic processes and context.

As organizations seek to be more accountable and measure the impact of learning interventions, they will need facilitation skills to help people think more deeply about what they believe and what they need to do to improve. Measuring change is part of good evaluation, but unless we can get leaders in an organization to learn from that data and apply that knowledge to improving performance, what’s the point?  

 

Strategy Questions

Nothing is as useful as a good question. Organizational evaluation and performance improvement are essentially about asking the right questions. So I appreciate it when thought-leaders come up with a set of questions that make us think about what we are doing and how to do it better. Robert Simons does just that in his new book, Seven Strategy Questions: A Simple Approach for Better Execution. An exerpt appears in the November 22^nd issue of Harvard Business School’s Working Knowledge. In that article, Simons summarizes the seven key questions:

1. Who is your primary customer? The emphasis here is on “primary”. Most organizations have many different customers but resources should be dedicated to the most important customer. 
2. How do your core values prioritize shareholders, employees, and customers? Business leaders can’t make everyone happy. They need to be clear about who comes first when tough decisions have to be made.
3. What critical performance variables are you tracking? Organizations often track too many measures which dilutes the value of the process. They should identify a few, key variables that indicate progress toward success.
4. What strategic boundaries have you set? Employees need to know when their initiatives are aligned with the strategic direction of the organization and when they are not.
5. How are you generating creative tension? The organization needs onogoing innovation. Employees must be motivated to continually move out of their “comfort zones” and try new things that will help the company strategically.
6. How committed are your employees to helping each other? Commitment to shared goals or commitment to self-interest – this is a topic that should be explored openly in the organization.
7. What strategic uncertainties keep you awake at night? Change is constant and adaptation is critical. Survival depends on making employees aware of those changes and making them partners in the adaptation process.

To this list of questions, I would add, “What are you doing to align employee learning to strategic goals?” If you want employees to be customer-focused, values-based, data-driven, innovative, and team-oriented, then they need learning interventions that will help them develop these strategic behaviors. The organization needs to be constantly providing the kinds of training, coaching, mentoring, and experiences that will contribute to development of competencies needed for execution of business strategy. 

 

McKinsey Discovers What We Already Knew About Training Impact

Only a fourth of respondents to a McKinsey survey said that their training programs made a measurable difference in business performance. This survey was conducted online in January 2010 and received responses from 1,440 executives “representing the full range of regions, industries, functional specialties, and seniority.”

I wrote about this lack of training impact almost 20 years ago in a book that I co-authored with Robert O. Brinkerhoff titled, “The Learning Alliance: Systems Thinking in Human Resource Development”. At the time, we were using the estimate of only 10% to 20% of training having an impact on business results. In 20 years, that measly percentage hasn’t changed much.

The key to increasing impact, according to McKinsey, is to have the direct, active support of senior leaders. The McKinsey investigators write:

Given how important skilled workers are, companies must do better at creating them. When senior leaders focus on making training work—and get personally involved—improvement can come rapidly. The content of the training itself is not the biggest issue, though many companies could certainly improve it (see sidebar, “Getting training content right”). The most significant improvements lie in rethinking the mindsets that employees and their leaders bring to training, as well as the environment they come back to afterward. These are tasks only senior leaders can take on.

I couldn’t agree more. I wrote about this in the early 90s and recently reiterated the importance of senior leader involvement in a book I wrote with Sean Murray titled, "The 5As Framework". Over and over again, studies of the business impact of training provide evidence that senior-leader involvement, as well as learner-manager involvement, are essential to achieving business results from training. So why are organizations resistant to involving senior leaders in supporting learning? Do these companies enjoy wasting 75% to 80% of their investment in employees?

 

Can a Workshop Change Behavior?

Companies invest billions of dollars on half-day, full-day, and multi-day workshops with the hope that these one-time events will somehow make a difference. Vendors, who would rather sell workshops than results, are eager to claim that their product will make a difference. So it is a marriage made in heaven. What companies don’t know much about is if these one-time workshops truly change behavior significantly.

In the latest issue of T+D, Laurie Carrick takes on this question as it relates to developing emotional
intelligence (EI) competencies in leaders. She writes about conducting an extensive evaluation of a four-hour, EI workshop for 11 people in a company. She describes her measures in this way:

…this study included pretest quantitative Emotional Quotient Inventory (EQ-i) assessments and qualitative behavioral event interviews. The participants completed the EQ-i assessment before participating in the one-hour interviews that took place 30 days prior to the learning initiative. The participants attended the training class (learning initiative), which included the distribution of the EQ-i pre-assessment results. Six weeks after the learning initiative, each participant received a 90-minute critical incident interview session combined with a 30-minute coaching session and program evaluation…Three months after the learning initiative, the participants completed an EQ-i self-assessment post-test.

Carrick concludes from her findings that while the workshop raised awareness of EI, it was the combination of the workshop and post-workshop coaching that resulted in competency development. I am not surprised by this conclusion but I think she has missed one of the key factors in learning: the evaluation activity. Her study brought unusual attention to the workshop and EI, heightening the perceived value within the company. The regular assessments caused participants to reflect on the meaning of EI, their learning, and application of that learning. The evaluation process must have shaped expectations and reinforced learning. I would have to conclude that it was not only the workshop and post-workshop coaching that made the difference, but it was also the evaluation process itself that contributed to competency development.

 

Online Tools for Evaluating Informal Learning

Learning occurs in "informal" (another term that I don’t like but is gaining traction) settings and online, as well as in the classroom. How can we know what is learned informally and what difference that learning makes to the success of an organization?  Some of the methods being used in informal science education could be applied more broadly to informal and online learning in business, nonprofit, and government settings. Kirsten Ellenbogen describes some of these methods on the American Evaluation Association blog. She writes about “embedded evaluation”:

Embedded evaluation integrates “invisible” tools for evaluation into the existing program activities. For example, in a youth program that has a focus on interactive media, the projects produced by youth are posted to online environment. http://info.scratch.mit.edu/ScratchR:_The_Online_Community The projects can be downloaded by others, modified, and reported. All activity in the online community can be tracked, and the ongoing development of the youth’s projects can be analyzed for more detail.

And she writes about videotaping users:

For example, videotaping a museum visitor using an exhibition (using IRB-approved informed consent as appropriate). In the post interview, after the initial set of questions, show the visitor a video segment of his or her interactions with the exhibition that was taped just moments before. Use a semi-structured interview approach and ask the visitor to narrate their video and tell you more about what they were doing. This approach can become somewhat automated using technologies like Video Traces. http://depts.washington.edu/pettt/projects/videotraces.html

She also suggests apps that can be used to log and document experiences and reactions to experiences on a mobile device (smart phone, iPad, etc.).

These evaluation tools could easily be applied in work settings. Employee teams could post the progress and results of their projects on a Web site (e.g., internal wiki). Managers could follow these results for the purpose of assessing impact of team learning. Employees could be videotaped during a applying a newly learned skill or in conversations with managers or co-workers and then those tapes could be reviewed with the employee to assess learning and impact. Workers could be asked to use an app to log their reactions to a benchmarking activity and then discuss that experience with their supervisors.  New technologies are offering many useful ways of monitoring and assessing the impact of “informal” learning.