The Performance Improvement Blog

St. Jude Trains Its Leaders

We could have been invited to speak about org charts, making tough decisions, managing finances, etc. Instead, we were asked to talk about vulnerability, trust, and engagement. This is what my co-presenters and I did at a one-day leadership program for St. Jude Children’s Research Hospital last week. This exceptional pediatric research and treatment center recognizes that it needs to keep learning and that the intra/interpersonal aspects of leadership are as important as the mechanics of managing hospital departments and labs.  

I facilitated a workshop on the leader’s role in ensuring the success of employee learning and development programs.  Brené Brown gave a keynote address about how everyone's sense of vulnerability and shame can be a barrier to leading. Greg McKeown had St. Jude managers 
thinking about how they can make the people around them smarter, more motivated, and more successful. Jim Ferrell talked about minimizing the self-deception we all do as both individuals and organizations. Rich Brandt emphasized the importance of leaders building trust and engagement. And Carl Cohen described the interpersonal dynamics of leading biomedical research teams effectively. The day provided a tremendous learning opportunity for St. Jude’s managers to become more effective leaders. 

 

Kirkpatrick's Four Levels of Training Evaluation: A Critique

At the end of Dan McCarthy’s blog post, “How to Evaluate a Training Program”, in which he explains his pre-post, survey approach to applying the Kirkpatrick four levels of training evaluation, he asks: Has anyone used a system like this, or something better? What do you think, is it worth the bother?  Since McCarthy asked, here is my response.

Yes. I’ve used a similar “system” of evaluation many times in my career as a program evaluator. However, I don’t recommend this approach any longer for most situations. While better than nothing, for many training programs (as well as for coaching, mentoring, simulations, self-directed learning, etc.), this approach does not produce the information needed to continuously improve performance and achieve business results. There are at least six reasons for this.

First of all, there is low correlation among the four levels (reaction, learning, behavior, and results). This takes nothing away from the contribution that the Kirkpatrick model has made to the field over the past 50 years. I hate to think where employee training would be today if we hadn’t been guided by Donald Kirkpatrick’s thinking. However, just because learners liked the training doesn’t mean that they learned anything new, or will apply what they learned, or will contribute to achieving business results. Knowing one level does not predict outcomes at the other levels.

Second, self-report surveys produce unreliable performance data. What people say they did or will do is often not what actually happens (I know this is not shocking news.). This is not to say that we shouldn’t ask; it just means that we have to interpret those findings very cautiously.

Third, it shouldn’t be a decision to evaluate or not to evaluate. Good training programs measure impact; they hold learners (and their organizations) accountable. That’s part of the learning process. And some form of evaluation should always be done because it reinforces learners learning and contributes to performance improvement.

Fourth, the outcomes of leadership training cannot be fully anticipated. To construct a useful survey, we have to be able to anticipate what will be learned, how it will be applied, and what difference it will make so that we know what questions to ask. However, especially with leadership training, that kind of prediction is nearly impossible. The only way to discover all of the outcomes is through direct observation or in-depth interviews.

Fifth, if we apply the Kirkpatrick model strictly, then we’re not asking if the training program is the right thing to be doing in the first place.  Shouldn’t we ask that question first, before we ask about reactions, learning, behavior, and results?

Sixth, performance improvement is never the result of training, alone. Many organizational factors (e.g., manager support) determine whether impact from that learning will occur. If, as McCarthy suggests, we only compare what people knew before training to what they know and do after training, we still will not know what to do to achieve better results.

In most learning interventions, especially for leadership and management training, Robert O. Brinkerhoff’s Success Case Method is a more useful approach to evaluation. I have written about this method in previous blog posts.

Thank you, Dan McCarthy, for asking. 

 

Five Steps to Business Impact From Learning

Five key organizational factors determine if there will be business results from learning: 1) alignment; 2) anticipation; 3) alliance; 4) application; and 5) accountability. Together, these factors make up a framework that contributes to employee success. Sean Murray and I wrote about this in our book, The 5As Framework. Sean will be talking about the framework in a session at the American Society for Training & Development Conference in Orlando, Florida on May 25th. If you go to the ASTD Conference, I encourage you to attend Sean's session.

In his session, Sean will describe the factors and tools participants can use to ensure that learning is a business driver in their organizations. Sean's goal is that participants leave the session understanding each of the 5As and how they impact learning. After attending, participants will be able to conduct a 5As audit to identify the greatest opportunities for increasing business success from training and education in their organizations. Participants will be able to create an action plan for achieving business results. They will hear how other organizations are using the 5As Framework to achieve business success.

Physician as Bully

Organizational culture affects the quality of decisions that are made by people who work in those environments. No place is this as obvious and as critical (life and death decisions) then in a hospital. Theresa Brown, oncology nurse and author, writes about the impact of hospital culture in a blog post titled, "Physician, Heel Thyself". She describes a recent situation in which a doctor blamed her for slow test results in front of a patient and how that kind of "bullying" is detrimental to patient care. She writes:

When a doctor thoughtlessly dresses down a nurse in front of patients or their families, it’s not just a personal affront, it’s an incredible distraction, taking our minds away from our patients, focusing them instead on how powerless we are.

That said, the most damaging bullying is not flagrant and does not fit the stereotype of a surgeon having a tantrum in the operating room. It is passive, like not answering pages or phone calls, and tends toward the subtle: condescension rather than outright abuse, and aggressive or sarcastic remarks rather than straightforward insults.

This kind of subtle workplace bullying, although acceptable behavior in many hospitals (and many other organizations), creates a culture of fear and intimidation that results in poor patient care and increased costs. Nurses are reluctant to share vital information they know about individual patients and their treatments. Then doctors make decisions based on incomplete data. And nurses are reluctant to confront doctors even when treatment decisions might be harmful to patients. Hospitals don't condone this doctor behavior but they don't do much to stop it either. Eventually, it becomes so ingrained in the way doctors work with staff that the behavior is accepted as the norm and rarely gets reported as abuse. 

 

Organizational Dashboards

Organizational dashboards have become a popular tool for monitoring performance.  They are being used by a wide variety of organizations. Here are some examples:

Healthcare

The Dashboard Spy presents some examples of performance measures that hospitals are tracking on their dashboards. He shows dashboards being used by Methodist Medical Center in Illinois, Dean Health System in Wisconsin, and Scripps Health system in California.

Nonprofit

BlueAvocado.org, in an article by Jeanne Bell, shows how nonprofit organizations are using dashboards for managing financial performance, program implementation, human resources, Board of Directors accountability, fundraising, compliance and risk management.

Government

The new governor of Michigan, Rick Snyder, has implemented a dashboard to track the State’s performance in the area of education. His administration solicited input from constituent interest groups throughout the State and then came up with five categories: 1) Student Outcomes; 2) School Accountability; 3) Culture of Learning; 4) Value for Money; and 5) Post-secondary Education.

The performance measures on the dashboard in each of these types of organizations are important because they get stakeholder attention and drive action priorities. However, it is the process of creating a dashboard that is the most powerful evaluation and learning experience for an organization. As I wrote in my book, Developing a Learning Culture in Nonprofit Organizations, the value of developing a dashboard (or scorecard) is in “…selecting outcomes to measure, identifying indicators of progress toward those outcomes, and observing effects of changes that are made to maximize organizational results. The [dashboard] is just the locus of energy for a process of evaluation and organizational learning that involves critical decisions about what is important to measure and what should be done about the results.” The content and look of the dashboard influence its usefulness, but it is the process of developing the dashboard that contributes to organizational learning.