The Performance Improvement Blog

Evaluating Informal Learning

Informal learning in organizations is finally being recognized as a key aspect of employee development and performance improvement. But how do you evaluate something that is as unpredictable and serendipitous as informal learning? The following post first appeared on this blog in November 2010. 

Learning occurs in "informal" (another term that I don’t like but is gaining traction) settings and online, as well as in the classroom. How can we know what is learned informally and what difference that learning makes to the success of an organization?  Some of the methods being used in informal science education could be applied more broadly to informal and online learning in business, nonprofit, and government settings. Kirsten Ellenbogen describes some of these methods on the American Evaluation Association blog. She writes about “embedded evaluation”:

Embedded evaluation integrates “invisible” tools for evaluation into the existing program activities. For example, in a youth program that has a focus on interactive media, the projects produced by youth are posted to online environment. http://info.scratch.mit.edu/ScratchR:_The_Online_Community The projects can be downloaded by others, modified, and reported. All activity in the online community can be tracked, and the ongoing development of the youth’s projects can be analyzed for more detail.

And she writes about videotaping users:

For example, videotaping a museum visitor using an exhibition (using IRB-approved informed consent as appropriate). In the post interview, after the initial set of questions, show the visitor a video segment of his or her interactions with the exhibition that was taped just moments before. Use a semi-structured interview approach and ask the visitor to narrate their video and tell you more about what they were doing. This approach can become somewhat automated using technologies like Video Traces. http://depts.washington.edu/pettt/projects/videotraces.html

She also suggests apps that can be used to log and document experiences and reactions to experiences on a mobile device (smart phone, iPad, etc.).

These evaluation tools could easily be applied in work settings. Employee teams could post the progress and results of their projects on a Web site (e.g., internal wiki). Managers could follow these results for the purpose of assessing impact of team learning. Employees could be videotaped while applying a newly learned skill and then those tapes could be reviewed with the employee to assess learning and impact. Workers could be asked to use an app to log their reactions to a benchmarking activity and then discuss that experience with their supervisors.  New technologies are offering many useful ways of monitoring and assessing the impact of “informal” learning.

How do you measure the impact of informal learning experiences?

Unified Organizational Strategy

This cartoon by Manu Cornet first made the internet rounds a couple of years ago but still has relevance today. It appeared again in a recent issue of The New York Times business section as part of a story about the reorganization of Microsoft. 

Satire aside, I think this cartoon has two messages. One is that regardless of how a company's structure looks on paper, it probably functions very differently. And two is that there are many different types of company culture and no one culture is right for every business. 

Adam Bryant, in his "News Analysis" column in the same issue of The New York Times as the Manu cartoon, poses a question about culture that vexes every CEO, whether the leader of a 100,000-employee company or a three-person law office: 

How do you get everyone to think in terms of “Us” — we’re all on the same team — rather than thinking of colleagues in other divisions as “Them”?

This is the challenge that all departmentalized organizations face - getting everyone on-board and rowing in the same direction.  A lack of alignment is represented by the Microsoft box in Manu's cartoon (only a slight exageration - they might use email instead of a gun).

The problem is trust. it's the tendency of human beings to trust the people to whom they are physically and operationally close and to distrust everyone else until something is done to actively build trust among individuals and teams. 

Bryant offers a few tips for overcoming this silo mentality and building trust. 

Create a ‘One Company’ Culture - Help employees see how what they do and the decisions they make on a day-to-day basis impact the entire company.

Simplify the Scoreboard - Develop a small set of simple-to-understand performance measures that everyone can work towards.

Communicate Relentlessly to the Entire Staff - Repeat the same, consistent message over and over again at every opportunity so that everyone hears the same thing and believes that you mean it.

This sounds simple but it's not. It takes earning the trust of your employees everyday and working hard everyday to gain their commitment to a shared organizational strategy.

Organizational Learning in Colleges and Universities

Recent newspaper stories have questioned the value of a college education (although it is clear that college graduates earn more money over their lifetimes) and the burden of students taking on debt to pay for that education. What isn’t discussed much is how colleges and universities need to change in order to increase value for students and society.

Jeffrey J. Selingo, editor at large for The Chronicle for Higher Education and author of the new book, College (Un)bound: The Future of Higher Education and What It Means for Students, says,“American higher education has lost its way.” He points to the high cost, the high number of non-completers, the proliferation of online courses available anywhere at any time, and a disconnect between what is taught and what employers need.

And there are other pressures on higher education institutions to change and improve. Stakeholders (board members, funders, employers, legislators, state and federal agencies, etc.) want their institutions to be entrepreneurial and create businesses from intellectual property, be responsive to the talent needs of the private sector, provide an ROI that justifies the high cost of a college education, and be more responsible for student access and success, all while fulfilling course requirements in the various disciplines.

In order to change, colleges and universities must first learn; that is, acquire new organizational knowledge and new organizational skills. They must create an organizational routine of feedback, reflection, and active social learning. They must develop a community in which administrators, faculty, and staff are constantly sharing information and seeking performance improvement through new knowledge, new skills, and new applications of knowledge and skills to achieving the goals of the institution.

They need to learn how to examine what they do, compare that to what needs to be done, reflect on what they have learned from their actions, and make the needed changes in the organization. They must be open to true transformation; not simply incremental improvements. 

Vision, Mission, and Guiding Principles...of a Nation

High performance organizations are clear about their mission, vision, and guiding principles. On this Independence Day weekend in the U.S., it seems fitting that we revisit the mission, vision, and guiding principles of America. As intended, this belief system has served the nation well for the past 237 years. Charter documents (Declaration of Independence, Constitution, and Bill of Rights) have been amended over the years but the fundamental values have held strong.

Paul Krugman, New York Times columnist and winner of the Nobel Prize in Economics, argues that although much has changed in the U.S. over that time, especially in terms of population and production, the nation’s commitment to democracy has not waivered. He writes:

Above all, we are still, at root, a nation that believes in democracy, even if we don’t always act on that belief. And that’s a remarkable thing when you bear in mind just how much the country has changed.

Notions about how democracy should be implemented vary widely among its citizens and their legislators. Sometimes what they do does not appear to be in the best interests of all the people and some people seem to be considered more equal than others. However, over two centuries later the conversation is still about aspiring to the underlying values of the founding fathers.

The “Declaration of Independence” (written in 1776) presents the vision in its opening paragraph:

We hold these truths to be self-evident, that all men are created equal, that they are endowed by their Creator with certain unalienable Rights, that among these are Life, Liberty and the pursuit of Happiness.--That to secure these rights, Governments are instituted among Men, deriving their just powers from the consent of the governed

The “Constitution” (written in 1787) presents the mission in its opening paragraph:

We the People of the United States, in Order to form a more perfect Union, establish Justice, insure domestic Tranquility, provide for the common defence, promote the general Welfare, and secure the Blessings of Liberty to ourselves and our Posterity, do ordain and establish this Constitution for the United States of America.

Guiding principles are spelled out in the “Bill of Rights”. Together, these three documents present the values of the founding fathers. Implementation of this belief system has been a struggle. But it’s the shared values, as in any sustainable organization, that have kept America from succumbing to the many internal and external threats that it has faced since July 4, 1776.