Black-box evaluation is measuring the effects of a program
or service without examining the nature of that program or service. It is
evaluating what people think and do after the intervention (e.g., training, classroom
education, e-learning, coaching, policy change) and then attributing that
behavior
change or lack of change to what happened in the “black box”.
Sometimes a comparison is made post-intervention to what people knew and did
before the intervention. However, nobody opens the
black box to see what’s inside. Because of this, we don’t know what aspects of
the program had the effect and why they had that effect. So, while we can be
accountable for results, black-box evaluation doesn’t help us understand how to
replicate those results in the future, we don’t know what should be continued
and what should be eliminated, and we don’t know what should be changed about
the program to get even better results.
My last two blog posts are about my recent work with Agahozo-Shalom Youth Village, a residential community for orphaned youth in Rwanda. That work provides a good example of the problem with a black-box approach to evaluation. If we only examine the output (e.g., number of graduates, national exam scores) and outcomes (e.g., enhanced self-confidence, enrolling in college or technical school, starting a business), staff would not have the information they need to replicate ASYV, to improve programs, and to possibly figure out how to get the same results at lower cost.
What ASYV needs is a way to link the contents and process of their programs and activities with outcomes. In addition, they need to know how circumstances of each child before coming to the Village interact with Village activities to affect results. And they need to look for the causes of those pesky unintended consequences that might become a predictable aspect of learning and change.
One of the methods we propose for this evaluation is in-depth interviews of a small sample of students at their time of graduation, then 18 months later, and then six years after that. Much like Brinkerhoff’s Success Case Evaluation Method, the Village staff will use these interviews to tell the stories of the youth. The interviewers will listen for links between what the students experienced at ASYV and how they’ve developed as self-confident, independent adults. They will listen for how the change has occurred from the time the youth came to the Village to how they are today. They will look into the “black box”, which is the Village experience of informal and formal education programs as well as the support of house mothers, counselors, social workers, and other staff, and examine how those experiences contribute to intended and unintended results.
