In my last blog post I mentioned my evaluation work with Agahozo-Shalom Youth Village in Rwanda. One of the tools we used to help the Village staff think about indicators of success was a model of the residents’ total Village experience. See below.
We asked the staff, “What indicates that you have been successful at each point in this model?” At "baseline", they said that it’s knowing students’ abilities and expectations when they first arrive at the Village. It’s understanding the circumstances they came from and how the Village is different? It’s knowing their hopes and dreams...if at that point they are able to hope and dream.
During the “process” stage, indicators of success include the level of participation of residents in their school and non-school programs. Measures of learning, examples of leadership, examples of volunteering to help others in the Village and outside, and examples of showing initiative were all considered evidence that youth are learning and developing into knowledgeable, self-sufficient, and productive adults.
Some of the indicators of “output” success are the number of students who stay in the Village for four years, the number who graduate from the High School, and the number who pass Rwanda’s national exams. A low incidence of health clinic visits and emotional problems are also indicators of success at this point.
“Outcomes” that staff think are indicators of success include attending the university or a technical institute, starting a business, getting a job, raising a family in a responsible way, and maintaining contact with other graduates from their cohort. Also, taking a leadership position in the community and “giving back” to their communities in other ways are indicators of success.
“Impact” success is apparent in graduates who complete a college education, who are running a successful business, who have an adequate income that is supporting a family, and who are giving back to their communities through teaching, service, and economic development projects.
What the ASYV staff and I explored together is that, given their theory of change, success is measured at a number of points in the learner’s experience. If they waited, as so often happens in organizations, until after a program (i.e., learning intervention) has ended for learners, it’s too late for continuous improvement.