Program Evaluation in the Wild, Part 2
In my last post, I introduced the outcomes workgroup, a learning community we have organized for several Boys and Girls Clubs who use KidTrax to manage their participant and program data. In this post, I describe the first meeting of this group.
We began the initiative by bringing together ten staff members from seven Clubs to the nFocus headquarters. The goal of the workshop was to provide an “introduction to logic models and outcomes measurement.” However, we knew that often, the language of program evaluation and outcomes measurement comes across to front-line staff as dry and disconnected from both the challenges and rewards of their day-to-day work. We therefore had a secondary goal of exploring whether we could bridge that gap, finding a way to map the staffers’ lived experiences serving in Boys and Girls Clubs to more standard, evaluation-oriented frameworks.
To set this exploratory tone, we began the meeting with storytelling, asking for a volunteer to tell a “success story” about an individual Club member. One Chief Professional Officer (CPO) told a story of a boy who had started coming to the Club with his older brother. The CPO remembered the boy was not particularly confident – he was hunched over, had bad skin, did not make eye contact, and was relatively quiet. Over time, however, as he started getting more engaged with the Club, the boy began to stand up straighter. He took on a leadership role in the Club and became Youth of the Year. He joined JROTC. When he graduated high school, he joined the military. After his service he came back to his community, got married, became a social worker, and he still volunteers for the Club.
This story resonated with all the other participants, reminding them of their own “success stories” and setting a positive, youth-centered tone for the workshop. The story also became the basis of an exercise to identify both long-term outcomes as well as shorter-term indicators or predictors in a high-level Club logic model. For example, if the successful outcome was a young man who was employed, had healthy relationships, and contributed back to his community, what signs did the CPO have that he was on that path? What did she think were the turning points in his Club experience, and how did she know? What experiences did he have at the Club that contributed to those turning points?
Debriefing the story in this way helped the participants take an intuitive, narrative way of describing their work and its purpose, then begin mapping it to the more typical logic model language of inputs, activities, outputs, and outcomes.
We began this mapping by having the participants break out into small groups (by site). Each group filled out four-column tables, based on trajectories/stories of young people from their own Clubs:
|Where did they start?||BGC Activities||Signs of Change||What Does Success Look Like?|
Coming back together, we reviewed the tables and compared them to models of youth development from well-known research and youth advocacy groups such as the Search Institute and the Forum for Youth Investment, noting the similarities and differences between those models and the “homegrown” ones we had just developed.
The groups then each picked a single BGC activity and tried to pull out the inputs, activities, and outputs associated with that activity, in order to link them to the short-term and long-term outcomes they had identified in creating the earlier table. While this exercise was useful as a way of practicing logic modeling and, as one participant put it, “learning to ask the right questions,” we realized that the participating Clubs were so diverse in size, geographic scope, Member demographics, and service approach that it was a challenge for us to identify lessons that cut across the Clubs’ models.
For our next meeting, then, we decided to focus on using the logic model of a single, common program to create a program evaluation action plan for each Club. Our next post will describe the work done in that meeting.