By Chandan Jain, Evaluation Specialist
Stuti Tripathi, Senior evaluation specialist
and Kirthi Rao, Evidence impact specialist
3ie is the independent evaluator of this unique initiative. As you know, the Saamuhika Shakti initiative adopts a multi-sectoral approach to collectively impact and improve the wellbeing of informal waste pickers in Bengaluru, a metropolitan city in India. Under this initiative, seven organizations are delivering a set of mutually reinforcing interventions to address different but interconnected problems facing the community. The initiative is funded by the H&M Foundation, while The/Nudge Institute is supporting and coordinating the interventions by BBC Media Action, CARE, Hasiru Dala, Sambhav Foundation, Save the Children, Social Alpha and WaterAid.
With growing urbanisation and a burgeoning informal economy in cities across the globe, the study, with its focus on the understudied waste picker community, will be extremely policy-relevant. Given the coordinated interventions that target systemic root causes rather than individual symptoms such as poor incomes or ill health in the waste pickers’ community, the findings of this evaluation will help inform the development of holistic systems-level solutions.
Besides, our preliminary review shows that very little is known about the effectiveness of collective impact approaches. Given this knowledge gap, the evaluation will generate learning that can potentially inform the design and implementation of collective impact programs in comparable contexts.
Our impact evaluation aims to assess if and how the initiative’s collective impact approach benefits waste picker households. Our approach is motivated by a theory-based framework, which means we need to understand the theory of change or how key activities planned under the initiative are expected to generate outputs and outcomes.
We will use mixed methods, that is, both quantitative and qualitative methods to understand the impact of the program on waste picker households and find the factors that help or hinder program implementation and participation. Unlike traditional impact evaluations of interventions delivered by a single implementing organization, this evaluation is different in that it needs to pay attention to the coordination and communication between different initiative partners. This aspect will be an important focus of the planned process evaluation.
The quantitative evaluation is based on rigorous econometric methods, and we are using a quasi-experimental technique (difference-in-difference approach) to evaluate the effect of the initiative on waste picker households. To be more specific, the evaluation is based on the triple difference framework wherein we will compare the changes in the outcomes of waste picker (eligible group) and non-waste picker (non-eligible group) households over time and across different localities to estimate the program effect. The triple difference framework will allow us to isolate the impact of the program from any other changes taking place in the same localities over time in a rigorous and robust way.
The evaluation will compare outcomes in localities with two or more implementation partners with those of localities which have at most one partner. The quantitative analysis will rely on data collected from baseline and end-line surveys, while the qualitative analysis will rely on program information from the implementation partners, key informant interviews (KIIs) and focus group discussions (FGDs).
A theory of change or ToC links inputs and activities of a project to intended outputs and outcomes in the short- and long-term. In diagram form, it maps out how inputs and activities will interact with the context and participants to trigger hypothesized changes (or the intended impact) in a specified timeframe.
When evaluators and the program implementers map out the theory of change collaboratively, it helps build a mutual understanding of how the program will work and the causal pathways that link inputs and activities to final outcomes. On occasion, the process of mapping out theory reveals where there might be unmitigated risks or hidden assumptions that threaten the causal link. The identified outputs and outcomes also serve as a guide for the implementation partners to identify indicators that can help to measure the progress of their activities towards achieving desired outcomes.
For the impact evaluation of the Saamuhika Shakti Initiative, 3ie, in consultation with implementation partners developed theories of change for each of the interventions under the initiative. The exercise was extremely insightful and helped us understand how each intervention was set up (its various constituent activities) and the outcomes it will likely lead to. In addition, the exercise helped us understand the effect of COVID-19 on the interventions.
These intervention-level theories of change became the building blocks for putting together an initiative-level ‘meta-Theory of Change’ that is guiding our approach to evaluating the impact of this initiative.
It has been one of a steep learning curve, owing to the uniqueness of the approach and the complexity of putting together interventions of multiple organizations working with the waste pickers community. Added to this were the unprecedented challenges that COVID-19 posed.
The pandemic limited our ability to hold in-person conversations with the program partners and almost all the communication had to be managed remotely. It also affected program timelines significantly and, in some cases, program implementation strategies as well. The uncertainties led the implementing partners to adopt a nimble approach and realign constantly. Keeping abreast with changing programmatic strategies proved to be the biggest challenge. It became critical for us to systematically document the dynamic reality we were working in.
The exciting part however has been to continuously engage and learn from the initiative partners, and leverag their knowledge and understanding of the context and issues that confront waste pickers in Bengaluru. This has been critical for designing the scope of the evaluation and developing strategies so the study and especially data collection, could be rolled out smoothly. It has also been critical for developing our stakeholder engagement and evidence uptake plan.
We all know that the link between research findings and development decisions is not uncomplicated, predictable or time-bound. However, research insights (including evidence from evaluations) are much more likely to contribute to change if researchers engage and communicate with stakeholders regularly. Recognising the importance of keeping potential evidence users and evidence champions informed, we have a living stakeholder engagement and evidence uptake plan (SEEP) for the Saamuhika Shakti Initiative. By a living SEEP, we mean that the plan is not a one-time exercise. We plan to revisit the SEEP to discuss and leverage opportunities for communication around policy and programmatic lessons from the evaluation. The plan is important if we want to meet the goal of informing and influencing actors who might be interested in learning from the Saamuhika Shakti initiative.
In the coming year, you can look forward to the findings from the on-going baseline survey. Once the baseline data collection is completed, we will put together the findings and share the baseline report with the various partners of this initiative. Besides, we are also working towards designing the process evaluation. We will be sharing the draft framework with the partners for their comments and suggestions soon.
