Позитивные изменения. Том 2, № 2 (2022). Positive changes. Volume 2, Issue 2 (2022)

The Use of the Contributive Analysis Methodology on the Case of the «Clean Games» Project

The Clean Games project started in St. Petersburg in 2014 as a simple volunteer initiative. The project leader Dmitry Ioffe once went hiking with his friends on Lake Vuoksa, but his vacation was spoiled by a landfill on one of the islands. That is how he came up with an idea for the first waste collection contest, which over eight years evolved into a series of eco-quests uniting 90,000 players in 26 countries. The participants collect up to 5 tons of waste in one game, most of which is then recycled. The project organizers had an ambitious goal: to raise the awareness and increase public engagement in the environmental agenda. How can one assess the social impact of the Clean Games and whether their ambitious goal has actually been met? One tool that can be used for this purpose is the qualitative approach based on analyzing the participants' contributions – the contribution analysis method.

Ivan Smekalin

Analyst, Positive Changes Factory

AN OVERVIEW OF THE SOCIAL PROJECT

Clean Games is a team competition that involves the clean-up of natural areas from garbage and the use of waste separation (Vainer & Nizametdinova, 2022). It is held annually in dozens of cities throughout Russia and around the world. Volunteers hold all-Russian and international garbage collection tournaments – the Spring and Fall Clean Cups. More than a hundred and fifty volunteers and tens of thousands of participants are engaged in the project.

The Clean Games have become an eco-franchise, with communities of organizers in the regions receiving free training and learning the project methodology after signing a non-profit agreement. In 2020, the project team raised 25.217 million rubles in grants and subsidies, corporate orders, government contracts, and donations from legal entities and individuals.

The project organizers say their mission is to give people an opportunity to see the problems of environmental pollution and raise the public awareness about environmental issues in game format. The project methodology allows the participants to see the problems of pollution and the benefits of waste separation and recycling. Learning to sort waste through the game, people start thinking about the environmental agenda.

The organizers have defined the following key performance indicators for the project: the number of participants and volunteers, the number of events held, social media coverage, mass media publication activity, and the funds raised (Vainer, 2021). These indicators are deemed to be useful for assessing the operational performance of the project, but they are only partially related to assessing the progress towards the project goal, namely raising awareness and increasing public engagement in the environmental agenda.

THE EXPERIENCES OF THE PARTICIPANTS AND ORGANIZERS OF THE CLEAN GAMES PROJECT: MOTIVES, ATTITUDES, EXPECTATIONS

In 2021, the Center for Public Initiatives Assessment of the National Research University Higher School of Economics conducted a research project "Motivations and Environmental Attitudes of Volunteers in the Clean Games Project" and published an analytical report.

More than two thirds of the Clean Games participants interviewed had prior volunteering experience, but those were generally one-off volunteer activities and targeted help. For more than a half of the respondents, this was their first experience participating in the project. As the researchers pointed out, the public engagement levels proved that participation in the Clean Games had become a broadly encouraging activity, something the participants were proud of and tended to tell their loved ones about.

Participation in the Clean Games had become a broadly encouraging activity, something the participants were proud of and tended to tell their loved ones about.

The regional project organizers were also interviewed. It turned out they were much more into environmental activism: all of them had past experience in social projects, and all of them were concerned about environmental issues.

However, the data collected in the pilot study could not be interpreted in statistical terms, due to the small and unrepresentative sample. Moreover, conducting a representative survey of all the Clean Games participants appears to be extremely time-consuming (the sample size would be comparable to nationwide surveys and would require zoning by participating teams). Assessing the perception of the environmental agenda also requires a more serious methodology, which drives the survey costs up. Therefore, one of the most rational ways to use qualitative data is through secondary analysis based on the contribution analysis methodology.

CONTRIBUTION ANALYSIS

Designing a study based on contribution analysis requires a focus on the change agent, with a description of the history of changes provided by the research subjects themselves. By interacting with the interviewer, the significance of the intervention for the object of social impact becomes apparent.

The contribution analysis methodology includes a method of logging the qualitative assessment of the impact (Copestake, 2014). The essence is to ask the informant what changes occurred in their life in the course of the project, without focusing on the project itself and its content. This way, the researchers can use biographical information to gain access to causal relationships, as perceived by the informant, which can be used to describe the qualitative effects of the social impact. For example, this method was used during an evaluation of development projects in African agricultural communities.

Along with the qualitative impact assessment protocol, there is also the "most significant change" method. This is a participatory qualitative method that, for example, has been used by the research team (Polet et al., 2015) to assess the social impact of affordable healthcare programs in the Philippines, Palestine, the Democratic Republic of Congo, and El Salvador. The method involves collecting personal stories of program participants, where they speak non-anonymously about their perception of the project and their role in it. This allows us to put the formal indicators in context by understanding the social significance and perceived meaning of the intervention.

Designing a study based on contribution analysis requires a focus on the change agent, with a description of the history of changes provided by the research subjects themselves.

THE MOST SIGNIFICANT CHANGES OF THE CLEAN GAMES

The results of the study of the Motives and Environmental Attitudes of the Clean Games Project Volunteers can be used to demonstrate the logic of the contribution analysis method. The report's authors highlight the following key motivations of the project organizers: game format, scale and fame of the project, and volunteering as a way of entertainment. Only a small part of the respondents said their goal was to develop environmental education in their region.

Additionally, a second look at the interviews reveals that most organizers are not aware of the motives of other organizers, noting that those motives could be different. The project’s social significance, in the opinion of the organizers, is limited to the actual garbage cleanup, without any focus on the cultural aspect. At the same time, when the topic of environmental education was mentioned, most respondents started talking about other projects, not about the Clean Games. The response closest to the intended result of the educational role of the Clean Games was formulated as follows:

"[This] could be called an aggressive enlightenment of sorts. That is to say, a person is taken out into the field, shown that there is too much human-caused garbage. Here, please clean it up – there's no shame in it."

This description of the educational task, however, leaves the following question: What values is a project participant expected to learn, besides the importance of garbage collection?

The project’s social significance, in the opinion of the organizers, is limited to the actual garbage cleanup, without any focus on the cultural aspect.

The organizers identify the following set of motives for joining the project: an interesting activity that benefits the society, receiving prizes/points, a desire for self-fulfillment, and a charismatic organizer. This information is meaningful on two levels. The lack of awareness of the importance of participating in environmental projects among the reasons for participation suggests a problem on the "demand side" – this motivation is excluded from the participants’ attitudes. However, this also describes the "supply side" situation – apparently, based on such attitudes, the organizers do not position the project as an opportunity for participants to be involved in the environmental agenda and do not themselves perceive this opportunity as the primary reason for participation.

In the organizers’ internal survey, when asked "What do you like about the Clean Games?" only 14 out of 93 respondents (15 %) indicated the project’s social significance. Some exemplary answers by a responsible eco-activist are presented below:

"The Clean Games teach children to treat nature with care and introduce the basics of separate garbage collection. Some people can no longer be educated. The Clean Games allow us to reach out to children and young people, make them love the place where they live. Still, I hope that those who have participated in the Clean Games will be more responsible for their behavior."

"Thanks to the Clean Games, the people change their attitudes and thinking about the environment and environmental issues, and everything is done in a playful and entertaining manner."

Most of the answers to this question mentioned the game format, competition and team building, leaving aside any environmental topics. For example:

"The excitement, the celebration, and the positive mood and fellowship that people begin to feel in the process."

"It is an interesting format. I feel like the organizer of a big and important event."

"It's a good thing. You make me feel like I'm part of a big, useful cause."

However, the project goal was not simply cleaning up the garbage or holding an entertainment event for the public, but rather increasing the people’s awareness and engagement in the environmental agenda. When even the organizers miss the social significance of the project and instead talk about its secondary attributes such as the format and atmosphere, it is clear that the social impact remains merely an accessory factor.

Thanks to the Clean Games, the people change their attitudes and thinking about the environment and environmental issues, and everything is done in a playful and entertaining manner.

The logic of contribution analysis is to look at the situation through the eyes of a project participant and to visualize the change in their attitude through their narrative. Based on the data available, it is hard to say that the project organizers in the field are actually aware of its social impact. Therefore, in the absence of an observable positive effect in attitudes, it makes no sense trying to identify its causes.

Thus, despite the limited quantity and quality of data, the findings are sufficient to say that the results obtained are not fully consistent with the stated social impact goals of the project.

THE PROSPECTS OF USING THE CONTRIBUTION ANALYSIS METHOD IN THE CLEAN GAMES CASE

What can be done to improve the assessment of the social impact of a project by means of the contribution analysis method?

According to the most significant changes methodology (Davies & Dart, 2005), for the case in question, the research design might look as follows:

1. Conceptualizing "the public awareness and engagement in the environmental agenda" as the object of change.

2. Developing a qualitative survey guide for organizers and participants to explore perceptions of the subject of change and the respondent's role in that change.

The proposed use of contribution analysis involves more than just a qualitative assessment of its results, but requires designing a participatory action research, which in itself is a tool for positive social change.

3. Defining the reporting period during which the change is expected to be achieved.

4. Non-anonymous interviews about the experience of participation in the project are collected at least at two points – before and after participating in the project. In them, the participants pronounce their perception of the subject of change, and the researcher tries to fix the changes in this perception that occurred during the project.

5. Selection of the most significant stories – those in which the positive changes in the project were the greatest, and which would serve as an expression of general trends.

6. Feedback from the participants to clarify the story details.

7. Content analysis of the interview materials based on the codes obtained from the stories. The most meaningful stories are expected to change the understanding of the project itself and lead to the formulation of new concepts that are used to interpret the participants' experiences.

8. Making organizational changes to the project based on the study findings.