Advanced technologies have become part of most aspects of our lives and evaluation studies are no exception. Motion sensors, cameras and GPS, among other technologies, present ways of measuring behaviour directly or indirectly. Our repertoire of data collection modes is indeed expanding. These technologies mean evaluation studies can thus be less dependent on self-reported behavioural data, which is often subject to recall and social desirability bias. Especially mundane everyday activities, like handwashing, are easily forgotten after the event, or respondents may be shy to admit that they don’t practice handwashing according to received knowledge.
Kantar Public in partnership with Emerson International recently conducted an evaluation on behalf of UNICEF Ghana, which aimed to assess the effectiveness of a handwashing intervention on changing children’s behaviour. The UNICEF intervention targeted 20 schools in Ghana. Innovative in itself, it used specifically developed interactive stories and games as part of a curriculum to improve hygienic behaviours. Hypotheses were built around the potential of the game to change knowledge, attitudes and practices. Specifically it set out to assess knowledge on the importance of handwashing with water and soap (HWWS), whether knowledge leads to a positive change in attitudes and perceptions of HWWS, and if changed attitudes actually result in healthier handwashing practices with water and soap. Innovative technologies were used to measure behaviour, and these data were triangulated with pupil surveys, focus group discussions, and observations to produce a very rigorous evaluation.
The study took a longitudinal approach and measured the levels of improvement in knowledge, attitude and practices among Ghanaian pupils in the ages of 7-11 years in both intervention and comparative schools. A total of 20 schools with about 150-250 pupils each – ten schools in each group – took part in the study.
Technologies for data collection were used in two different ways:
- Sensors were mounted on ‘tippy taps’ to measure the number of times and duration the water containers were tilted (a tippy tap is a conveniently mounted water container, using a stick and rope, which is then operated by a foot pedal for hygienic handwashing);
- Video cameras with a motion recognition algorithm were set up above the tippy taps and identified when handwashing movements took place.
These technologies were used to measure how often handwashing facilities were used and whether the correct handwashing procedures were employed. On their own they would not answer all evaluation questions, however. In order to assess knowledge and attitudes, and explore pupils’ experience of the handwashing curriculum, it was, unsurprisingly, necessary to collect data from pupils rather than taps. Hence, a pupil survey and focus group discussions formed part of the evaluation. Moreover, researchers observed the study schools on given days and manual measurements were also taken of the volume of water and soap used each day.
As mentioned above, the multiple lines of evidence ensured a high level of rigour and the innovative use of technologies provided high quality behavioural data that improved the validity of the conclusions. That said, it is essential that technologies are employed in an efficient manner, or stakeholders’ high expectations of what such technologies can contribute may not be realised. In this light, it is worth highlighting the following points:
- Conditions of experimental schools must be continuously monitored to be constant across the period of study. For example, volunteers needs to ensure continuous measurements and re-fill of water containers to ensure constant availability of water and soap. If not, there is a risk pupils start using alternative handwashing practices, which would contaminate the findings.
- Strong collaboration with IT and technical local teams is essential. Having the IT and technical teams onboard from an early, pre-installation stage is crucial so they can modify the systems or sensors as necessary. Clearly, retrieving of data from sensors and cameras is crucial and needs right expertise. It is important that the equipment is set up the right way in the first place.
- Collaborative efforts between all stakeholders from donor to school pupils is required for successful implementation and evaluation. In innovative interventions and complex evaluations such as this one, a lot can be at stake for those involved. Building in ‘cooling’ periods in the project design is therefore recommended to reduce biases due to context or setting influences.
Now that the project has concluded, the tippy tap is freed from the camera, and children may or may not still be singing the handwashing songs, we can say that the evaluation was largely successful. About 85% of schools successfully used the equipment without breakages and produced high quality data as a result. Innovative technologies seem to be here to stay.
As for the intervention itself, the analysis showed that the specifically developed games and song-based curriculum was popular with the children. The evidence suggests the intervention succeeded in improving knowledge, attitudes and practices with regards to handwashing with soap and water for many of the key indicators. There were, however, some differences by gender and urban/rural location. Overall the immediate follow up data show positive results from the intervention. Now this is a project that no one wants to wash their hands of!