Mobile technology: The future of evidence in development?

Written by: Alexandra Cronberg

The future is all about mobile technology, right? Well, perhaps, but in the context of real programme evaluations, it is worth examining and understanding the benefits and drawbacks of mobile data collection modes for gathering evidence, before waving goodbye to human interviewers.IMG-20180321-WA0002

In order to address this topic, Kantar Public and the British Council hosted a joint event which took place in London, covering three studies including one in partnership with RTI. The full slideset with the findings is available here. Read on for a brief summary.

  1. How efficient is mobile SMS vs other methods of collecting evidence from teachers in the Connecting Classroom programme?

In order to answer this question, we carried out a pilot survey among participants who had attended the British Council’s Connecting Classroom training in Ethiopia. The study collected progress data using an SMS survey, rather than using the alternatives of paper or telephone. This study showed that SMS has many benefits and some challenges: It is a cost-efficient and viable option for collecting progress data among a target population among known participants, although the response rate is lower compared to telephone and paper. Moreover, there were some unexpected challenges during the pilot, including internet downtime and a change in the MNO’s airtime bundles, which affected the administration of incentives. This highlighted the importance of allowing plenty of time for testing and piloting the survey.

The second study addressed the following question:

  1. What is the potential for using Interactive Voice Response (IVR), compared with SMS, telephone surveys (CATI) and face-to-face surveys of collecting information in the general population?

This study, which Kantar Public conducted in partnership with RTI, compared response rates and representativeness of mobile data collection modes (i.e. SMS, CATI, and IVR) with that of face-to-face interviews. All of these studies targeted the adult general population in Nigeria aged 18 to 64. The results showed that the response rate of SMS and IVR are very, very low, and even for CATI it is much lower than for face-to-face. This would not be such a problem if the achieved samples were representative of the general population. That is, unfortunately, not nearly the case. The study showed that the achieved samples using SMS and IVR are very much skewed towards better educated and younger people, and also towards men. People often think that applying statistical weights to improve sample representativeness is the solution to this problem. However, the findings showed that weighting does not solve the problem, and least when looking at voting behaviour. In fact, weighing actually increased the bias. Finally, with respect to cost, once we adjusted for questionnaire length and sample size, SMS and IVR are more expensive than CATI on a question-by-question basis.

The third study addressed the this question:

  1. At the classroom learning outcome level, what is the role that mobile play? Can mobile improve the immediacy of outcome data collection?

This part of the presentation related to a pilot study that the British Council carried out to test how new technologies – in this case a mobile phone app – can be used to the assess core skills at a classroom level. This app enabled assessment at the “point of learning” by teachers, peers, and by students themselves through self-reflection. It proved to be a useful, easy-to-use tool with scope for further roll-out.

In sum, these studies showed that SMS and IVR have some potential for use in survey data collection, but that representativeness is a serious concern when these data collection modes are targeting the general population. Perhaps the future isn’t quite yet what we think it is.

 

Global Thinking and Local Scones: Experience of Doing an MA in Development Studies

Written by: Alexandra Cronberg

What is it like doing an MA in Development Studies? More specifically, what is it like doing an MA in Development Studies when you are born and bred Nigerian, and have already spent fifteen years working with organisations such as the World Bank, USAID, and Lagos State Government?

Mariam Fagbemi, Head of Kantar Public Nigeria and recently graduated from her MA at Sussex University, tells us of her experience.

“I’ve been working in research-based consulting and evaluation for many years, so I was really excited to get the opportunity to do a MA in Development Studies at the Institute of Graduating classDevelopment Studies, Sussex University.  What we do as Kantar Public is very hands-on giving us a deep understanding of the “field”, whereas some of our clients have a PhD and quite a theoretical approach, so that can cause a bit of a clash!  While I realized I already knew a lot practical terms of what was being taught, this helped me put it all into theoretical frameworks. In that sense I feel I have now got both sides – the practical knowledge from my experience, as well as the theoretical approach. The MA certainly did meet my expectations in what I’d learn.

What was it like? Well, aside from the cold weather and expensive costs, it was a great experience. It was pretty intense, though. There was a lot to take in and process during the course, and I also had to keep an eye on the work going on back in Nigeria. And that’s not even mentioning trying to stay in touch with my family and friends…

Thankfully my cousin traveled down from Milton Keynes with a pile of jackets and duvets just in time for the weather change, so at least I could wrap up against the icy seaside winter. I wasn’t quite so insulated, unfortunately, against the cost of living. The un-timely fall of the Naira just before the start of my MA meant my scholarship and savings suddenly lost a third of their value. I had to live on scones and Pepsi to cope!! While the university cafeteria made them very nice and fresh, admittedly it was a bit of a slog sometimes.

In terms of the course and my classmates, it was a mix between students freshly graduated from their first degree and people like me with plenty of practical work experience from the field, including people working for NGOs and donors, and a journalist. The professors were all active practitioners, as well as doing their own academic research. We certainly had some interesting discussions in the seminars. The less experienced members of the class were keen to hear about practical experiences from developing countries, which I and a few others willingly shared. So I both learnt a lot and shared my own learnings a lot.

Specifically, I took a course on development studies generally, and then specific courses on climate change, gender & development, governance, globalisation, and poverty Picture1programming, which mostly covered social protection and microfinance. My thesis was on donor-funded “graduation programmes”[1] and how they enable people to graduate out of poverty. For example they often include entrepreneurial support, consumption support, and other types of support. Generally these programmes last 18 months. I learnt there is no silver bullet to solving development issues, but interventions stand better chances of succeeding if they don’t discount local knowledge or the lived realities of the programme beneficiaries.

Related to that, it seems like a contradiction that the importance of local ownership and involvement in development programmes is repeated like a mantra, yet most academic departments offering these courses are actually based in the global north. Studying for this masters made me realise that much more research should be done in the global south, rather than north. Why is it that there are far more options for doing an MA in Development Studies in Europe or America than in Africa, or at least outside of Nigeria?My question is, why are universities and other stakeholders not more commonly establishing e.g. Centres for Excellence of African Research at African universities, and giving scholarships for interested people to attend these centres? Why are the “best” schools of development research all based in developed countries?

Aside from concentrating relevant knowledge in the “wrong” place, it also makes it very expensive for students from the global south to learn and participate in these studies, which I experienced in a pretty visceral way, literally!

Although saying this, I should also mention that I just learnt that Kantar Public is setting up a scholarship for evaluation learning in Nairobi soon, so that is one step towards building evaluation capacity locally.

The masters degree was well worth the effort, though. The MA in Development Studies opened my eyes to how broad the field really is, indeed it is way bigger than I thought. After a year-long course I’ve only scratched the surface of development studies. I’m glad, however, I didn’t do a master’s degree straight out of university. I think you need to be a bit more seasoned, need a bit more life experience and insight into human experiences to get the most out of an MA. I’d say that is especially true for development studies.

[1] See this page, for example for further details: http://www.worldbank.org/en/results/2013/04/04/graduation-program-creating-pathways-out-of-extreme-poverty-into-sustainable-livelihoods