Mobile technology: The future of evidence in development?

Written by: Alexandra Cronberg

The future is all about mobile technology, right? Well, perhaps, but in the context of real programme evaluations, it is worth examining and understanding the benefits and drawbacks of mobile data collection modes for gathering evidence, before waving goodbye to human interviewers.IMG-20180321-WA0002

In order to address this topic, Kantar Public and the British Council hosted a joint event which took place in London, covering three studies including one in partnership with RTI. The full slideset with the findings is available here. Read on for a brief summary.

  1. How efficient is mobile SMS vs other methods of collecting evidence from teachers in the Connecting Classroom programme?

In order to answer this question, we carried out a pilot survey among participants who had attended the British Council’s Connecting Classroom training in Ethiopia. The study collected progress data using an SMS survey, rather than using the alternatives of paper or telephone. This study showed that SMS has many benefits and some challenges: It is a cost-efficient and viable option for collecting progress data among a target population among known participants, although the response rate is lower compared to telephone and paper. Moreover, there were some unexpected challenges during the pilot, including internet downtime and a change in the MNO’s airtime bundles, which affected the administration of incentives. This highlighted the importance of allowing plenty of time for testing and piloting the survey.

The second study addressed the following question:

  1. What is the potential for using Interactive Voice Response (IVR), compared with SMS, telephone surveys (CATI) and face-to-face surveys of collecting information in the general population?

This study, which Kantar Public conducted in partnership with RTI, compared response rates and representativeness of mobile data collection modes (i.e. SMS, CATI, and IVR) with that of face-to-face interviews. All of these studies targeted the adult general population in Nigeria aged 18 to 64. The results showed that the response rate of SMS and IVR are very, very low, and even for CATI it is much lower than for face-to-face. This would not be such a problem if the achieved samples were representative of the general population. That is, unfortunately, not nearly the case. The study showed that the achieved samples using SMS and IVR are very much skewed towards better educated and younger people, and also towards men. People often think that applying statistical weights to improve sample representativeness is the solution to this problem. However, the findings showed that weighting does not solve the problem, and least when looking at voting behaviour. In fact, weighing actually increased the bias. Finally, with respect to cost, once we adjusted for questionnaire length and sample size, SMS and IVR are more expensive than CATI on a question-by-question basis.

The third study addressed the this question:

  1. At the classroom learning outcome level, what is the role that mobile play? Can mobile improve the immediacy of outcome data collection?

This part of the presentation related to a pilot study that the British Council carried out to test how new technologies – in this case a mobile phone app – can be used to the assess core skills at a classroom level. This app enabled assessment at the “point of learning” by teachers, peers, and by students themselves through self-reflection. It proved to be a useful, easy-to-use tool with scope for further roll-out.

In sum, these studies showed that SMS and IVR have some potential for use in survey data collection, but that representativeness is a serious concern when these data collection modes are targeting the general population. Perhaps the future isn’t quite yet what we think it is.


Co-operation or freeloading: What is the effect of conditional versus unconditional incentives in an SMS survey?

Written by: Alexandra Cronberg


Gifts can be a tricky business. While they may stem from pure generosity and care, they often come with sticky strings. Just ask all the companies that tightly regulate the receipt of gifts from, say, potential clients or partners. Such are human relationships that obligation and reciprocity often govern behaviour and interactions, for better or worse.

In survey research we may draw on the same deep-seated human traits of obligation and reciprocity to get respondents to complete our questionnaires. We can do this by giving an unconditional gift, i.e. incentive, in advance of asking for participation. Indeed, several studies[1] on postal surveys have shown that unconditional incentives do lead to higher response rates compared to giving a gift conditional upon completing the survey, which arguably treats the questionnaire more like a transactional exchange.

The use and administration of incentives is a particularly relevant issue for surveys making use of self-completion questionnaires, such as postal and SMS surveys: These data collection modes do not have the benefit of an interviewer who can coax respondents to take part and therefore need to rely on incentives to a greater extent.

Now, the same studies showing that unconditional incentives in postal surveys lead to higher response have also shown that unconditional incentives are actually not cost efficient. This can be due to undelivered letters or the absence of eligible respondents. Some respondents will also take the incentives, e.g. a voucher attached to an advance letter, without completing the questionnaire. Consequently, in practice there are few postal surveys that actually administer incentives unconditionally.

With the increasing popularity of SMS surveys, it is pertinent to ask whether unconditional incentives have the same effect on SMS as on postal surveys, and whether it is cost efficient or not. In particular, SMS has the advantage over postal surveys that respondents can easily opt in, meaning cost efficiency may well be improved.

In order to seek the answer to these questions, Kantar Public carried out a small experimental study together with the British Council. Read on to find out the results.

This study

The study involved an SMS survey with an experimental design to test the effect of administering conditional versus unconditional incentives. The study also sought to test the feasibility more broadly of using SMS as data collection mode to gather feedback and progress updates from British Council course participants, but that question is the topic for another blog post.

The survey was carried out among course participants in a British Council teacher training course in Ethiopia and the questionnaire comprised 16 questions. The sample consisted of 434 respondents with valid telephone numbers. Respondents were randomly allocated into one of two groups, Group A and Group B. The initial message was successfully delivered to 390 respondents (Group A: 199 resp. and Group B: 191 resp.). Each group was administered the survey as shown in the diagram below.

Group A & B

At the beginning of fieldwork, respondents were sent a message alerting them to the survey. A day later they were then sent another message asking them participate. In order to participate, respondents were instructed to first opt in by responding to the message. For Group A, the questions were then sent out followed by the incentive, provided the respondent completed all 16 questions. For Group B, the incentive was sent immediately after the respondent opted in, which was followed by the questions. The incentive consisted of airtime worth 15 Ethiopian Birr, equivalent to 0.55 US dollars.


The findings from the study suggest that offering the incentive in advance yields a slightly higher response rate compared to an incentive conditional on the respondent completing all the questions. As shown in the table below, among Group B, 25% completed all the questions whereas in Group A the equivalent figure was 21%.

These figures are broadly in line with surveys of this nature. That said, it is clear that response is still fairly low even among Group B.

How does this impact on cost efficiency? As mentioned above, one advantage of SMS surveys over postal ones is that respondents can easily opt in before any other message or incentive is sent to them. This means that unconditional incentives are only sent to respondents who have a valid telephone number and who are eligible, thus minimising loss. There is, however, still the potential issue of respondents taking the incentive without completing the questionnaire. This problem turned out to be quite a notable one in our SMS survey. Among respondents who opted in, nearly half of Group B (48%) did not complete the questionnaire. That means a large share of respondents took the incentive but ditched the questionnaire. The equivalent proportion who opted in but failed to answer all questions was somewhat higher for Group A (56%). Yet the resulting cost for the airtime incentives overall (and per completed interview) was lower for Group A since we did not allow for any freeloaders.

Putting monetary values to the incentives given to Group A and B, we can see that the total cost for Group B was ETB 15*93=ETB 1,395 (USD 59.30), equivalent to an average of ETB 29 per completed interview. This compares to a total cost of ETB 15 per completed interview among Group A, resulting in a total cost of ETB 15*42=ETB 630 (USD 26.77). Consequently, we might draw the conclusion that cost efficiency is a major concern also for SMS surveys when administering unconditional incentives.



Based on the results from this experimental SMS study among teachers in Ethiopia, we can see that unconditional incentives yielded slightly higher response compared to administering incentives conditional upon completion of the questionnaire. This finding is line with other studies, and re-affirms the view that drawing on respondents’ sense of obligation and reciprocity is more productive than treating survey participation as something of a transactional exchange.

That said, it is clear that a large share of respondents are not that bothered about reciprocity in the face of a free gift, even when first asked for their active participation. In this light, administering unconditional incentives in an SMS survey is arguably not cost efficient, with the average cost of unconditional incentives per completed interview nearly double that of the conditional alternative.

Hence, the sense of obligation and reciprocity may well be part of deep-seated human traits and behaviour, but it seems that in a context of technology and faceless interactions, many respondents will turn into freeloaders. Unfortunately for us social researchers, free airtime does not seem to come with sticky strings.


[1] See for example Simmons, E. and Wilmot, A. ‘Incentive payments on social surveys: a literature review’, published by the Office for National Statistics in the UK, 2004. See also Abdulaziz K, Brehaut J, Taljaard M, et al. ‘National survey of physicians to determine the effect of unconditional incentives on response rates of physician postal surveys’. BMJ Open 2015;5: 007166.doi:10.1136/bmjopen-2014-007166

From snoring camels to product diversification: A gendered analysis of internet participation in Ghana, Kenya, Nigeria and South Africa

Written by: Alexandra Cronberg

It is hard to find anything that offers so much hope and potential as increased internet access across Africa. The internet offers a whole new world of information, ideas, tools, and ways of connecting people as well as providing sources of entertainment and distractions, certainly with silly kittens and camels galore. Importantly, it offers revolutionising ways of accessing and delivering services, including vital ones such as finance. Recent discussions with jua kali, or informal sector producers in Kenya, showed enormous potential to diversify their product lines provided they had access to and knowledge of the internet. Enabling people at the bottom of the pyramid, who currently have little or limited internet access, to make use of all of this will be life changing.

Or so we like to think. In reality, the picture is more complex. While internet access itself may be binary, just like the data it holds, the users are intricate, inconsistent and often contradictory human beings. Indeed, internet participation cannot be reduced to zeros and ones. A paper by Kantar Public, presented at the African ITS Conference in Accra in March 2016, sheds light on the complexity of internet engagement and the factors that underpin it. The paper, authored by Nicola Marsh, is based on analysis of a global annual study of internet use conducted by Kantar TNS in a wide range of countries[1]. This particular piece of analysis focuses on Ghana, Kenya, Nigeria and South Africa.

Gender is a key part of this picture. Fewer women than men use the internet in most African countries, and these four countries are no exception. By way of example, 19% of men in Ghana have access to the internet, whereas the figure for women is a measly 9%. In South Africa, which has the highest level of internet access among the four countries, 41% of men use the internet whereas only 29% of women do so[2]. Consequently the door to the digital world remains shut for many women.

Figure 1. Internet access by country and sex, 2012


Source: Research In Africa, 2012

The KP paper analysed different levels of internet engagement and factors that underpin different types of usage. First, an overall “internet participation” composite score was created based on a bunch of common online activities and their frequency. The findings show that greater access for women, or indeed disadvantaged men, does not imply online engagement. In fact, the countries with higher levels of access tend to have lower levels of participation. Within the countries, men consistently have higher levels of engagement than women.

Figure 2. Mean score of internet participation by country and sex, 2015


Source: Kantar TNS Connected Life Survey, 2015

Second, this overall score was then broken down into three main factors or categories of usage, capturing some of the nuance of internet engagement. The categories are:

  • Popular activities. This includes instant messaging, social networking, uploading photos, playing games, reading news/sports/weather.
  • Sophisticated activities. This includes mobile payments[3], streaming/downloading shows/movies, streaming music/radio, watching videos, internet banking
  • Text heavy activities. This includes blogging, visiting blogs/forums, and emails.

The gender gap is further highlighted when looking at these different categories of internet usage, with sophisticated activities having the greatest gap.

Other factors in addition to gender that lead to greater internet participation overall are younger age, better education, and higher socio-economic group. However, different life stages, defined as student status, marital status and having kids, have no consistent impact on online participation across the four countries.

Lower education and social class have less of an impact on the popular online activities. If we want to get women and people who are less well educated to participate more, the starting point should arguably therefore be data light services.

These findings show that as online participation increases and people lower down the pyramid gain access, proportionately more people engage with the internet in lighter ways. Women are often among those who are late to join the online party. Indeed, across the four countries the gender gap for internet participation is inversely related to the level of internet access.  For example, in South Africa a more similar proportion of men and women access the internet, but among those who are online, women have a lower level of participation than men.  In contrast, in Ghana where the gender gap in access is large, the men and women who do have access have more similar levels of engagement.

In sum, this analysis makes it clear that for the internet to be a truly useful tool for disadvantaged groups of people, much more ought to be done to get women in particular to develop more technical skills and online literacy, as well as solving other affordability and access issues. If not, many of the most vulnerable people will remain excluded from the digital possibilities including access to services, information, networks and ideas. While a few tentative steps online might mean people tumble into Facebook and other social networks, it is essential they don’t just get sucked into the whirlpool of singing dogs, snoring camels and other people’s dinner from which they may or may not emerge. Rather, people need to engage with more sophisticated online activities if they are to click their way onwards and upwards. A snoring camel ain’t gonna help with that.

The full version of the paper is available on request.

[1] The analysis was based on the data from the annual, multi-country survey conducted by Kantar TNS, called “Connected Life”. The survey covers technology and internet behaviours amongst internet users. All those interviewed use the internet at least once a week, and the sample for each country is weighted to be nationally representative of weekly internet users aged 16+. The data was collected between June and August 2015.

[2] Source: Research In Africa, Gillwald et al (2012),

[3] Note that in Kenya mobile payments are commonly done using Mpesa, but the level of penetration of mobile money is much lower in other countries.