You are here: Is Acumen’s Lean Data approach a “revolution” in social impact assessment?

29/08/2017

Can the use of mobile phones radically simplify the process of collecting social impact data and cut costs? Some people claim it can. Rebecca Berry, a post-graduate student from the University of Edinburgh, spent May and June 2017 testing the usefulness of telephone surveys for gathering data on customers of solar energy companies. The research was undertaken during a study placement with Energy 4 Impact in Nairobi.

Face-to-face interviews have long been regarded as the ‘gold standard’ of measuring social impact, offering additional observational data to verbal responses. However, collecting social impact data in-person is often deemed expensive and logistically challenging for small social enterprises.

Acumen’s Lean Data approach aims to overcome the shortfalls of face-to-face interviews by utilising low-cost technology, such as short message services, interactive voice response or telephone calls. Acumen promotes Lean Data as an innovative, efficient and affordable approach to uncovering social performance data which gives a voice to rural customers. But how effective are Lean Data tools?

Practical Challenges and Limitations:

Although Lean Data telephone surveys were undoubtedly cheaper than face-to-face interviews, the study conducted in Narok, Kenya with Solar Kiosk revealed that using mobile phone technology has unexpected practical limitations and adverse consequences, which are understated by Acumen.

Lean Data’s dependence on mobile phone technology pre-determines a research audience. The very bottom of the pyramid, who may not own phones and are likely to be using ‘unclean’ sources of energy, are excluded. In particular, women may be omitted. Several women spoken to during face-to-face interviews said that they did not own a phone, but their husband did. Furthermore, observational notes reveal that it is mainly men who use phone charging facilities, confirming academic research which suggests that the use of mobile phones in Africa is not gender neutral. Technology ultimately acts as a barrier to accessing customer social impact data, as opposed to being a gateway.

 All you need to understand your social performance data is a mobile phone and the will to start. What could be simpler?

- Acumen

This research revealed that Acumen has underplayed the complexities of collecting mobile telephone numbers. For a newly established business which does not have a customer database and operates in the off-grid solar energy market, collecting customer telephone numbers is a significant challenge. Acumen recommends hosting radio campaigns or distributing flyers to collect phone numbers. However, small social enterprises may not have the finances, resources or time to do this.

Mobile penetration is relatively new in East Africa, making things like robotic-calls and scam calls less prevalent than in geographies such as India and Pakistan.

- Acumen

Acumen underestimated the power of this external factor in hindering the success of Lean Data becoming a ‘breakthrough’ in social impact assessment. A recent report by Truecaller states that Kenya has the highest number of fraudulent telephone calls in the world. This external factor lowered the response rate in Energy 4 Impact’s research, with 22% of customers not willing to complete the telephone survey.

Response rates were also hindered due to customers living in remote rural locations with poor phone signal or limited phone charging facilities. This increased the number of telephone call attempts, thus decreasing response rates and the efficiency of Lean Data.

By asking customers directly for feedback, suggestions, or complaints, we are able to gather a richer data set that provides color and context to quantitative data.

- Acumen

Acumen’s philosophy of ‘less is more’ and strive for efficiency limits the effectiveness of Lean Data at revealing social impact information with ‘color and context’. This research revealed that misunderstandings were common during telephone surveys, forcing the enumerator to prompt the customer and thus limiting the validity of the customer’s response. In contrast, visual cues during face-to-face interviews provide contextual data that help ascertain how knowledge is being created and shared.

Observing customers in their natural helps understand customer motives and causes for behaviour. In turn, this created a customer story that portrays the social impact of Solar Kiosk. These case studies are an invaluable tool, not only for communicating Solar Kiosk’s social impact to donors and investors, but also to help Solar Kiosk understand the socio-demographic of their customers. This in-depth qualitative information was missing from telephone surveys due to the absence of a visual encounter and lack of rapport between the enumerator and customer.

Conclusions

The use of mobile phone technology disguises Lean Data as a “revolution”. However, Lean Data is a supplement to existing social impact methodologies. It is not a “revolution” that can be exclusively used in social impact assessment. Lean Data’s dependence on technology excludes the voices of those who need to be heard the most: customers at the bottom of the energy ladder using ‘unclean’ sources of energy such as kerosene.

In contrast, face-to-face interviews reveal a wealth of qualitative data which are simply irreplaceable in understanding social impact. The data gathered from face-to-face interviews is considered more valid, reliable and respectful, confirming that in-person techniques hold a justified superiority in social impact assessment.