Here at Panoply Digital, we work with a number of different clients on the monitoring and evaluation of their mobile services: start-ups / clients from the private sector, clients from the development sector such as NGOs, or with mobile operators. I’ve written before not only about the challenges of doing monitoring and evaluation (M&E – also known as MEL: monitoring, evaluation and learning) for mobile products and why a lot of organisations hate doing it, but also why it is so important to try and build the evidence base and proof points of what works and what doesn’t in this (relatively) new field. Based on our experience, here are 5 questions to ask yourself when doing M&E with a mobile or ICT product or service:
1. Have you planned (and budgeted) for it?
A lot of the funding from pilots and testing out mobile solutions in development comes from donors – and donors want to see evidence and results. Unfortunately, for a lot of start ups or mobile operators, this means doing M&E in a way that they’ve not done before, and often weren’t expecting to have to do – so they don’t plan for it and it gets shoe-horned in at the last minute. If it’s planned (and budgeted for) from the very beginning, when the project is being mapped out, then it is much more likely to be done well. Sounds obvious, but you’d be surprised at the number of organisations that don’t – and then M&E becomes a nightmare.
2. Are you being pragmatic?
One of the great things about ICT4D is that it brings together development organisations and the private sector – who can both learn so much from each other. One of the challenging things about ICT4D is that these sectors are so different – and have very different ways of doing things. The development partner may want to do a large-scale, academic, longitudinal study over a number of years with users of their mobile service, whereas the private sector partner just wants to do one quick and dirty survey with the users to tick the box. My course of action is usually a pragmatic one – design a methodology that is robust and stands up to academic scrutiny (within reason), but doesn’t require a sample size of 10,000 users and half the total project budget. There is a middle ground and you do need to be pragmatic – having an M&E advisor who understands both the development and the commercial side is very useful in helping the two parties reach that compromise.
3. Can you borrow anything from the private sector?
That being said, I think that the development sector has a lot of things it could learn from the private sector’s way of doing things. There was a great blog post recently on ICT Works on doing M&E the Cisco way – incorporating customer satisfaction surveys and feedback loops into your M&E to understand what users think, and gauge their satisfaction levels, something more common to the private sector than the traditional development sector style of M&E. It’s something I incorporate into all of my M&E – and it tells you so much about your product (and also helps private sector partners measure their own internal KPIs).
4. Are you fully utilising your data analytics?
Along a similar line, I do think that mobile and ICT projects actually have a great advantage when it comes to M&E compared to more ‘traditional’ development sectors – the fact that there is a backend system to draw more detailed insights. From a backend dashboard you can understand things like user engagement, user retention or pain points in real time, which give extremely valuable information about how your mobile service is performing and allow you to course correct – and yet it’s something that actually isn’t taken advantage of as much as you’d think, and is what GSMA Mobile for Development Impact calls a ‘missed opportunity’. Interestingly, it’s actually NGOs that are leading the charge here: a great example is HNI, who are using big data to evaluate and improve their 321 mobile service for women.
5. Have you incorporated qualitative insights?
That being said, customer satisfaction scores and data analytics will give you great insight into what is happening, but not insights into why those things are happening. It’s all very well knowing that user engagement drops off after the first IVR module, but until you understand why that is, there may not be much you can do about it to improve this situation. If you build in opportunities to talk to your users, through focus group discussions, individual interviews or even customer feedback loops, then that will give you the depth of understanding as well as the breadth. One of the pushbacks I get from clients against qualitative work is that it can be very expensive and time-consuming. My counter-argument is yes, it can be (if you hire a research agency, for example) – but a) it doesn’t have to be if you hire an independent freelance researcher to conduct interviews rather than a research agency and b) the payoff is immense. To use the example of of HNI again, they conducted interviews amongst female users of their gender-based violence service in Madagascar – the case studies they were able to draw out of those interviews enabled them to demonstrate social impact to donors and the wider ICT4D community, which in turn enabled them to get more funding to scale up and expand their mobile service to East Africa. It may have cost a bit more upfront – but it paid dividends in the long run.
What do you think? Is there anything there you agree with, or don’t agree with? What other questions need to be asked when it comes to M&E?