Evaluating in the Developing World: A Canadian Evaluator’s Perspective
This article is rated as:
Up until one month ago my evaluation experience was firmly rooted in Canada. From my academic endeavours to my career as an evaluator, I carried a perspective very specific to Canada and the developed world.
Like many young (ish) evaluators, I wanted to get international experience – more specifically I wanted to take on the challenge of doing evaluation work in a developing country. I’m a Credentialed Evaluator, how hard could it be? Would it really be that different? Would I apply the same approach and methods? Would evaluation be welcomed with open arms? These were the types of questions I optimistically asked myself as I packed what I thought would be practical clothes and savvy travel gadgets before my long flight to Siem Reap, Cambodia. I would soon find out that I was incorrect in my optimism to copy and paste my evaluation perspective as well as my packing list.
One month later, as I cross my fingers and hope that the power stays on long enough for me to send some emails, I am finally able to reflect on my experience so far. In no particular order, I am sharing six of my learnings to date. Although some may come across as cynical or potentially naïve, I share them as I am confident they will make me a better evaluator in my current role as well as when I am back home in Canada.
Lesson #1
Context is Everything
Cambodia and the Khmer people captured my heart as a traveller, so naturally I jumped at the opportunity to volunteer at a healthcare NGO based in Siem Reap. Learning about a country and its history has always been of interest to me, but now I understand that it is also imperative to be able to do any meaningful evaluation work.
Cambodia is a country with a very dark, recent history. One doesn’t have to go too far back in time to hear the horrific stories of the Khmer Rouge, genocide, and war. And there are constant reminders of the obstacles that this country continues to face – from landmines to the loss of a generation. For example, the country was left with only 32 physicians for a population of approximately 6.6 million after the genocide (Santini, 2002) essentially forcing them to restart their medical education system.
The impact of these events is evident in the way people communicate, build relationships and set priorities. For example, health care appointments are often done with an open door and multiple family members in attendance – a sign of the slow process to rebuilding trust with both individuals and the organizations they represent. Likewise, when conducting interviews in villages the majority of time is spent being welcomed into a home or establishment and offered food, then once a relationship is established the actual gathering of information can begin. Again, I can’t emphasize enough that as an evaluator it is vital to understand the context. I started to realize this on day one and feel that it may be the biggest “take home” from this entire experience.
As evaluators we are trained to ask questions – we need to do this before we even get close to starting our work. Don’t be afraid to ask questions and continue to learn. In my case I was rewarded with a country that is happy to share their stories and perspectives which has given me the foundation to now ask better evaluation questions.
Lesson #2
Feasible, Not Perfect
Capacity, does it even exist? This question is not unique to the developing world, I think it is inherent to most organizations – but I am reminded of it as I navigate my way through my current project. For example, it is great to recommend a new data collection tool, but is it actually possible to implement? Or worse, if implemented will it take up more time than it should for frontline staff?
Coming from a Consultant position prior to this project, I aim to submit watertight deliverables. I still have this perfectionist mentality, but it isn’t always in the best interest of low resource projects. I am learning that in a low resource setting a finished product is often better than a perfect product.
Lastly, it is important to create or implement changes that are not only feasible but sustainable. NGO’s, particularly in developing countries, have quite a bit of turnover due to short term expat contracts. A question I am trying to ask myself regularly is “Will the documents or processes I implement be easily understood by future staff and volunteers?”
Lesson #3
Champion Evaluation
In my experience so far I rarely hear about evaluation without it being tied to monitoring. Monitoring & Evaluation (M&E) is often thrown into funding agreements or annual reports but I would argue that it isn’t really evaluation, but rather the presentation of inputs and outputs.
There is a desire to present the impact of the work NGOs are doing, but in reality they rarely get past transactional or process data. Again, this is an issue in both developed and developing countries but one I experience daily in my current role.
Rather than correcting definitions (output versus outcome, anyone?), I have decided to adopt an educational approach. I think that when evaluators find themselves in a setting where evaluation literacy is low, it is our job to explain the difference between research, monitoring and evaluation. We need to champion the use of our profession to help developing countries communicate what they are doing (and at my NGO it really is impressive!) and then hopefully this will help them land more funding and do more good.
Lesson #4
Be Patient (but seriously)
I already mentioned the inconsistent electricity – one prime example of something I took for granted. I was not the best version of myself the first few times I sat in the dark wondering if it would be hours or days until I could charge my laptop. Fortunately, I work with people that learn to use that time to grab a coffee, stretch and catch up with other staff. They are rock stars when it comes to having patience and I really hope it will rub off on me.
Patience isn’t just required to live in a developing country, it is needed for every project and every task. I have learned that often it is two steps back before ever moving forward. NGOs in the developing world have big goals and are eager to make a difference (a lot like NGOs back home). However, it doesn’t always line up with what processes are in place. For example, I was asked to help my NGO better understand their donors. They have a donor database so I figured it would be a ‘quick win’. Wrong – data entry processes appeared to have changed every year and excel spreadsheets became preferable to the actual database. So it was two steps back, which in this case was creating a process map to understand data entry processes.
Lesson #5
Unexpected Influence
I’d like to think that as an Evaluator I am pretty aware of bias. However, I have been introduced to potential biases that I have never considered in the past. For example, Westerners or ‘Barang’ as Cambodian’s say, introduce a lot of bias when engaging in research or M&E. We can be mistaken as a potential donor or funder – this might put people on edge or cause them to only show the good stuff (which isn’t great when you are trying to improve a program).
Unsurprisingly, we also come with our own set of ideas and beliefs. Again, not a revelation but I am more aware of this than I have ever been in the past. I have to actively try and ignore the KPIs and evaluation questions I became accustomed to at home as chances are they are not nearly as meaningful for this population. For example, most health care measurement in developed countries revolves around topics like quality, efficiency, and safety. I initially took this same approach but quickly learned that access and availability of expertise were more important to understand before diving in to the more traditional KPIs. Moreover, the larger concern is getting health services to rural communities and training local staff to be able to sustain access.
I am fortunate in that I am being exposed to methods that help minimize our bias. For example, Participatory Action Research or PAR is actively used at my NGO (Baum et al., 2006). It may not be a new method, but it is definitely one that I am eager to learn more about.
Lesson #6
Take It All In!
I saved my favourite lesson for last. If you are lucky enough to work with an NGO in a developing country – take it all in (seriously). Do as the locals do, try the food, get lost in markets, visit local art galleries, and try to pick up a few phrases (even if it is embarrassing like my Khmer). It can be overwhelming, such as dodging scooters in 38°C whenever I need to get food, but it is worth it. I have never been so out of my comfort zone, but I’m learning to love it.
I would argue that engaging in the culture will not only give you some great party stories but it will make you a better evaluator.
Written by: Samantha Larose
Sources
Baum, F., MacDougall, C., & Smith, D. (2006). Participatory action research. Journal of epidemiology and community health, 60(10), 854–857. https://doi.org/10.1136/jech.2004.028662
Santini, H. (2002). Rebirth of the health-care system in Cambodia. The lancet, 360, s57-s58. Retrieved from: https://www.thelancet.com/pdfs/journals/lancet/PIIS0140-6736(02)11824-1.pdf