Evaluating In The Global South: A Canadian Evaluator’s Perspective
March 2020
This article is rated as:
Up until one month ago, my evaluation experience was firmly rooted in Canada. From my academic endeavours to my career as an evaluator, I carried a perspective very specific to Canada and the Global North.
Like many young (ish) evaluators, I wanted to get international experience – more specifically, I wanted to evaluate a program in the Global South. Would it really be that different? Would I apply the same approach and methods? These were the types of questions I optimistically asked myself as I packed what I thought would be practical clothes and savvy travel gadgets before my long flight to Siem Reap, Cambodia. I would soon find out that I was incorrect in my assumption to copy and paste my evaluation perspective, as well as my packing list.
One month later, as I cross my fingers and hope that the power stays on long enough for me to send some emails, I am finally able to reflect on my experience so far. In no particular order, I am sharing six of my learnings to date. Although some may come across as cynical or potentially naïve, I share them as I am confident they will make me a better evaluator in my current role, as well as when I am back in Canada.
Lesson #1
Context is Everything
Cambodia and the Khmer people captured my heart as a traveller, so naturally, I jumped at the opportunity to volunteer at a healthcare NGO based in Siem Reap. Learning about a country and its history has always been of interest to me, but now I understand that it is also imperative to be able to do any meaningful evaluation work.
Cambodia is a country with a very recent history. One doesn’t have to go too far back in time to hear the horrific stories of the Khmer Rouge, genocide, and war. And there are constant reminders of the obstacles that this country continues to face – from landmines to the loss of a generation. For example, the country was left with only 32 physicians for a population of approximately 6.6 million after the genocide (Santini, 2002), essentially forcing them to restart their medical education system.
The impact of these events is evident in the way people communicate, build relationships, and set priorities. For example, health care appointments are often done with an open door and multiple family members in attendance – a sign of the process of rebuilding trust with both individuals and the organizations they represent. Likewise, when conducting interviews in villages, the majority of time is spent being welcomed into a home or establishment and offered food, then once a relationship is established, the actual gathering of information can begin. Again, I can’t emphasize enough that as an evaluator, it is vital to understand the context. I started to realize this on day one and feel that it may be the biggest “take-home” from this entire experience.
As evaluators, we are trained to ask questions – we need to do this before we even get close to starting our work. Don’t be afraid to ask questions and continue to learn.
Lesson #2
Feasible, Not Perfect
Capacity, does it even exist? This question is not unique to the Global South, I think it is inherent to most organizations – but I am reminded of it as I navigate my way through my current project. For example, it is great to recommend a new data collection tool, but is it actually possible to implement? Or worse, if implemented, will it take up more time than it should for frontline staff?
Coming from a Consultant position before this project, I aim to submit watertight deliverables. I still have this perfectionist mentality, but it isn’t always in the best interest of lower-resourced projects. I am learning that in a low-resource setting, a finished product is often better than a perfect product.
Lastly, it is important to create or implement changes that are not only feasible but sustainable. NGO’s, particularly in the Global South, have quite a bit of turnover. A question I am trying to ask myself regularly is “Will the documents or processes I implement be easily understood by future staff and volunteers?”
Lesson #3
Champion Evaluation
In my experience so far, I rarely hear about evaluation without it being tied to monitoring. Monitoring & Evaluation (M&E) is often thrown into funding agreements or annual reports, but I would argue that it isn’t really evaluation, but rather the presentation of inputs and outputs.
There is a desire to present the impact of the work NGOs are doing, but in reality, they rarely get past transactional or process data. Again, this is an issue in both the Global South and North, and one I experience daily in my current role.
Rather than correcting definitions (output versus outcome, anyone?), I have decided to adopt an educational approach. I think that when evaluators find themselves in a setting where evaluation literacy is low, it is our job to explain the difference between research, monitoring, and evaluation. We need to champion the use of our profession to help programs communicate what they are doing (and at my NGO, it really is impressive!), and then hopefully, this will help them land more funding and do more good.
Lesson #4
Be Patient (but seriously)
I already mentioned the inconsistent electricity – one prime example of something I took for granted. I was not the best version of myself the first few times I sat in the dark wondering if it would be hours or days until I could charge my laptop. Fortunately, I work with people that learn to use that time to grab a coffee, stretch, and catch up with other staff. They are rock stars when it comes to having patience, and I really hope it will rub off on me.
NGOs have big goals and are eager to make a difference. However, it doesn’t always line up with what processes are in place. For example, I was asked to help my NGO better understand their donors. They have a donor database, so I figured it would be a ‘quick win’. Wrong – data entry processes appeared to have changed every year, and Excel spreadsheets became preferable to the actual database. So it was two steps back, which in this case, was creating a process map to understand data entry processes.
Lesson #5
Unexpected Influence
I’d like to think that as an Evaluator I am pretty aware of bias. However, I have been introduced to potential biases that I have never considered in the past. For example, Westerners or ‘Barang’ as Cambodian’s say, introduce a lot of bias when engaging in research or M&E. Unsurprisingly, we also come with our own set of ideas and beliefs. Again, not a revelation, but I am more aware of this than I have ever been in the past. I have to actively try and ignore the KPIs and evaluation questions I became accustomed to at home, as chances are, they are not nearly as meaningful for this context. For example, most health care measurement in the Global North revolves around topics like quality, efficiency, and safety. I initially took this same approach, but quickly learned that access and availability of expertise were more important to understand before diving into the more traditional KPIs. Moreover, the larger concern is getting health services to rural communities and training local staff to sustain access.
I am fortunate in that I am being exposed to methods that help minimize our bias. For example, Participatory Action Research or PAR is actively used at my NGO (Baum et al., 2006). It may not be a new method, but it is definitely one that I am eager to learn more about.
Lesson #6
Take It All In!
I saved my favourite lesson for last. If you are lucky enough to work with an NGO in the Global South – take it all in (seriously). Get lost in markets, visit local art galleries, and try to pick up a few phrases (even if it is embarrassing, like my Khmer). I have never been so out of my comfort zone, but I’m loving it.
I would argue that engaging in the culture will not only give you some great stories, but it will also make you a better evaluator.
Sources
Baum, F., MacDougall, C., & Smith, D. (2006). Participatory action research. Journal of epidemiology and community health, 60(10), 854–857. https://doi.org/10.1136/jech.2004.028662
Santini, H. (2002). Rebirth of the health-care system in Cambodia. The lancet, 360, s57-s58. Retrieved from: https://www.thelancet.com/pdfs/journals/lancet/PIIS0140-6736(02)11824-1.pdf