Dear Journalists: I know I’ve been hard on you in the past (and yes, you kinda did deserve it). But I’m turning over a new leaf. For the second time (not than anyone’s counting), I’m going to try to be helpful. Here is the first in a series that I’ll add more posts to over time about how you can get the most of your visit to an aid program or project:
Understand that you cannot evaluate a project, program or organization during one-day visit. I’m going to say it again: relief and development work are complicated. Just as it takes us concerted effort over time to understand a context, so it also takes an outsider (e.g. you) time and effort to understand what is going on in a program in the field. You can certainly gain impressions, and below I’ll share a few tips on what to look for that will help you gain a more informed opinion. But in general I’ll say that to spend one day or even one whole week with an organization’s relief team in the field and then print declarative statements about whether they’re doing well or doing poorly, or whether the overall relief effort is succeeding or flagging – and whether your statements are positive or negative – is plain ludicrous. Things are very often not what they seem to outsiders. Things that look chaotic might simply be complex. And in the context of a short visit, both success and failure can, to an untrained eye, appear as the other.
It’s fair to ask for copies of evaluation documents, but be aware that many organizations will demure from providing this outright. Ask whether the evaluation was internal or external. Also be aware of where you are in the chronology of a program or disaster response: there may not be an evaluation available.
Ask about the learning. Good aid learns from experience. A good relief manager or program director or media spokesperson should be able to articulate how the current program or relief effort is based on learning gained from previous experience. Don’t be surprised if the learning expressed is incremental – development and relief evolve slowly over time through a series of tweaks and fine-tuning from one program or relief effort to the next. An NGO that can’t articulate specifically what’s been learned in the past that’s being applied now should raise a red flag in your mind, as should one that claims to have it all the way right, now.
Ask whether learnings have been published or if there are plans to publish them. Ask whether the organization has already or has plans to participate in a multi-agency learning event or interagency evaluation (fairly common following large disaster responses).
Ask about the process used to design the program, project, emergency response, etc.: Here you are specifically looking for evidence a couple of things: First, you want to hear that local people were involved in articulating the need as well as the design of the project or relief program. Second, you want to hear that there was, in fact, a process: there was an actual assessment (not just a sort of willy-nilly mix of observation and the odd interview), there was an actual program design exercise that involved the analysis of assessment data.
Larger organizations may have their own in-house organization-specific process(es) and an accompanying set of tools (available online in many cases). They’ll most probably be happy to provide this, but be aware that your media colleagues won’t have the documents at their fingertips – they’ll have to track down a programs person for this. Smaller organizations may not have their own model/tools, but should still be able to talk to you in specific terms about the processes that they follow for assessment, program design, monitoring and evaluations.
Ask about outcomes. Someone should be able to tell you what the expected outcomes of a relief or development program are. Or, in hindsight, what they were. In development programs this is frequently expressed as a percentage of change in something (infant mortality decreasing, literacy rates increasing, etc.), while in relief programs this is typically expressed in what we call outputs – numbers of something (number of transitional shelters put up, the number of families with access to clean water, etc.).
It’s fair to ask why proposed outcomes or outputs may have been underachieved (if your conversation is after-the-fact), but don’t assume that underachievement equals failure or incompetence. Hear the explanation. Prices change, overall context evolves, and aid worker’s understanding of situations deepens – all of which potentially affect the outcomes of an aid project or program.
It is becoming increasingly common for organizations to make their evaluation documents available externally. It’s fair to ask. If they won’t share a full evaluation document, ask for an executive summary.
Use logic. Understand Pythagorean logic. Understand that correlation does not equal causation. Know the difference between issues that humanitarian aid providers can fairly be expected to address or be held responsible for, and those that they cannot (e.g. cholera in Haiti = not the fault of NGOs). Don’t confuse anecdotes with data, or data with evidence, or evidence with proof. Understand the difference between simple and simplistic. Beware of magik bullets and one-size-fits all solutions: Anyone who’s selling their widget or approach as the thing that will solve the world’s problems deserves closer questioning around learning, and process. Watch out for solutions in search of problems.
Ask the “why this, not that?” question. It’s fair for an NGO to define it’s focus or niche in a particular sector or place. I tend to be skeptical, though, of an organization that only does one thing (only one tool in it’s tool box). There should be a logical explanation for why they’ve chosen to do what they’re doing in the place where they’re doing it.
Understand ambiguity. With few exceptions, neither successes nor failures are total. Even the most stellar, most widely acclaimed aid organization or approach or program has areas that didn’t work well, aspects for improvement. See also “ask about learnings”: Any organization or project that presents a 100% rosy picture probably need deeper scrutiny. On the other hand, despite bold headlines, few aid programs are unmitigated failures. Thing are never cut-and-dried.
Ask for descriptions of the context. Where programs seemed very successful, it’s fair to ask about challengs and learnings. Where programs seemed to have failed or been extremely marginal, it’s fair to ask why. Also ask about learnings. Also ask about areas where there may have been unexpected successes (although these might not have substantially affected the overall outcome).
Understand that things are almost never the way they seem at first blush. Recipients of aid, local authorities, and local partners can all have many reasons for telling you that they loved project X when in fact they hated it, or vice versa, none of which are related to the reality of what project X is or does or did. Understand that there is a ubiquitous dynamic called “screw the outsider”, and understand – further – that you may be variously treated an outsider or an insider due to factors over which you have precisely zero influence. In very simple language, just because a local person goes on about how much they love project Z, isn’t proof that project Z works or is “good” or even that that person really loves it.
Frustrating? Yes, welcome to our world. And also, see the very first point: it takes time to triangulate information; it takes experience and expertise and also time to be able to figure out what’s going on.