ACADEMICS: Please stop making pronouncements (btw, thanks for nothing…)

26 Jun

I try. I really do try to not go negative and make things personal. But lately my patience is wearing rather thin.

Despite a few random comments in support of it, Chris Blattman’s opinion as shared in this incredibly unhelpful and – frankly – offensive post is a classic example of what often feels wrong to me with the international relief and development industry.

Out of all of it, out of all the well-intentioned but utterly clueless mom-n-pop charities who run amuck, of all the containers of donated used clothing and old computers, of all of the NGO culture that makes it hard to call lame ideas just that, of all the faffing strategies around ethereal and nearly impossible to measure concepts, of all the dumb-ass ideas from individual donors (ideas that usually somehow involve their own personal involvement in the field), the thing that drives me around the bend about the fastest is this:

Professional evaluators, often academics, frequently with CVs showing a glut of UN or WB or for-profit sector experience and a dearth of NGO experience, pitching up to spend exactly five minutes (alriiiight… maybe an exaggeration) evaluating a program run by an NGO. This person, with the coat-of-arms of some bastion of higher learning on her or his business card (and/or perhaps also a book with a catchy title to her or his name) will conduct a few cursory “focus group interviews”, review a few documents, and then retreat to the white Landcruiser to tweet a few pithy/ironic/in-joke-humorous comments from her or his iPhone.

Don’t misunderstand: I am absolutely not against the rigorous external evaluation of aid and development programs. Properly done evaluations are needed on a number of levels. Nor am I saying that “training” is by definition a good thing. Any intervention or activity can be planned and/or carried out poorly.

What I am against is some guy from Yale (ooohhh aaaahhh) bouncing into town for a quick look-see and then pronouncing for the world to hear that training is a bad thing and that NGOs should just stop doing it. Because anyone in the industry and who is also half awake knows that this is an absolutely ridiculous conclusion. Maybe the program that Chris is evaluating is horrible and should be shut down immediately. Maybe his implication is correct: perhaps in the instance that he’s looking at all that effort should be put into infrastructure instead (are you suggesting cash-for-work, Chris?). Or maybe he’s just being a smart-ass and in the process and falling for what Andrew Brody, host of my favorite podcast from… Princeton would call a “part-to-whole comparison flaw.”

There is another dimension as well that I’ll go ahead and be vulnerable and share. It particularly irks me to hear this kind of pronouncement from someone who, as far as I can tell, has never had to actually manage a development program or project. There are always multiple facets to every development or relief project, regardless of the overall conclusion of the evaluation. Someone who has never been faced with the task of writing a logframe that both met donor requirements and also meaningfully reflected reality on the ground; who has never had to personally implement and report on the outcomes of a program planned by someone else; someone who has never worked for an NGO; someone who has never her or himself had her or his program evaluated by some interloper with the coat-of-arms of some bastion of higher learning on his business card… is necessarily blind to huge pieces of the overall picture.

(And by the way, I see the “has never worked for an NGO” part to be particularly important. Many who have never worked for NGOs seem to assume that UN and WB staff as well as people who just happen to have read a lot of books about aid work and get consultancies to evaluate programs are “aid workers” in the same sense that NGO staff are. In fact they are worlds apart. The fact this post was ever written underscores that point.)

Maybe I totally misjudge Chris Blattman. It would be easy enough for him to prove me wrong by sharing on his blog exactly how many years he’s spent with an NGO in some kind of program implementation role.

* * *

I guess what bothers me the most, though, is not just a particular post on a particular blog.

What bothers me is that so many supposedly credible voices seem to put so much effort into sharing with the world what, in their opinion, does not work. Unfortunately, for most of us actually in the aid industry (as opposed to commenting on it from the outside) such analysis is largely redundant. Speaking personally and for those within my own circle of aid-worker friends and close colleagues, I can say that we’re crystal clear on what doesn’t work and in many cases even why it doesn’t work. More blanket criticism, even criticism that tries to be witty, is just plain not helpful.

Can somebody, preferably somebody who has actually implemented something, preferably a practitioner rather than a professional evaluator or skeptic, talk about what does work?

9 Responses to “ACADEMICS: Please stop making pronouncements (btw, thanks for nothing…)”

  1. Vasco Pyjama 28 June, 2009 at 3:39 pm #

    From a practitioner, I have to confess to being rather surprised by aid bloggers who have strong opinions about what works or what doesn’t. Not necessarily specific to Chris Blattman’s post (though I can see both your point and his), but in general.

    You ask what works. I don’t know. My experiences are more about various shades of grey. And it is all so context specific. One can do everything according to ‘best practice’ standards, and it would perfectly in one place, and fail disasterously in another. So much about development and humanitarian practice is about ability to learn and be adaptable. To realise that there are no absolutes.

    • J. 28 June, 2009 at 8:42 pm #

      Thank you very much for commenting, Vasco. You’ve described almost verbatim my own perspectives on what does work. In fact, “Shades of Grey” is (or now probably was…) the tentative title of an already partially written post on this very subject.

      No absolutes; everything is context-specific…

  2. Phil 30 June, 2009 at 4:57 am #

    Two thoughts.
    I have done lots of training, mostly in Afghanistan, mostly in local languages etc. And most of it has been pretty good. But there is a largely unquestioned assumption that training leads to changed behaviour. Not so, I now think. I now work on the arithmetic that training + access to resources (ie, whatever is needed to implement the change) + motivation = changed behaviour. While NGOs can provide the first two in this equation, the third, motivation, cannot be externally delivered. Impediments to enthusiasm can be removed (ie, insecurity, local stressors), but you can’t whip up motivation to change for your target group.
    Second, having done plenty of evaluations in Afghanistan, I am convinced that the best evaluations are done by people with nearly parallel experience in the subject (not just academic, but practical), plus local language and cultural knowledge. I have passed up some lovely evaluations in East Timor, because I don’t know enough about it. Now sure, some evaluation by truly external people who have never been there, never done it, don’t know the language or culture but who are innately smart or well educated can be useful, but rarely are they critical enough to really reveal programmatic flaws, discern new directions or assist in strategy.

    • J. 30 June, 2009 at 10:50 am #

      Great points, Phil. Thanks so much for commenting!

  3. Texas in Africa 1 July, 2009 at 5:54 am #

    Okay, let me get this straight. I spent nine years learning the language and developing expertise in evaluating programs – and did fieldwork over a period of two years to actually do so through various means, including collecting statistics, getting local perspectives on how well (or poorly) organizations do their job, and developing a scale to measure programs against one another fairly – but I don’t know what I’m talking about because I’m not a practitioner? Who should be evaluating these programs? You clearly need someone who’s outside of the process and able to give an unbiased evaluation that isn’t based on the NGO’s intentions or the donor’s interests. And most practitioners unfortunately aren’t trained to do these things in a rigorous, scientific manner.

    I totally understand the frustration with an outsider telling you how to do your job. But don’t dismiss all of us as “people who fly in for one week” and then claim to understand the region. Academic training doesn’t work that way, especially outside of economics. I don’t know Chris Blattman personally, but he’s not one of those academics who pretends to know all about ending poverty based on a one week visit. From everything I can glean from his professional work (which, if you haven’t had the chance, it would be good to read before dismissing everything he says), he spends a lot of time working with local agencies, getting to know their staff and the way they’re perceived by the community, and implementing studies that actually measure what they claim to measure. How is that bad for the people you seek to serve?

    • J. 1 July, 2009 at 9:28 am #

      Hey Texasinafrica – thank you for reading. And thank you very much for commenting. You make some excellent points, ones with which I do agree. Let me further clarify that I completely agree with the need for a variety of external voices to be involved in the evaluation of aid programs, for all of the reasons you give (and then some). I absolutely do not think (and did not mean to imply) that non-practitioners have nothing to add.

      This said, I do feel very strongly that:

      All of that analysis and critique – and I mean this broadly, not just Chris Blattman’s one post – has got to lead somewhere. At the end of the day it has give the practitioners something that is, you know, practical. Using Chris’ post as an example, it’s all good and well to declare training or particularly “peace training” to be a bad thing that all NGOs everywhere should immediately cease forthwith. Maybe that’s a correct analysis. But if so, then what? Which leads to…

      There needs to be some balance, some proportionality. Whether online or in print there is certainly no shortage of material out there on what’s wrong with aid, from high-level philosophical critiques right on down to picking apart data collection at the project level. Every time I turn around, it seems, there is another book or article or blog post up about how some dumb NGOs totally fubared something. But there is comparatively very little out there about what is working and what should be replicated. I’m not saying that academics should shut up and go away. I am saying – again – that those of us actually doing it need proportionally more discussion about what to do. Further, although anyone with knowledge of the industry can theoretically offer insight on what to do, I’m not quite sure I agree that The Academy is the best source of such insight.

      Finally, I see a responsibility element at play here as well. Perhaps this is where I have the biggest issue with the Chris Blattman’s Blog post now under discussion. Not so much that he is for or against a particular type of intervention, and not even that I disagree with what he’s saying (overall, philosophically, I think he and I would converge more than we would diverge). But it is a reality that he speaks from a very high platform, and also that his voice will be carried farther than those of many others. When Chris Blattman, associate professor at Yale, former management consultant and accomplished researcher speaks, people listen. And they take him seriously precisely because he is Chris Blattman, associate professor at Yale… And so when someone with that kind of high visibility makes pronouncements like “NGOs: Please stop training…”, I’m sorry – that needs to be called out.

  4. Ian 2 July, 2009 at 8:12 pm #

    Well said. I’m generally a fan of Cris Blattman’s writing but was also a bit put out by this particular post.

    I think you are right that when a well known aid researcher makes a pronouncement on a particular issue it can carry a lot of weight, even if it’s not that well thought out, so I think he has a greater responsibility not to make sweeping judgements of this type.

    I think this type of statement comes also from the current tone of the aid debate which apart from being (for the the most part) focused on the negative aspects of aid also tends to polemics and generalizations overall (aid works vs aid doesn’t work, bednets should always be free vs always paid, private vs. public etc.) probably because “it depends” and “it’s a bit of both” aren’t as eyecatching and emotive as debates between absolutes.

    I fully agree that we need to be doing more to identify things that work, and then to disseminate them so they can be adapted (not adopted) by others. This is something we are trying to do in a modest way in the organization where I work – but in my case as an outsider (from the UN in headquarters) but working closely with “practitioners” to help understand, document and share what they know. The reason we are supporting this work “from afar” though is that practitioners are often too busy, or do not have the right skills to document and disseminate what they do without help.

    • J. 2 July, 2009 at 8:25 pm #

      Hey there – thanks for commenting. Truth be told, I’m a fan as well. Or at least like his blog enough to keep it in my blogroll. We all say the wrong thing sometimes.

  5. Alanna 8 July, 2009 at 12:38 am #

    I agree completely with Vasco about the shades of grey. It’s very hard to see anything that works and can be made into some kind of universal guidance. I’ve been trying to write about that on my blog lately – ways to do this better – and it’s hard thinking and slow going.

Pearls of wisdom

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: