Yes, we should measure impact - but HOW?

The desire for outcome-based measures or judgments of non-profit effectiveness is increasing, but non-profits seem sceptical. We need to develop a clear picture of what this would involve for the non-profits, and why they do not have to fear a reductionist "one-size-fits-all" indicator that sets "one reading child" equal to a fifth of a saved hurricane victim. We also need to make it clear why such measures are desirable - and finally, we need to get them involved in moving the process forward. And the time is now...

The desire for outcome-based indicators of effectiveness from nonprofits seems to be increasing. Some recent examples/indicators:

• Anonymous blogger "Don't tell the donor" wrote in a new column in "NonProfit Times" that "Currently, donors who thirst for information on whether their donation is being spent efficiently or effectively are too often left without enough information to make those decisions. Is it any surprise that faced with an absence of any real alternative, too many donors turn to simplistic (and often misleading) fundraising ratios?"
• The interest generated by GiveWell, founded by two hedgefund managers who wanted to analyze the impact of non-profits rigorously. Unfortunately, GiveWell, which was featured extensively in the media (see, for instance, this December article from New York Times article), was a couple of weeks ago by the news that both (!!) founders had pretended to be someone else on the web in order to direct people to their service (see New York Times here)
• The lively blog "Tactical Philanthropy" by Sean Stannard-Stockton repeatedly returns to this theme (as in this post). He also started an interesting discussion about the appearance of Non-profits in Google Finance concerning how the template should be modified to be more appropriate to non-profits

The question I want to raise today is how to measure impact. It is clear that one major fear of non-profits is an oversimplified universal indicator - as evidenced by two answers from Red Cross employees given to Stannard-Stockton's question on the Red Cross Google Finance page. They compare it to "choosing a favorite child" and are evidently afraid that valuable things will be left undone because they are low on some universal metric. "How would we compare the community impact of, say, a unit of blood, a person learning a lifesaving skill like CPR, a person sheltered in a disaster, a meal delivered to someone cleaning up after a disaster, etc.?"

The first thing we should tell the non profits is that, of course, there is no universal metric! Teaching a child to read is important. Giving a pregnant mother nutritional supplements to avoid premature childbirth is important. Providing disaster relief after a hurricane is important. That's not the point. The point is: Given that you want to teach children to read - how do you know that you are doing a good job, that you are spending your funds in a wise manner and using the appropriate teaching methods, partnering with communities in a way that makes new procedures stick long term, and so on? Given that you want to provide nutrititional supplements to pregnant women, how do you evaluate whether you are doing it in the best way possible and how do you find out in what areas you could do better? Are you targeting the geographies where the problem is worst? Are you focusing only where infrastructure makes it easy to assist? And so on.

In other words: The question is not "what is the equivalent of a universal profit measure for non-profits?" (How much goodness-per-dollar are you providing?) - the question is how can we measure or evaluate or judge our success in achieving the outcomes we are aiming for - however narrowly these need to be defined to make sense?

Imagine two non-profits both trying to teach children in the same area how to read, but one using volunteer teachers from Canada and the other spreading a new teaching method to local teachers using local volunteers. Both strategies can have their strengths and weaknesses, we can have our personal convictions concerning what we think is a good strategy - but how can we find out? What data or whose judgment could be used for these two charities to discuss their alternative strategies for achieving the same purpose? What would it take to (rationally) convince one to abandon its approach and switch to the other?

Put differently: We cannot assume out of hand that every idea someone has funded is a sensible way of approaching a problem, and even good approaches can be improved. What do we use to compare two approaches and to evaluate suggested improvements?

The second thing we should tell the non-profits is that if the demand for outcome-based indicators or judgments is growing - now is the time to act. Otherwise, we risk getting a long, drawn out period with a confusing array of overlapping and poorly thought out indicators. This will also waste resources, as organizations devote time to parallell development of indicators, and administrative and personnel time in collecting, aggregating and publishing information that ends up being less valuable than it could have been.

My suggestion for what we should try to achieve:

• End state: A taxonomy of different outcomes, a Dewey-decimal system for goodness if you will, a catalogue that you can use to look up the cause you are interested in and find well-thought out and thoroughly discussed indicators or ways of evaluating outcome and effectiveness.
• Process: An open, collaborative, transparent, discussion-based process involving the non-profits themselves! This involvement is crucial if they are to adopt the measures - their concerns and knowledge will be critical to making the measures practical and relevant. At the same time, I don't believe they can do this alone - they need to get new ideas, be pushed, etc.
  
• Next step: The next thing to do would be to define a simple structure for the discussion/process. It might be to create a WIKI and define a hierarchy of users so that some features can be locked in as time goes by (a continually disrupted way of organizing causes would just be confusing), while also providing a way for any question to be raised and rediscussed. All that would need to be done is to
  
• Define a simple set of rules for operation (who is in charge and determines the hierarchy of users, what are the rules of discussion and conflict resolution, etc.)
  
• Set up the WIKI
• Make it known and start contributing - working especially hard to make sensible, valid indicators for some of the more common aims - and working especially hard to involve some of the larger, international organizations working towards those aims

Maybe this could be a new focus for GiveWell that would allow it a useful role in a project perceived less "in your face" and judgmental by non-profits (one of their major problems was getting answers to their requests for information - such requests may be seen as less of a burden if that information was collected to isolate a few, sensible parameters that could be the focus going forward).

Anyway - this is an idea, not a finished solution. Let me know what you think.