[ Attachment content not displayed ]

[ Attachment content not displayed ]

[ Attachment content not displayed ]

[ Attachment content not displayed ]

[ Attachment content not displayed ]

I've made changes to the agenda posted at
http://www.givewell.net/research-agenda to reflect the changes we've been
making to our plans as we progress through research.  Here's a quick summary
of what's changing and why.
Education programs are taking a back seat to health and economic
empowerment.  From what I've seen, health has by far the most compelling
track record of improving lives.  We have to do economic empowerment because
we've got $250k to regrant in that area.  Education is a potentially
promising area but not in the same category as those two.  We will "do
education" (get a sense of the literature, try to find good programs) if we
have time but only health and ec empowerment are considered must-haves right
now.

This would change if there were an education program with a strong record of
success (measured in terms of standard-of-living type outcomes, not just
increasing school enrollment/completion/performance), so if you know of a
program or charity we should consider, let us know.

No more comprehensive charity database.  This work is too time-consuming and
not valuable enough.  We are going to quickly scan charities' websites
looking for those that focus on top interventions, but we are not creating a
directory of every charity's activities.  This was discussed at our most
recent Board meeting (audio to come in a few weeks).  We have yet to set the
criteria we'll use for deciding whether to check a charity's website, but we
plan to be relatively comprehensive: e.g., all U.S.-based charities working
internationally with annual expenses of at least $1m. We will record and
publish that charity-by-charity assessment.

I've reframed the "general aid questions."  Although we are still trying to
explore all the questions at
http://www.givewell.net/wiki/index.php?title=General_Aid_Questions, I've
written down the 5 questions that I think we really need to answer from a
donor's perspective: should I donate, what sorts of programs are best, what
countries are best to focus on, what kinds of vehicles are best, and bottom
line - where should I give.  All of these questions are basically best
answered through the "general aid questions" but putting them in this way
makes it easier for us to focus on what's important.

I've also created a wiki (
http://www.givewell.net/wiki/index.php?title=DWDA_Report_Outline)<http://www.givewell.net/wiki/index.php?title=DWDA_Report_Outline>
that
is the same structure re: these questions, but also has the provisional
answers (the things I believe right now) and lists things (in bold) that we
particularly need to investigate further.  This is publicly available but it
is intended for close followers of the project; it's notes to ourselves
rather than a presentation.

I've cut the part about visiting regions from the agenda summary - we still
intend to do this but it just doesn't fit in particularly well and isn't one
of the essential points.

A note on division of labor.  I am currently focused on questions 1-4, which
will be primarily answered through academic literature and conversations
with experts.  Question 5 requires in-depth examination of charities and is
Elie's current focus.  (We share work with each other throughout each day
and discuss each other's findings, but each of us is primarily focused on
one area.)

Some general comments:


"Education programs are taking a back seat to health and economic
empowerment.
...
This would change if there were an education program with a strong
record of success "

I don't have a problem with this, but I encourage you to "show your
work" behind the decision in an appropriate way on the website. 
Education is a popular area for donors to give to.  If you have
concluded that evidence for education donation effectiveness is weak,
document it so that others can see.


"No more comprehensive charity database. "
I think this is a good decision, for now anyways.  You can always
revisit this down the road if/when your resources expand.


"I've reframed the "general aid questions.
...
I've written down the 5 questions that I think we really need to
answer from a donor's perspective: should I donate, what sorts of
programs are best, what countries are best to focus on, what kinds of
vehicles are best, and bottom line - where should I give."

The link you provide still shows 16 questions.  I don't know how (or
if) you're planning to link the 5 new questions to the 16 old ones. 
I'm not crazy about the new list:

"should I donate" - For a charity guide website, the answer seems sort
of self-evident.  Anybody reading it is going to expect (and probably
receive) a certain sort of answer.  Seems a bit like a waste of
pixels, at least in that framing.

"what sorts of programs are best" - Reasonable

"what countries are best" - Seems to assumes overseas donation.  While
I lean this way and think you lean this way, I think you've skipped an
important step (Should I give locally/nationally/globally?)  Also, how
much ability to donors have to micro-select target nations - to donate
to Tanzania instead of Kenya?

"what kinds of vehicles are best" - I don't really know what this means

"where should I give" - Obviously, the big question

[Quick follow up]
I didn't see when I wrote the above that you a bit further down, you
provided a link to your draft of this.  

I think much of the information is good, but, as mentioned above, I'm
not sure the framing is quite right.  I'm not even sure if the many
questions (16 or more) that reasonably interested donors might have
should be stuffed into 5 questions, regardless of exactly what those 5
are.

In particular, I'd really like to see space allocated to the important
question of "Does aid work in general?" and the related "If we've been
giving aid in the past, why aren't things better?"  (Maybe those
aren't the exact phrasings - but something like that).  Even if you
can't provide super-clear answers (i.e. the research is nebulous), I
think you need to present the questions and, if need be, SAY that
there is a lot of research around the issue but not a lot of clear
answers.  Better to address the question and say that the answers are
unclear than to leave the question unasked (because a lot of donors
will be asking it in their minds).

I came across this short piece here:

http://kristof.blogs.nytimes.com/2009/01/01/your-comments-on-my-slavery-column/

(If the link breaks, It's Nicholas Kristof's 1/1/09 piece in the NY Times)

The piece is about sex trafficking/slavery, in Cambodia and elsewhere,
and mentions some groups that are combating it.

It occurred to me that:

A) Reducing/eliminating this practice would be a very good thing
B) Charities involved in this are likely operating at the advocacy
level rather than actually, say, sending guerrilla teams into the
field to break up these operations
C) The value and impact of advocacy is hard to measure
D) That doesn't necessarily mean that advocacy charities (for this or
other issues) aren't worthwhile or effective

More generally, there are a lot of areas where charity MIGHT have a
significant impact, but that impact is hard to measure in
quantifiable, objective terms.

If GiveWell, at some point, wished to delve into these kinds of areas
a bit, one way to do it might be to solicit essentially op/ed pieces
from interested parties - be they folks working for charities in these
areas, academics, or others, making the case for their charity (or
type of charity).

Of course, you can find these kinds of pieces elsewhere on the 'net
(including the op/ed that I linked to), but GiveWell might add some
value in this area by creating a central, structured place where a
donor could browse a number of such pieces.  

Downsides:
1) Would want/need to segregate these op/eds from primary GiveWell
content.  Op/eds written by biased parties and may lack objective
support for their positions.

2) Takes time - may be hard to get the right folks motivated to
contribute to this.

===

Perhaps file this into ideas for more community involvement at some
point to GiveWell...

In my previous message, I'd mentioned 3 books on this topic.  I'd
meant to get around to adding a few more names to the list.  I haven't
read these books - I have them on my shelf and just wanted to provide
a quick mention of the names and the kinds of info they seem to
provide (by just skimming through the pages quickly).

Don't feel obligated to read these books (I haven't read them myself)
- I just wanted to add them to the list of POSSIBLE reference material.

The Fate of Africa : A History of Fifty Years of Independence, by
Martin Meredith
As the title would suggest - basically a history of Africa in the last
50 years.  Lengthy (688 pages before the chapter notes and
bibliography).  Not specifically aid/charity focused.

The Trouble with Africa : Why Foreign Aid Isn't Working, by Robert
Calderisi
Relatively slim (230 pages before notes and bibliography).  Written by
a former World Bank official.  A fair number of personal anecdotes and
some history of efforts in the region.  Seems to have little in the
way of academic-type research behind it (not that that's bad - just
the kind of book it is.

African Development : Making Sense of the Issues and Actors, by Todd
J. Moss
Kind of a guide-book to these issues.  Includes things like a lengthy
list of acronyms (and their meanings) of African development-related
terms (CMA = Common Monetary Area, IGAD = Intergovernmental Authority
on Development), some stats and charts, and a high level overview of a
lot of development related issues.  There's a slightly-more-than-half
-page blurb on 'The "Poverty Trap" Idea' for instance (among many
other topics dealt with briefly.  Probably useful as a quick text to
bring a newbie up-to-speed on a lot of different concepts. 

Making Aid Work, by Abhijit Vinayak Banerjee
Slim book that is basically a collection of essays by a variety of
authors/experts (including the listed author) on the title subject. 
The main text of the book is only about 160-170 pages, and the pages
are small and not very word-dense.  This is probably a 2 hour or less
read, total.  A quick way perhaps to assess a range of expert opinion.  

The Bottom Billion : Why The Poorest Countries Are Failing And What
Can Be Done About It, by Paul Collier
Not footnoted/endnoted (though there is a short list of some research
at the end).  Seems maybe a bit in the style of Easterly's most recent
book, though I don't know where Collier's opinions stand.  His bio
looks good.

===

Also, one more book that I *have* read but can't locate my copy of -
Blue Clay People: Seasons on Africa's Fragile Edge by William Powers.
 Powers was a senior aid worker in Liberia for a food program during a
time where Liberia was mainly a mess.  A good first hand account of
the problems of delivering aid to a country in tremendous
distress/civil war.  



Previously mentioned (in my earlier post):
> "Does Foreign Aid Really Work?" by Roger Riddell.  Blurbs on the back
> make it sound like the author takes a middle view (neither wildly
> optimistic nor pessimistic).  Extensive endnotes and references.
> 
> "Africa In Chaos" by George Ayittey.  From backcover and VERY quick
> flip-through, seems pessimistic about Africa in general.  Seems less
> aid focused and more about general failures in Africa, especially
> political (but seems to include some discussion of aid).  Appears to
> include reasonable number of references in-line in the text, with
> "Literature Cited" at the end.
> 
> The Riddell book looks quite on-point for this topic.  The Ayittey
> book is probably more marginal, but perhaps worth consideration.

Thanks.  Just a couple notes:

   - I've read Collier's book and discussed it in the first email in this
   thread.
   - I've also read Banerjee's book - it was short and general and my notes
   are few, but I will send them later.
   - In the middle of the Moss book.


On Thu, Jan 15, 2009 at 12:56 PM, psteinx <psteinmeyer@...> wrote:

>   In my previous message, I'd mentioned 3 books on this topic. I'd
> meant to get around to adding a few more names to the list. I haven't
> read these books - I have them on my shelf and just wanted to provide
> a quick mention of the names and the kinds of info they seem to
> provide (by just skimming through the pages quickly).
>
> Don't feel obligated to read these books (I haven't read them myself)
> - I just wanted to add them to the list of POSSIBLE reference material.
>
> The Fate of Africa : A History of Fifty Years of Independence, by
> Martin Meredith
> As the title would suggest - basically a history of Africa in the last
> 50 years. Lengthy (688 pages before the chapter notes and
> bibliography). Not specifically aid/charity focused.
>
> The Trouble with Africa : Why Foreign Aid Isn't Working, by Robert
> Calderisi
> Relatively slim (230 pages before notes and bibliography). Written by
> a former World Bank official. A fair number of personal anecdotes and
> some history of efforts in the region. Seems to have little in the
> way of academic-type research behind it (not that that's bad - just
> the kind of book it is.
>
> African Development : Making Sense of the Issues and Actors, by Todd
> J. Moss
> Kind of a guide-book to these issues. Includes things like a lengthy
> list of acronyms (and their meanings) of African development-related
> terms (CMA = Common Monetary Area, IGAD = Intergovernmental Authority
> on Development), some stats and charts, and a high level overview of a
> lot of development related issues. There's a slightly-more-than-half
> -page blurb on 'The "Poverty Trap" Idea' for instance (among many
> other topics dealt with briefly. Probably useful as a quick text to
> bring a newbie up-to-speed on a lot of different concepts.
>
> Making Aid Work, by Abhijit Vinayak Banerjee
> Slim book that is basically a collection of essays by a variety of
> authors/experts (including the listed author) on the title subject.
> The main text of the book is only about 160-170 pages, and the pages
> are small and not very word-dense. This is probably a 2 hour or less
> read, total. A quick way perhaps to assess a range of expert opinion.
>
> The Bottom Billion : Why The Poorest Countries Are Failing And What
> Can Be Done About It, by Paul Collier
> Not footnoted/endnoted (though there is a short list of some research
> at the end). Seems maybe a bit in the style of Easterly's most recent
> book, though I don't know where Collier's opinions stand. His bio
> looks good.
>
> ===
>
> Also, one more book that I *have* read but can't locate my copy of -
> Blue Clay People: Seasons on Africa's Fragile Edge by William Powers.
> Powers was a senior aid worker in Liberia for a food program during a
> time where Liberia was mainly a mess. A good first hand account of
> the problems of delivering aid to a country in tremendous
> distress/civil war.
>
> Previously mentioned (in my earlier post):
>
> > "Does Foreign Aid Really Work?" by Roger Riddell. Blurbs on the back
> > make it sound like the author takes a middle view (neither wildly
> > optimistic nor pessimistic). Extensive endnotes and references.
> >
> > "Africa In Chaos" by George Ayittey. From backcover and VERY quick
> > flip-through, seems pessimistic about Africa in general. Seems less
> > aid focused and more about general failures in Africa, especially
> > political (but seems to include some discussion of aid). Appears to
> > include reasonable number of references in-line in the text, with
> > "Literature Cited" at the end.
> >
> > The Riddell book looks quite on-point for this topic. The Ayittey
> > book is probably more marginal, but perhaps worth consideration.
>
>  
>

To clarify, we're still planning to address the 16 questions explicitly and
say what we know of the research on each.
The "stuffing into 5" is more for agenda/priority-setting purposes.  In my
opinion, the 16 are important to the extent - and only to the extent - that
they pertain to the 5.  (And all do, to some extent.)

Also, this is going to be our report on developing-world aid specifically -
the 5 questions should be read as "Within developing-world aid, should I
give? / what programs are best? / what countries are best? / what vehicles
are best? / therefore what organizations are best?"  It is important to give
a general sense of whether international aid is beneficial on net (many
people question whether it is, and at least in informal conversations there
are arguments that it's net harmful), but also to discuss where one should
give within international aid.

Hopefully the wiki (
http://www.givewell.net/wiki/index.php?title=DWDA_Report_Outline#How_big_are_the_differences_between_different_giving_options.3F)
clarified the meaning of the "vehicles" question. Let me know if it didn't.

On Thu, Jan 15, 2009 at 11:22 AM, psteinx <psteinmeyer@...> wrote:

>   Some general comments:
>
>
> "Education programs are taking a back seat to health and economic
> empowerment.
> ...
> This would change if there were an education program with a strong
> record of success "
>
> I don't have a problem with this, but I encourage you to "show your
> work" behind the decision in an appropriate way on the website.
> Education is a popular area for donors to give to. If you have
> concluded that evidence for education donation effectiveness is weak,
> document it so that others can see.
>
> "No more comprehensive charity database. "
> I think this is a good decision, for now anyways. You can always
> revisit this down the road if/when your resources expand.
>
> "I've reframed the "general aid questions.
> ...
> I've written down the 5 questions that I think we really need to
> answer from a donor's perspective: should I donate, what sorts of
> programs are best, what countries are best to focus on, what kinds of
> vehicles are best, and bottom line - where should I give."
>
> The link you provide still shows 16 questions. I don't know how (or
> if) you're planning to link the 5 new questions to the 16 old ones.
> I'm not crazy about the new list:
>
> "should I donate" - For a charity guide website, the answer seems sort
> of self-evident. Anybody reading it is going to expect (and probably
> receive) a certain sort of answer. Seems a bit like a waste of
> pixels, at least in that framing.
>
> "what sorts of programs are best" - Reasonable
>
> "what countries are best" - Seems to assumes overseas donation. While
> I lean this way and think you lean this way, I think you've skipped an
> important step (Should I give locally/nationally/globally?) Also, how
> much ability to donors have to micro-select target nations - to donate
> to Tanzania instead of Kenya?
>
> "what kinds of vehicles are best" - I don't really know what this means
>
> "where should I give" - Obviously, the big question
>
> [Quick follow up]
> I didn't see when I wrote the above that you a bit further down, you
> provided a link to your draft of this.
>
> I think much of the information is good, but, as mentioned above, I'm
> not sure the framing is quite right. I'm not even sure if the many
> questions (16 or more) that reasonably interested donors might have
> should be stuffed into 5 questions, regardless of exactly what those 5
> are.
>
> In particular, I'd really like to see space allocated to the important
> question of "Does aid work in general?" and the related "If we've been
> giving aid in the past, why aren't things better?" (Maybe those
> aren't the exact phrasings - but something like that). Even if you
> can't provide super-clear answers (i.e. the research is nebulous), I
> think you need to present the questions and, if need be, SAY that
> there is a lot of research around the issue but not a lot of clear
> answers. Better to address the question and say that the answers are
> unclear than to leave the question unasked (because a lot of donors
> will be asking it in their minds).
>
>  
>

This is an update on my progress on general aid Q's, including the most
important gaps in what I know and what I'm planning to do to address them.

--

I've added a "Holden's take" in bold to most of the questions here (link
below) - says what my basic "conclusion" in progress is for each question,
including what I still don't know.

http://www.givewell.net/wiki/index.php?title=General_Aid_Questions<http://www.givewell.net/wiki/index.php?title=General_Aid_Questions#How_much_has_been_spent.2C_where.2C_how.2C_and_by_whom.3F>

There is still a ton that I don't know.  I spent much of this week going
through major think tanks' papers in the hopes that I'd find new
perspectives on the questions I've checked out and any perspectives on the
info I'm still missing.  My notes on think tanks are at
http://www.givewell.net/wiki/index.php?title=Think_tanks_and_multilaterals
<http://www.givewell.net/wiki/index.php?title=Think_tanks_and_multilaterals#OECD-DAC>-
although I haven't read the papers I've downloaded yet, based on titles &
abstracts I don't think my answers to most of these questions are going to
change substantially.  I will be reading these papers, along with books
about aid (discussed earlier), casually as I work on more targeted stuff.

At this point I want to focus on filling in the most important gaps.  There
are many gaps, but to me the most important ones by far are:

1. Track record of aid / success stories (a) outside of Africa (how much of
success is attributed to aid?) (b) outside of health (what exactly was the
Green Revolution and why isn't it working in Africa?) (c) outside of govt's
(does anyone know anything about what NGOs have accomplished)?

2. A couple harms of aid I've seen nothing on:
a. Concern about aid sucking up talent that could be being entrepreneurial
b. Overpopulation?  Are there any *good* studies of whether lowering infant
mortality lowers fertility?

3. The "How can we tell when an intervention is funded to capacity?"
question, probably not an academic research question

4. Is there any good track record / success stories for horizontal aid /
health system strengthening?  Are there instances of aid getting sustainable
/ being taken over by govt or is that a fantasy?  (So far I've seen
nothing.)

And the plan is:

1. There are a couple things I know I need to check out in a targeted way: I
need to learn more about the Green Revolution (the main non-health success
story that is constantly cited), and I need to do a thorough review of the
list of randomized-controlled-trial-supported interventions.

2. After that, I think it will be time to get in touch with scholars and/or
grad students personally.  Now that I've picked the low hanging fruit from
publicly available docs and gotten my own basic grounding, I feel that I
have more specific (and credible) questions, and I feel that personal
conversations will be more illuminating than the earlier ones I had.

The Gates Foundation awarded these 4 organizations its annual Award for
Global Health (past recipients include The Carter Center, BRAC, and The
Aravind Eyecare System), so we thought it'd be good to look at them and see
if any seemed like potential recommendees. This is part of our work to
identify potential organizations that are likely worth a "deep dive"
investigation that I talked about at
http://groups.yahoo.com/group/givewell/message/39.

*In brief*: ICCDR,B is somewhat promising (in the 2nd group in my email
linked above), but would take a long time to evaluate, going through the
docs on their website. I think AMREF is even less likely (in the fourth
group from my email above) we'd recommend (though possible), and evaluating
it would require spending a lot of time going through their site. I doubt
we'd recommend Rotary, given the type of organization it is. PCDA could be
good (it's unclear what role they played in Thailand's successful HIV
reduction program), but as far as I can tell they don't provide any
monitoring of current activities directly on their website.

*Org summaries
*

   1. *ICCDR,B* - not clear to me whether they're primarily a research
   organization or an implementer that conducts good research and evaluation of
   its activities. There's a lot of information and citation on the website (in
   publications and the annual report), but I haven't found anything reasonably
   concise that summarizes their activities.
   2. *Rotary* - Rotary is a fundraising and service community organization.
   Members contribute money (to polio eradication or other projects)and
   volunteer (e.g., helped coordinate distribution of supplies after Katrina.)
   They have and continue to contribute money to polio eradication (through
   fundraising, volunteering (unclear exactly what they did), and advocacy (
   http://www.rotary.org/en/ServiceAndFellowship/Polio/RotarysWork/Pages/ridefault.aspx).
   It's unlikely we'd recommend Rotary, so not worth pursuing further.
   3. *AMREF* -  It seems like a "do everything, everywhere" NGO. They run
   lots of relatively small-scale projects to attack substantially different
   problems (e.g., HIV/AIDS prevention or addressing gender-based violence). I
   can't find a clear approach (like the Carter Center has); similarly, the
   connection between AMREF activities and "proven" interventions isn't
   obvious. They definitely publish a lot of reports on their activities;
   what's available on their website is a lot better than what's available on
   almost any other website we've seen. And, the reports provide negative
   assessments of programs. But, we'd have to read all the reports and assess
   the results in aggregate (ourselves) to evaluate whether we have confidence
   in the organization as a whole since, without a clear orgnaizational
   strategy, it's not clear how they'll use additional resources.
   4. *PCDA* - They provide all types of services, health and povery
   reduction, (BRAC-like) to Thailand. Gates credits them to some degree with
   Thailand's successful AIDS reduction program (
   http://www.gatesfoundation.org/gates-award-global-health/Pages/2007-population-and-community-development-association.aspx),
   but PCDA isn't mentioned in either the DCP or the Center for Global
   Development case study (http://www.cgdev.org/doc/millions/MS_case_2.pdf),
   though a paper of his (I think) is cited as footnote 21 there. They list all
   their projects, but no links to project reports. It's possible that they
   have information in-house that they could provide us with, but doesn't
   appear to be on their website.

*More detail on ICCDR,B and AMREF*

*ICCDR,B *(aka Centre for Health and Population Research, aka International
Centre for Diarrhoeal Disease Research)

A. Summary

Research organization and hospital based in Bangladesh. *Bottom line*: it's
really hard to tell the degree to which they're primarilly a research and
evaluation organization or whether they're an implemented who heavily
researches and monitors what they do/could do. They definitely do both, to
some degree. There's a ton of papers, citations, information on their
website which we would have to go through to get a sense for what they're
doing.

A. What they do

ICDDR,B: International Centre for Diarrhoeal Disease Research, Bangladesh,
is a non-profit, international research, training and service institution
based in Dhaka, Bangladesh. Originally established as the Cholera Research
Laboratory in 1960, today the Centre's work encompasses a full spectrum of
issues related to child health, infectious diseases and vac­cine sciences,
reproductive health, nutrition, population sciences, health systems
research, poverty and health, HIV-AIDS and safe water. The Centre remains
the only international health research centre based in a developing country.
http://www.icddrb.org/pub/publication.jsp?classificationID=59&pubID=2359

"Services provided to the Bangladeshi community, particu­larly the poor, at
the Centre's Dhaka and Matlab Hospitals."
http://www.icddrb.org/pub/publication.jsp?classificationID=59&pubID=2359

B. Track record

   1. Credited with in developing and testing Oral Rehydration Therapy (
   http://rehydrate.org/ors/25years-saving-lives.htm), an effective
   intervention to prevent deaths from diarrhea (
   http://www.givewell.net/node/38#OralRehydrationTherapyORT)
   2. Developed, tested and implemented a family planning method MCH-FP (
   http://www.icddrb.org/pub/publication.jsp?classificationID=59&pubID=2359)
   cited by the DCP (Pg 1084) as an experimental program
   3. Claims: "The Centre's combined programmes in child health and family
   planning have contributed to a 75% reduction in the annual number of
   childhood deaths in the last 25 years in its Matlab field area."  (
   http://www.icddrb.org/pub/publication.jsp?classificationID=59&pubID=2359)

C. Information on ongoing activities

ICCDR has programs (information from the 2006 annual report, available at
http://www.icddrb.org/pub/publication.jsp?year=2006&classificationID=46),
which focus on providing services and conducting research in many health
areas (e.g., child health, nutrition, HIV/AIDS, etc.).

The annual report lists the details of all the individual projects/research
studies in each category, see for example Pgs 71-74. Many of these programs
are just research ("Risk factors for sclerema in infants with diarrhoeal
disease"); others are trials evaluating a specific program ("A double-blind,
randomized, placebo-controlled, parallel group study to assess the efficacy,
safety, and tolerability of crofelemer (SP 303) in the treatment of cholera
in adults"); it also provides some information on ongoing acivities ("The
MCHS runs the largest, fixed-site immunization centre in the country to
provide 6 EPI vaccines and hepatitis B vaccine to children aged less than 2
years and tetanus toxoid to girls/women aged 15– 45 years.") Though, it's
not clear to me whether we could find the technical papers or more
monitoring detail of these programs.

The annual report lists all the papers published by ICCDR,B staff. It's
unclear whether these are evaluating ICCDR,B activities or researchers
evaluating programs implemented by other organizations.

D. Other

Were we to still do the "regions" approach, I think the research available
here would be very helpful. For example, ICCDR's papers on demographic and
health trends in Bangladesh over the last 40 years provide insight into
changes in some primary factors: fertility rate, child mortality, and
contraception use (
http://www.icddrb.org/pub/publication.jsp?classificationID=64&pubID=9407).*

AMREF - *African Medical and Research Foundation

AMREF seems to be a "do everything, everywhere" NGO. They run lots of
relatively small-scale projects to attack substantially different problems.
I can't find a clear approach (like the Carter Center has); similarly, the
connection between AMREF activities and "proven" interventions isn't clear
to me. *Bottom line:* They definitely publish a lot of reports on their
activities; what's available on their website is a lot better than what's
available on almost any other website we've seen. And, the reports provide
negative assessments of programs. But, we'd have to read all the reports and
assess the results in aggregate (ourselves) to evaluate whether we have
confidence in the organization as a whole since, without a clear
orgnaizational strategy, it's not clear how they'll use additional
resources.

A. Monitoring and evaluation of ongoing activities

They provide the following types of information on their website:

   - List of technial papers from their projects:
   http://amref.org/info-centre/technical-briefing-papers/
   - Annual report provides stories from some subset of projects:
   http://amref.org/silo/files/amref-annual-report-2007.pdf
   - Searchable database of publications: The following link is a list of 32
   "progress reports" from various programs -
   http://196.207.17.140/ics-wpd/exec/icswppro.dll. There are many (500+, at
   least) different documents here.

The technical papers do provide negative assesments. The one I opened (an
adolescent health program to prevent HIV/AIDS
http://amref.org/silo/files/impact-evaluation-mema-kwa-vijana.pdf) found
(this paper is also cited by the DCP on pgs 1115-6):

There was no evidence of any consistent impact of the intervention on
biological outcomes in either direction.... The incidence of HIV in the
comparison communities was lower at 2.21/1,000 person years, than had been
estimated in advance. Overall, there were 45 incident cases, of which only 5
were in males. The adjusted incidence of HIV in females was 24% lower in the
intervention communities, but this was not statistically significant
(RR=0.75, 95% CI 0.34,1.66).

I spent ~10 hours trying to get a basic understanding of the Green
Revolution, which is constantly cited as a major international-aid success
story, though one that people have been trying unsuccessfully to extend to
Africa, the only place where it hasn't caught on.
Other relevant success stories (the health ones) are basically collected in
one place (the Levine book by Center for Global Development); it was
trickier to get a handle on exactly what this one was, what the role of
philanthropy was, and what the obstacles are to taking it further.  But it
seemed very important since the Green Revolution is not only held up as an
example of what philanthropy can accomplish, but is also widely seen as a
key to the macro-scale emergence of countries.

There's still a lot I don't understand and plan to ask people about, but
I've got a basic picture now.  My understanding is that philanthropy
(Rockefeller and Ford Foundations) is given a lot of credit for funding
research on improved agricultural seeds & techniques, which led to huge
increases in agricultural productivity, which in turn led to changes in
government policy and infrastructure and is widely (though not unanimously)
seen as the "trigger" for the emergence of many countries in Asia and
Central/South America.  That said, there are many reasons to believe that
the same success story cannot be repeated in Africa, and it's unclear what
role non-research interventions have played or can play.

In health, I believe there are interventions (like mass vaccination, and
food fortification) that have been responsible for big accomplishments in
the past and can be reasonably expected to do so again.  From what I know, I
wouldn't say the same of any particular agriculture/Green-Revolution-focused
programs.  I think our focus within this area
(agriculture/economic-empowerment programs) should be on programs that have
good track records at the micro level (e.g., randomized controlled trials
such as those carried out by the Poverty Action Lab).

More at the Green Revolution wikipage I've created.
http://www.givewell.net/wiki/index.php?title=Green_revolution

I've created a new wikipage devoted to listing what we see as interventions
that have some track record of success.  Right now it's just a skeleton.
http://www.givewell.net/wiki/index.php?title=Successful_interventions

A micro success is a well-documented, fairly rigorously established instance
where the intervention changed lives.  The randomized controlled trials
performed by http://www.povertyactionlab.com/ are this sort of evidence.
 (For example, children were randomly selected for deworming; those that
were selected had higher school attendance.)  Many medical interventions
also have strong "micro" track records, and we are currently compiling
information on these track records (this work is being done mostly by
part-time workers).

A macro success is a less rigorously established, but larger-scale and more
impressive, success, like those discussed in the Center for Global
Development's "success stories" (see my notes at
http://groups.yahoo.com/group/givewell/message/25).  For example, a major
immunization program coincided with a large drop in mortality from the
disease being immunized against (most extreme case: smallpox eradication).

There is a large amount of micro evidence on health interventions, so much
that we are currently focusing our part-time employees on gathering and
summarizing it.  From our initial scan of both micro and macro evidence,
we've prioritized the interventions currently listed at
http://www.givewell.net/research-agenda .

We've done much less with economic-empowerment interventions, and that's my
current focus.  I've gotten a basic sense of the most commonly cited "macro"
success story, the Green Revolution (discussed in the previous email).  Now
I'm looking for interventions that have "micro" support.  I believe they are
much rarer than in health, and that I'll be able to identify most of them
through two research institutions devoted to this sort of evidence (see
http://blog.givewell.net/?p=219) as well as a couple of literature reviews
(see
http://givewell.net/node/278#Question9Whataretheinterventionsthataresupportedbyrigorousevidence
).

Once I've gone through these, I'm going to do a little more work on the
remaining major gaps in my understanding of general aid Q's (harms of aid;
allocation of funding) and then focus on getting in touch with people.  To
reiterate, the goal of my current work is to read the
basic/classic/low-hanging-fruit literature, putting us in position to ask
more informed, specific, targeted questions of people who know more.

I'm taking a closer look at Jeffrey Sachs's book (before I did a 30-min scan
in the bookstore; now I actually have a copy).  On pgs 260-265 he lists
"several significant examples of programs that have been scaled up massively
to remarkable success."
So far our only source for major/macro "success stories" has been  the
Levine book published by Center for Global Development (discussed earlier:
http://groups.yahoo.com/group/givewell/message/25).  Sachs has points of
overlap but also adds some.

Cases where Sachs's characterization matches Levine's:

   - Eradication of smallpox (Levine case study #1)
   - Control of river blindness in sub-Saharan Africa (#7)

Campaigns that Sachs sees as broader than how Levine presents them:

   - Polio control: Levine case study #5 points to elimination of polio in
   Latin America & the Caribbean by the Pan American Health Organization; Sachs
   credits the Global Polio Eradication Initiative and says "Today, thanks to
   massive efforts by official institutions such as WHO, UNICEF, and the U.S.
   Centers for Disease Control and Prevention, as well as actions within poor
   countries and a remarkable and tireless effort by Rotary International,
   polio remains in only six countries ... Only 784 cases were reported
   worldwide in 2003, compared with 350,000 in 1988."
   - Family planning: Levine case study #13 discusses a fertility reduction
   program in Bangladesh, but Sachs says "Modern contraception ha contributed
   to a dramatic reduction in total fertility rates, from a world average of
   5.0 children per woman in the period 1950 to 1955 to 2.8 children per woman
   in the period 1995 to 2000 ... The United Nations Population Fund (UNFPA)
   ... has helped to spur a massive increase in the use of modern contraception
   among couples in developnig countries, rising from an estimated 10 to 15
   percent of couples in 1970 to an estimated 60 percent in 2000"

More success stories from Sachs:


   - Global Alliances for Vaccines and Immunization (GAVI): "As of 2004, the
   alliance reported 41.6 million children vaccinated against hepatitis B; 5.6
   million children vaccinated against ... Hib; 3.2 million children vaccinated
   against yellow fever; and 9.6 million children vaccinated with other basic
   vaccines."  As we've noted before, we are suspicious of these #'s until we
   get a better sense of GAVI's data quality audit process.  The #'s are
   reported by governments that receive direct funding from GAVI.
   - Campaign for Child Survival, launched in 1982 by UNICEF: "The campaign
   promoted a package of interventions known as GOBI: growth monitoring of
   children; oral rehydration therapy to treat bouts of diarrhea; breastfeeding
   for nutrition and immunity to diseases in infancy; and immunization against
   six childhood killers ... Child mortality rates fell sharply in all parts of
   the low-income world, including Africa ... The Campaign was estimated to
   have saved around twelve million lives by the end of the decade."
   - WHO campaign against malaria in 1950s and 1960s: "Sometimes judged to
   have been a failure, since malaria was certainly not eradicated, these
   efforts can be seen as a stunning success for certain parts of the world ...
   Well over half o the world's population living in endemic regions in teh
   1940s were largely freed of malaria transmission and mortality as a result
   of WHO's concentrated efforts, mainly in the areas where disease ecology
   favored the control measures.  Africa, alas, was neither part of the program
   at the time, nor a beneficiary of its results until today."  This program
   depended on DDT and other pesticides, and chloroquine and other
   antimalarials (both of which the disease has more recently developed some
   resistance to).


Less relevant success stories:

   - Green Revolution - discussed previously
   - Export Processing Zones in East Asia - purely a policy matter (setting
   up zones where "special tax, administrative, and infrastructure conditions
   are applied in order to encourage foreign companies to set up
   export-oriented manufacturing facilities"
   - Cellphones in Bangladesh - "Grameen Telecom went into the business of
   mobile phones in 1997, reaching half a million subscribers by 2003, roughly
   equal to the total number of landlines.  It used that mainly urban base of
   operations to launch a village phones program ... With 9400 villages covered
   by early 2004, the estimated access would be on the order of 23 million
   villagers."  No discussion of impact on life outcome / standard of living
   (and we've previously looked for and failed to find such discussion for the
   same program).

*Re: success of GAVI. *Does Sachs mention a change in mortality
rates/incidence from the diseases for which GAVI has successfully vaccinated
people? GAVI's been around since 2000, so I'd expect that any significant
increase in vaccination coverage would lead to improved outcomes re
incidence and mortality.

The three vaccines mentioned here are Hib, Hep B, and yellow fever, so I
looked at those. The latest immunization monitoring report from the WHO
(2007) is available at
http://whqlibdoc.who.int/hq/2007/WHO_IVB_2007_eng.pdf(6mb PDF).

*Vaccine coverage rates are up.*

   - Hep B (3rd dose = full immunization) - 32% -> 60%, Pg 12
   - Hib - 14% -> 22%, Pg 14
   - Yellow fever - 35% -> 73%, Pgs 26-7.

*What about incidence? *

For yellow fever, the report also provides incidence. Since 1999, annual
incidence is as follows (no clear positive trend):

'99 - 214
'00 - 684
'01 - 620
'02 - 705
'03 - 672
'04 -1'344
'05 - 588
'06 - 356

(The report doesn't note whether these are in '000s. I'm guessing they are
because Pg 26 says that WHO estimated 30k deaths from yellow fever in 2002.)

I don't know why the WHO only presents incidence for yellow fever and not
Hib or Hep B, but the disconnect between the strong increase in vaccination
coverage but no decrease in incidence (the 356 in '06 oculd be somethign
real, or could be noise) makes me wonder what's going on.

*What about data accuracy?*

There's some possibility that the GAVI/WHO focus on immunuzation rates led
to heavier monitoring, leading to a measured *increase* in incidence.
Incidence in the Africa region since 1997 is:

'97 - 47
'98 - 33
'99 - 8
'00 - 593
'01 - 572
'02 - 610
'03 - 498
'04 - 1'253
'05 - 474
'06 - 272

The report states, "In 2004 the Democratic Republic of the Congo reported
1'192 cases, however no case was confirmed in the laboratory." I have no
idea what to make of this other than that I find it hard to have confidence
in the accuracy of this data.

I'd like to do more systematically look at changes in vaccination coverage
rates and incidence/mortality rates for the associated diseases, but what
I've seen here makes me wonder whether the data is accurate enough for that
excercise to be worthwhile.

On Thu, Jan 22, 2009 at 4:09 PM, Holden Karnofsky <Holden@...>wrote:

>   I'm taking a closer look at Jeffrey Sachs's book (before I did a 30-min
> scan in the bookstore; now I actually have a copy).  On pgs 260-265 he lists
> "several significant examples of programs that have been scaled up massively
> to remarkable success."
>
> So far our only source for major/macro "success stories" has been  the
> Levine book published by Center for Global Development (discussed earlier:
> http://groups.yahoo.com/group/givewell/message/25).  Sachs has points of
> overlap but also adds some.
>
> Cases where Sachs's characterization matches Levine's:
>
>    - Eradication of smallpox (Levine case study #1)
>    - Control of river blindness in sub-Saharan Africa (#7)
>
> Campaigns that Sachs sees as broader than how Levine presents them:
>
>    - Polio control: Levine case study #5 points to elimination of polio in
>    Latin America & the Caribbean by the Pan American Health Organization; Sachs
>    credits the Global Polio Eradication Initiative and says "Today, thanks to
>    massive efforts by official institutions such as WHO, UNICEF, and the U.S.
>    Centers for Disease Control and Prevention, as well as actions within poor
>    countries and a remarkable and tireless effort by Rotary International,
>    polio remains in only six countries ... Only 784 cases were reported
>    worldwide in 2003, compared with 350,000 in 1988."
>    - Family planning: Levine case study #13 discusses a fertility
>    reduction program in Bangladesh, but Sachs says "Modern contraception ha
>    contributed to a dramatic reduction in total fertility rates, from a world
>    average of 5.0 children per woman in the period 1950 to 1955 to 2.8 children
>    per woman in the period 1995 to 2000 ... The United Nations Population Fund
>    (UNFPA) ... has helped to spur a massive increase in the use of modern
>    contraception among couples in developnig countries, rising from an
>    estimated 10 to 15 percent of couples in 1970 to an estimated 60 percent in
>    2000"
>
> More success stories from Sachs:
>
>
>    - Global Alliances for Vaccines and Immunization (GAVI): "As of 2004,
>    the alliance reported 41.6 million children vaccinated against hepatitis B;
>    5.6 million children vaccinated against ... Hib; 3.2 million children
>    vaccinated against yellow fever; and 9.6 million children vaccinated with
>    other basic vaccines."  As we've noted before, we are suspicious of these
>    #'s until we get a better sense of GAVI's data quality audit process.  The
>    #'s are reported by governments that receive direct funding from GAVI.
>    - Campaign for Child Survival, launched in 1982 by UNICEF: "The
>    campaign promoted a package of interventions known as GOBI: growth
>    monitoring of children; oral rehydration therapy to treat bouts of diarrhea;
>    breastfeeding for nutrition and immunity to diseases in infancy; and
>    immunization against six childhood killers ... Child mortality rates fell
>    sharply in all parts of the low-income world, including Africa ... The
>    Campaign was estimated to have saved around twelve million lives by the end
>    of the decade."
>    - WHO campaign against malaria in 1950s and 1960s: "Sometimes judged to
>    have been a failure, since malaria was certainly not eradicated, these
>    efforts can be seen as a stunning success for certain parts of the world ...
>    Well over half o the world's population living in endemic regions in teh
>    1940s were largely freed of malaria transmission and mortality as a result
>    of WHO's concentrated efforts, mainly in the areas where disease ecology
>    favored the control measures.  Africa, alas, was neither part of the program
>    at the time, nor a beneficiary of its results until today."  This program
>    depended on DDT and other pesticides, and chloroquine and other
>    antimalarials (both of which the disease has more recently developed some
>    resistance to).
>
>
> Less relevant success stories:
>
>    - Green Revolution - discussed previously
>    - Export Processing Zones in East Asia - purely a policy matter
>    (setting up zones where "special tax, administrative, and infrastructure
>    conditions are applied in order to encourage foreign companies to set up
>    export-oriented manufacturing facilities"
>    - Cellphones in Bangladesh - "Grameen Telecom went into the business of
>    mobile phones in 1997, reaching half a million subscribers by 2003, roughly
>    equal to the total number of landlines.  It used that mainly urban base of
>    operations to launch a village phones program ... With 9400 villages covered
>    by early 2004, the estimated access would be on the order of 23 million
>    villagers."  No discussion of impact on life outcome / standard of living
>    (and we've previously looked for and failed to find such discussion for the
>    same program).
>
>  
>

I typed everything Sachs says that's of interest.  He only has a paragraph
on each success story, and no references.

On Thu, Jan 22, 2009 at 5:08 PM, Elie Hassenfeld <ehassenfeld@...>wrote:

>   *Re: success of GAVI. *Does Sachs mention a change in mortality
> rates/incidence from the diseases for which GAVI has successfully vaccinated
> people? GAVI's been around since 2000, so I'd expect that any significant
> increase in vaccination coverage would lead to improved outcomes re
> incidence and mortality.
>
> The three vaccines mentioned here are Hib, Hep B, and yellow fever, so I
> looked at those. The latest immunization monitoring report from the WHO
> (2007) is available at
> http://whqlibdoc.who.int/hq/2007/WHO_IVB_2007_eng.pdf (6mb PDF).
>
> *Vaccine coverage rates are up.*
>
>    - Hep B (3rd dose = full immunization) - 32% -> 60%, Pg 12
>    - Hib - 14% -> 22%, Pg 14
>    - Yellow fever - 35% -> 73%, Pgs 26-7.
>
> *What about incidence? *
>
> For yellow fever, the report also provides incidence. Since 1999, annual
> incidence is as follows (no clear positive trend):
>
> '99 - 214
> '00 - 684
> '01 - 620
> '02 - 705
> '03 - 672
> '04 -1'344
> '05 - 588
> '06 - 356
>
> (The report doesn't note whether these are in '000s. I'm guessing they are
> because Pg 26 says that WHO estimated 30k deaths from yellow fever in 2002.)
>
> I don't know why the WHO only presents incidence for yellow fever and not
> Hib or Hep B, but the disconnect between the strong increase in vaccination
> coverage but no decrease in incidence (the 356 in '06 oculd be somethign
> real, or could be noise) makes me wonder what's going on.
>
> *What about data accuracy?*
>
> There's some possibility that the GAVI/WHO focus on immunuzation rates led
> to heavier monitoring, leading to a measured *increase* in incidence.
> Incidence in the Africa region since 1997 is:
>
> '97 - 47
> '98 - 33
> '99 - 8
> '00 - 593
> '01 - 572
> '02 - 610
> '03 - 498
> '04 - 1'253
> '05 - 474
> '06 - 272
>
> The report states, "In 2004 the Democratic Republic of the Congo reported
> 1'192 cases, however no case was confirmed in the laboratory." I have no
> idea what to make of this other than that I find it hard to have confidence
> in the accuracy of this data.
>
> I'd like to do more systematically look at changes in vaccination coverage
> rates and incidence/mortality rates for the associated diseases, but what
> I've seen here makes me wonder whether the data is accurate enough for that
> excercise to be worthwhile.
>
>
> On Thu, Jan 22, 2009 at 4:09 PM, Holden Karnofsky <Holden@...>wrote:
>
>>   I'm taking a closer look at Jeffrey Sachs's book (before I did a 30-min
>> scan in the bookstore; now I actually have a copy).  On pgs 260-265 he lists
>> "several significant examples of programs that have been scaled up massively
>> to remarkable success."
>>
>> So far our only source for major/macro "success stories" has been  the
>> Levine book published by Center for Global Development (discussed earlier:
>> http://groups.yahoo.com/group/givewell/message/25).  Sachs has points of
>> overlap but also adds some.
>>
>> Cases where Sachs's characterization matches Levine's:
>>
>>    - Eradication of smallpox (Levine case study #1)
>>    - Control of river blindness in sub-Saharan Africa (#7)
>>
>> Campaigns that Sachs sees as broader than how Levine presents them:
>>
>>    - Polio control: Levine case study #5 points to elimination of polio
>>    in Latin America & the Caribbean by the Pan American Health Organization;
>>    Sachs credits the Global Polio Eradication Initiative and says "Today,
>>    thanks to massive efforts by official institutions such as WHO, UNICEF, and
>>    the U.S. Centers for Disease Control and Prevention, as well as actions
>>    within poor countries and a remarkable and tireless effort by Rotary
>>    International, polio remains in only six countries ... Only 784 cases were
>>    reported worldwide in 2003, compared with 350,000 in 1988."
>>    - Family planning: Levine case study #13 discusses a fertility
>>    reduction program in Bangladesh, but Sachs says "Modern contraception ha
>>    contributed to a dramatic reduction in total fertility rates, from a world
>>    average of 5.0 children per woman in the period 1950 to 1955 to 2.8 children
>>    per woman in the period 1995 to 2000 ... The United Nations Population Fund
>>    (UNFPA) ... has helped to spur a massive increase in the use of modern
>>    contraception among couples in developnig countries, rising from an
>>    estimated 10 to 15 percent of couples in 1970 to an estimated 60 percent in
>>    2000"
>>
>> More success stories from Sachs:
>>
>>
>>    - Global Alliances for Vaccines and Immunization (GAVI): "As of 2004,
>>    the alliance reported 41.6 million children vaccinated against hepatitis B;
>>    5.6 million children vaccinated against ... Hib; 3.2 million children
>>    vaccinated against yellow fever; and 9.6 million children vaccinated with
>>    other basic vaccines."  As we've noted before, we are suspicious of these
>>    #'s until we get a better sense of GAVI's data quality audit process.  The
>>    #'s are reported by governments that receive direct funding from GAVI.
>>    - Campaign for Child Survival, launched in 1982 by UNICEF: "The
>>    campaign promoted a package of interventions known as GOBI: growth
>>    monitoring of children; oral rehydration therapy to treat bouts of diarrhea;
>>    breastfeeding for nutrition and immunity to diseases in infancy; and
>>    immunization against six childhood killers ... Child mortality rates fell
>>    sharply in all parts of the low-income world, including Africa ... The
>>    Campaign was estimated to have saved around twelve million lives by the end
>>    of the decade."
>>    - WHO campaign against malaria in 1950s and 1960s: "Sometimes judged
>>    to have been a failure, since malaria was certainly not eradicated, these
>>    efforts can be seen as a stunning success for certain parts of the world ...
>>    Well over half o the world's population living in endemic regions in teh
>>    1940s were largely freed of malaria transmission and mortality as a result
>>    of WHO's concentrated efforts, mainly in the areas where disease ecology
>>    favored the control measures.  Africa, alas, was neither part of the program
>>    at the time, nor a beneficiary of its results until today."  This program
>>    depended on DDT and other pesticides, and chloroquine and other
>>    antimalarials (both of which the disease has more recently developed some
>>    resistance to).
>>
>>
>> Less relevant success stories:
>>
>>    - Green Revolution - discussed previously
>>    - Export Processing Zones in East Asia - purely a policy matter
>>    (setting up zones where "special tax, administrative, and infrastructure
>>    conditions are applied in order to encourage foreign companies to set up
>>    export-oriented manufacturing facilities"
>>    - Cellphones in Bangladesh - "Grameen Telecom went into the business
>>    of mobile phones in 1997, reaching half a million subscribers by 2003,
>>    roughly equal to the total number of landlines.  It used that mainly urban
>>    base of operations to launch a village phones program ... With 9400 villages
>>    covered by early 2004, the estimated access would be on the order of 23
>>    million villagers."  No discussion of impact on life outcome / standard of
>>    living (and we've previously looked for and failed to find such discussion
>>    for the same program).
>>
>>
>  
>

I thought this was an interesting read:
http://www.gatesfoundation.org/annual-letter/Pages/2009-annual-letter-introduction.aspx
(10
short pages).

   - Buys into the conventional wisdom that lowering child mortality lowers
   fertility, but the support given is the same charts I've seen before that
   leave lots of alternate hypotheses open.  I'd guess that the conventional
   wisdom is right, but it'd be nice to find a more thorough discussion (even
   just country-by-country charts to see if the fertility drop is mostly in
   countries that have "emerged", which I don't think it is).
   - On the Green Revolution: "Africa jumps out as the only case where this
   increase has not taken place. A big reason is that African countries have
   widely varying climate conditions, and there hasn't been the same investment
   in creating the seeds that fit those conditions. Because agriculture is an
   essential part of economic growth for most African countries, we are working
   with others to fund a "Green Revolution for Africa" and other areas that
   could benefit from this kind of investment."  Is it true that Africa
   hasn't seen the "same investment"? My impression is that Norman Borlaug has
   been working on Africa for longer than he spent on Mexico, India and
   Pakistan combined. It would be interesting to get #'s on how much was spent
   on successful Green Revolution initiatives and how much has been spent
   trying to bring it to Africa. It might suggest either a huge opportunity
   (underinvestment in Africa) or provide a pretty vivid failure story.
   - He's very interested in eradicating polio.  Does this really make sense
   given that the disease is now extremely rare?  I'm guessing (though can't
   find any cost-effectiveness analysis in the Disease Control Priorities
   report) that the initiatives he's talking about - reaching the parts of
   India where children need 8+ vaccinations - is a high cost per person
   affected, and the Q is whether complete eradication would yield some other
   benefit like being able to stop vaccinating people in the rest of the world.
    Disease Control Priorities report discusses this on 1170-1173 ... it's
   pretty unclear.
   - In general, he seems most interested in R&D, particularly for vaccines
   (malaria, AIDS - as well as rolling out the rotavirus vaccine).  This seems
   like a pretty reasonable thing to be focused on.  Most of aid's success
   stories seem to involve the mass rollout of extremely effective technologies
   (seeds, drugs, vaccines, surgeries - things that required technical research
   to create) rather than the "teaching" of softer skills such as
   entrepreneurship, community development, etc.

This quote is consistent with some other stuff I've read and can't source in
giving me an impression about how Gates spends money:
Last year, Melinda and I met with our polio team to get an update on
progress against the disease. The team was asking us to approve the same
amount of money we had been spending for years, but they kept talking about
the many challenges of eradicating polio. Melinda and I probed to understand
if they were saying that the world needed to spend more, and whether our
leading by example could help make it happen. They said yes, and within a
month they had put together a more aggressive plan that involved us spending
hundreds of millions more and getting other donors to step up as well. We
approved the plan. Rotary International and other donors are doing a great
job so far coming up with the extra resources that are needed.

ie, they see themselves as "funding leaders" - creating initiatives but not
funding them all the way to capacity, hoping that others will come in.  I'd
really like to talk to someone at the Foundation about this and see: do they
try to fund as much as they can of a good thing?  Or be a "funding leader?"
 Do they have estimates of how much is needed, total, for their various
initiatives?

One of the questions we really haven't made any progress on is the question
of whether some initiatives are "more in need of funding" than others.  For
example, even if malaria control is a great program, it may be a "sexier"
cause than onchocerciasis control and thus might be likely to attract (or
might have already attracted) all the funding that can reasonably be used.
I really don't have a lot of ideas about how to investigate this sort of
thing short of asking around.  But I did take a quick scan through the World
Health Organization's lists of programs and projects (
http://www.who.int/entity/en/) to see which ones have estimated total costs
(and/or total committed to date, to produce a "funding gap").  My notes are
at http://www.givewell.net/wiki/index.php?title=Funding_gaps

...Some of these programs have large-scale programs with price tags & some
don't.  All claim funding gaps, though the sizes vary a lot and it's hard to
say whether a "funding gap" is a real need for funds.  (For example, the
polio eradication initiative claims a big gap for 2009 but also says they
just closed the 2008 gap thanks to Gates.)

I'd like to get in touch with the people behind at least a couple of these
large-scale programs and see if I can grill them on what exactly these #'s
mean and what will happen if they fall far short (see tuberculosis for an
example).

Thoughts on this are appreciated.

I'm not 100% done scanning literature on general aid Q's, but I'm ready to
stop searching for papers ... for each question I either pretty much know
where I stand or know exactly what I have to read/do to know where I stand.
 As a reminder, by "where I stand" I mean "based on prominent & available
literature, what I see as the different viewpoints held by scholars and the
evidence for each."

My focus now is going to be on contacting experts, both to run my
interpretation by them and to try to fill in some specific gaps (on
questions that I think are particularly important and where I'm particularly
unsatisfied with what I've found).

   - The major sources I've found for each general aid question, as well as
   my 2-sentence summaries for the different viewpoints/evidence on each, are
   at http://www.givewell.net/wiki/index.php?title=General_Aid_Questions
   - Notes on what I've read/scanned (at least what I used as starting
   points to find other papers of interest) are at
   http://www.givewell.net/wiki/index.php?title=DWDA_reading_list
   - My top questions for scholars, as well as the scholars I most want to
   get in touch with, are listed at
   http://www.givewell.net/wiki/index.php?title=Questions_for_contacts

Should have mentioned that while trying to get in touch with experts, I'm
going to be writing up the info at the "general aid q's" wiki more formally,
i.e., producing draft "reports" on each question (while modifying them to
incorporate anything I learn from contacts).

On Sat, Jan 31, 2009 at 12:04 AM, Holden Karnofsky <Holden@...>wrote:

> I'm not 100% done scanning literature on general aid Q's, but I'm ready to
> stop searching for papers ... for each question I either pretty much know
> where I stand or know exactly what I have to read/do to know where I stand.
>  As a reminder, by "where I stand" I mean "based on prominent & available
> literature, what I see as the different viewpoints held by scholars and the
> evidence for each."
>
> My focus now is going to be on contacting experts, both to run my
> interpretation by them and to try to fill in some specific gaps (on
> questions that I think are particularly important and where I'm particularly
> unsatisfied with what I've found).
>
>    - The major sources I've found for each general aid question, as well
>    as my 2-sentence summaries for the different viewpoints/evidence on each,
>    are at
>    http://www.givewell.net/wiki/index.php?title=General_Aid_Questions
>    - Notes on what I've read/scanned (at least what I used as starting
>    points to find other papers of interest) are at
>    http://www.givewell.net/wiki/index.php?title=DWDA_reading_list
>    - My top questions for scholars, as well as the scholars I most want to
>    get in touch with, are listed at
>    http://www.givewell.net/wiki/index.php?title=Questions_for_contacts
>
>
>

Some comments:

FUNGIBILITY
I think fungibility is an important enough consideration that it
deserves its own sub-topic - perhaps as a way that aid could be either
net harmful or at least provide close to zero net benefit.  If foreign
entities pay for a poor countries health care and that allows the
government to spend what would have gone to health care on weapons,
that's not so good.  I don't know quite how much of an issue
fungibility is, but I think it's an important enough topic that you
should collect the research on it in a single place, as you have done
with other topics.

Fungibility should probably also go in the list of questions for
experts somewhere.


HOW MUCH HAS BEEN SPENT...
I'm not satisfied with what I see in this section.  Parts of it don't
make much sense.  The amount of aid is supposedly vanishingly small,
yet it's only 2.78% of "what would be needed to close the gap"?  I'm
not sure what the gap is here, but if we are expected to scale up from
2.78% of something to 100% of something, that's a 100/2.78= ~36 fold
increase.  It may (or may not) be reasonable to expect some future
increases in aid, but 36 fold seems quite a reach.

I think we should be able to get at least the recent run rates for
funds from several sources (aid by governments at the national level,
and some measure of charitable aid (by individuals/foundations) as
well - I'm pretty sure I've seen this stuff floating around.  Maybe we
won't get it going back to 1955, but we should at least know what it
was a year or two ago.

====

In your questions for contacts, I like your question for Bono. :) 

Thanks for the comments.
Re fungibility: my understanding is that all relevant literature (and there
isn't much - I think this is a few papers, not a major debate /
mini-literature like "aid's effect on growth") looks at aid that is
given directly
to governments.  Estimating the impact of projects run by nonprofits (except
when they themselves grant govt's, as for example the GAVI alliance does)
would basically be an exercise in extrapolation/guesswork.  For example, I'd
guess that a primary care initiative (i.e., building a hospital), which
provides services similar to what the Ministry of Health is expected to
provide, would be a bigger fungibility concern than a vertical program aimed
at eliminating a particular disease, though both would ultimately put some
downward pressure on health care costs.  And I wouldn't expect either to
free up government funding in the same way that a check written to the
government does.

I've added fungibility as a possible "harm of aid" and listed the refs I
have (including a couple that weren't there before - found them in some of
my old notes): <http://www.givewell.net/wiki/index.php?title=General_Aid_Questions#Freeing_up_government_funds_for_bad_uses_such_as_military_spending>
http://www.givewell.net/wiki/index.php?title=General_Aid_Questions#Freeing_up_government_funds_for_bad_uses_such_as_military_spending

On the "how much has been spent" question, could you clarify what you're
looking for from this question and why?  I think the two of us have
different interpretations of it.

We have official data going back to 2001; I had been thinking of this time
period as still sort of conceptually belonging in the "What is the current
allocation?" question.

On Sat, Jan 31, 2009 at 3:25 PM, psteinx <psteinmeyer@...> wrote:

>   Some comments:
>
> FUNGIBILITY
> I think fungibility is an important enough consideration that it
> deserves its own sub-topic - perhaps as a way that aid could be either
> net harmful or at least provide close to zero net benefit. If foreign
> entities pay for a poor countries health care and that allows the
> government to spend what would have gone to health care on weapons,
> that's not so good. I don't know quite how much of an issue
> fungibility is, but I think it's an important enough topic that you
> should collect the research on it in a single place, as you have done
> with other topics.
>
> Fungibility should probably also go in the list of questions for
> experts somewhere.
>
> HOW MUCH HAS BEEN SPENT...
> I'm not satisfied with what I see in this section. Parts of it don't
> make much sense. The amount of aid is supposedly vanishingly small,
> yet it's only 2.78% of "what would be needed to close the gap"? I'm
> not sure what the gap is here, but if we are expected to scale up from
> 2.78% of something to 100% of something, that's a 100/2.78= ~36 fold
> increase. It may (or may not) be reasonable to expect some future
> increases in aid, but 36 fold seems quite a reach.
>
> I think we should be able to get at least the recent run rates for
> funds from several sources (aid by governments at the national level,
> and some measure of charitable aid (by individuals/foundations) as
> well - I'm pretty sure I've seen this stuff floating around. Maybe we
> won't get it going back to 1955, but we should at least know what it
> was a year or two ago.
>
> ====
>
> In your questions for contacts, I like your question for Bono. :)
>
>  
>

I stumbled across this short debate transcript between William Easterly and
Stephen Radelet (a cautious "aid optimist" and general backer of the current
World Bank approach) and thought people might be interested.  I found it
broadly representative of debates over "has aid worked and how should we be
approaching it?" debates, but more compact/readable than academic papers and
more informative than debates between Easterly and Sachs (which seem to
degenerate more quickly into rhetoric).
http://www.cfr.org/publication/12077/ - read from the bottom.

BTW, this should make clear that Easterly's reputation as "the guy who
thinks aid doesn't work" is a bit exaggerated.  What he's advocating
basically sounds like the work we're trying to do: focus on identifying
projects with demonstrable impact and scaling them up, rather than on
raising more money for enormous-scale plans to end poverty.

I'm going back through the Copenhagen Consensus to try to get very specific
about which interventions they do and don't endorse.  This is for purposes
of completing the table linked at the top of
http://www.givewell.net/wiki/index.php?title=DWDA_intervention_writeups
CC's top-level recs are listed at
http://www.copenhagenconsensus.com/Default.aspx?ID=953

For now I'm gathering info on education interventions even though I'm not
sure yet whether we're going to end up really covering that.  I am skipping
legislation-only and research-only stuff.

Having gone through these, I note that the Copenhagen Consensus seems
overwhelmingly concerned with cost-effectiveness calculations and not very
concerned at all with rigor/track record of interventions.  Throughout their
papers is an enormous amount of discussion of their assumptions,
methodology, etc. for cost-effectiveness calcs, and although they cite
papers in support of interventions' effectiveness they almost never discuss
the rigor of these papers, potential alternative hypotheses, etc.  I think
the DCP report (whose scholars overlap with Copenhagen Consensus's) has the
same tendency.

This is good to keep in mind: an endorsement from one of these groups can be
taken to mean that the intervention is one of the most cost-effective when
its measured effects are taken at face value, but without an independent
confirmation that it has a track record of success (rigorous micro evidence
or a large-scale success story to point to) it's appropriate to be cautious.
 Especially for non-medical interventions whose effectiveness is very much
in question.  I think this explains why there is pretty limited overlap
between the interventions that these guys recommend and the ones that
Abhijit Banerjee (who is much more focused on rigor) recommends.

------------

Women (http://www.copenhagenconsensus.com/Default.aspx?ID=1153) - top-level
recs are "Increase and improve girls' schooling" (#8), "Provide support for
women's reproductive role" (#10), Microfinance (#22),

pg 13: discussion of "Option 1 - Increase and improve girls' schooling"
Lots of refs on returns to education - looks like a pretty broad (not just
women-specific) lit review of the matter - may want to revisit at some pt

   - 15-16: not strong on building new schools.  however, ""the supply of a
   close and culturally appropriate school for girls can have sizable impacts
   on enrollments in some contexts provided that the school is quite close (not
   necessitating girls to travel more than half a kilometer on their own), a
   female teacher is present, and there are adequate sanitary facilities for
   girls (e.g.  Hill and King, 1995; Alderman, Orazem and Paterno, 2001; Orazem
   and King, 2007; Herz and Sperling, 2004).  In these contexts, specific
   supply-side interventions could help."
   - Endorses vouchers (16) and conditional cash transfers (17).

pg 21: Option 2 - reduce women's financial vulnerability through
microfinance
Refs arguing that "women who receive credit may command greater bargaining
power in the household" ... "women's access to credit resources also tends
to increase labor force participation.
CC lists difft kinds of microfinance but doesn't seem to distinguish between
them in its endorsement.

pg 28: Option 3 - Provide support for women's reproductive role
CC endorses (28):

   - Family planning programs for young women
   - Support for safe births
   - Emergency contraception and related services

The endorsement of #2 appears fairly "blanket."  29: "Experts recommend the
best way  to combat such risk is to ensure that delivery services are
provided by professionals skilled in obstetrics, both in health facilities
and in homes. Health centers providing primary care are needed to provide
prenatal care (including managing abortion complications), postpartum care,
and care of newborns (Graham et al. 2006).  Routine prenatal care includes
screening and treatment of syphilis, immunization with tetanus toxoid,
prevention and treatment of anemia, and prophylaxis or bed nets for
preventing and treating malaria.  Basic emergency obsetric care (BEmOC)
should also be available but is highly dependent on the availability of
supplies, drugs, infrastructure, and skilled health care providers. In case
of need,  a rapid referral communication chain is needed between
district-level hospitals and the primary-care level. District hospitals must
be able to provide surgical interventions and blood bank services.  Lastly,
routine physical examinations of postpartum women are critical, a difference
from the focus on education."

Nutrition supplements too (28): "Appropriate interventions could include the
provision of multivitamins, minerals, or macronutrient supplements, such as
protein-energy supplements as well as iron and folic acid to combat anemia.
While evidence of the impacts of such policies has been limited, Graham et
al. (2006) find  that interventions in addressing maternal health are more
cost-effective if nutritional supplementation is included."

More on 31-32 on what sorts of programs they picture.  I'm putting them down
for general support for maternal mortality programs for now.

Option 4 is affirmative action, outside our scope (and not in the final
Copenhagen Consensus list).

Water and sanitation (
http://www.copenhagenconsensus.com/Default.aspx?ID=1150) - top-level recs
are biosand filters (#15), "rural water supply" (#16), total sanitation
campaign (#20), "large multipurpose dam in Africa" (#24).

This paper is written in a very hedgy tone.  It doesn't flat out say "X is
better than Y" but it often implies that X > Y, and of course the master
Copenhagen Consensus table (
http://www.copenhagenconsensus.com/Default.aspx?ID=953) lists as
"recommended solutions" things that the paper might call "illustrations"
(more on this below).  My strategy here is to take the things that match
what the master table lists and call them "recommendations," even though the
paper often concedes that other things might work better under different
circumstances.

Pg 48 makes fairly clear which interventions these are.

   - Rural water supply = borehole well construction with hand pump.
    Mentions that these only make sense when the water source is below-ground
   (57); the cost-effectiveness depends on # users (>500 gets overly crowded
   and inconvenient); the value of labor (since one of the main benefits of
   boreholes is to save people time); the incidence of diarrhea; and the
   availability of water from other sources.  57-58 concede that rural water
   supply programs have had significant failures, but claims that a new
   "demand-driven" approach is working.  Cites refs for both (refs for the
   latter are all from 2007 - so we may want to check this out).
   - Total sanitation campaign = behavior-focused campaign along the lines
   of Community-Led Total Sanitation Program (we discussed this prog briefly
   at http://blog.givewell.net/?p=261).  76-90 gives a couple arguments for
   the superiority of behavior focus over focus on building latrines.
   - Biosand filters: CC says We selected the biosand filter for
   illustrative purposes; we do not argue that it is the "best" of the
   available POU technologies (90).  However, it then goes on to give args
   for why it's good: has been demonstrated to be safe & effective, is widely
   in use, is convenient/simple to install (and uses easily available
   materials).
   - Multipurpose dam in Africa: In recent years large multipurpose dams
   have been among the most controversial infrastructure projects in both
   industrialized and developing countries (World Commission on Dams, 2000).
   Proponents cite several types of direct economic benefits: hydroelectric
   power generation, domestic and industrial water supply, drought mitigation,
   recreation, irrigation, and flood control. They also claim a variety of
   indirect benefits (e.g., increased employment, better diplomatic
   relationships between riparians on international rivers, reduced risk of
   conflict over water resources, improved trade, and enhanced economic
   integration).18 On the other hand, critics believe that these benefits are
   overstated or nonexistent, that the high construction and resettlement costs
   are underestimated, and that negative side effects, especially environmental
   and cultural losses, are high (Duflo and Pande, 2007). Table 24 presents a
   list of the types of costs and benefits typically associated with dam
   projects.  (103)

Education (http://www.copenhagenconsensus.com/Default.aspx?ID=1147) -
top-level recs are deworming and other school nutrition programs (#6),
lowering the price of schooling (#7), conditional cash transfers (#17).

Education paper opens with review of returns to schooling literature.

Favors primary schooling interventions (11-12) as opposed to programs
targeted at later education.  Favors demand-side over supply-side
interventions (discussion begins on pg 25) - this means that they want to
focus on getting kids to attend rather than on improving/building schools.

   - They provide pretty much a blanket recommendation for nutrition
   programs aimed at school-age children, with scattered refs on why this can
   matter (31-34; mostly attendance effects but some cognition/performance
   stuff).
   - "Lowering the price of schooling" means capitation grants to school
   operators (36), vouchers (37-38), and after-school tutoring programs (38).
   - Conditional cash transfers: this is the first place I've seen refs for
   programs other than (though still including) PROGRESA (39-41).

Diseases (http://www.copenhagenconsensus.com/Default.aspx?ID=1146) -
top-level recs are "expanded immunization coverage for children" (#4), heart
attack acute management (#11), malaria prevention & treatment (#12),
tuberculosis case finding & treatment (#13), HIV "combination prevention"
(#19), improving surgical capacity at district hospital level (#21), tobacco
tax (#28).

This paper is written by the DCP2 lead author and explicitly is aiming to
identify the "best of the DCP."  Has a long discussion of why disease
interventions are a good idea, including macro argument that technology is
responsible for improving health and some refs (starting on 16) for the link
between health/productivity.

Pg 51 summary table gives slightly more clarity on the above listed.  When
it was still unclear, I hunted down the discussion to figure out exactly
what they were talking about.

A couple confusing things:

1. The paper lists a lot of under-5 health interventions on pgs 29-33, not
all of which make the final table on pg 51.  It seems likely to me that the
field was narrowed based on issues of scalability (33-35); this isn't fully
spelled out, but the table on pg 51 includes a "level of capacity required"
column.  I've listed the interventions that aren't in the final table, below
(immediately after the ones that are in the final table).

2. The rank-order given by the paper's authors is not the same as the one
given by the Copenhagen Consensus.  For example, the authors state that "TB
[tuberculosis] treatment stands out as perhaps the most important
investment" (53) and, consistent with this, rank it first; but in the
overall Copenhagen Consensus, TB treatment is ranked 13th, below 3 of the
other interventions in the table (vaccines is #4).

Just so we don't lose any info, what I've done is created 2 columns in our
intervention summary table.  One holds the official Copenhagen Consensus
rank for an intervention; the other, labeled "Jamison/Jha/Bloom", lists the
ranking given in this paper, "honorable mention" for those that are listed
in the paper but not in the final table.

INTERVENTIONS IN THE FINAL TABLE

   - Expanded immunization coverage (from pg 32) - Copenhagen Consensus #4
   of 30, Jamison/Jha/Bloom #4 of 7
      - Expanding the traditional Expanded program on Immunization
      (diphtheria-tetanus-pertussis vaccine, BCG vaccine for tuberculosis and
      meningitis, polio vaccine, measles vaccine - see
      http://dcp2.org/pubs/DCP/20/Section/2680)<http://dcp2.org/pubs/DCP/20/Section/2680>
      - HiB vaccine
      - Hepatitis B vaccine
      - Rotavirus vaccine
      - Streptococcus (pneumococcal disease) vaccine
   - Heart attack acute management = acute management with low-cost drugs (I
   assume this is the "aspirin and beta-blockers" from DCP and conversation
   with Prabat Jha) (43-46).  Copenhagen Consensus #11 of 30, Jamison/Jha/Bloom
   #2 of 7.
   - Malaria prevention & treatment = "prevention and ACT treatment package"
   = (from pg 32).  Copenhagen Consensus #12 of 30, Jamison/Jha/Bloom #3 of 7.
      - Insecticide-treated bednets
      - Drug treatment specifically for pregnant women (intermittent
      preventive treatment for pregnant women)
      - Indoor residual spraying with DDT
   - Tuberculosis case finding & treatment = DOTS strategy (41-42).
    Copenhagen Consensus #13 of 30, Jamison/Jha/Bloom #1 of 7.
   - HIV combination prevention = lists lots of things, without recommending
   a particular combination.  Copenhagen Consensus  #19 of 30,
   Jamison/Jha/Bloom #6 of 7.  "prevention efforts appear to work best when
   there is national leadership and simultaneous, sustained investment in
   multiple approaches to prevention, including efforts to reduce
   stigmatization of vulnerable groups" (38).  39-41 argues for caution in ART
   (blogged about this at http://blog.givewell.net/?p=329).  Components
   listed on pgs 36-38:
      - Peer interventions among sex workers (such as the one conducted in
      Thailand, condom distribution & promotion)
      - Treatment for sexually transmitted infections (other than HIV/AIDS;
      they increase risk)
      - Voluntary counseling & testing
      - Prevention of mother-to-child transmission through antiretroviral
      therapy
      - Needle safety & blood exchange programs
      - Improving surgical capacity at district hospital level =
   specifically for difficult childbirths and injuries (52).  Copenhagen
   Consensus #21 of 30, Jamison/Jha/Bloom #7 of 7.
   - Tobacco tax (46-51).  Copenhagen Consensus #28 of 30, Jamison/Jha/Bloom
   #5 of 7.

JAMISON/JHA/BLOOM HONORABLE MENTIONS

   - Stillbirth and neonatal interventions - not in table - refers to
   "Newborn survival" chapter of DCP report. (30)  Checked out that chapter and
   there are an enormous # of interventions; for now I'm just entering a
   "Newborn survival" category in the table.
   - Education interventions (30-31) - listed elsewhere in Copenhagen
   Consensus - argues that education is associated with lower infant mortality
   rates.  I am skeptical that there is any causative (as opposed to
   correlative) linkage here.
   - Breastfeeding promotion (32) - not in table
   - Expand the use of the simple and low cost but highly effective
   treatments for diarrhea and child pneumonia through integrated management of
   childhood illness or other mechanisms (32) - not in table
   - Micronutrient distribution (esp Vitamin A, Zinc, iron) (32) - listed
   elsewhere in Copenhagen Consensus.

I've updated the summary table of interventions - top link at
http://www.givewell.net/wiki/index.php?title=DWDA_intervention_writeups
The idea of this table is to list every intervention that

   - Carries a recommendation from an expert explicitly aiming to compare
   interventions and recommend particularly strong ones.  The comparisons I've
   used (and these are all the appropriate ones I know of, i.e., from people
   trying to make broad comparisons using consistent criteria and not from
   advocates for particular charities or programs):
      - Copenhagen Consensus (highly focused on cost-effectiveness, as my
      last email mentions)
      - Jamison/Jha/Bloom: 3 Copenhagen Consensus authors who had their own
      take on top interventions (as my last email mentions)
      - Abhijit Banerjee, whose paper "Making Aid Work" (chapter 2 in this
      book http://www.cgdev.org/content/publications/detail/16446
<http://www.cgdev.org/content/publications/detail/16446>-
      summary table available at
      givewell.net/files/Analysis/banerjee%20table.doc) lists interventions
      supported by randomized controlled trials

and/or

   - Has a "track record of success" of some kind, including
      - Major (country-level) aid success story, such as those discussed at
      http://groups.yahoo.com/group/givewell/message/25
      - Rigorous (ideally randomized) micro-level evaluation, such as those
      discussed by Banerjee

And "flag" the most promising ones to (a) investigate further (using reports
such as
http://www.givewell.net/wiki/index.php?title=Intervention_writeup:_Prevention_of_mother-to-child_HIV_transmission_through_antiretroviral_therapy)<http://www.givewell.net/wiki/index.php?title=Intervention_writeup:_Prevention_of_mother-to-child_HIV_transmission_through_antiretroviral_therapy>,
so we can be clear on the evidence for them; (b) finalize our list of
priority interventions (currently at http://www.givewell.net/research-agenda)
so that we can flag charities that focus on these interventions.

I'm not quite sure yet exactly what our criteria should be for "flagging" an
intervention.  I'm thinking something along the lines of at least 1 expert
recommendation + at least 1 strong piece of past success (either randomized
controlled trial or country-level success story).

ClinicalTrials.gov appears to be a repository of studies, nearly all
health-related and nearly all using randomized-controlled-trial design.  For
health interventions that are associated with lots of studies, we're
generally using other sources (like the Disease Control Priorities report
and the Cochrane library) that summarize literature, but this could be a
good resource for finding recent studies (which our lit-review sources
aren't recent enough to catch) on programs that don't have a long history of
evaluation.
--

For example, two strong-looking studies imply that circumcision can cut HIV
risk in half.
http://clinicaltrials.gov/ct2/show/NCT00098319?spons=gates&rank=54&flds=Xabn
 and http://clinicaltrials.gov/ct2/show/NCT00059371?term=circumcision&rank=6
 and http://clinicaltrials.gov/ct2/show/NCT00425984?term=circumcision&rank=4.
 The DCP lists these studies as ongoing but doesn't give their results.

Searching for "gates" and "usaid" in the "funders" field turns up a lot of
interesting-looking studies, particularly on vitamin supplementation and
anti-malaria campaigns.  Most of these don't seem to be completed/available,
though.  (Two particularly interesting ones: comparison of 3 types of
micronutrient supplements in Ghana
http://clinicaltrials.gov/ct2/results/rss.xml?spons=usaid&rcv_d=14&count=1000and
effect of folate on cleft palatt deformity
http://clinicaltrials.gov/ct2/show/NCT00098319?spons=gates&rank=54&flds=Xabn)<http://clinicaltrials.gov/ct2/show/NCT00098319?spons=gates&rank=54&flds=Xabn>

Awesomely, you can get an RSS feed for last-14-days results for any search,
so I've subscribed to these two searches.

I've posted a doc that gives a proposed structural outline for the report
we're working on.
There's no substantive info in there about aid; the aim of the doc is to
make clear how information will be presented.  If you'd like to provide
feedback on it, we'd appreciate it.  To view it, click the top link on
http://www.givewell.net/wiki/index.php?title=Links_for_2009_report (this is
a new page I just made that just links to a bunch of other pages).

We're finalizing intervention reports for Carter Center programs and one
thing we haven't found discussed consistently in the DCP (Disease Control
Priorities Project) is the clinical evidence of effectiveness for drugs
(i.e. that a given drug has a *medical* impact on the condition it treats).
For example, in the case of albendazole to treat soil-transmitted helminths
(worms), the studies we've seen cited discuss weight gains adn cognitive
performance but not infection rates directly. I went looking for a
comprehensive source that presents the criteria the WHO uses to approve
drugs.

*Bottom line*:* *I didn't find anything we can point to as *the source* that
WHO recommended drugs work medically. For each drug though, WHO cites its
source. In the case of albendazole, it's Cochrane reviews. I haven't checked
others to see how they'll play out but would guess it'll be similar.

*Details:*

   - There is a WHO Model List of Essential Medicines (Children:
   http://www.who.int/childmedicines/publications/EMLc%20(2).pdf<http://www.who.int/childmedicines/publications/EMLc%20%282%29.pdf>;
   Adults:
   http://www.who.int/medicines/publications/08_ENGLISH_indexFINAL_EML15.pdf).
   The report says, "The core list presents a list of minimum medicine needs
   for a basic health care system, listing the most efficacious, safe and
   cost$B!>(Beffective medicines for priority conditions." (Children, Pg 1) But, it
   never says anything about the criteria used to evaluate "efficacious."
   Poking around on the WHO Medicines site (where I found this list), I didn't
   find anything either.
      - This report includes albendazole. It's marked for review:. "Review
      evidence of efficacy and safety of use of
      anthelminth/antifilarial/antischistosomal and antitrematode medicines in
      children below the specified age in current licences."
(Chidlren, Pg 5) But,
      that seems to only be for children under the age of 24m.
   - WHO has a report on the necessary guidelines for a good clinical trial
   and how to evalaute drug effectiveness:
   http://www.who.int/medicinedocs/en/d/Jwhozip13e/1.html. It's not clear
   that this ties back to recommended drugs. Maybe it does; I didn't find it.
   - The WHO has an Essential Medicines Library, where you can search for a
   specific drug and see how, when, why it was included:
   http://www.who.int/emlib/. Here's the search for albendazole:
   http://www.who.int/emlib/MedicineDisplay.aspx?Language=EN&MedIDName=10%40albendazole.
   It was initially included for a reason unrelated to STHs, though it's
   currently used for STHs. This page seems like the best aggregate source
   we'll find. The page lists references to papers at the bottom.
      - Ref 1:
      http://www.who.int/medicinedocs/en/d/Js4953e/5.2.html#Js4953e.5.2:
      Seems like only evidence is nutritional outcomes (though the
fact that they
      cite cognitive outcomes is weird/worrisome given that we don't think
      anyone's (Miguel/Kremer or Cochrane) really claiming that
effect: "A review
      of available data on health benefits of treating
soil-transmitted helminth
      infections in these young children concluded that treatment reduced the
      likelihood of growth stunting and favourably influenced nutritional and
      cognitive outcomes. The conference also reviewed human and animal
      toxicological data and concluded that in children as young as 12 months
      there are no reasons for exclusion from treatment with albendazole or
      mebendazole according to existing literature and company drug information
      (1). Hence, a recommendation was made that children from one year old
      onwards should be included in systematic deworming programmes."
      - Ref 2: http://www.who.int/wormcontrol/documents/en/pvc_20024full.pdf:
      Major review of whether some drugs can be given to children under age of
      24m. Gets into nitty gritty medical details -- "Albendazole binds to
      intracellular tubulin, selectively affecting helminths and inhibiting
      essential absorptive functions in the organism." I can't figure out if
      section 6.2 on Pg 16 is telling me that it's effective or not.
It does say,
      "Unmetabolized ALB [albendazole] is directly effective against intestinal
      nematodes in the alimentary tract." (Pg 24 -- there's no ref her, but
      there were paper refs back in section 6.2, so maybe those are
the studies)
      Pg 27 reports on 8 studies in Africa of albendazole in children under 24m
      and the results. It's a little confusing. Sometimes it talks
about another
      drug (mebendazole) and sometimes it talks about impact when combined with
      iron supplements. Finally, it says that it resulted in "reduction in
      parasite prevalence and/or intensity after treatment;" but also
that "Most
      of the studies that have been done involved small sample sizes
and in some
      cases parasitological data, including measures of intensity of infection,
      were lacking."
      - Ref 3 and 4: Cochrane papers we've read (one for albendazole for LF
      and one is the earlier version of the paper Ron read for STH)
      - Ref 5: cite with no link or seemingly unrealted to albendazole for
      STHs
      - Ref 6: WHO guidelines paper thing from 1990:
      http://whqlibdoc.who.int/trs/WHO_TRS_796.pdf. Page 11 does say, "Only
      those drugs should be selected for which sound and accurate data
on efficacy
      and safety are available from clinical studies."

I skimmed this kinda quick, and I'm a bit tired, so take it for what
it's worth:

I like the starting approach.  What Givewell.net has now - a
reasonably short, punchy 'story' on the front page (bottom left area)
seems like a good model.  For each point - one key sentence/statement,
then a few sentences of quick support and a clear link to details.

Hierarchical approaches seem good to me.  Putting your basic
conclusions up front allows readers to skim them and see what you're
about.  For the ones that particularly interest the reader, they can
click through.

I don't know what the exact right amount of hierarchy (layers) is,
BUT, it is important that the reader can follow a continuous set of
clicks to drill down from high level to low level and eventually to
strong supporting research, linking where appropriate to academic
research and the like.  You want to make clear that the short
statements on the front page are not the rambling opinions of a
blogger, but rather, well researched pieces relying on the best
readily available information.  If a reader checks you on one area and
finds strong support, they're more likely to accept what you have to
say on other areas even if they don't drill down for all the details.

Anyways, back to your proposed structure:

I didn't quite understand how 'Detail pages' fit in - where are they
linked from?  Again, it may have been explained and I missed it.

You may already be planning to do so, but you'll probably want to list
the charities in several ways.  From the program side, you'll want to
link to the relevant charities, but you'll probably want to make a
list of rated charities of all types directly accessible in some way.

It's a little hard to visualize what things will look like based on a
Word document, but from what I can tell it seems like a reasonable start.

Thanks for the thoughts.
In response to your question: the "Detail" pages (in the "Issues" section)
are linked to by the "Headline" and "Question" pages.  Each of the bullet
points under "Headlines" and "Questions" specifies which "Detail" pages are
linked to for support.  "Detail" pages are the footnoted pages that try to
give more of a full picture.

Basic idea: one page with a bunch of blurbs; each blurb links to a page
giving a couple key charts and the big-picture overview (Headlines,
Questions); each of those links to one or more pages with referenced
summaries of relevant research (Detail).

We should be able to send out samples of some of these pages in the coming
weeks (the individual Charity, Intervention, etc. pages - not yet the "main
pages" that aggregate the information).

On Thu, Feb 12, 2009 at 9:23 PM, psteinx <psteinmeyer@...> wrote:

>   I skimmed this kinda quick, and I'm a bit tired, so take it for what
> it's worth:
>
> I like the starting approach. What Givewell.net has now - a
> reasonably short, punchy 'story' on the front page (bottom left area)
> seems like a good model. For each point - one key sentence/statement,
> then a few sentences of quick support and a clear link to details.
>
> Hierarchical approaches seem good to me. Putting your basic
> conclusions up front allows readers to skim them and see what you're
> about. For the ones that particularly interest the reader, they can
> click through.
>
> I don't know what the exact right amount of hierarchy (layers) is,
> BUT, it is important that the reader can follow a continuous set of
> clicks to drill down from high level to low level and eventually to
> strong supporting research, linking where appropriate to academic
> research and the like. You want to make clear that the short
> statements on the front page are not the rambling opinions of a
> blogger, but rather, well researched pieces relying on the best
> readily available information. If a reader checks you on one area and
> finds strong support, they're more likely to accept what you have to
> say on other areas even if they don't drill down for all the details.
>
> Anyways, back to your proposed structure:
>
> I didn't quite understand how 'Detail pages' fit in - where are they
> linked from? Again, it may have been explained and I missed it.
>
> You may already be planning to do so, but you'll probably want to list
> the charities in several ways. From the program side, you'll want to
> link to the relevant charities, but you'll probably want to make a
> list of rated charities of all types directly accessible in some way.
>
> It's a little hard to visualize what things will look like based on a
> Word document, but from what I can tell it seems like a reasonable start.
>
>  
>

I didn't attend the whole event, but saw talks by William Easterly (whom
I've referred to repeatedly on this research list), Esther Duflo
(co-Director of J-PAL, see http://blog.givewell.net/?p=219), Nancy Birdsall
(President of the Center for Global Development, a think tank whose research
we've read and written about a lot), and Dennis Whittle (Chairman/CEO of
GlobalGiving.com).  I also made met a couple of people including a guy from
the World Bank's Independent Evaluation Group, whom I'm following up with to
request a phone conversation.

Most of what they said was familiar from the reading I've done ... here are
notes on what wasn't.  These are just the notes I took - not transcriptions.

William Easterly:

   - "When I was at the World Bank, when we were really feeling the heat is
   when we would start naming goals that didn't mean anything and couldn't be
   measured.  Like 'Gender empowerment' and 'Community-driven development.'"
    This seems good to keep in mind since we see these goals named often.
   - Wants to see more "Decentralized accountability, like a Consumer
   Reports for aid organizations."
   - Great story about USAID.  Says that they used to get heat for having so
   much "tied aid" (aid that has to be spent on US products).  Their response
   was to stop reporting tied aid to OECD-DAC, even though it's required.
    Easterly and his researchers tried to find data on US tied aid but found
   ranges from ~30% to ~95%.  Found a doc on the USAID site bragging that 80%
   of US aid is spent in the US.  Later, the doc had been removed.  Asked the
   audience to commit verbally to holding USAID accountable for not reporting
   tied aid.
   - Recommends "Voices of the Poor" by the World Bank, a giant collection
   of survey responses from people in extreme poverty.
   - Cited a paper by Reineke and Svenson about how monitoring reduces
   corruption.
   - I went to the mic and said even though I agree with the idea of the
   World Bank's being more accountable, in my experience looking at private
   NGOs I often find myself wishing they were as transparent and accountable as
   the World Bank.  (At which point he interrupted to say "That's a scary
   thought.")  Asked if he had the same experience and what he thought of NGOs
   vs. official aid agencies.  He said he thinks NGOs often sound good but
   aren't necessarily doing anything well, and that they need to be held much
   more accountable than they are now.  "OECD-DAC has its limitations but
   there's no database for NGO money flows."  Told a horror story about an
   email he had just received (last couple days) where a key number an NGO had
   been citing turned out to be completely fabricated.

Dennis Whittle of GlobalGiving:

   - Had the audience vote via cellphone on whether we wanted a bridge,
   pump, or clinic for our hypothetical village.  Says he's looking into how to
   do this for real villages.  (Or maybe GG already does it, can't remember for
   sure.)
   - From this and White Man's Burden (which mentions GlobalGiving), I see
   that there is a big concern with "how do we get the money to the poor, for
   projects they want?" - a question that's similar to, but different in
   emphasis, our focus on demonstrably improving life outcomes (health, income,
   etc.)  There was an implication that so much of aid gets swallowed up in
   bureaucracy and/or govt and/or programs that nobody wants, that just getting
   those steps right is a lot.  The cool thing about the voting exercise is
   that it isn't just asking the poor "Do you like this project?" (that's too
   low of a bar, as Easterly observed - people usually will give a thumbs-up to
   something that's free, but that doesn't mean it was a good use of funds) -
   it's finding a project that they prefer to another project of similar costs.
   - Global Giving has "moved" $17 million total over its history.  (It was
   founded in 1997.)

Nancy Birdsall, Director of CGDev: presented a "cash on delivery aid"
proposal (I've seen this paper on their website but not looked carefully at
it).  The only thing really relevant to us is that it's again coming from
the mentality of "how can we get money to the poor instead of their horrible
governments."  She said something like "Stop giving money to these crooks"
in reference to direct-to-govt aid.

I am paritosh from pune, India.
We are a group of people who support free and open source software.
Free software can save lot of money for NGO's and also bring complete
 transparency if they want to.
More the transparency more confident the donors would be.
And more people could study the model discuss and give ideas.

Free software also reduces the need to outsource and creates more
local jobs.

regards
paritosh

A quick note:
We prefer that list emails be restricted to substantive feedback/updates on
our research.  We don't have any hard rule in place for this, but recommend
that new people read the list for a while (and/or read the history) to try
to get a sense for the kind of conversation that goes on, before posting.
 To encourage this and reduce clutter, going forward new list members will
be moderated until they've been on for a while.

On Mon, Feb 16, 2009 at 2:36 PM, paritoshpungaliya <
paritoshpungaliya@...> wrote:

>   I am paritosh from pune, India.
> We are a group of people who support free and open source software.
> Free software can save lot of money for NGO's and also bring complete
> transparency if they want to.
> More the transparency more confident the donors would be.
> And more people could study the model discuss and give ideas.
>
> Free software also reduces the need to outsource and creates more
> local jobs.
>
> regards
> paritosh
>
>  
>

I'm now transitioning my in-progress findings on general aid issues from
"wiki" (my informal thoughts with scattered links) to "blog" (still somewhat
informal, but trying to be more careful about stating exactly where we stand
and what the most relevant links are).  Part of the aim is to get things in
better shape for people to give feedback on.

I'm doing this over a series of blog posts on the broad topics I've looked
at; the first one is up now at http://blog.givewell.net/?p=344 .  Any
thoughts on these (both on the content and on whether you're able to
follow/engage with them) are much appreciated.

Now linked from the front page of GiveWell.net <http://www.givewell.net/> is
a preview of our 2008-2009 report. The main content of the report so far is
a review of the Carter Center as well as information on the track records of
the programs it runs and the diseases it targets.

There is much more on the way, but for now, the review of the Carter Center
(and accompanying materials) will give a strong sense of our basic
structure, approach, and criteria, which have changed significantly since
our 2007-2008 report. We are eager for feedback.

Thought this was interesting.

- Brian

-------

For malaria, we just can’t afford to use cheap drugs

"There are two ways to take anti-malarial drugs: the expensive way,
which helps the world; and the cheap way, which helps only the
patient. Most Africans cannot afford the expensive way and, as a
result, the world’s most effective anti-malarial drug may lose its
potency."

http://timharford.com/2009/03/for-malaria-we-just-can%E2%80%99t-afford-to-use-cheap-drugs/

I've been working my way through a large set of developing-world
organizations to identify those we might recommend in the upcoming report.
We're interested in your thoughts on this process and the output, so please
send feedback. Let me know if you have any questions or if any of this is
unclear.

This email includes:

   - Methodology we've used to identify top charities
   - How we generated the list of charities to consider
   - Findings so far

My work is in this file:
http://www.givewell.net/files/Analysis/Charity%20scan%202009%2003%2018.xls

*Methodology we're using to identify top charities*

We're choosing charities based on:

   - *Program selection. *Run programs that are on our top interventions
   list. (Roughly the programs listed in this file:
   http://www.givewell.net/files/Analysis/top%20interventions%202009%2002%2003.xls<http://www.givewell.net/wiki/index.php?title=DWDA_intervention_writeups>)
   Organizations that run many programs must run programs that are
*mostly*on our list.
   - *Monitoring.* Charities that publish reports which detail the *outputs
   *of their work (e.g., pills* *distributed, classes taught, surgeries
   performed) on their website.
   - *Financial transparency.* When applicable, charities that break down
   their finances beyond what the IRS Form 990 requires, detailing costs for
   each program or country.
   - *Anything thing else of note.* I've highlighted charities that
   publish/cite academic articles; charities that have large sets of
   publications; and, anything else that positively distinguishes that
   organization.

Also, I've flagged any charity that works entirely on Health programs (which
are often well-proven) as opposed to providing health along with many other
types of programs (like community empowerment, disaster relief, gender
focus) for which evidence of effectiveness and cost-effectiveness is less
strong.

The goal at this point is to relatively quickly scan this list and find
those organizations that are worth evaluating at a deeper level.

*How we generated the list*

Our list has 321 charities. (We'll expand this list if you have additional
charities that you think meet our criteria, so pleases send them our way.)
We created this list by:

   1. Charities submitted through our charity submission form (
   http://givewell.net/submitcharity) that work in the developing world. -
   93 charities
   2. Charities that Charity Navigator tags as working in International
   Relief and Development with budgets over $13.5m, the largest budget filter
   CN provides - 53 charities (We may look at additional charities from CN,
   depending on time)
   3. Charities that partner with Innovators for Poverty Action or JPAL - 40
   charities
   4. Global Health Partnerships (organziations like the Global Fund, GAVI,
   Global Alliance to Eliminate Lymphatic Filariasis) - 107 organizations
   5. Other miscellaneous organizations we found or were recommended, of
   which I've emailed about before - 28 charities

*Findings so far*

I've scanned 233 organizations thus far. I've given each charity a rank
(Column O in the file) based on the criteria defined above. I think that all
charities with a score of 25 or better -- lower is better -- are worth
another look. At the moment, there are 45 charities with that meet that
standard, about 5 of which we've already looked at somewhat closely.

I was relatively liberal as I went through these, so I believe that once I'm
finished with this scan and we have a better sense of what types of
organizations are out there, we'll be able to narrow this list down further.

How do you derive the rank?

The rank isn't determined systematically at this point. At this point, I'm
just trying to quickly flag charities that I want to go back to, so there's
no strong distinction between, for example, a "5" and a "25".

There are really 3 meaningful groups: (the additional detail was more for my
own use so I could sort them in the order that I wanted to go when
re-evaluating organizations)
1. < 25 = needs another close look
2. 26 - 100 = probably not going to work out, but possible
3. > 100 = out

It's roughly a combination of a) the degree of monitoring an organization
has, b) their implementation of our priority programs, and c) other factors
(e.g., publishing research on their website related to, but not directly
monitoring, their programs). For example:

   - MVP gets a high rank because I think there's enough information
   available on their website to do a relatively comprehensive evaluation of
   their program.
   - The Global Alliance to Elimination Lymphatic Filariasis and Deworm the
   World get high ranks because they only implement one program, and both are
   high on our priority list.
   - Doctors without Borders is a) all health b) has some monitoring, and c)
   publishes what they learn in peer-reviewed journals, but d) doesn't provide
   comprehensive monitoring on their website, and d) doesn't exclusively
   implement priority interventions.

The goal of the scan is to quickly sort through a lot of charities and get a
better sense of the number of potential recommendees. The next step is to go
back to the ones we flagged, take a closer look at their materials and be
specific about our remaining questions.

On Wed, Mar 18, 2009 at 5:43 PM, psteinx <psteinmeyer@...> wrote:

>   How do you derive the rank?
>
>  
>

We recently completed a preliminary report on Stop TB, and we're interested
in any feedback or questions. Details on our blog:
http://blog.givewell.net/?p=359

Insecticide-treated net (ITN) distribution programs are among those that
have strong independent evidence of effectiveness (see our report at
http://givewell.net/node/329).

One unanswered question in that report is: when are programs successful? Do
they require marketing and education or is distribution enough? Does
distribution through health clinics work or is mass distribution along with
vaccination campaigns better?

I tried to answer these questions by looking at the three examples of
large-scale, successful (as measured by reduced mortality) ITN programs in
Gambia, Tanzania, and Kenya.

In brief, there's not much to go on. None of the summary papers details the
distribution at a level that allows us to clearly state what factors lead to
success (and even if they did, there's no way to know that those factors are
either necessary or sufficient for success). Nevertheless, this is what we
gleaned:

   - All 3 programs had some degree of education/marketing connected with
   distribution, so distribution, alone, may not be adequate
   - All 3 programs had relatively high ITN-coverage rates in the target
   areas, so saturation might matter.

Relevant quotes from the papers follow. Full citations are on our bednets
report page: http://givewell.net/node/329#Sources

------------------------------------------------------------------------------------------------------------------------------------------

*Kenya*

(From Noor 2007 Pg 1342)

Phase 1: "In January 2002 the UK Department for International Development
(DFID) awarded PSI-Kenya US$33 million over 5 y to socially market partially
subsidised ITN within the existing retail sector. The programme, named PSI
CoveragePlus, was the only major operational ITN distribution initiative
between 2002 and 2004 and aimed to target urban and rural retail outlets
with Supanet ITNs across all malariaendemic districts in Kenya. A two-tier
pricing system of 350 Kenya Shillings (KES) (equivalent to US$4.7) in urban
settings versus KES100 (US$1.3) in rural settings was implemented."

Phase 2: "In June 2004, DFID approved an additional US$19 million to PSI to
establish a parallel distribution system of heavily subsidised ITNs to
children and pregnant women through Maternal and Child Health (MCH) clinics,
recognizing that these vulnerable groups might not be able to access
socially marketed commercial sector nets. The programme began in October
2004, and during the first 6 mo Supanet ITNs were bundled with separate
Powertab net treatment tablets (for every 6 mo) and distributed to MCH
attendees."

Phase 3: "The implementation of the free mass distribution of LLINs was
arranged in two phases during 2006. During the first phase, 21 of Kenya’s 70
districts were selected for distribution of LLINs from 8 to 12 July 2006 and
integrated with the national measles catch-up vaccination campaign. Health
facilities and centralised non-health facility posts were identified by the
Kenya Expanded Programme on Immunisation and used as delivery points of both
measles vaccine and LLINs to each child under the age of 5 y. A second mass
distribution of LLINs, not integrated with any other intervention, took
place from 25 to 27 September 2006 in 24 additional districts using previous
mass vaccine campaign delivery centres as distribution points."

"By the end of September 2006, the three principal net distribution
strategies (retail social marketing, heavily subsidized clinic distribution,
and free mass distribution) were all operating in parallel, providing an
opportunity to examine socioeconomic targeting of each of the delivery
mechanisms (Table 2)." Noor 2007, Pg 1345

There was also PSI marketing in mass media -
http://www.psi.org/resources/pubs/kenya-ITN.pdf, Pgs 1-2

Bottlenecks: "What we can say is that if funding is not secured for clinic
supply and catch-up mass campaigns for LLIN delivery beyond 2008 the
impressive, rapid progress toward the RBM target of 80% coverage by 2010 in
Kenya will be lost." Noor 2007, Pg 1347

"The effect of ITNs on mortality when delivered under operational conditions
has only been measured in The Gambia, as part of a national campaign,4 and
in one district in Tanzania after the promotion of socially marketed
ITNs.5,6" Fegan 2007, Pg 1035

*Tanzania*

"In an assessment of the only national programme of net treatment in Africa
to date, d’Alessandro and colleagues8 reported from The Gambia that communal
net treatment distributed free of charge by regional health teams led to
improved child survival. However, people were unwilling to pay for services
that had once been free, and mortality rates returned to their previous
values after the introduction of a cost-recovery programme." Armstrong 2001,
Pg 1241

"In 1996, we developed a social marketing programme, known as KINET, for
insecticide-treated nets in two rural districts of Tanzania (Kilombero and
Ulanga), with the aim of achieving substantial and sustainable use of such
nets in young children and pregnant women. The positive effect of
insecticide-treated nets on malaria and anaemia in children is described
elsewhere.12" Armstrong 2001, Pg 1242

"The social marketing was phased in from May, 1997, to June, 1999, starting
in the 25 villages covered by the demographic surveillance system (figure 1)
and reaching one or two more divisions every few months (figure 2). Treated
nets and insecticide for net treatment were introduced together in each
area. After sensitisation meetings in 1996, formative research studied
householders’ perceptions of causes of child death, mosquito nets, net
treatment, and malaria.18 Details of the social marketing programme are
given elsewhere.19 Briefly, treated nets (pretreated with 20 mg/m2
deltamethrin, supplied by Siamdutch, Bangkok, Thailand; A to Z, Arusha,
Tanzania; or TMTL, Dar es Salaam, Tanzania) and insecticide for net
treatment at home (lambdacyhalothrin, Icon, Zeneca, Haslemere, UK) were
packaged and branded according to local preferences. Sales agents in each
village included health workers, shopkeepers, religious leaders, and village
government members. At first, different agents were chosen for nets and
insecticide, but, over time, many agents started selling both products.
Successful agents were generally shopkeepers and a few health personnel.
Every division had a wholesale agent." Armstrong 2001, Pg 1242

"A comprehensive information, education, and communication campaign was
developed and implemented. Increased emphasis was given to the insecticide
when it became clear that insecticide-treated nets were more popular than
net treatment. Retail prices were set at around US$5 for a treated net and
$0·42 for insecticide treatment kits. Retail prices of nets remained the
same throughout the study period. Higher-dose net treatment kits were sold
at $0·50 from February, 2000. In 1997, ex-factory prices were subsidised by
about 25% for nets and 90% for treatment kits. By 2000, ex-factory prices
had reduced: nets were sold without subsidy and treatment kits had a 40%
subsidy. The cost of the information, education, and communication
campaign,  and distribution to wholesalers was about $1·70 per treated net
or insecticide kit." Armstrong 2001, Pg 1242

"We present results from the first assessment of a large scale social
marketing programme of insecticide-treated nets on child survival in Africa,
in an area of high-intensity malaria transmission. We have shown that social
marketing with a high cost-recovery level is an effective way to deliver
insecticide-treated nets. More than half of all infants in the DSS area were
sleeping under evertreated nets in mid-2000—ie, 3 years after the start of
the social marketing activities. Overall coverage of evertreated nets in the
two districts was 18% in children younger than 5 years in mid-1999."
Armstrong 2001, Pg 1246

*Gambia*

"National impregnated bednet programme In 1981, The Gambian Government
initiated a national Primary Health Care (PHC) programme; all villages with
a population of 400 or more were invited to join the scheme. Each
participating village selected a village health worker and a traditional
birthattendant who received 6 and 8 weeks’ training, respectively. In
June-July, 1992, the NIBP was implemented in about half (221) of the PHC
villages. Dipping of bednets was organised by Regional Health Teams and done
by a village health worker assisted by a traditional birth-attendant and the
head of the women’s group, supervised by community health nurses. Before the
intervention, people were asked to wash their nets. 40 mL insecticide (20%
permethrin) needed to treat each net was poured into a large plastic bowl
and 2 litres of water added to give a permethrin concentration on nets of
about 200 mg/m2." D'alessandro 1995, Pg 480

"The difficulties of distributing insecticide at the right time of the year,
of ensuring that insecticide is used at the correct dilution, and of
treating all the nets in a village are substantial. The NIBP employed a
manager (MKC), responsible for coordinating the programme, purchasing the
insecticide, organising the health education campaign, and liasing with the
different RHTs. However, at the local level, the NIBP was organised and
implemented mainly by rural health teams. Despite the many logistical
problems encountered, mortality in children was reduced significantly in
villages where bednets were used and treated with insecticide, indicating
that in these age groups, malaria is the most important cause of death, a
view supported by review of postmortem questionnaires. D'alessandro 1995

We've been struggling to find strongly evidence-backed programs in the
general domain of "programs that aim directly at raising incomes (as opposed
to targeting health or education").  I found a literature review from the
World Bank on rural extension services (i.e., working with farmers to share
knowledge) that more or less confirms that there is very thin evidence in
this area.  Key quotes below.
Link to the review: http://go.worldbank.org/5W9F4C57N0
--


Definitions (from page 2):

The goals  of extension include  the  transferring  of  knowledge  from
researchers  to  farmers, advising  farmers  in their decision making  and
educating  farmers on how to make better decisions,  enabling  farmers  to
clarify their own  goals and possibilities,  and stimulating desirable
agricultural  developments  (vander Ban and Hawkins,  1996). While
extension  agents often also provide services  that are not directly related
to farm  activities  (e.g.,  health, non-farm  business management,  home
economics and nutrition),  the  focus of discussion  in this paper is on
agricultural  and farm management knowledge  dissemination  (which may
include  financial  and marketing  information).

Re impact (From pages 22-24):

Birkhaeuser,  Evenson and Feder (1991)  made an  early review  of studies of
extension impact  and  found few studies  of systematic  comparison  of
costs and benefits with and without a project.  Systematic  social
experiments  comparing different methods of extension  in similarly situated
areas have  yet to be carried out. Where  extension programs have been
evaluated  by comparing  outcomes  in similar contiguous  areas,  the
results have been nuanced.  Thus,  careful work by Feder and Slade  (Feder
and Slade,  1986; Feder,  Lau and Slade,  1987)  comparing productivity
differentials  in Haryana and Uttar Pradesh  suggested  that T&V had no
significant impact  on rice production but yielded economic  returns of  at
least  15  percent  in wheat growing areas.  Similar work in Pakistan
(Hussain, Byerlee  and Heisey,  1994)  found even smaller impacts in wheat
areas,  although  the effect of T&V  in increasing  the quantity of
extension  contact was documented.  Although evaluations  of  extension
investments  have criticized  the observed  low levels of efficiency  and
frequent  lack of equity  in service provision,  they have  in the past
reported  relatively high benefit/cost  ratios  (e.g.,  Perraton et al.,
1983).

More recent  studies of extension  impacts have  also  shown significant  and
positive effects(e.g.,  Bindlish Evenson  and Gbetibouo,  1993; Bindlish and
Evenson,  1993)  and  intemal  rates ofreturn on extension  investments  in
developing  countries have  reportedly ranged  from 5%  to over50%  (Table  3)
(Evenson,  1997).  The overriding  lesson from Evenson's review of  57
studies  ofthe economic  impact  of agricultural  extension  is,
however,  that
impacts vary widely-manyprograms have been highly  effective, while
others  have
not. Extension  systems seem to havebeen most effective  where  research  is
effective  and have  the highest pay-off where  farmers havehad good  access
to schooling,  although doubtless  other factors also play key roles.

The most comprehensive  review of  impacts  is found  in a recent meta-study
of  289  studies of economic  returns  to agricultural  research and
extension.  This study  found median rates of return  of 58 percent  for
extension  investments,  49 percent  for research,  and 36 percent  for
combined  investments  in research  and extension  (Alston et al.,  2000).'
Similar  success has been documented  even  for Sub-Saharan Africa
alone  (e.g.,
Oehmke, Anandajayasekeram  and Masters,1997).  Economic  analysis has thus
provided fairly strong  justification  for many past extension investnents,
but does not tell  the  full story.


Concern over data quality along with difficult methodological issues
regarding causality and quantification of all benefits must, however, be
important qualifiers to the prevailing evidence of good economic returns
from extension. In Kenya, perhaps (from Leonard 1977, to Gautam 2000) the
most closely studied case in developing countries, although previous
evaluations had indicated remarkably high positive economic returns to
extension investments, a comprehensive evaluation based on improved and new
data revealed a disappointing performance of extension, with a finding of an
ineffective, inefficient, and unsustainable T&V-based extension system and
no measurable impact on farmer efficiency or crop productivity (Gautam,
2000). Such findings help to pose dilemmas for policy makers whose
skepticism(reinforced by observations such as those of Hassan, Karanja and
Mulamula, 1998) about getting returns to investment in public extension that
are actually rather low, seems more than well justified. It is not our
intention to end this survey on a note so salutary but evidently more
evaluative work is called for to better assist policy insights and
investment decisions.

We've found that most of the strongest charities - just based on scanning
websites - are in the area of health.  Because we want to recommend at least
one "directly aiming for economic benefits" charity if at all possible,
we're planning on using an approach more similar to last year's process -
i.e., a grant application - for this area.
Before we put together the application, we're trying to get a broad sense of
what different types of charities there are in this area and what sorts of
information will be reasonable to request.  We're talking to representatives
of various charities to try to get at this.  Below is a (conceptual, not
verbatim) transcript of a conversation I had last week with TechnoServe (
technoserve.org).  The representative has signed off on this transcript.

*Holden: I see you have 5 programs listed on your website under the Work and
Impact section.  Business plan competition, entrepreneurship training,
building businesses and industries, capital access, Aspen Network of
Development Entrepreneurs.  Could we get a breakdown of expenses, past
present and future, by these areas?*

TechnoServe representative: That information can probably be gathered but I
don't believe it's how we currently organize the financial information.  I
don't work closely with the financial department, but I believe they track
it by country and segment it by program and there's something like 65
programs, and the programs are largely segmented based on who's funding
them.

*Holden: informally, could you tell me which of the 5 programs are bigger
and which are smaller?  I'm guessing that Aspen Network for Development
Entrepreneurs doesn't take up much of TechnoServe's funding while the
business plan competition, entrepreneurship training, building businesses
and industries might be bigger.*

TechnoServe representative: the Aspen Network is tiny as far as our
resources, it's basically a small commitment for one staff person.
Entrepreneurship
training (outside of business plan competitions) is fairly small at this
point I believe.  I also believe that capital access is fairly small, some
pilot projects (I'm referring to capital access-specific projects although
other projects involve assistance with capital access).  The business plan
competition category is bigger.  Building Businesses and Industries is the
biggest by far.

*Holden: What about within the programs, are we going to be able to get a
breakdown by for example how much is spent on staff vs. capital costs vs.
vouchers for business services?*

TechnoServe representative: I don't know whether it would be easy to break
it down that way.  So you know, business vouchers are almost always provided
by companies in that country and are not funded or even funneled by
TechnoServe.  I'm not sure we'd count them even in our revenue.

*Holden: So it sounds like the main programs involve fairly intensive
assistance for businesses.  Does TechnoServe ever or often take a stake in
these businesses or recoup funds?*

TechnoServe representative: There have been a few isolated incidences where
some kind of fees are recouped, perhaps as a percentage of profits, but that
may have happened in somewhere between 1% and 5% of cases.  By no means is
it standard.



*Holden: so it sounds like basically TechnoServe's main activity is
providing free business assistance funded by donations.*

TechnoServe representative: That's right.

*Holden: so how do you track the impact of these activities and see how
you're doing?  What are your metrics?*

TechnoServe representative: Basically, for the enterprises that we work
with, we track them to see how much revenue they're generating and how many
people they're employing, and how much money they're paying in wages, and
how much they're buying from small-scale suppliers, so basically the
financial bases of the enterprises that we're helping.

*Holden: And do you track that information before and after TechnoServe
steps in or is it just after?  *

TechnoServe representative: I think someone else will have to answer that
question.  I'm not sure how much of a baseline is done beforehand.

*Holden: And do you have any information or analysis on the question of the
counterfactual, how would these businesses be doing if not for TechnoServe?
Stats on comparable businesses or businesses that didn't quite make the cut?
*

TechnoServe representative: We will tend to have some general information
along those lines, for example, for cashew farming, how much the average
cashew farmer in Mozambique makes as a comparison for how much
TechnoServe-assisted cashew farmers are making.

*Holden: Do you track information on the standard of living for
entrepreneurs and employees?*

TechnoServe representative: We have info on the standard of living in an
area, how much people make per day, but I'm not sure that we measure the
work on a standard of living basis directly, except in an anecdotal fashion.


More in-depth evaluations have also been done that try to get at this
question qualitatively in certain areas.  Larger-scale surveys.

*Holden: OK so to summarize, the full picture of how TechnoServe is tracking
its impact is (1) tracking financial information for businesses you work
with, (2) having general information about the areas you work in, both
overall incomes and incomes for people who might be considered comparable to
the people you're helping, (3) more in-depth evaluations using survey data
to spot check how things are going.*

TechnoServe representative: That's right.

*Holden: Now I'd like to know how much of this information we might expect
you to share with (a) GiveWell (b) the public at large, i.e., give us
permission to make the materials public.  First on (1) the financial
information on businesses?*

TechnoServe representative: This information is in the Annual Report in the
flowchart.  *[Holden's note: this flowchart gives # businesses, total
revenues, total profits, total wages, total product purchases from
small-scale producers, # families who "benefited from these income sources"]
*We wouldn't be able to share it at an individual business level because
that would be disclosing businesses' financials which could hurt their
competitive position.  It should be straightforward to share this
information on a country level.

*Holden: (2) general info about the area, standard of living?*

TechnoServe representative: I'm not sure.  This isn't one of our required
core indicators, so it may not be standardized.  I could look into how much
could be shared.

*Holden: What are the required core indicators?*

TechnoServe representative: The numbers given in the annual report are the
required core indicators.

*Holden: What about sharing (3) in-depth evaluations?*

TechnoServe representative: I can find out how much of that is available and
how much people feel comfortable sharing that.  That may not be a big issue.
**

Sounds to me like there's very little info here that could offer meaningful, statistically reliable insight into this group's effectiveness.

Apologies for the long silence on general aid issues.  Because I was
traveling a lot, I started throwing notes in a Word doc instead of in wikis
and emails, and I just now got the chance to sort it all out, so this is a
massive dump of new info.
I've done a lot more reading, and I've thoroughly revised two pages:

1. Page listing our planned writeups on general questions about aid -
http://www.givewell.net/wiki/index.php?title=Issues_pages .  At this point I
think we have enough information available to answer nearly all of these
questions (that means we know where the information is - not necessarily
that we've analyzed it to our satisfaction in all cases).  Now the challenge
is going to be prioritizing appropriately and getting a good set of writeups
that doesn't take too much time.

2. Page with notes on the particular merits and risks of different program
types -
http://www.givewell.net/wiki/index.php?title=Notes_on_not-yet-written-up_interventions.
 Here
there are still some major information gaps that we are looking for help
with - particularly in the area of "economic empowerment" interventions'
track record.

These pages should both be fairly straightforward to read (and it should be
easy to determine which parts to skip).  Each section has a "summary" with
our up-to-date take, and then notes on what's still to be done and notes to
assist with writing up what we've found.  The format for references is
completely inconsistent; some papers are linked, some are given as
citations, and some are listed using code words that I'll recognize.  I
didn't want to take the time to clean this all up now, but I put all
non-linked references in italics so they don't distract from the content,
and if you'd like to see one that isn't linked just let me know.

Finally, there were a bunch of notes that I thought were worth keeping
around even though they didn't fit in anywhere on our wikis - I just threw
these onto a third wiki,
http://www.givewell.net/wiki/index.php?title=Misc_notes

I skimmed through the 3 linked pages.

One specific point - under "Harms of aid", you briefly mention the concept of "if it isn't doing enough good, it's doing harm."  To be clear, from my perspective, this is a very different thing.  Trying to get donations to do the most good is basically the overall concept of GiveWell, and to try to bring that concept into this sub-area muddies things up.  

As a donor, I am genuinely concerned about the potential for my donations to do net harm.  I'm far less worried about the potential to give some money to charity A now but to discover later on that charity B is 20% more effective.  I take it as basically a given that I will probably not find the absolute best charity - I'd be happy to be in the ballpark.

What I worry about is that some charity that seems appealing now is in actuality either worthless or actively harmful.  Perhaps I would make a donation now and discover this later, perhaps I would not discover it.  Either way, it's a situation to be avoided, for me.

===

More generally, I would like to see a greater push to translate the research you appear to be doing into content formatted for the web and slotted into the right place (roughly, anyways) on your website.  I see references to blog posts and of course there are these rough notes themselves, but I worry that if you wait and try to do one big epic document and push it out on the web, it will be overwhelming, and a less effective approach than tackling areas one or two at a time and updating the website* on a more frequent basis.  I realize there are interdependencies in your research and things you find out while researching issue B may impact your ideas on issue A as well, but still, if you don't publish anything about A until you've also fully researched issues B through Z, I think you will find your task more difficult.

*The polished, non-blog, non-wiki website.

I have to agree here, and furthermore would like to see some very concise,
high-level bullets published on the web, catering to busy people who want
the bottom line with LINKS to research and analysis but not the whole
research dump.

On Fri, Apr 17, 2009 at 5:36 PM, psteinx <psteinmeyer@...> wrote:

>
>
> I skimmed through the 3 linked pages.
>
> One specific point - under "Harms of aid", you briefly mention the concept
> of "if it isn't doing enough good, it's doing harm." To be clear, from my
> perspective, this is a very different thing. Trying to get donations to do
> the most good is basically the overall concept of GiveWell, and to try to
> bring that concept into this sub-area muddies things up.
>
> As a donor, I am genuinely concerned about the potential for my donations
> to do net harm. I'm far less worried about the potential to give some money
> to charity A now but to discover later on that charity B is 20% more
> effective. I take it as basically a given that I will probably not find the
> absolute best charity - I'd be happy to be in the ballpark.
>
> What I worry about is that some charity that seems appealing now is in
> actuality either worthless or actively harmful. Perhaps I would make a
> donation now and discover this later, perhaps I would not discover it.
> Either way, it's a situation to be avoided, for me.
>
> ===
>
> More generally, I would like to see a greater push to translate the
> research you appear to be doing into content formatted for the web and
> slotted into the right place (roughly, anyways) on your website. I see
> references to blog posts and of course there are these rough notes
> themselves, but I worry that if you wait and try to do one big epic document
> and push it out on the web, it will be overwhelming, and a less effective
> approach than tackling areas one or two at a time and updating the website*
> on a more frequent basis. I realize there are interdependencies in your
> research and things you find out while researching issue B may impact your
> ideas on issue A as well, but still, if you don't publish anything about A
> until you've also fully researched issues B through Z, I think you will find
> your task more difficult.
>
> *The polished, non-blog, non-wiki website.
>
>  
>



-- 
Lindy Miller Crane

**We are the ones we have been waiting for.**

I'd appreciate some clarification on this, and should probably start with
some clarification on our end.
There are 3 possible levels of detail for these writeups:

1. Lowest level of detail: highlights. These will be pages along the lines
of those currently on the front page of givewell.net (on the left). They
will present key points as engagingly as possible. They will not be
footnoted, but will link to more detailed writeups. A working bullet-point
summary of these highlights is currently available via the top link on this
page: http://www.givewell.net/wiki/index.php?title=Links_for_2009_report.
The 5-6 "best" (most engaging) will go on the front page of the report as
links.
2. Medium level of detail: these will essentially be the blog posts we've
made so far (linked from section 2 of this page:
http://www.givewell.net/wiki/index.php?title=Issues_pages). Changes will be
very minor: things like replacing "I" with "we," as well as updating to
incorporate new information we've found (and cutting comments like "we're
still looking for more"). Aside from these minor changes and the visual look
(text, colors), these pages will be identical to the ones we currently have
on the blog.

3. Highest level of detail: Phil mentioned at one point that he would like a
higher level of detail on a blog post such as
http://blog.givewell.net/?p=344 - giving our view of the specific merits and
flaws in various studies, rather than simply pointing to others' literature
reviews and stating things like "Some believe in a moderate positive
relationship, often with the caveat that aid works better where existing
institutions are stronger (more below) or that aid has diminishing returns.
Others believe that there is no relationship or that there is insufficient
evidence." I think it is likely that we will not create pages along these
lines.

The blog posts we have are closest to #2; putting them on the main website
would involve extremely minor changes.  The "finished product" will include
#1, but I don't believe this is the right time to do more work on this.
 Before we discuss further, we should clarify (1) which of these 3 levels of
detail you're looking to see in its finished form and (2) whether you've
seen all the material we already have, particularly the Word doc I referred
to as a preview of #1 (and which addresses the question of what the
high-level bullet points will be).


On Sat, Apr 18, 2009 at 7:21 AM, Lindy Miller Crane <hellolindy@...>wrote:

>
>
> I have to agree here, and furthermore would like to see some very concise,
> high-level bullets published on the web, catering to busy people who want
> the bottom line with LINKS to research and analysis but not the whole
> research dump.
>
>
> On Fri, Apr 17, 2009 at 5:36 PM, psteinx <psteinmeyer@...> wrote:
>
>>
>>
>> I skimmed through the 3 linked pages.
>>
>> One specific point - under "Harms of aid", you briefly mention the concept
>> of "if it isn't doing enough good, it's doing harm." To be clear, from my
>> perspective, this is a very different thing. Trying to get donations to do
>> the most good is basically the overall concept of GiveWell, and to try to
>> bring that concept into this sub-area muddies things up.
>>
>> As a donor, I am genuinely concerned about the potential for my donations
>> to do net harm. I'm far less worried about the potential to give some money
>> to charity A now but to discover later on that charity B is 20% more
>> effective. I take it as basically a given that I will probably not find the
>> absolute best charity - I'd be happy to be in the ballpark.
>>
>> What I worry about is that some charity that seems appealing now is in
>> actuality either worthless or actively harmful. Perhaps I would make a
>> donation now and discover this later, perhaps I would not discover it.
>> Either way, it's a situation to be avoided, for me.
>>
>> ===
>>
>> More generally, I would like to see a greater push to translate the
>> research you appear to be doing into content formatted for the web and
>> slotted into the right place (roughly, anyways) on your website. I see
>> references to blog posts and of course there are these rough notes
>> themselves, but I worry that if you wait and try to do one big epic document
>> and push it out on the web, it will be overwhelming, and a less effective
>> approach than tackling areas one or two at a time and updating the website*
>> on a more frequent basis. I realize there are interdependencies in your
>> research and things you find out while researching issue B may impact your
>> ideas on issue A as well, but still, if you don't publish anything about A
>> until you've also fully researched issues B through Z, I think you will find
>> your task more difficult.
>>
>> *The polished, non-blog, non-wiki website.
>>
>>
>
>
> --
> Lindy Miller Crane
>
> **We are the ones we have been waiting for.**
>
>  
>

Speaking only for myself, I am looking more for a higher level of polishing and publishing rather than a particular level of detail.  i.e. Rather than putting effort into blog posts and detailed e-mails to this list, I'd rather see you guys make more of an effort to put the research and content into publishable form.  

IMO, neither blog entries nor e-mails on this list should be the final goal of GiveWell research.  

Perhaps you are a bit gunshy about putting content onto the main website until it is in "final" form.  Personally, I'd like to see content go up sooner, and if it gets revised later, so be it. 

By having so many stages to your research (you read a bunch, take some rough notes, eventually collect the rough notes and e-mail this list, maybe make a blog post, and finally, perhaps weeks/months later, put content on the main site), I think you are creating more work for yourselves.

As for level of detail - clearly there needs to be some cascading there.  I think bulletpoints (with links) are needed at the high level, and don't have a strong opinion at this time on the exact shape of the more detailed levels.  

I *do* think that when there is research on both sides of an issue (some researchers say "A", others say "not A"), that you should provide some color on your opinions of the strength of the relevant research, rather than simply throwing in links to the research and letting the reader decide.  The links are nice (and important), but most readers won't go through the research in detail - that's GiveWell's job.  You don't have to make an absolute pronouncement (A is *clearly* correct) - feel free to shade your opinion according to the strength of the arguments and research.  And sometimes, an issue will be quite unresolved or the two sides will be equally strong - it's ok to say that too.  But one way or another, I think you should express some opinion about the relative strength of the arguments on contested issues.

--- In givewell@yahoogroups.com, Holden Karnofsky <holden0@...> wrote:
>
> I'd appreciate some clarification on this, and should probably start with
> some clarification on our end.
> There are 3 possible levels of detail for these writeups:
> 
> 1. Lowest level of detail: highlights. These will be pages along the lines
> of those currently on the front page of givewell.net (on the left). They
> will present key points as engagingly as possible. They will not be
> footnoted, but will link to more detailed writeups. A working bullet-point
> summary of these highlights is currently available via the top link on this
> page: http://www.givewell.net/wiki/index.php?title=Links_for_2009_report.
> The 5-6 "best" (most engaging) will go on the front page of the report as
> links.
> 2. Medium level of detail: these will essentially be the blog posts we've
> made so far (linked from section 2 of this page:
> http://www.givewell.net/wiki/index.php?title=Issues_pages). Changes will be
> very minor: things like replacing "I" with "we," as well as updating to
> incorporate new information we've found (and cutting comments like "we're
> still looking for more"). Aside from these minor changes and the visual look
> (text, colors), these pages will be identical to the ones we currently have
> on the blog.
> 
> 3. Highest level of detail: Phil mentioned at one point that he would like a
> higher level of detail on a blog post such as
> http://blog.givewell.net/?p=344 - giving our view of the specific merits and
> flaws in various studies, rather than simply pointing to others' literature
> reviews and stating things like "Some believe in a moderate positive
> relationship, often with the caveat that aid works better where existing
> institutions are stronger (more below) or that aid has diminishing returns.
> Others believe that there is no relationship or that there is insufficient
> evidence." I think it is likely that we will not create pages along these
> lines.
> 
> The blog posts we have are closest to #2; putting them on the main website
> would involve extremely minor changes.  The "finished product" will include
> #1, but I don't believe this is the right time to do more work on this.
>  Before we discuss further, we should clarify (1) which of these 3 levels of
> detail you're looking to see in its finished form and (2) whether you've
> seen all the material we already have, particularly the Word doc I referred
> to as a preview of #1 (and which addresses the question of what the
> high-level bullet points will be).
> 
> 
> On Sat, Apr 18, 2009 at 7:21 AM, Lindy Miller Crane <hellolindy@...>wrote:
> 
> >
> >
> > I have to agree here, and furthermore would like to see some very concise,
> > high-level bullets published on the web, catering to busy people who want
> > the bottom line with LINKS to research and analysis but not the whole
> > research dump.
> >
> >
> > On Fri, Apr 17, 2009 at 5:36 PM, psteinx <psteinmeyer@...> wrote:
> >
> >>
> >>
> >> I skimmed through the 3 linked pages.
> >>
> >> One specific point - under "Harms of aid", you briefly mention the concept
> >> of "if it isn't doing enough good, it's doing harm." To be clear, from my
> >> perspective, this is a very different thing. Trying to get donations to do
> >> the most good is basically the overall concept of GiveWell, and to try to
> >> bring that concept into this sub-area muddies things up.
> >>
> >> As a donor, I am genuinely concerned about the potential for my donations
> >> to do net harm. I'm far less worried about the potential to give some money
> >> to charity A now but to discover later on that charity B is 20% more
> >> effective. I take it as basically a given that I will probably not find the
> >> absolute best charity - I'd be happy to be in the ballpark.
> >>
> >> What I worry about is that some charity that seems appealing now is in
> >> actuality either worthless or actively harmful. Perhaps I would make a
> >> donation now and discover this later, perhaps I would not discover it.
> >> Either way, it's a situation to be avoided, for me.
> >>
> >> ===
> >>
> >> More generally, I would like to see a greater push to translate the
> >> research you appear to be doing into content formatted for the web and
> >> slotted into the right place (roughly, anyways) on your website. I see
> >> references to blog posts and of course there are these rough notes
> >> themselves, but I worry that if you wait and try to do one big epic document
> >> and push it out on the web, it will be overwhelming, and a less effective
> >> approach than tackling areas one or two at a time and updating the website*
> >> on a more frequent basis. I realize there are interdependencies in your
> >> research and things you find out while researching issue B may impact your
> >> ideas on issue A as well, but still, if you don't publish anything about A
> >> until you've also fully researched issues B through Z, I think you will find
> >> your task more difficult.
> >>
> >> *The polished, non-blog, non-wiki website.
> >>
> >>
> >
> >
> > --
> > Lindy Miller Crane
> >
> > **We are the ones we have been waiting for.**
> >
> >  
> >
>

This is an update on our status finding and reviewing potential recommended
charities for our 2008-09 report. This email deals with charities which
focus on activities aside from economic empowerment.

(Economic empowerment is a particularly thorny cause -- in terms of evidence
of effectiveness for interventions and the information organizations in the
cause make available on their website -- so we're treating it separately.)

*Overview*

There are currently 44 non-economic empowerment charities we're considering
recommending in the upcoming report.

We've separated the remaining charities into three tiers:

   - *Tier 1 (8 organizations)*: Charities focusing on a) priority
   interventions (see
   http://www.givewell.net/files/Analysis/top%20interventions%202009%2002%2003.xls)
   and b) provide monitoring of their activities on their website.* *We can
   more or less review these organizations based on publicly available
   information without contacting them for more information.
      - Completed review: 3 organizations (PSI, Stop TB, and The Carter
      Center)
      - Currently reviewing: 1 organization (GAVI -- some notes on GAVI will
      follow this email)
      - Waiting to review: 4 organizations (African Programme for
      Onchocerciasis Control (APOC), VillageReach, Aravind Eye Care,
and Pratham)
   - *Tier 2 (20 organizations)*: Charities focusing on all (or almost all)
   of their activities on priority interventions but don't provide enough data
   on their website to review them without further information.
      - Completed review: 1 organization (Interplast)
      - Contacted already: 6 organizations (Global Alliance to Eliminate
      Lymphatic Filariasis, Measles Initiative, Fistula Foundation, Deworm the
      World, International Council for the Control of Iodine
Disorders, and Global
      Alliance for Improved Nutrition)
      - To contact as soon as possible: 1 organization (Tam Tam Africa --
      their website is under construction and no contact information
is available
      right now)
      - Waiting to contact: 12 organizations (see Excel file linked below)
      - *Tier 3 (16 organizations):* Charities that *do* *provide *relatively
   strong monitoring information on their website, but are not working on our
   priority interventions. We may review these, though we've prioritized
   organizations above, which work on the interventions we independently
   identified as most cost-effective. Names are in the Excel file linked below.

*The plan*

Our primary focus now is finalizing the list of charities that we might
ultimately review in depth. To that end, we're focusing on contacting the
Tier 2 organizations to see what type of information they may be able to
share with us (thereby moving them into Tier 1). We've chosen to contact
first the 6 organizations that a) we'd guess have the best chance of
ultimately receiving a recommendation and b) work on different priority
interventions (ideally we'd be able to recommend a charity for multiple
priority programs).

Time permitting, we'll continue to scan more charities to broaden our scope,
but we think the process we've used so far is reasonable and we have covered
most of our bases.

*Quick note on priority programs*

We've added a couple of interventions to the list in the file linked here (
http://www.givewell.net/files/Analysis/top%20interventions%202009%2002%2003.xls)
because we think they're (a) potentially more straightforward to monitor
than some of the more "vertical" programs with fairly elaborate theories of
change; (b) particularly appealing to certain donors.  This doesn't mean
that we endorse these interventions at this point, but that we think
charities working on them are worth investigating further.  Those new
interventions are: a) charities that run homes/shelters for orphans/street
children who would otherwise be homeless and b) charities that run local
health clinics, an approach similar to Partners in Health.

*Excel file with my work*

Link: http://www.givewell.net/files/DWDA 2009/Analysis/Charity scan 2009 04
22.xls

This file has details on the charities we're considering. The 'Top non-EE
charities' has summary information for the organizations I'm discussing in
this email. The 'All charities' tab has information on all charities we've
scanned, that I mentioned in this email:
http://groups.yahoo.com/group/givewell/message/73

On the 'Top non-EE charities' tab, columns B:E are probably most useful to
look at:

   - Column C is a 2-3 word description of what program the charities runs.
   - Column D provides the current status for that organization (e.g.,
   reviewed, to review, to contact)
   - Column E provides my very rough guess at how likely the charities is to
   be recommended. (These are rough and are based on my gut instinct, not any
   formula. We hope to recommend 5-10 charities in this report, so this % is
   useful in checking whether we're on track to meet that goal.)

The GAVI Alliance is one of the most promising charities we've come across
based on a) its singular focus on immunization, a proven cost-effective
intervention and b) its commitment to transparency, as evidenced by the
information available on its website. For more, see Holden's email to this
list: http://groups.yahoo.com/group/givewell/message/10

I spent a couple of hours yesterday beginning to review GAVI because we
think there's a good chance that we'd ultimately recommend them. After
reviewing the documents on their website, however, I'm now less sure. These
are the most significant questions we have:

*1. What does GAVI primarly spend its money on? *The program that GAVI is
most well-known for is their Immunisation Services Support (ISS) program in
which they provide funds to countries that demonstrate increased
immunization coverage relative to the set benchmark. However, in 2007, this
program accounted for an extremely small part of their total expenses.

The bottom line is that approximately 40% of GAVI's funds are granted to
external programs/organizations, such as the Global Polio Eradication
Initiative, which may indicate that GAVI is at capacity for their
implementing their primary activities of directly increasing basic
immunization coverage. An additional, ~30% of GAVI's funds go to cash or
in-kind donations for vaccination materials. ~20% goes to flexible money for
health system strengthening. < 10% goes to GAVI's "flagship" ISS program.

Roughly 60% of these funds (and all of the funds that go to the external
intiatives) come through a mechanism know as the IFFIm (more at
http://www.iff-immunisation.org/index.html). It's not totally clear whether
it's appropriate to treat these funds as funds given to GAVI through other
channels. I've contacted someone at GAVI to ask about this issue. Details on
their spending are below.

Assuming that the IFFIm funds are conceptually separate from general
spending, we might still assess GAVI on the basis of its non-IFFIm spending,
which leads us to question 2.

 *2. How do countries utilize GAVI's funds? *GAVI's NUS and INS programs
provide either a) in-kind donations of vaccines and related materials or b)
cash to purchase these materials. GAVI's website says that GAVI requires a
relatively high standard of evidence that countries use the funds provided
to purchase the expected materials (see
http://www.gavialliance.org/support/what/nvs/cofinancing/index.php).

It's not clear that GAVI monitors that these materials are ultimately used
properly (as opposed to sold by the countries for cash, which is a concern
we've read about). However, they do a) use WHO data to track the number of
people immunized and b) perform data quality audits (DQA) to assess that the
data provided is accurate. Assuming that GAVI funds/materials account for a
substantial portion of the people immunized in a country, these two checks
together would provide a relatively strong case that GAVI funds are used for
immunizations. To check this, we'd need to look at a) the number of
immunizations GAVI expects to provide and b) the number of total
immunizations implemented in that country.

The HSS funds are a different case.  These, which account for ~20% of
expenses, can be used for many uses. I didn't see a summarized report on
what countries used these funds for. (It may be - and I haven't yet checked
- in the individual country-level reports -- though, as far as I know, GAVI
does not perform any monitoring of these reports aside from the DQA, so it's
unclear how trustworthy these are, on their own.)

===========================================

* Details on GAVI spending:*

   - Total program expenses: $1.1 billion (Audited financial statements:
   http://www.gavialliance.org/resources/GAVI_2007_financial_statements___non_A_133.pdf,
   Pg 3)
   - "Investment cases": $428 million (GAVI Alliance Progress Report 2007,
   Pg 77).
   - New and underused vaccine program: $345 million (GAVI Alliance Progress
   Report 2007, Pg 77 for IFFIm money and subtracting cumulative spending
   through 2006 (2006 Progress Report, Pg 32 available at
   http://www.gavialliance.org/resources/2006_Progress_Report.pdf) from
   cumulative spending through 2007 (2007 Progress report, Pg 9). [I wasn't
   able to find annual spending broken out by area.]
   - Health systems strengthening: $206 million (Same as above)
   - Immunization services support: $92 million (Same as above)
   - Injection safety and support: $11 million (Same as above)

GAVI's website provides information about what each of these program areas
are. Here's my summary.

   - *Investment cases:* in 2007, these consisted of a $190m to the Global
   Polio Eradication Initiative, $139m to the Measles Initiative, $49m to the
   Maternal and Neonatal Tetanus Initiative, and $48m to the Yellow Fever
   Initiative.
   - *New and underused vaccines: *(NUS) "provides support to developing
   countries to introduce the following vaccines and associated vaccine
   technology. GAVI's support aims to accelerate their uptake and to improve
   vaccine supply security. "
   - *Immunisation services support: *(ISS) "flexible cash which countries
   can use as they choose to improve immunisation performance." Funding comes
   in two phases: an "investment" phase of two years granted on the basis of an
   approved application and a "reward" phase, where funding is continued on the
   basis of meeting pre-defined targets for immunization coverage.* *
   - *Injection safety support:* (INS) funding used to purchase equipment
   which increases vaccination safety for a pre-determined set of vaccines.*
   *
   - *Health system strengthening:* (HSS) funds used for
   non-vaccine-specific health services aiming to ultimately increase
   immunization coverage

We've discussed this a bit.  A couple of thoughts:

   - Our charity-specific work is already being rolled out to the website.
    We recently published the Stop TB review and will soon be publishing a
   couple more.
   - I've been doing the "issues pages" as blog posts, but I think at this
   point I'm going to switch over to publishing them directly on the website
   and just announcing/linking them on the blog.  The difference is more or
   less one of formatting (and to a lesser extent tone), but this seems
   appropriate.
   - We're going to experiment with using the blog for more conversational,
   less tightly referenced points - the big picture more than the details, with
   the aim of provoking conversation even before we have all the references in
   order.  Eventually the two will meet - we'll have "highlights" pages on the
   polished website that are also well-referenced - but putting those pages
   together now would not be efficient.


On Tue, Apr 21, 2009 at 11:59 AM, psteinx <psteinmeyer@...> wrote:

>
>
> Speaking only for myself, I am looking more for a higher level of polishing
> and publishing rather than a particular level of detail. i.e. Rather than
> putting effort into blog posts and detailed e-mails to this list, I'd rather
> see you guys make more of an effort to put the research and content into
> publishable form.
>
> IMO, neither blog entries nor e-mails on this list should be the final goal
> of GiveWell research.
>
> Perhaps you are a bit gunshy about putting content onto the main website
> until it is in "final" form. Personally, I'd like to see content go up
> sooner, and if it gets revised later, so be it.
>
> By having so many stages to your research (you read a bunch, take some
> rough notes, eventually collect the rough notes and e-mail this list, maybe
> make a blog post, and finally, perhaps weeks/months later, put content on
> the main site), I think you are creating more work for yourselves.
>
> As for level of detail - clearly there needs to be some cascading there. I
> think bulletpoints (with links) are needed at the high level, and don't have
> a strong opinion at this time on the exact shape of the more detailed
> levels.
>
> I *do* think that when there is research on both sides of an issue (some
> researchers say "A", others say "not A"), that you should provide some color
> on your opinions of the strength of the relevant research, rather than
> simply throwing in links to the research and letting the reader decide. The
> links are nice (and important), but most readers won't go through the
> research in detail - that's GiveWell's job. You don't have to make an
> absolute pronouncement (A is *clearly* correct) - feel free to shade your
> opinion according to the strength of the arguments and research. And
> sometimes, an issue will be quite unresolved or the two sides will be
> equally strong - it's ok to say that too. But one way or another, I think
> you should express some opinion about the relative strength of the arguments
> on contested issues.
>
>
> --- In givewell@yahoogroups.com <givewell%40yahoogroups.com>, Holden
> Karnofsky <holden0@...> wrote:
> >
> > I'd appreciate some clarification on this, and should probably start with
> > some clarification on our end.
> > There are 3 possible levels of detail for these writeups:
> >
> > 1. Lowest level of detail: highlights. These will be pages along the
> lines
> > of those currently on the front page of givewell.net (on the left). They
> > will present key points as engagingly as possible. They will not be
> > footnoted, but will link to more detailed writeups. A working
> bullet-point
> > summary of these highlights is currently available via the top link on
> this
> > page:
> http://www.givewell.net/wiki/index.php?title=Links_for_2009_report.
> > The 5-6 "best" (most engaging) will go on the front page of the report as
> > links.
> > 2. Medium level of detail: these will essentially be the blog posts we've
> > made so far (linked from section 2 of this page:
> > http://www.givewell.net/wiki/index.php?title=Issues_pages). Changes will
> be
> > very minor: things like replacing "I" with "we," as well as updating to
> > incorporate new information we've found (and cutting comments like "we're
> > still looking for more"). Aside from these minor changes and the visual
> look
> > (text, colors), these pages will be identical to the ones we currently
> have
> > on the blog.
> >
> > 3. Highest level of detail: Phil mentioned at one point that he would
> like a
> > higher level of detail on a blog post such as
> > http://blog.givewell.net/?p=344 - giving our view of the specific merits
> and
> > flaws in various studies, rather than simply pointing to others'
> literature
> > reviews and stating things like "Some believe in a moderate positive
> > relationship, often with the caveat that aid works better where existing
> > institutions are stronger (more below) or that aid has diminishing
> returns.
> > Others believe that there is no relationship or that there is
> insufficient
> > evidence." I think it is likely that we will not create pages along these
> > lines.
> >
> > The blog posts we have are closest to #2; putting them on the main
> website
> > would involve extremely minor changes. The "finished product" will
> include
> > #1, but I don't believe this is the right time to do more work on this.
> > Before we discuss further, we should clarify (1) which of these 3 levels
> of
> > detail you're looking to see in its finished form and (2) whether you've
> > seen all the material we already have, particularly the Word doc I
> referred
> > to as a preview of #1 (and which addresses the question of what the
> > high-level bullet points will be).
> >
> >
> > On Sat, Apr 18, 2009 at 7:21 AM, Lindy Miller Crane <hellolindy@
> ...>wrote:
> >
> > >
> > >
> > > I have to agree here, and furthermore would like to see some very
> concise,
> > > high-level bullets published on the web, catering to busy people who
> want
> > > the bottom line with LINKS to research and analysis but not the whole
> > > research dump.
> > >
> > >
> > > On Fri, Apr 17, 2009 at 5:36 PM, psteinx <psteinmeyer@...> wrote:
> > >
> > >>
> > >>
> > >> I skimmed through the 3 linked pages.
> > >>
> > >> One specific point - under "Harms of aid", you briefly mention the
> concept
> > >> of "if it isn't doing enough good, it's doing harm." To be clear, from
> my
> > >> perspective, this is a very different thing. Trying to get donations
> to do
> > >> the most good is basically the overall concept of GiveWell, and to try
> to
> > >> bring that concept into this sub-area muddies things up.
> > >>
> > >> As a donor, I am genuinely concerned about the potential for my
> donations
> > >> to do net harm. I'm far less worried about the potential to give some
> money
> > >> to charity A now but to discover later on that charity B is 20% more
> > >> effective. I take it as basically a given that I will probably not
> find the
> > >> absolute best charity - I'd be happy to be in the ballpark.
> > >>
> > >> What I worry about is that some charity that seems appealing now is in
> > >> actuality either worthless or actively harmful. Perhaps I would make a
> > >> donation now and discover this later, perhaps I would not discover it.
> > >> Either way, it's a situation to be avoided, for me.
> > >>
> > >> ===
> > >>
> > >> More generally, I would like to see a greater push to translate the
> > >> research you appear to be doing into content formatted for the web and
> > >> slotted into the right place (roughly, anyways) on your website. I see
> > >> references to blog posts and of course there are these rough notes
> > >> themselves, but I worry that if you wait and try to do one big epic
> document
> > >> and push it out on the web, it will be overwhelming, and a less
> effective
> > >> approach than tackling areas one or two at a time and updating the
> website*
> > >> on a more frequent basis. I realize there are interdependencies in
> your
> > >> research and things you find out while researching issue B may impact
> your
> > >> ideas on issue A as well, but still, if you don't publish anything
> about A
> > >> until you've also fully researched issues B through Z, I think you
> will find
> > >> your task more difficult.
> > >>
> > >> *The polished, non-blog, non-wiki website.
> > >>
> > >>
> > >
> > >
> > > --
> > > Lindy Miller Crane
> > >
> > > **We are the ones we have been waiting for.**
> > >
> > >
> > >
> >
>
>  
>

Here's a very informal, loose summary of my "big picture overview" of our
upcoming research report.  This is based less on any particular vision for
the website than on the conversations I have with people and the points I
usually emphasize in those conversations.  It's not referenced.  I wrote it
up as something to throw out there and get people's initial thoughts on, as
well as a reference point to see how well our report in progress reflects
the big picture.

--

International aid is a great cause, but also a dangerous one.
Great because it involves helping by far the world's lowest-income people.

   - That means your donation - whatever size it is - means far more to
   them, bottom line, than it can mean to anyone in the U.S.
   - Charities in the U.S. are mostly trying to untangle very
   hard-to-understand dynamics of poverty such as the achievement gap in
   education.  Charities in Africa are often giving basic medical treatment to
   someone who can't afford it.
   - We've got a table that shows just how big the difference is, in the US
   the best program we've found is $10k/yr for a 2-year program that has
   positive effects but not huge effects on people's lives, in Africa we're in
   the range of $1000 to save a person's life.


Dangerous because it involves a completely different part of the world.

   - The distance makes it very hard for you as a donor to really tell what
   an organization's doing or hold it accountable, unless it's voluntarily
   providing a ton of information to help you do so.  There are horror stories
   of money simply getting swallowed up and not reaching the people and
   programs it was supposed to.  There are also stories of programs where the
   plan in the brochure just looks nothing like the action on the ground.  The
   World Bank once pointed to a "success story" of an Internet-access program
   in an area where computers couldn't even stay on because of the lack of
   reliable electricity.
   - And the cultural distance means that you have to be careful and humble
   about how much you know about these communities and what you can do for
   them.
      - We are always seeing charities (and donors) convinced that they know
      the "root causes of poverty" and that a well or a community mobilization
      program will catalyze a whole community's pulling itself out of
poverty for
      good.  But the track record of projects like these - and larger scale
      efforts to attack poverty at its roots - is not encouraging.
We've had 50
      years of huge coalitions of international aid organizations trying one
      theory or another of the "root causes of poverty" and they haven't solved
      poverty.
      - Understanding the local economy seems like a very difficult
      undertaking but that doesn't stop lots of charities from being convinced
      that they do.  They try to train farmers to grow cash crops, but
what if the
      market collapses due to export restrictions (as it did in one
      well-documented case)?  Or what if the people aren't even farmers (this
      happened in another well-documented case)?  Similar concerns apply to
      education - how much do we really know about the impact of math
skills on a
      person's life outcomes in rural Rwanda?
   - And going into a foreign community and imposing your plans on it isn't
   just risking "no benefit," it's risking "harm."  Set salaries too high and
   you'll pull away people who could be doing perfectly productive things in
   their own economies, driven by local needs.  Throw too much cash into an
   area without accountability and you could be reducing the government's
   accountability.

So what should you do?  Find a charity such that

   - What they're doing is doable.  It's worked before.  We've got enough
   giant international aid agencies trying to be the first to crack the root
   causes of poverty.  You don't have their ability to understand the theories
   and the history and you don't have to throw your money into that pot.  You
   can fund something that's got a long history of working and that still has a
   lot of room to be scaled up elsewhere.
   - You can tell what they're doing.  Lots of charities do enormous numbers
   of unconnected projects and many seem to be essentially driven by their
   funders.  You don't need to be funding that.
   - They're obsessively documenting that the money is reaching its intended
   target and having its intended outcomes.  You can look at the information
   and see that this is happening the way it's supposed to and it's helping
   people.

In looking for charities that work this way, we've found that:

   - Health is the strongest sector.  It could be because health is easier
   to measure and document than other areas.  It could be because Western
   medical knowledge translates better to another culture than Western
   knowledge of business/community mobilization/education.  Whatever reason -
   health is something that donors can help with.  You may care more about
   other problems, but you should also consider what you can do about those
   problems and in many cases there isn't evidence that you can do very much
   (as a donor).
   - Many of the "sexy" health interventions are overrated.  ART (AIDS
   treatment) costs many times as much as tuberculosis treatment, and
   tuberculosis treatment permanently cures tuberculosis.  Water projects are
   popular but there are many cases of wells falling into disrepair and few
   cases of charities that provide the followup to check whether this is
   happening.  Even if a project successfully improves water quality, it may
   not improve health much - the major waterborne diseases are transmitted in
   many other ways, and water without other sanitation improvements doesn't
   necessarily make a dent.  Insecticide-treated bednets are an exception to
   this point; their use has a strong track record, though it isn't entirely
   clear how much a standard net distribution program resembles the programs
   that are known to have worked.
   - Vaccines are simple, doable, and have worked countless times before.
    Slightly less straightforward but also with excellent track records and low
   costs are mass drug administration and tuberculosis treatment.  Vitamin
   fortification is promising.
   - The charities that best meet our criteria are StopTB and PSI (hopefully
   more forthcoming).  Both of these charities are doing relatively
   straightforward, life-saving things and monitoring all the info you'd wnat
   to monitor to be confident they're actually saving lives.  With either of
   these charities we're ballparking around $1000 to save a life.

Overall I think this is a really good summary. It definitely resonates with
my experience here in Sudan, where many of the aid projects are complete
garbage, but the clear success story is the health sector - where NGOs
basically run the entire health system.

One thing I think you omit from this discussion is the value of
experimentation when it is properly evaluated. Organisations should be
congratulated for doing work which hasn't been proven to be effective *if
they are rigorously evaluating it and therefore creating new evidence about
what is and is not effective.*

Admittedly there probably isn't that much of this going on, but the message
to NGOs should be either
A) do something that we know works, or
B) do something and properly find out if it works, and then share that
information, even if the project is proven not to work

hmmm, I'm just reading that back and it sounds like a difficult sell to
potential individual donors.


cheers,


Lee

--
Lee Crawfurd

Economist (ODI Fellow)
Ministry of Finance and Economic Planning
Government of Southern Sudan

lee.crawfurd@...
Zain: +249 (0)914897740
Gemtel: +256 (0)477256753


2009/5/1 Holden Karnofsky <Holden@...>

>
>
> Here's a very informal, loose summary of my "big picture overview" of our
> upcoming research report.  This is based less on any particular vision for
> the website than on the conversations I have with people and the points I
> usually emphasize in those conversations.  It's not referenced.  I wrote it
> up as something to throw out there and get people's initial thoughts on, as
> well as a reference point to see how well our report in progress reflects
> the big picture.
>
> --
>
> International aid is a great cause, but also a dangerous one.
> Great because it involves helping by far the world's lowest-income people.
>
>    - That means your donation - whatever size it is - means far more to
>    them, bottom line, than it can mean to anyone in the U.S.
>    - Charities in the U.S. are mostly trying to untangle very
>    hard-to-understand dynamics of poverty such as the achievement gap in
>    education.  Charities in Africa are often giving basic medical treatment to
>    someone who can't afford it.
>    - We've got a table that shows just how big the difference is, in the
>    US the best program we've found is $10k/yr for a 2-year program that has
>    positive effects but not huge effects on people's lives, in Africa we're in
>    the range of $1000 to save a person's life.
>
>
> Dangerous because it involves a completely different part of the world.
>
>    - The distance makes it very hard for you as a donor to really tell
>    what an organization's doing or hold it accountable, unless it's voluntarily
>    providing a ton of information to help you do so.  There are horror stories
>    of money simply getting swallowed up and not reaching the people and
>    programs it was supposed to.  There are also stories of programs where the
>    plan in the brochure just looks nothing like the action on the ground.  The
>    World Bank once pointed to a "success story" of an Internet-access program
>    in an area where computers couldn't even stay on because of the lack of
>    reliable electricity.
>    - And the cultural distance means that you have to be careful and
>    humble about how much you know about these communities and what you can do
>    for them.
>       - We are always seeing charities (and donors) convinced that they
>       know the "root causes of poverty" and that a well or a community
>       mobilization program will catalyze a whole community's pulling itself out of
>       poverty for good.  But the track record of projects like these - and larger
>       scale efforts to attack poverty at its roots - is not encouraging.  We've
>       had 50 years of huge coalitions of international aid organizations trying
>       one theory or another of the "root causes of poverty" and they haven't
>       solved poverty.
>       - Understanding the local economy seems like a very difficult
>       undertaking but that doesn't stop lots of charities from being convinced
>       that they do.  They try to train farmers to grow cash crops, but what if the
>       market collapses due to export restrictions (as it did in one
>       well-documented case)?  Or what if the people aren't even farmers (this
>       happened in another well-documented case)?  Similar concerns apply to
>       education - how much do we really know about the impact of math skills on a
>       person's life outcomes in rural Rwanda?
>    - And going into a foreign community and imposing your plans on it
>    isn't just risking "no benefit," it's risking "harm."  Set salaries too high
>    and you'll pull away people who could be doing perfectly productive things
>    in their own economies, driven by local needs.  Throw too much cash into an
>    area without accountability and you could be reducing the government's
>    accountability.
>
> So what should you do?  Find a charity such that
>
>    - What they're doing is doable.  It's worked before.  We've got enough
>    giant international aid agencies trying to be the first to crack the root
>    causes of poverty.  You don't have their ability to understand the theories
>    and the history and you don't have to throw your money into that pot.  You
>    can fund something that's got a long history of working and that still has a
>    lot of room to be scaled up elsewhere.
>    - You can tell what they're doing.  Lots of charities do enormous
>    numbers of unconnected projects and many seem to be essentially driven by
>    their funders.  You don't need to be funding that.
>    - They're obsessively documenting that the money is reaching its
>    intended target and having its intended outcomes.  You can look at the
>    information and see that this is happening the way it's supposed to and it's
>    helping people.
>
> In looking for charities that work this way, we've found that:
>
>    - Health is the strongest sector.  It could be because health is easier
>    to measure and document than other areas.  It could be because Western
>    medical knowledge translates better to another culture than Western
>    knowledge of business/community mobilization/education.  Whatever reason -
>    health is something that donors can help with.  You may care more about
>    other problems, but you should also consider what you can do about those
>    problems and in many cases there isn't evidence that you can do very much
>    (as a donor).
>    - Many of the "sexy" health interventions are overrated.  ART (AIDS
>    treatment) costs many times as much as tuberculosis treatment, and
>    tuberculosis treatment permanently cures tuberculosis.  Water projects are
>    popular but there are many cases of wells falling into disrepair and few
>    cases of charities that provide the followup to check whether this is
>    happening.  Even if a project successfully improves water quality, it may
>    not improve health much - the major waterborne diseases are transmitted in
>    many other ways, and water without other sanitation improvements doesn't
>    necessarily make a dent.  Insecticide-treated bednets are an exception to
>    this point; their use has a strong track record, though it isn't entirely
>    clear how much a standard net distribution program resembles the programs
>    that are known to have worked.
>    - Vaccines are simple, doable, and have worked countless times before.
>     Slightly less straightforward but also with excellent track records and low
>    costs are mass drug administration and tuberculosis treatment.  Vitamin
>    fortification is promising.
>    - The charities that best meet our criteria are StopTB and PSI
>    (hopefully more forthcoming).  Both of these charities are doing relatively
>    straightforward, life-saving things and monitoring all the info you'd wnat
>    to monitor to be confident they're actually saving lives.  With either of
>    these charities we're ballparking around $1000 to save a life.
>
>  
>

I'd love to hear more about your thoguhts so far on a few topics:

1. Organizations such as  http://www.ideorg.org/ and Kickstart, which
are supposed to be more bottom-up: identifying specific needs (in
water/irrigation) of the localities and help to develop the
technologies / market to fit the local needs. I read your thoughts on
Kickstart a couple years back but wonder if you have any update since
then.

2. What do you think the value of those projects that are successful
in its short-term outcome but not in the long-term outcome yet? E.g.,
you mention some successful water projects (in improving water
quality) but improving water quality alone isn't moving the dent.


3. What's your thoughts about the organizations experimenting new
approach? It's a chicken-and-egg problem, as a donor, I of course love
to see the organization I support does make a real differenc, but if
those organizations don't get sufficient funding to try out, they
won't get a chance to show that it works (or not).

One live example is Literacy Bridge ( http://www.literacybridge.org/
), developing a low cost (US $5) audio / information sharing devices.
It seems to be a powerful idea at 10,000 feet level. However, it'll
definitely take a while to know whether it works or not, requiring
time to develop and refine the technologies, and to have the local
partners make use of the devices.


- sam



On Fri, May 1, 2009 at 5:26 AM, Holden Karnofsky <Holden@...> wrote:
>
>
> Here's a very informal, loose summary of my "big picture overview" of our
> upcoming research report.  This is based less on any particular vision for
> the website than on the conversations I have with people and the points I
> usually emphasize in those conversations.  It's not referenced.  I wrote it
> up as something to throw out there and get people's initial thoughts on, as
> well as a reference point to see how well our report in progress reflects
> the big picture.
> --
> International aid is a great cause, but also a dangerous one.
> Great because it involves helping by far the world's lowest-income people.
>
> That means your donation - whatever size it is - means far more to them,
> bottom line, than it can mean to anyone in the U.S.
> Charities in the U.S. are mostly trying to untangle very hard-to-understand
> dynamics of poverty such as the achievement gap in education.  Charities in
> Africa are often giving basic medical treatment to someone who can't afford
> it.
> We've got a table that shows just how big the difference is, in the US the
> best program we've found is $10k/yr for a 2-year program that has positive
> effects but not huge effects on people's lives, in Africa we're in the range
> of $1000 to save a person's life.
>
> Dangerous because it involves a completely different part of the world.
>
> The distance makes it very hard for you as a donor to really tell what an
> organization's doing or hold it accountable, unless it's voluntarily
> providing a ton of information to help you do so.  There are horror stories
> of money simply getting swallowed up and not reaching the people and
> programs it was supposed to.  There are also stories of programs where the
> plan in the brochure just looks nothing like the action on the ground.  The
> World Bank once pointed to a "success story" of an Internet-access program
> in an area where computers couldn't even stay on because of the lack of
> reliable electricity.
> And the cultural distance means that you have to be careful and humble about
> how much you know about these communities and what you can do for them.
>
> We are always seeing charities (and donors) convinced that they know the
> "root causes of poverty" and that a well or a community mobilization program
> will catalyze a whole community's pulling itself out of poverty for good.
>  But the track record of projects like these - and larger scale efforts to
> attack poverty at its roots - is not encouraging.  We've had 50 years of
> huge coalitions of international aid organizations trying one theory or
> another of the "root causes of poverty" and they haven't solved poverty.
> Understanding the local economy seems like a very difficult undertaking but
> that doesn't stop lots of charities from being convinced that they do.  They
> try to train farmers to grow cash crops, but what if the market collapses
> due to export restrictions (as it did in one well-documented case)?  Or what
> if the people aren't even farmers (this happened in another well-documented
> case)?  Similar concerns apply to education - how much do we really know
> about the impact of math skills on a person's life outcomes in rural Rwanda?
>
> And going into a foreign community and imposing your plans on it isn't just
> risking "no benefit," it's risking "harm."  Set salaries too high and you'll
> pull away people who could be doing perfectly productive things in their own
> economies, driven by local needs.  Throw too much cash into an area without
> accountability and you could be reducing the government's accountability.
>
> So what should you do?  Find a charity such that
>
> What they're doing is doable.  It's worked before.  We've got enough giant
> international aid agencies trying to be the first to crack the root causes
> of poverty.  You don't have their ability to understand the theories and the
> history and you don't have to throw your money into that pot.  You can fund
> something that's got a long history of working and that still has a lot of
> room to be scaled up elsewhere.
> You can tell what they're doing.  Lots of charities do enormous numbers of
> unconnected projects and many seem to be essentially driven by their
> funders.  You don't need to be funding that.
> They're obsessively documenting that the money is reaching its intended
> target and having its intended outcomes.  You can look at the information
> and see that this is happening the way it's supposed to and it's helping
> people.
>
> In looking for charities that work this way, we've found that:
>
> Health is the strongest sector.  It could be because health is easier to
> measure and document than other areas.  It could be because Western medical
> knowledge translates better to another culture than Western knowledge of
> business/community mobilization/education.  Whatever reason - health is
> something that donors can help with.  You may care more about other
> problems, but you should also consider what you can do about those problems
> and in many cases there isn't evidence that you can do very much (as a
> donor).
> Many of the "sexy" health interventions are overrated.  ART (AIDS treatment)
> costs many times as much as tuberculosis treatment, and tuberculosis
> treatment permanently cures tuberculosis.  Water projects are popular but
> there are many cases of wells falling into disrepair and few cases of
> charities that provide the followup to check whether this is happening.
>  Even if a project successfully improves water quality, it may not improve
> health much - the major waterborne diseases are transmitted in many other
> ways, and water without other sanitation improvements doesn't necessarily
> make a dent.  Insecticide-treated bednets are an exception to this point;
> their use has a strong track record, though it isn't entirely clear how much
> a standard net distribution program resembles the programs that are known to
> have worked.
> Vaccines are simple, doable, and have worked countless times before.
>  Slightly less straightforward but also with excellent track records and low
> costs are mass drug administration and tuberculosis treatment.  Vitamin
> fortification is promising.
> The charities that best meet our criteria are StopTB and PSI (hopefully more
> forthcoming).  Both of these charities are doing relatively straightforward,
> life-saving things and monitoring all the info you'd wnat to monitor to be
> confident they're actually saving lives.  With either of these charities
> we're ballparking around $1000 to save a life.
>
> 

Pretty solid summary, overall. 

Of course, I'd like to see it extensively referenced and/or with the ability to click through on specific points to get more detailed information.  In particular, you may need to document some of the failures and in general go to greater lengths to pop some of the ideas that SEEM appealing on the surface to donors, but don't necessarily hold up well when implemented.

I'm also interested in the details of the charities that work, and a little surprised that The Carter Center didn't make your (very) short list, but willing to look at the details.  


  ----- Original Message ----- 
  From: Lee Crawfurd (MoFEP) 
  To: givewell 
  Sent: Saturday, May 02, 2009 11:09 AM
  Subject: Re: [givewell] Informal summary of thoughts on international aid





  Overall I think this is a really good summary. It definitely resonates with my experience here in Sudan, where many of the aid projects are complete garbage, but the clear success story is the health sector - where NGOs basically run the entire health system. 

  One thing I think you omit from this discussion is the value of experimentation when it is properly evaluated. Organisations should be congratulated for doing work which hasn't been proven to be effective if they are rigorously evaluating it and therefore creating new evidence about what is and is not effective.

  Admittedly there probably isn't that much of this going on, but the message to NGOs should be either
  A) do something that we know works, or
  B) do something and properly find out if it works, and then share that information, even if the project is proven not to work

  hmmm, I'm just reading that back and it sounds like a difficult sell to potential individual donors. 


  cheers, 


  Lee

  --
  Lee Crawfurd

  Economist (ODI Fellow)
  Ministry of Finance and Economic Planning
  Government of Southern Sudan

  lee.crawfurd@...
  Zain: +249 (0)914897740
  Gemtel: +256 (0)477256753




  2009/5/1 Holden Karnofsky <Holden@...>





    Here's a very informal, loose summary of my "big picture overview" of our upcoming research report.  This is based less on any particular vision for the website than on the conversations I have with people and the points I usually emphasize in those conversations.  It's not referenced.  I wrote it up as something to throw out there and get people's initial thoughts on, as well as a reference point to see how well our report in progress reflects the big picture.


    --

    International aid is a great cause, but also a dangerous one.


    Great because it involves helping by far the world's lowest-income people.
      a.. That means your donation - whatever size it is - means far more to them, bottom line, than it can mean to anyone in the U.S.  

      b.. Charities in the U.S. are mostly trying to untangle very hard-to-understand dynamics of poverty such as the achievement gap in education.  Charities in Africa are often giving basic medical treatment to someone who can't afford it.  

      c.. We've got a table that shows just how big the difference is, in the US the best program we've found is $10k/yr for a 2-year program that has positive effects but not huge effects on people's lives, in Africa we're in the range of $1000 to save a person's life.



    Dangerous because it involves a completely different part of the world.  
      a.. The distance makes it very hard for you as a donor to really tell what an organization's doing or hold it accountable, unless it's voluntarily providing a ton of information to help you do so.  There are horror stories of money simply getting swallowed up and not reaching the people and programs it was supposed to.  There are also stories of programs where the plan in the brochure just looks nothing like the action on the ground.  The World Bank once pointed to a "success story" of an Internet-access program in an area where computers couldn't even stay on because of the lack of reliable electricity.

      b.. And the cultural distance means that you have to be careful and humble about how much you know about these communities and what you can do for them.  
        a.. We are always seeing charities (and donors) convinced that they know the "root causes of poverty" and that a well or a community mobilization program will catalyze a whole community's pulling itself out of poverty for good.  But the track record of projects like these - and larger scale efforts to attack poverty at its roots - is not encouraging.  We've had 50 years of huge coalitions of international aid organizations trying one theory or another of the "root causes of poverty" and they haven't solved poverty.

        b.. Understanding the local economy seems like a very difficult undertaking but that doesn't stop lots of charities from being convinced that they do.  They try to train farmers to grow cash crops, but what if the market collapses due to export restrictions (as it did in one well-documented case)?  Or what if the people aren't even farmers (this happened in another well-documented case)?  Similar concerns apply to education - how much do we really know about the impact of math skills on a person's life outcomes in rural Rwanda? 
      c.. And going into a foreign community and imposing your plans on it isn't just risking "no benefit," it's risking "harm."  Set salaries too high and you'll pull away people who could be doing perfectly productive things in their own economies, driven by local needs.  Throw too much cash into an area without accountability and you could be reducing the government's accountability. 
    So what should you do?  Find a charity such that
      a.. What they're doing is doable.  It's worked before.  We've got enough giant international aid agencies trying to be the first to crack the root causes of poverty.  You don't have their ability to understand the theories and the history and you don't have to throw your money into that pot.  You can fund something that's got a long history of working and that still has a lot of room to be scaled up elsewhere. 
      b.. You can tell what they're doing.  Lots of charities do enormous numbers of unconnected projects and many seem to be essentially driven by their funders.  You don't need to be funding that.
      c.. They're obsessively documenting that the money is reaching its intended target and having its intended outcomes.  You can look at the information and see that this is happening the way it's supposed to and it's helping people. 
    In looking for charities that work this way, we've found that:
      a.. Health is the strongest sector.  It could be because health is easier to measure and document than other areas.  It could be because Western medical knowledge translates better to another culture than Western knowledge of business/community mobilization/education.  Whatever reason - health is something that donors can help with.  You may care more about other problems, but you should also consider what you can do about those problems and in many cases there isn't evidence that you can do very much (as a donor). 
      b.. Many of the "sexy" health interventions are overrated.  ART (AIDS treatment) costs many times as much as tuberculosis treatment, and tuberculosis treatment permanently cures tuberculosis.  Water projects are popular but there are many cases of wells falling into disrepair and few cases of charities that provide the followup to check whether this is happening.  Even if a project successfully improves water quality, it may not improve health much - the major waterborne diseases are transmitted in many other ways, and water without other sanitation improvements doesn't necessarily make a dent.  Insecticide-treated bednets are an exception to this point; their use has a strong track record, though it isn't entirely clear how much a standard net distribution program resembles the programs that are known to have worked. 
      c.. Vaccines are simple, doable, and have worked countless times before.  Slightly less straightforward but also with excellent track records and low costs are mass drug administration and tuberculosis treatment.  Vitamin fortification is promising. 
      d.. The charities that best meet our criteria are StopTB and PSI (hopefully more forthcoming).  Both of these charities are doing relatively straightforward, life-saving things and monitoring all the info you'd wnat to monitor to be confident they're actually saving lives.  With either of these charities we're ballparking around $1000 to save a life. 



  

Thanks for the thoughts, everyone.  A few responses:
Lee: our focus is on finding successful charities (in keeping with our
mission of helping donors), but we agree about the importance of
experimentation and the desirability of funding it.  You may be interested
in this blog post: http://blog.givewell.net/?p=311

Sam:

   - The organizations you mention are both on our list for a closer look,
   but from what we've seen to date, they don't appear to have compelling
   evidence of impact.  I also don't think that "bottom-up" is necessarily a
   good way to distinguish them from the organizations we recommend.  (PSI and
   Stop TB both integrate local people and leave room for local practices in
   their programs, and both are responding to needs they've identified.  I'm
   not sure whether these things are more or less true of KickStart/IDE.)
   - We consider water quality to be an "intermediate" outcome - encouraging
   in that it shows that part of the project has gone well, but not the same as
   demonstrating something like reduced diarrhea, which we see as a worthwhile
   end in itself.  (By contrast, we consider clean water valuable only insofar
   as it leads to significantly better health and/or quality of life -
   something that may not be the case in areas that have serious
   non-water-related sanitation problems.)
   - New, experimental projects may be worthwhile, but we think that they
   should ideally be funded only by those who have strong relevant expertise
   and the ability to evaluate them up close, and that they should stay small
   when possible until their viability and impact can be demonstrated.  We're
   going to make a blog post on this topic shortly.

Phil: we are working on writing up the details of the points you mention.
 We find it most efficient to merge the high-level summary with the
well-referenced details as a last step.  The Carter Center is not currently
on our short list because we have still not gotten the information we've
requested about their financials, and without this information, we can't
assess the weight of their stronger vs. weaker programs.  The programs range
from quite strong to quite weak, in my opinion.

On Mon, May 4, 2009 at 11:37 AM, Phil Steinmeyer <psteinmeyer@...>
wrote:
>
>
> Pretty solid summary, overall.
>
> Of course, I'd like to see it extensively referenced and/or with the
ability
> to click through on specific points to get more detailed information.  In
> particular, you may need to document some of the failures and in general
go
> to greater lengths to pop some of the ideas that SEEM appealing on the
> surface to donors, but don't necessarily hold up well when implemented.
>
> I'm also interested in the details of the charities that work, and a
little
> surprised that The Carter Center didn't make your (very) short list, but
> willing to look at the details.
>
>
>
> ----- Original Message -----
> From: Lee Crawfurd (MoFEP)
> To: givewell
> Sent: Saturday, May 02, 2009 11:09 AM
> Subject: Re: [givewell] Informal summary of thoughts on international aid
>
> Overall I think this is a really good summary. It definitely resonates
with
> my experience here in Sudan, where many of the aid projects are complete
> garbage, but the clear success story is the health sector - where NGOs
> basically run the entire health system.
>
> One thing I think you omit from this discussion is the value of
> experimentation when it is properly evaluated. Organisations should be
> congratulated for doing work which hasn't been proven to be effective if
> they are rigorously evaluating it and therefore creating new evidence
about
> what is and is not effective.
>
> Admittedly there probably isn't that much of this going on, but the
message
> to NGOs should be either
> A) do something that we know works, or
> B) do something and properly find out if it works, and then share that
> information, even if the project is proven not to work
>
> hmmm, I'm just reading that back and it sounds like a difficult sell to
> potential individual donors.
>
>
> cheers,
>
>
> Lee
>
> --
> Lee Crawfurd
>
> Economist (ODI Fellow)
> Ministry of Finance and Economic Planning
> Government of Southern Sudan
>
> lee.crawfurd@...
> Zain: +249 (0)914897740
> Gemtel: +256 (0)477256753
>
>
> 2009/5/1 Holden Karnofsky <Holden@...>
>>
>>
>> Here's a very informal, loose summary of my "big picture overview" of our
>> upcoming research report.  This is based less on any particular vision
for
>> the website than on the conversations I have with people and the points I
>> usually emphasize in those conversations.  It's not referenced.  I wrote
it
>> up as something to throw out there and get people's initial thoughts on,
as
>> well as a reference point to see how well our report in progress reflects
>> the big picture.
>> --
>> International aid is a great cause, but also a dangerous one.
>> Great because it involves helping by far the world's lowest-income
people.
>>
>> That means your donation - whatever size it is - means far more to them,
>> bottom line, than it can mean to anyone in the U.S.
>> Charities in the U.S. are mostly trying to untangle very
>> hard-to-understand dynamics of poverty such as the achievement gap in
>> education.  Charities in Africa are often giving basic medical treatment
to
>> someone who can't afford it.
>> We've got a table that shows just how big the difference is, in the US
the
>> best program we've found is $10k/yr for a 2-year program that has
positive
>> effects but not huge effects on people's lives, in Africa we're in the
range
>> of $1000 to save a person's life.
>>
>> Dangerous because it involves a completely different part of the world.
>>
>> The distance makes it very hard for you as a donor to really tell what an
>> organization's doing or hold it accountable, unless it's voluntarily
>> providing a ton of information to help you do so.  There are horror
stories
>> of money simply getting swallowed up and not reaching the people and
>> programs it was supposed to.  There are also stories of programs where
the
>> plan in the brochure just looks nothing like the action on the ground.
 The
>> World Bank once pointed to a "success story" of an Internet-access
program
>> in an area where computers couldn't even stay on because of the lack of
>> reliable electricity.
>> And the cultural distance means that you have to be careful and humble
>> about how much you know about these communities and what you can do for
>> them.
>>
>> We are always seeing charities (and donors) convinced that they know the
>> "root causes of poverty" and that a well or a community mobilization
program
>> will catalyze a whole community's pulling itself out of poverty for good.
>>  But the track record of projects like these - and larger scale efforts
to
>> attack poverty at its roots - is not encouraging.  We've had 50 years of
>> huge coalitions of international aid organizations trying one theory or
>> another of the "root causes of poverty" and they haven't solved poverty.
>> Understanding the local economy seems like a very difficult undertaking
>> but that doesn't stop lots of charities from being convinced that they
do.
>>  They try to train farmers to grow cash crops, but what if the market
>> collapses due to export restrictions (as it did in one well-documented
>> case)?  Or what if the people aren't even farmers (this happened in
another
>> well-documented case)?  Similar concerns apply to education - how much do
we
>> really know about the impact of math skills on a person's life outcomes
in
>> rural Rwanda?
>>
>> And going into a foreign community and imposing your plans on it isn't
>> just risking "no benefit," it's risking "harm."  Set salaries too high
and
>> you'll pull away people who could be doing perfectly productive things in
>> their own economies, driven by local needs.  Throw too much cash into an
>> area without accountability and you could be reducing the government's
>> accountability.
>>
>> So what should you do?  Find a charity such that
>>
>> What they're doing is doable.  It's worked before.  We've got enough
giant
>> international aid agencies trying to be the first to crack the root
causes
>> of poverty.  You don't have their ability to understand the theories and
the
>> history and you don't have to throw your money into that pot.  You can
fund
>> something that's got a long history of working and that still has a lot
of
>> room to be scaled up elsewhere.
>> You can tell what they're doing.  Lots of charities do enormous numbers
of
>> unconnected projects and many seem to be essentially driven by their
>> funders.  You don't need to be funding that.
>> They're obsessively documenting that the money is reaching its intended
>> target and having its intended outcomes.  You can look at the information
>> and see that this is happening the way it's supposed to and it's helping
>> people.
>>
>> In looking for charities that work this way, we've found that:
>>
>> Health is the strongest sector.  It could be because health is easier to
>> measure and document than other areas.  It could be because Western
medical
>> knowledge translates better to another culture than Western knowledge of
>> business/community mobilization/education.  Whatever reason - health is
>> something that donors can help with.  You may care more about other
>> problems, but you should also consider what you can do about those
problems
>> and in many cases there isn't evidence that you can do very much (as a
>> donor).
>> Many of the "sexy" health interventions are overrated.  ART (AIDS
>> treatment) costs many times as much as tuberculosis treatment, and
>> tuberculosis treatment permanently cures tuberculosis.  Water projects
are
>> popular but there are many cases of wells falling into disrepair and few
>> cases of charities that provide the followup to check whether this is
>> happening.  Even if a project successfully improves water quality, it may
>> not improve health much - the major waterborne diseases are transmitted
in
>> many other ways, and water without other sanitation improvements doesn't
>> necessarily make a dent.  Insecticide-treated bednets are an exception to
>> this point; their use has a strong track record, though it isn't entirely
>> clear how much a standard net distribution program resembles the programs
>> that are known to have worked.
>> Vaccines are simple, doable, and have worked countless times before.
>>  Slightly less straightforward but also with excellent track records and
low
>> costs are mass drug administration and tuberculosis treatment.  Vitamin
>> fortification is promising.
>> The charities that best meet our criteria are StopTB and PSI (hopefully
>> more forthcoming).  Both of these charities are doing relatively
>> straightforward, life-saving things and monitoring all the info you'd
wnat
>> to monitor to be confident they're actually saving lives.  With either of
>> these charities we're ballparking around $1000 to save a life.
>
> 

We've completed a few write-ups recently, and though we're not ready to make
them entirely public, we'd like to share them with anyone who's interested
in reading them. At the moment, they're password protected, so if you'd like
to read them, email me and I'll send you the login info.

*Charity reviews:*

   - *VillageReach* - a relatively small organization that aims to increase
   vaccination coverage by improving logistics for health centers. We are still
   waiting to speak with VillageReach representatives. We think this
   organization is very promising.
   - *African Programme for Onchocerciasis Control* - a Global Health
   Partnership which funds, monitors and supports river blindness control
   programs in Africa. We don't think that we'll ultimately recommend this
   organization.

*Intervention reports:*

   - A report on the effectiveness of water supply programs at improving
   health.
   - A report on the effectiveness of condom promotion and distribution
   programs (Population Services International's primary program)

*Failures of aid:*

   - The Poverty Action Lab evaluated an HIV/AIDS education program in Kenya
   and found no impact on HIV infection.

Let me know if any of these interest you, and I'll send you the information
you need to read them.

Like the conversation Holden had with Technoserve last month (
http://groups.yahoo.com/group/givewell/message/79), we recently spoke with a
representative from VisionSpring, a potential fit for our economic
empowerment area. A paraprahsed transcript of our conversation is below. The
representative has signed off on this transcript.

-------------------------*
*

*I'd like to start by seeing if I have the right basic understanding of what
your organization does.  The basic model is to provide entrepreneurs in the
developing world with the "business in a bag," which is eyeglasses along
with training to help them sell effectively.  Is that right?**
*
That is the basic business model.  We actually have 3 channels of
distribution.  What you just described is what we call our Direct channel –
we have about 150 entrepreneurs in India and El Salvador (total) that we
train, manage and support ourselves, meaning that we have staff on the
ground actually doing that work.

But throughout the rest of the world, we partner with local organizations –
our Franchise Partner channel.  So we'll work with a microfinance
organization or health institution, typically it's an organization that has
local men or women that they're already working with, and we teach that
organization how to implement our "business in a bag" model, and we do a
feasibility study and once that phase is over, the NGO is in charge of the
business model and then it's fully their own, and we provide support.

The third channel is Wholesale: we supply our glasses to retail
organizations that are targeting poorer consumers.

*Are you giving glasses to NGOs and entrepreneurs free of charge or are you
selling them?  **
*
Selling.  Our model is fully market-based.  The goal is to build markets for
eyeglasses in the developing world.

I'll give you a quick background because it does kind of highlight what
we're doing.  Our founder is an optometrist and he spent many years in the
developing world and he saw that people generally bring used reading glasses
with them to give away, and he thought it was very inefficient to have an
eye doctor giving away glasses that go for under $10 in a US drugstore, and
he thought we could create employment for local people if we created a fully
sustainable supply chain for entrepreneurs.

This way, if somebody misses the distribution campaign or their glasses
break, or they need another pair, they have the ability to purchase those
glasses on an ongoing basis.  The mission-based distribution model doesn't
provide that aspect and it's not scalable, and it's impossible to really
match the need of the problem, which is 400 million people who need low-cost
eyeglasses.  We do charge a price for the glasses – in our network we give
the glasses to the entrepreneurs on consignment, and once they sell the
glasses they pay us back for the cost.  As for the partner organizations, we
sell them the glasses and they do that consignment model with their
entrepreneurs. The other important benefit to a market-based model is that
it creates jobs and builds the local economy.


*Why are donor subsidies needed for your organization?**
*
The answer, to me, is that we're creating markets in communities where there
is no way to get a pair of glasses on a sustainable basis to people. There
are no pharmacies, the nearest optical shop tends to be several bus rides
away, and the shop's glasses might cost $40-100 plus the travel time getting
there the exam fee.  Our model actually brings the glasses out to the rural
communities and at the same time creates jobs.  That sort of job creation is
expensive at the moment.  As we expand and we get more and more glasses, the
costs get driven down.

You might be interested to read our prospectus, which is available on our
website.  It gives some of the information you're talking about.  For
example, in 2007, when we first created this prospectus document, it cost us
about $17 to deliver a pair of glasses in the developing world.  Already,
the cost per pair of glasses is down to $11.  As we continue to scale, it is
our intention that the revenues that we bring in from the sale of the
glasses will fully cover the costs of the organization.  And so additional
donor investment will eventually go to expanding to new areas.  We're pretty
far from that right now – we're at about 20% costs covered – so there is a
large role for donations at this point.

*Are those numbers, the $17 and the $11, are those just total VisionSpring
expenses divided by total eyeglasses sold, is that how that number is
reached?**
*
Yes, we could do a channel breakdown but I don't have it right now.

*Do you have a sense of whether donor subsidies are going more to some of
your three channels than others?**
*
There is a good amount of donor subsidization of the first channel, the
Direct channel.  We look at it as sort of our innovation or R&D channel.  We
do a lot of testing in that channel.  For example, with the support of
USAID, we're doing a test to provide glasses to children.  We can only serve
adults right now, but we'd like to be able to expand our services to other
communities and other market segments.  We're investigating whether we can
have our vision entrepreneurs screen children and provide them with low-cost
glasses.  We're testing glasses that get dark in the sun and if they're
successful then we'll be able to provide them to our partners around the
world.

The Wholesale channel is the lowest-cost channel right now.  Population
Services International is a customer through that channel, they have a
network of retail pharmacy chains throughout sub-Saharan Africa, and we're
selling our glasses to them.

The other channel, the Franchise Partner channel – it is the most efficient
channel in terms of expenses, but it's a big source of revenue.  Our
partners actually pay us for the feasibility study.  And of course the
ongoing sales bring us revenue as well.

*When you talk about a feasibility study, what does feasibility mean and how
are you assessing it?**
*
On our website we have a section called Build a Franchise, which includes a
few documents that describe that.  Our biggest criterion is that the partner
has an existing network.  It's very difficult and expensive to build a
network of entrepreneurs from nothing, and we're trying to make this happen
as quickly and efficiently as possible.  To do that, we're trying to find
networks that are large and organized and that have rural reach.

The best example of this is BRAC in Bangladesh.  They have a network of
70,000 community health workers.  They have microfinance borrowers who've
also been trained to diagnose and treat the 10 most common diseases and on
top of that they sell basic products like oral rehydration salts.  We taught
50 of these women to sell eyeglasses to start, and then scaled up to 500
women. These pilots were so successful that we’ve recently signed an
agreement with BRAC to scale up to all 70,000 of these women.  BRAC is a
great partner because they're highly organized, they have a great
reputation, and they have a lot of infrastructure to be able to roll out a
program like this.  They're sort of top of the line in terms of partners –
but there's really only one BRAC.  Our goal is to find organizations that
are as much like them as possible.  For example, a microfinance institution
that has 10,000-20,000 borrowers, with an infrastructure in which someone
from the organization might go out to the village to meet with borrowers
once a week or once a month. We can tap into that infrastructure to add our
Business in a Bag model pretty easily.

Overall, we look for networks, we look for rural reach because we try to
work mostly in rural areas, and we look for organizations with similar
missions to our own.

*How do you measure your impact?**
*
The biggest measure of impact is the number of glasses sold. This is because
each pair of glasses improves the productivity and quality of life of the
person who wears glasses. We estimate that each pair of glasses generates
over $100 in additional income to the wearer. We try to choose partners that
can eventually ramp up to selling a certain number of glasses per year.  For
example, right now we're looking for partners that have the potential to
sell 10,000 glasses per year.  So we look at whether an organization is
holding up their end of the bargain and whether we're holding up our end of
the bargain.

We also look at the income of our vision entrepreneurs.  They need to be
earning not necessarily a full income, but enough of a supplemental income
that they are motivated to stay in the program and continue to sell
glasses.  It becomes inefficient to manage hundreds of vision entrepreneurs
selling a small number of glasses each month.

*Would you be able to provide us with the source/analysis for the $100 in
additional income estimate?*

A page from our prospectus explains that impact calculation in more
detail. [See
Pg 13, online at
http://visionspring.org/downloads/docs/VisionSpring_Prospectus.pdf] The
Aravind Eye Hospital study that we refer to on this page is titled the “Impact
of Uncorrected Vision - A Health Economics Perspective,” and was presented
at the 8th General Assembly of the International Agency for the Prevention
of Blindness. This study found a 10% increase in productivity thanks to
reading glasses (which we strongly believe to be a conservative number),
which we based our impact calculation on. Before publishing our prospectus,
we had this calculation vetted by Prof. Greg Dees at Duke University and
Prof. Ted London at the University of Michigan, who is leading the impact
assessment we discussed.
*Do you track the different numbers of glasses sold by channel and by
region?**
*
For our direct channels, we do track by region, and also by type.  We mostly
sell reading glasses and sunglasses.  We sell sunglasses partly because it
helps to boost the entrepreneurs' incomes, and there is a health benefit to
sunglasses – they help to prevent cataracts.  However, most of our sales are
reading glasses – I don't have the exact breakdown but I think it's less
than 25% sunglasses.

For partners, by-region breakdowns could get a little complicated.  Some of
our partners send updates on how many glasses they've sold and how many of
their entrepreneurs are still in the program, but we can't ask them to give
us super detailed information.

*Could you share a by-partner breakdown of sales?  **
*
Yes.

*Do you have written reports on how partners are progressing?  **
*
We do written reports on progress with partners in terms of sales of
glasses.  Eventually we'll make a decision of whether or not to keep going
with a partner.  As for sharing that information, I'm not sure it's fair to
report on partners in the middle of the process.

*Might you be able to share the initial reports on feasibility that you do
for partners?  **
*
Yes.

*Do you have data available on the incomes and standards of living for
entrepreneurs?**
*
We don't have very specific information about who's earning what.  We do
have data on average sales.  We generally work in low-income areas.  In
India, for example, we're working in the rural or semi-rural areas where the
average income is going to be about a dollar a day.

There have been a couple of case studies on us that have information on
entrepreneur income.  They're a little older – the University of Michigan
one came out last fall.  We've been working with the business school at the
University of Michigan over the last few years to do an impact study and
we're in the second year of the impact study.  Last summer we did the
baseline survey, looking at both entrepreneurs and customers.  In the summer
we're going to be collecting the followup surveys. The full impact study
should be available this fall, and we will be happy to share it at that
time. That will provide a robust picture of our entrepreneurs and customers,
at least within our Direct channel in India.

Under "Make an impact" on our website, there's a section called "Learn," and
the two case studies can be downloaded there.

*Do you have data available on the professions of your entrepreneurs and/or
customers?**
*
We do track professions of a lot of our customers.  Again, we don’t require
that level of detailed information from our partners, but  I can share
information on the areas in which we work directly, such as India.

*Would you share the information on standards of living?**
*
It's not necessarily easily accessible. We have a very small team so it’s
tough for us to do that level of research at this stage in our
organizations’ life. When I send you information on glasses sold by
partners, I can tell you who those partners are and what areas they're in.
One could then research the standards of living in those areas.

*But the professions of customers would be easily accessible?**
*
We do collect that data in India, which I believe would be easy for me to
send.  I think when you see the impact study as soon as that's done in
October, that's going to be a really robust picture of who the people are
that we're serving.  Generally jewelers, mechanics, goldsmiths, those sort
of professions.

*How do reading glasses help farmers increase their income?**
*
With farmers it's not as direct an impact as you'd see with a tailor.
Speaking anecdotally – we've seen some interesting stories of farmers, such
as one who prior to getting glasses had trouble seeing his seeds, so he
would just plant all the seeds and once the plants grew he'd be able to pick
out the ones he actually wanted.

*How do you determine the price that you choose to sell the glasses for?**
*
The price is set around 10% of the average monthly income for the region.

*That's the price for the entrepreneur or for the customer?**
*
For the customer.

*How do you determine the margin for the entrepreneur?**
*
The easiest case to talk about is India.  We sell glasses for $3-4 and the
entrepreneur makes $1-1.50 per pair.  The remaining margin helps us cover
our costs.  It's important for the entrepreneur to make at least $1 per pair
in India.  If an entrepreneur could sell 30 per month, that's another $1 per
day for them.  That's a number that in our experience is high enough to keep
the entrepreneur in the program.

*How do you monitor that the entrepreneur is selling for the price that you
determined?**
*
We try to ensure that as best as we can.  For example, all the marketing
materials that we have say the prices on them, so if an entrepreneur wanted
to sell for a higher price they'd have to change the materials.  We also
have supervisors going out into the field and checking up on them.  We have
set the price for each region so people generally know that that's what they
cost. Also, as part of their training they learn some of the reasons behind
why the price is set, and they learn to follow that protocol. We also find
that many of our entrepreneurs do the work from partially a social
motivation – they believe in our mission to provide low-cost glasses that
are affordable to the poor.

*Going back a little – you said that USAID is funding you to determine
whether you can be successful in selling glasses to children.  What is the
definition of success you're looking for?**
*
When we applied for the grant with them, we set a variety of benchmarks for
success.  They are very strict in their reporting requirements.  I'd be
happy to share those benchmarks with you – they are mostly about the number
of children who were screened, referred for comprehensive care, and who
receive distance glasses.

I spoke with William Easterly on Tuesday, and recorded the conversation with
his permission.  The audio file is here:
http://givewell.net/files/Research%20interviews/William%20Easterly%202009%2005%2012%20(trimmed).mp3

Highlights:

   - Agrees with us that health has a much stronger track record than other
   areas
   - Agrees with us that more measurement/evaluation is needed
   - Generally negative on the way we identified "priority interventions."
    On our sources:
      - Millions Saved (set of 20 success stories, discussed at
      http://groups.yahoo.com/group/givewell/message/25) - says it
      purposefully "cherry-picked" success stories (this was its
explicit aim) but
      thinks it did a generally good job finding the ones with good
evidence for
      impact
      - Poverty Action Lab and related (randomized controlled trials) -
      believes randomized controlled trials are good to test general
theories of
      human behavior, but not for identifying promising programs since
they only
      demonstrate (at most) that a program worked in a particular place at a
      particular time.  Contrasts with view of Poverty Action Lab that these
      studies can identify promising programs.
      - Very negative on Copenhagen Consensus - thinks cost-effectiveness
      analysis is only useful for identifying enormous differences (gives the
      example of antiretroviral therapy, far less cost-effective than other
      options).  (For the record we agree with this approach to using
      cost-effectiveness analysis, but still use Copenhagen Consensus
because a)
      it identifies interventions that are believed to be "in the range" of the
      most cost-effective; b) it represents a consensus of a large number of
      experts doing an explicitly comparative study.)
      - Suggests that we focus on interventions that "just work" - contrasts
      vaccinations (you can see it happening and be confident that the child is
      vaccinated) with "community development" (vaguer goal).  (We've gone back
      and forth on using a similar concept to designate additional "priority
      interventions" - see  http://blog.givewell.net/?p=278)
      - My note: because of the specific way we're using "priority programs"
      in our process (to flag charities/programs for further
investigation), we're
      trying to err on the side of including more rather than fewer in
our list of
      priority interventions
   - Says that "mechanical evaluation" (i.e., formal evaluation) is
   important and has its merits, but he would like to see a process that makes
   more use of person-to-person trust, which he sees as important.  Says the
   only charities he would personally get behind, currently, are small projects
   he's been able to see in person.
   - Generally negative on megacharities such as Save the Children and CARE
   (his examples) - doesn't like where they stood on the food aid debate (see
   http://www.nytimes.com/2007/08/16/world/africa/16food.html - note that
   CARE specifically was on the "right" side of this debate), says many are
   essentially contractors for USAID, which he has a low opinion of.  Also
   specifically negative on UNICEF or "anything that starts with the letters
   UN."
   - Wary of global health partnerships and other large "vertical" health
   programs - made a general statement that they are putting money and
   expertise into the "Top of an enormous funnel and they all have to go
   through the same choke point at the bottom of the funnel which is the local
   labor force."  Says "I would look in health for someone who's creative
   not only about targeting a disease and finding the right low-cost technical
   solutions and assembling all the medicines and the needles but is also
   thinking about how can they creatively get this implemented on the ground"

The link doesn't work for me.  Actually - I'm not sure if its the link or an 
issue downloading mp3s in my browser.

In any case, the weird symbols in your link title (%20), are, I think, some 
sort of odd substitution for spaces.  I would try renaming the file using 
underscores, which is more conventional and less problematic for filenames

i.e.

http://givewell.net/files/Research_interviews/William_Easterly_2009_05_12_(trimmed).mp3


----- Original Message ----- 
From: Holden Karnofsky
To: givewell@yahoogroups.com
Sent: Friday, May 15, 2009 11:11 AM
Subject: [givewell] Conversation with William Easterly






I spoke with William Easterly on Tuesday, and recorded the conversation with 
his permission.  The audio file is here:


http://givewell.net/files/Research%20interviews/William%20Easterly%202009%2005%2012%20(trimmed).mp3



Highlights:
Agrees with us that health has a much stronger track record than other areas
Agrees with us that more measurement/evaluation is needed
Generally negative on the way we identified "priority interventions."  On 
our sources:
Millions Saved (set of 20 success stories, discussed at 
http://groups.yahoo.com/group/givewell/message/25) - says it purposefully 
"cherry-picked" success stories (this was its explicit aim) but thinks it 
did a generally good job finding the ones with good evidence for impact
Poverty Action Lab and related (randomized controlled trials) - believes 
randomized controlled trials are good to test general theories of human 
behavior, but not for identifying promising programs since they only 
demonstrate (at most) that a program worked in a particular place at a 
particular time.  Contrasts with view of Poverty Action Lab that these 
studies can identify promising programs.
Very negative on Copenhagen Consensus - thinks cost-effectiveness analysis 
is only useful for identifying enormous differences (gives the example of 
antiretroviral therapy, far less cost-effective than other options).  (For 
the record we agree with this approach to using cost-effectiveness analysis, 
but still use Copenhagen Consensus because a) it identifies interventions 
that are believed to be "in the range" of the most cost-effective; b) it 
represents a consensus of a large number of experts doing an explicitly 
comparative study.)
Suggests that we focus on interventions that "just work" - contrasts 
vaccinations (you can see it happening and be confident that the child is 
vaccinated) with "community development" (vaguer goal).  (We've gone back 
and forth on using a similar concept to designate additional "priority 
interventions" - see  http://blog.givewell.net/?p=278)
My note: because of the specific way we're using "priority programs" in our 
process (to flag charities/programs for further investigation), we're trying 
to err on the side of including more rather than fewer in our list of 
priority interventions
Says that "mechanical evaluation" (i.e., formal evaluation) is important and 
has its merits, but he would like to see a process that makes more use of 
person-to-person trust, which he sees as important.  Says the only charities 
he would personally get behind, currently, are small projects he's been able 
to see in person.
Generally negative on megacharities such as Save the Children and CARE (his 
examples) - doesn't like where they stood on the food aid debate (see 
http://www.nytimes.com/2007/08/16/world/africa/16food.html - note that CARE 
specifically was on the "right" side of this debate), says many are 
essentially contractors for USAID, which he has a low opinion of.  Also 
specifically negative on UNICEF or "anything that starts with the letters 
UN."
Wary of global health partnerships and other large "vertical" health 
programs - made a general statement that they are putting money and 
expertise into the "Top of an enormous funnel and they all have to go 
through the same choke point at the bottom of the funnel which is the local 
labor force."  Says "I would look in health for someone who's creative not 
only about targeting a disease and finding the right low-cost technical 
solutions and assembling all the medicines and the needles but is also 
thinking about how can they creatively get this implemented on the ground"
 

Good suggestion, thanks.
New link:
http://givewell.net/files/ResearchInterviews/William_Easterly_2009_05_12_trimmed.mp3

On Fri, May 15, 2009 at 12:31 PM, Phil Steinmeyer
<psteinmeyer@...>wrote:

>
>
> The link doesn't work for me. Actually - I'm not sure if its the link or an
>
> issue downloading mp3s in my browser.
>
> In any case, the weird symbols in your link title (%20), are, I think, some
>
> sort of odd substitution for spaces. I would try renaming the file using
> underscores, which is more conventional and less problematic for filenames
>
> i.e.
>
> http://givewell.net/files/Research_interviews/William_Easterly_2009_05_12_
> (trimmed).mp3
>
>
> ----- Original Message -----
> From: Holden Karnofsky
> To: givewell@yahoogroups.com <givewell%40yahoogroups.com>
> Sent: Friday, May 15, 2009 11:11 AM
> Subject: [givewell] Conversation with William Easterly
>
> I spoke with William Easterly on Tuesday, and recorded the conversation
> with
> his permission. The audio file is here:
>
>
> http://givewell.net/files/Research%20interviews/William%20Easterly%202009%2005%2012%20
> (trimmed).mp3
>
> Highlights:
> Agrees with us that health has a much stronger track record than other
> areas
> Agrees with us that more measurement/evaluation is needed
> Generally negative on the way we identified "priority interventions." On
> our sources:
> Millions Saved (set of 20 success stories, discussed at
> http://groups.yahoo.com/group/givewell/message/25) - says it purposefully
> "cherry-picked" success stories (this was its explicit aim) but thinks it
> did a generally good job finding the ones with good evidence for impact
> Poverty Action Lab and related (randomized controlled trials) - believes
> randomized controlled trials are good to test general theories of human
> behavior, but not for identifying promising programs since they only
> demonstrate (at most) that a program worked in a particular place at a
> particular time. Contrasts with view of Poverty Action Lab that these
> studies can identify promising programs.
> Very negative on Copenhagen Consensus - thinks cost-effectiveness analysis
> is only useful for identifying enormous differences (gives the example of
> antiretroviral therapy, far less cost-effective than other options). (For
> the record we agree with this approach to using cost-effectiveness
> analysis,
> but still use Copenhagen Consensus because a) it identifies interventions
> that are believed to be "in the range" of the most cost-effective; b) it
> represents a consensus of a large number of experts doing an explicitly
> comparative study.)
> Suggests that we focus on interventions that "just work" - contrasts
> vaccinations (you can see it happening and be confident that the child is
> vaccinated) with "community development" (vaguer goal). (We've gone back
> and forth on using a similar concept to designate additional "priority
> interventions" - see http://blog.givewell.net/?p=278)
> My note: because of the specific way we're using "priority programs" in our
>
> process (to flag charities/programs for further investigation), we're
> trying
> to err on the side of including more rather than fewer in our list of
> priority interventions
> Says that "mechanical evaluation" (i.e., formal evaluation) is important
> and
> has its merits, but he would like to see a process that makes more use of
> person-to-person trust, which he sees as important. Says the only charities
>
> he would personally get behind, currently, are small projects he's been
> able
> to see in person.
> Generally negative on megacharities such as Save the Children and CARE (his
>
> examples) - doesn't like where they stood on the food aid debate (see
> http://www.nytimes.com/2007/08/16/world/africa/16food.html - note that
> CARE
> specifically was on the "right" side of this debate), says many are
> essentially contractors for USAID, which he has a low opinion of. Also
> specifically negative on UNICEF or "anything that starts with the letters
> UN."
> Wary of global health partnerships and other large "vertical" health
> programs - made a general statement that they are putting money and
> expertise into the "Top of an enormous funnel and they all have to go
> through the same choke point at the bottom of the funnel which is the local
>
> labor force." Says "I would look in health for someone who's creative not
> only about targeting a disease and finding the right low-cost technical
> solutions and assembling all the medicines and the needles but is also
> thinking about how can they creatively get this implemented on the ground"
>
>
>  
>

I previously discussed the "Millions Saved" set of "success stories in
global health" - see http://groups.yahoo.com/group/givewell/message/25
We've seen this work cited repeatedly in reference to "success stories in
international aid," and have factored it into our process for choosing
priority interventions (more on this later).  Since its importance to us has
been increasing, we decided that I should "vet" it to some degree - i.e.,
subject the claims of "success" to the same scrutiny we give to charities'
claims, and get an idea of how convincing these stories are from our
perspective.  So I did three things:

1. Went through all the case studies to see how they address two key
questions: (a) what sort of data supports the claim of "success" and how
reliable is it?  (b) what analysis implies that the "success" can be
attributed to the project in question, as opposed to other factors such as a
general/unrelated improvement in living standards?  My notes on this are
below.

2. Picked one to look at more closely - following its references to see
whether the picture from primary sources matches the picture given by the
case study.  I chose the one on tuberculosis control since one of our top
charities (Stop TB Partnership) focuses in this area.  My notes on this are
at http://givewell.net/node/371

3. After all this was done, I spoke with Jessica Gottlieb, who worked
directly on the revised edition of Millions Saved.  Audio recording
forthcoming.

My conclusion is that this is a fairly strong set of case studies.  None
have the sort of rigor that can be had at the micro level with randomized
controlled trials, but most (not all) have what I consider reasonably
convincing answers to the two key questions above.

--

I asked 2 major questions of each of the 20 stories:
1. What data is the claim of impact based on?  Were the data collected
through direct observation or through estimation/projection?  Should they be
considered reliable?

17 of the 20 answered this in a way that I found reasonably (if not
overwhelmingly) convincing.

   - 6 of the studies were on projects targeting elimination or
   near-elimination of a particular disease.  They refer to data collection by
   "surveillance," sometimes giving details and sometimes not.  Generally it
   seems to refer to requiring medical care centers to report directly observed
   cases (see
   http://globalhealth.change.org/blog/view/what_is_surveillance_anyway).
    With elimination programs, the incentive (unless there's a highly explicit
   and organized attempt to falsify success) is not to underreport but rather
   to make sure as many cases as possible are found (this is an integral part
   of the control strategy).
   - 1 study (Chagas in South America) was control rather than elimination
   but used the same "surveillance" terminology.
   - 3 studies (caries in Jamaica, tuberculosis in China, HIV/AIDS in
   Thailand) explicitly discussed sampling and directly testing the population.
    HIV/AIDS was done by external evaluators, China was a govt survey; Jamaica
   was a survey performed by doctors involved in the project.
   - Maternal mortality in Sri Lanka and diarrhea in Egypt both relied on
   death registers.  Vitamin A in Nepal was a demographic and health survey
   including mortality.
   - Surgery in India relied on local reporting of vision conditions.
    Fertility in Bangladesh was periodic national surveys.
   - Conditional cash transfers in Mexico were evaluated through an
   intensive study (randomized controlled trial) in a sample of districts.
   - 3 case studies (onchocerciasis control in Africa; salt iodization in
   China; tobacco regulation in Poland) were not clear on this point.

2. How was the possible counterfactual addressed?

I felt reasonably persuaded by 11 of the 20; 5 were more iffy but at least
addressed the question.

   - 6 of the studies were elimination or near-elimination of a disease;
   they did not address the counterfactual question, but presumably the idea
   that the diseases "went away by themselves" (or due to changes in standard
   of living) was fairly straightforward to dismiss in these cases.   Jessica
   Gottlieb confirmed this reasoning.
   - Conditional cash transfers in Mexico were evaluated through an
   intensive study (randomized controlled trial) in a sample of districts.
   - Tuberculosis in China compared districts that got extra funding to
   districts that didn't - reasons to feel fairly (not totally) confident are
   spelled out at http://givewell.net/node/371
   - 3 others used what I would consider common-sense persuasion.
    Discussion of HIV/AIDS in Thailand was mostly based on timing, as well as
   the coincidence of reported condom use and measured HIV/AIDS prevalence (and
   the extremeness of how the numbers changed).  Discussion of maternal
   mortality in Sri Lanka focused on a study claiming that the types of death
   that were targeted for reduction had fallen more than other types of death
   (looking at this in a variety of ways).  Neural tube defects in Chile used a
   combination of methods I found pretty convincing.
   - 2 others used regression analysis controlling for observable data such
   as changes in income.  This sort of analysis is fairly common but fairly
   controversial; I personally fall on the skeptical side.
   - 2 others explicitly addressed the counterfactual issue and said
   "studies" had addressed it, but didn't elaborate.  Tobacco regulation in
   Poland used Hungary as a "comparison group."
   - The remaining 4 did not address this at all.

The weakest on these two questions were ORS in Egypt and Chagas disease in
South America (which didn't address counterfactual at all); IDD in China and
oncho in Africa (neither of which were clear about how data was collected or
how the counterfactual was addressed).

Jessica Gottlieb told me that the counterfactual question had been
explicitly brought up and at least discussed by the working group for each
of these cases - and that it had been the main reason for rejection of many
other possible "success stories" -  though she didn't provide specifics.

She also stated that

   - The case studies were intended to be "representative" and that if there
   were several success stories for a single program type (for example,
   tuberculosis control), only one was used.  This works well for us since our
   main aim with these was to identify priority programs.
   - The most common reason for dismissing a case study was that it hadn't
   had clear and demonstrable impact.  I had been worried about missing success
   stories with clear impact but failure to meet one of the other various
   criteria; she said there were very few of these.

Interesting discussion, I especially liked the idea about some kind of
person-to-person referral network. I have no idea how this would work in
practice but it might be interesting to think about engaging with some kind
of new media facebook-type application or something similar by which you
could potentially create direct interaction with local NGOs on the ground,
and/or somehow aggregate individual recommendations.

What do you think?

--
Lee Crawfurd

Economist (ODI Fellow)
Ministry of Finance and Economic Planning
Government of Southern Sudan

lee.crawfurd@...
Zain: +249 (0)914897740
Gemtel: +256 (0)477256753


2009/5/15 Holden Karnofsky <holden0@...>

>
>
> Good suggestion, thanks.
>
> New link:
> http://givewell.net/files/ResearchInterviews/William_Easterly_2009_05_12_trimmed.mp3
>
>
> On Fri, May 15, 2009 at 12:31 PM, Phil Steinmeyer <psteinmeyer@...
> > wrote:
>
>>
>>
>> The link doesn't work for me. Actually - I'm not sure if its the link or
>> an
>> issue downloading mp3s in my browser.
>>
>> In any case, the weird symbols in your link title (%20), are, I think,
>> some
>> sort of odd substitution for spaces. I would try renaming the file using
>> underscores, which is more conventional and less problematic for filenames
>>
>> i.e.
>>
>> http://givewell.net/files/Research_interviews/William_Easterly_2009_05_12_
>> (trimmed).mp3
>>
>>
>> ----- Original Message -----
>> From: Holden Karnofsky
>> To: givewell@yahoogroups.com <givewell%40yahoogroups.com>
>> Sent: Friday, May 15, 2009 11:11 AM
>> Subject: [givewell] Conversation with William Easterly
>>
>> I spoke with William Easterly on Tuesday, and recorded the conversation
>> with
>> his permission. The audio file is here:
>>
>>
>> http://givewell.net/files/Research%20interviews/William%20Easterly%202009%2005%2012%20
>> (trimmed).mp3
>>
>> Highlights:
>> Agrees with us that health has a much stronger track record than other
>> areas
>> Agrees with us that more measurement/evaluation is needed
>> Generally negative on the way we identified "priority interventions." On
>> our sources:
>> Millions Saved (set of 20 success stories, discussed at
>> http://groups.yahoo.com/group/givewell/message/25) - says it purposefully
>>
>> "cherry-picked" success stories (this was its explicit aim) but thinks it
>> did a generally good job finding the ones with good evidence for impact
>> Poverty Action Lab and related (randomized controlled trials) - believes
>> randomized controlled trials are good to test general theories of human
>> behavior, but not for identifying promising programs since they only
>> demonstrate (at most) that a program worked in a particular place at a
>> particular time. Contrasts with view of Poverty Action Lab that these
>> studies can identify promising programs.
>> Very negative on Copenhagen Consensus - thinks cost-effectiveness analysis
>>
>> is only useful for identifying enormous differences (gives the example of
>> antiretroviral therapy, far less cost-effective than other options). (For
>> the record we agree with this approach to using cost-effectiveness
>> analysis,
>> but still use Copenhagen Consensus because a) it identifies interventions
>> that are believed to be "in the range" of the most cost-effective; b) it
>> represents a consensus of a large number of experts doing an explicitly
>> comparative study.)
>> Suggests that we focus on interventions that "just work" - contrasts
>> vaccinations (you can see it happening and be confident that the child is
>> vaccinated) with "community development" (vaguer goal). (We've gone back
>> and forth on using a similar concept to designate additional "priority
>> interventions" - see http://blog.givewell.net/?p=278)
>> My note: because of the specific way we're using "priority programs" in
>> our
>> process (to flag charities/programs for further investigation), we're
>> trying
>> to err on the side of including more rather than fewer in our list of
>> priority interventions
>> Says that "mechanical evaluation" (i.e., formal evaluation) is important
>> and
>> has its merits, but he would like to see a process that makes more use of
>> person-to-person trust, which he sees as important. Says the only
>> charities
>> he would personally get behind, currently, are small projects he's been
>> able
>> to see in person.
>> Generally negative on megacharities such as Save the Children and CARE
>> (his
>> examples) - doesn't like where they stood on the food aid debate (see
>> http://www.nytimes.com/2007/08/16/world/africa/16food.html - note that
>> CARE
>> specifically was on the "right" side of this debate), says many are
>> essentially contractors for USAID, which he has a low opinion of. Also
>> specifically negative on UNICEF or "anything that starts with the letters
>> UN."
>> Wary of global health partnerships and other large "vertical" health
>> programs - made a general statement that they are putting money and
>> expertise into the "Top of an enormous funnel and they all have to go
>> through the same choke point at the bottom of the funnel which is the
>> local
>> labor force." Says "I would look in health for someone who's creative not
>> only about targeting a disease and finding the right low-cost technical
>> solutions and assembling all the medicines and the needles but is also
>> thinking about how can they creatively get this implemented on the ground"
>>
>>
>>
>  
>

Summary of the top-contender charities and their status is here:
http://givewell.net/node/393

Here's an idea that Elie and I have been kicking around.  We'd appreciate
input.
One of the things that we've become gradually more convinced of is that in
international aid, it's really hard to have high confidence in a charity
without actually seeing how its activities look on the ground.  The fact
that it's expensive and difficult to get a look only makes it more important
to do so - see our thoughts on this at http://blog.givewell.net/?p=381

To date, our approach has been to find the charities that we can have the
highest confidence in possible without visiting, and then "spot-check" them
with visits if and when we're able to.  However, the charities that we can
have the highest confidence in without visiting tend to be large
bureaucracies, as these types are most likely to be able to provide thorough
documentation.  And large bureaucracies are in some ways the hardest to
evaluate on the ground, as they do so many different things in so many
different places.

What we're thinking about is making a separate list of "Charities that are
most worth visiting in order to get a read on."  We couldn't recommend these
charities without visiting them, but they're the ones we feel are most
likely to be a good bet for a donor if, after a visit, it appears that their
operations on the ground are running roughly as described/intended.

Then, we would personally visit the top couple on the list, and post as much
photos/video as possible.  We would also encourage donors to do the same,
and send photos/video (and comments) to us for public posting.

The criteria for making this list would be pretty different from the
criteria for making our current list of recommendations.  We've only thought
about this a little, but here are some things I would look for for a charity
to make this list.

1. Activities: proven, cost-effective, straightforward.  If a charity is
providing medical treatment or vaccinating people, a personal visit could
verify whether vaccinations/treatments are occurring, diagnoses are
accurate, etc.  More ambitious economic development initiatives would be
impossible to assess without formal outcomes tracking and evaluation.
 (Again, see our blog post on this subject: http://blog.givewell.net/?p=381
.)

2. Narrow geographic and programmatic focus.  The more different locations
and activities a charity has, the more difficult it is to personally see how
everything is going.

3. Size: not too big, not too small.  Very large organizations would fail on
point #2 ... I'd need years to do enough visits to see a representative set
of CARE's activities, for example.  But very small organizations are not
appropriate for public recommendations for individual donors.  Ability to
absorb funds and scale up activities beyond a single location are important.

This list would take some time to make; we would also have to spend extra
time on site visits to these charities, unless we decided to skip site
visits for the larger ones; and if we published recommendations hinging
heavily on visits, we would be asking people to "trust us" more than we
currently do (currently nearly everything we say is based on publicly
available materials; in this case people would have to feel that the
photos/video we were posting were representative).  On the other hand, we
feel that this task is doable in a reasonable time frame and might generate
recommendations that we - and at least some of our donors - could be more
confident in than our current recommendations.

Thoughts?

Have you considered crowdsourcing some of these visits to people already in the area?  E.g. backpackers or students studying abroad, who might be energized to help out.  This couldn't quite replace your own visits, but it could supplement them, and at very least could reveal some flagrant failures to deliver advertised services.

I think Seth's idea is an excellent one. If you can figure out how to
disperse the criteria you'd like to use, and you could reasonably insure
against bias (i.e. preventing someone with close ties to the said NGO
"evaluating" it with a visit), it might be a great, cheap way to assess
along these lines.

On Thu, May 21, 2009 at 12:58 AM, seth.blumberg <sblumberg@...> wrote:

>
>
> Have you considered crowdsourcing some of these visits to people already in
> the area? E.g. backpackers or students studying abroad, who might be
> energized to help out. This couldn't quite replace your own visits, but it
> could supplement them, and at very least could reveal some flagrant failures
> to deliver advertised services.
>
>  
>



-- 
Lindy Miller Crane

**We are the ones we have been waiting for.**

Assuming it is realistic for people to see the programs on the ground
(which I won't be surprised if there are various difficulties), I also
think the idea carries merits:
1. there is some chance that in-depth analysis will come out
2. short of in-depth analysis, I'd think some anecdotal stories could
be interesting to lots of people, possibly helping givewell to reach
to new audiences (those who can be convinced).

Givewell could possibly reach out to some of the relevant travel
agencies specializing in trips focusing on development.

If it somewhat takes off, there will be some additional cost of
content management down the road. But it'd be a good problem to have!
:)

- sam

On Thu, May 21, 2009 at 4:42 AM, Lindy Miller Crane
<hellolindy@...> wrote:
>
>
> I think Seth's idea is an excellent one. If you can figure out how to
> disperse the criteria you'd like to use, and you could reasonably insure
> against bias (i.e. preventing someone with close ties to the said NGO
> "evaluating" it with a visit), it might be a great, cheap way to assess
> along these lines.
>
> On Thu, May 21, 2009 at 12:58 AM, seth.blumberg <sblumberg@...> wrote:
>>
>>
>> Have you considered crowdsourcing some of these visits to people already
>> in the area? E.g. backpackers or students studying abroad, who might be
>> energized to help out. This couldn't quite replace your own visits, but it
>> could supplement them, and at very least could reveal some flagrant failures
>> to deliver advertised services.
>>
>
>
>
> --
> Lindy Miller Crane
>
> **We are the ones we have been waiting for.**
> 

In general, I like Holden's idea.  

Key questions are:

1) To what extent can "did it happen" be readily verified on the ground?  i.e. In what percentage of cases would this be important?
2) How difficult are such visits?  Effort, time, cost...  How cooperative would charities be?

Finding a local to report, more cheaply, SEEMS (at first blush) like a good idea, but on further reflection, there are some problems:

Is our goal to find someone with a donor mindset and experience to report on the charity activities?  If so, the likelihood that we would find many donors with ready access to rural Africa or Asia or Latin America seems low.

We might have better luck finding either Western NGO workers or local residents close to the action.

But there are problems with each of these.  I suspect that in a given area, the community of Western NGO workers is generally small and somewhat tight, which would interfere with the ability/willingness of an NGO worker to be brutally honest in reporting on the activities of others.

As for locals, they would probably be much harder to find.  If charities are working in areas where folks earn a dollar or two a day, how many of those locals are going to have internet access, English fluency, and be reachable by someone in, say, New York?  Even if we did find such folks, I foresee possible conflicts and simply communication gaps in trying to get the kind of information that I think Holden is talking about.

That said, finding locals would certainly be nice, but I wouldn't want to see TOO much effort put in that direction if it early efforts didn't seem fruitful.

===

A few years ago, when I was first ramping up my charitable giving, I had vague thoughts of traveling to Africa or other poor areas to get a feel for charity activities and the general dynamics of poor nations from the ground level.  I have not followed through with it, and haven't really thought about it recently.  Still, I like the concept, and would at least consider doing something like this personally in the right scenario.

 

  

Thanks for the feedback.

We're leaning toward an incremental approach.  First step would be to
identify the charities where a visit would be most worth doing.  At that
point we would invite anyone who cared to to submit photos, video, and
comments, but we wouldn't sit back and wait for it – we would do visits
ourselves and hopefully come back with a better sense of what we're looking
for and a better sense of who else could help add to our knowledge base (and
how).  From there we might more actively push others (locals?  Students?  Aid
workers?  Donors?  No need to decide this yet) to participate.  If we were
getting loads of useful data, we would then think about how to
process/display it more systematically.

This would all start post 7/1 (except perhaps the "making a list" part), so
it would be unlikely to become a major info source in the short term.


On Thu, May 21, 2009 at 8:44 PM, Phil Steinmeyer <psteinmeyer@...>wrote:

>
>
>  In general, I like Holden's idea.
>
> Key questions are:
>
> 1) To what extent can "did it happen" be readily verified on the ground?
> i.e. In what percentage of cases would this be important?
> 2) How difficult are such visits?  Effort, time, cost...  How cooperative
> would charities be?
>
> Finding a local to report, more cheaply, SEEMS (at first blush) like a good
> idea, but on further reflection, there are some problems:
>
> Is our goal to find someone with a donor mindset and experience to report
> on the charity activities?  If so, the likelihood that we would find many
> donors with ready access to rural Africa or Asia or Latin America seems low.
>
> We might have better luck finding either Western NGO workers or local
> residents close to the action.
>
> But there are problems with each of these.  I suspect that in a given area,
> the community of Western NGO workers is generally small and somewhat tight,
> which would interfere with the ability/willingness of an NGO worker to be
> brutally honest in reporting on the activities of others.
>
> As for locals, they would probably be much harder to find.  If charities
> are working in areas where folks earn a dollar or two a day, how many of
> those locals are going to have internet access, English fluency, and be
> reachable by someone in, say, New York?  Even if we did find such folks, I
> foresee possible conflicts and simply communication gaps in trying to get
> the kind of information that I think Holden is talking about.
>
> That said, finding locals would certainly be nice, but I wouldn't want to
> see TOO much effort put in that direction if it early efforts didn't seem
> fruitful.
>
> ===
>
> A few years ago, when I was first ramping up my charitable giving, I had
> vague thoughts of traveling to Africa or other poor areas to get a feel for
> charity activities and the general dynamics of poor nations from the ground
> level.  I have not followed through with it, and haven't really thought
> about it recently.  Still, I like the concept, and would at least consider
> doing something like this personally in the right scenario.
>
>
>   
>

I happened onto an interesting article
<http://informationincontext.typepad.com/good_intentions_are_not_e/2009/\
05/bad-donor-advice-perpetuates-bad-aid-practices.html>  at the blog
"Good Intentions are not Enough" rebutting theWSJ's advice for smarter
giving
<http://online.wsj.com/article/SB124311556111550123.html#articleTabs%3Da\
rticle> .  Perhaps some of you have already seen it, or know the
blog.  Saundra Schimmelpfennig, the author, started D-TRAC, an
organization to track and hold accountable post-tsunami aid in Thailand,
and which seems to have evolved into ChildTRAC, a charity to provide
children with I'm-not-quite-sure-what.  Now, according to her blog,
she's looking to publish a book she's written called "Beyond Good
Intentions: How to Make Your Donation Dollars Do the Good You Intended".
Anyway, she takes issue with each one of the questions the WSJ suggests
donors to ask, and suggests that they instead:1) Ask for a copy of last
year's audit findings.2) Request the results of several independent
evaluations of the aid agency's work.3) Request the results of the
agency's needs assessment.
She also gives advice on some resources to learn more about good giving.
 (No need for me to reprint them, but I'll note that Givewell is not
among them.)  She's quite conscious that the above three things may
be difficult for a donor to understand, but she says that their mere
existence and frequency will convey useful information.
I'm less interested in her or the WSJ's specific suggestions (though to
anyone who is, please comment) than the mission question of how GiveWell
should prioritize the results of our own research of the best charities
to give to vs. advising donors how to research for themselves.  To
a large extent, the blog and our explanations of our ratings do serve
the latter purpose, educating the public and fostering debate.  But
our focus is, and I believe should be (for now), on the former.
However, the WSJ article mentions a group of friends who pooled together
their donations and started a group to support local nonprofits in
Nebraska, and it suggests forming such Giving Circles as one way to
donate, even if you don't have much money.  I wonder how popular such
groups are, and if they could take off the way CSAs (Community Supported
Agriculture organizations - which pay in advance for locally-grown
produce) have.  Isn't such a group the genesis story of GiveWell?
 If we or others lowered the bar for the effort and initiative it
takes to start a giving circle, perhaps they could become more popular.
 I think American society is becoming more socially conscientious of
issues like international poverty, so maybe this isn't a crazy
prediction.
Also, if we provided such groups with guidelines on how to evaluate
charities, perhaps they could do some of our research for us, enabling
us to expand our scope (particularly to domestic charities in many
localities).  I don't know how much of charitable giving is local,
but it's probably a lot, and it's probably nearly as in need of a better
system than international aid.
Since we already do lots of thinking about how to evaluate charities and
have experience with how hard it can be to get useful information,
making guidelines for outsiders would build on existing strengths.
 (Though it would be somewhat labor-intensive.)  Such a feature
would represent a more developed stage of our evaluation methodology,
but the exercise would probably also help us develop and simplify
those guidelines.
Well, this is another long-term crazy idea, like my last post about
travelers conducting site visits, but this seems like the right venue to
share these ideas.  Thoughts?
-Seth

We're working on something along these lines - a rough generic guide
for how a donor can/should investigate a charity.  We probably won't
have a draft before 7/1 (we're focused on our international aid report
until then) but there's a good chance that we will have a draft by
7/30 (for the next meeting of the Alliance for Effective Social
Investing - www.alleffective.org)


On Tue, Jun 2, 2009 at 4:13 AM, seth.blumberg <sblumberg@...> wrote:
>
>
> I happened onto an interesting article at the blog "Good Intentions are not
> Enough" rebutting theWSJ's advice for smarter giving.  Perhaps some of you
> have already seen it, or know the blog.  Saundra Schimmelpfennig, the
> author, started D-TRAC, an organization to track and hold accountable
> post-tsunami aid in Thailand, and which seems to have evolved into
> ChildTRAC, a charity to provide children with I'm-not-quite-sure-what.  Now,
> according to her blog, she's looking to publish a book she's written called
> "Beyond Good Intentions: How to Make Your Donation Dollars Do the Good You
> Intended".
> Anyway, she takes issue with each one of the questions the WSJ suggests
> donors to ask, and suggests that they instead:
> 1) Ask for a copy of last year's audit findings.
> 2) Request the results of several independent evaluations of the aid
> agency's work.
> 3) Request the results of the agency's needs assessment.
> She also gives advice on some resources to learn more about good giving.
>  (No need for me to reprint them, but I'll note that Givewell is not among
> them.)  She's quite conscious that the above three things may be difficult
> for a donor to understand, but she says that their mere existence and
> frequency will convey useful information.
> I'm less interested in her or the WSJ's specific suggestions (though to
> anyone who is, please comment) than the mission question of how GiveWell
> should prioritize the results of our own research of the best charities to
> give to vs. advising donors how to research for themselves.  To a large
> extent, the blog and our explanations of our ratings do serve the latter
> purpose, educating the public and fostering debate.  But our focus is, and I
> believe should be (for now), on the former.
> However, the WSJ article mentions a group of friends who pooled together
> their donations and started a group to support local nonprofits in Nebraska,
> and it suggests forming such Giving Circles as one way to donate, even if
> you don't have much money.  I wonder how popular such groups are, and if
> they could take off the way CSAs (Community Supported Agriculture
> organizations - which pay in advance for locally-grown produce) have.  Isn't
> such a group the genesis story of GiveWell?  If we or others lowered the bar
> for the effort and initiative it takes to start a giving circle, perhaps
> they could become more popular.  I think American society is becoming more
> socially conscientious of issues like international poverty, so maybe this
> isn't a crazy prediction.
> Also, if we provided such groups with guidelines on how to evaluate
> charities, perhaps they could do some of our research for us, enabling us to
> expand our scope (particularly to domestic charities in many localities).  I
> don't know how much of charitable giving is local, but it's probably a lot,
> and it's probably nearly as in need of a better system than international
> aid.
> Since we already do lots of thinking about how to evaluate charities and
> have experience with how hard it can be to get useful information, making
> guidelines for outsiders would build on existing strengths.  (Though it
> would be somewhat labor-intensive.)  Such a feature would represent a more
> developed stage of our evaluation methodology, but the exercise
> would probably also help us develop and simplify those guidelines.
> Well, this is another long-term crazy idea, like my last post about
> travelers conducting site visits, but this seems like the right venue to
> share these ideas.  Thoughts?
> -Seth
>
> 

Micro-lending has been the "hot" form of economic aid to the very poor for some time now, based largely, I think, on the purported success of Grameen Bank.

I recently came across an article that is admittedly quite old (November 2001) that calls into question a basic premise of the micro-lending success story - that repayment rates are very high.

Here's a link to the article:

http://online.wsj.com/public/resources/documents/pearl112701.htm

I looked briefly for something more recent and found something that appears to be by Muhammed Yunus himself, here:

http://www.grameen-info.org/index.php?option=com_content&task=view&id=30&Itemid=0

I just skimmed part of the latter (it's long), but it seems to confirm that Grameen went through major changes circa 2002.  While this latter article emphasizes the impact of natural disasters, particularly a 1998 flood, it also mentions problems with borrowers starting in 1995, and in any case, we might expect a somewhat more positive spin on things from the person most commonly associated with micro-lending.

Now, it's possible that the bad luck of the 1998 flood was a major issue and/or that the 2002 changes fixed things for Grameen, but I was still a bit surprised to see that a concept that was begun in 1976 and heralded as a major success nonetheless had to, apparently, go through such an overhaul in 2002.

It goes in hand with some other issues I've had with micro-lending.  Here's a quick thought experiment.  The numbers are wild guesstimates, but till...

Let's say that about areas containing about 1 billion people are reasonable candidates for micro-lending projects.  Let's say that in those areas, 10% of the population are reasonable candidates.  (Not everyone in a micro-lending area will want or be suitable for this sort of thing).  Let's say that the average loan needed is $100.  Do the math, and it equals $10 billion in capital needed for this.  If repayment rates are, say, 98%, then there is scarcely any need for additional capital after the first $10 billion is allotted.  

Now, that $10 billion may be very well spent.  But as hot as micro-lending is, and as long as it has been a hot topic, I would suspect that it has probably raised something of this level or more.  Remember that Gates and Buffett alone have more going into philanthropy, now or in the relatively near future, than this, and if the evidence were really strong and there was a funding gap, I suspect they (or someone else) would have filled it.

Some of my numbers may be off - maybe the average loan is or should be $200, but some numbers may be high - maybe only 5% of the population is really able to use these loans effectively.  

===

OK, so that's a lot of supposition, wild guesses, and pessimism by me.  Anyways, it's another data point or two for the discussion on micro-finance.

BTW, GiveWell looks at micro-finance here:
http://www.givewell.net/node/154

I've provided you the direct link.

BUT, I challenge you to find your way to that article by conventional website navigation.  Go ahead, start at www.givewell.net, with the idea "I wonder if GiveWell has anything to say about micro-finance", and see how easily you can answer your question. 

Overview of developing-world schooling (and why we aren't recommending any
charities in this area) is up at http://givewell.net/node/396
Some minor notes follow re: things I left out.

   - Under "scholarships": very little info is given about the Bangladesh
   study; it would be good to look at it and get the details.
   - When discussing the impact of education on life outcomes, I left out
   Glewwe's discussion of education's impact on health.  It notes that as with
   income, past studies look at simple correlations (implying that they aren't
   isolating the causal effect).  It cites two studies that Glewwe feels are
   superior; one found no impact, and the other found an impact using the kind
   of regression analysis that I criticize in the same section (when discussing
   income studies).  I didn't think that going into this would add much so I
   left it out for now.
   - I originally included 3 studies on Israel, but cut them after looking
   at the details and determining that there's no reason to consider them
   relevant for the "developing world."  The first footnote addresses this.

http://givewell.net/node/413
Lays out the very limited information on the track records of economic
empowerment programs (i.e., programs focused on raising incomes directly as
opposed to targeting health or education) and our framework for evaluating
such programs.

Includes links to our agriculture page (http://givewell.net/node/410) and
our current review of microfinance (http://givewell.net/node/154), although
we may be making significant revisions to the latter.

See http://blog.givewell.net/?p=396

After reading through your charity reviews, I put together some initial
thoughts on your rankings.  Here's how I'd think about ranking
the charities, coming from the perspective of a donor:

I want my donation to save/improve lives in a cost effective way.  To
have confidence that it will, I want to see empirical data and strong
intuitive logic to support a given program/charity.  I know there's
so much measurement error around any metrics and data we see that that
I'm mostly concerned about big downside risk, i.e. that there's
a material risk my money is simply wasted; it's just not worth
taking that risk when there's other charities available that
don't have it.

With that in mind, some of the charities have enough of that downside
risk to warrant being in the second tier.  It's certainly possible
that further research could assuage these concerns.

Global Fund:  There are enough red flags that it's tough to get
excited about the Global Fund.  There's the lack of clarity about
where the marginal dollar goes, the reliance on procedure-less
independent auditors, no indication of that ineffective programs (of
which there are undoubtedly many) are being shut down, and the
significant resources dedicated to relatively cost-ineffective programs
(anti retroviral treatment.)

GAVI: Without a good explanation for the outside grants and a list of
unfunded projects, I see no reason to risk a donation to a charity that
might can't provide a good reason why they need the funds.

AMF: The lack of utilization data is precisely the type of thing I'm
worried about.  Given the other alternatives, I don't see any reason
to incur the risk that the nets aren't being used or maintained. 
While this concern also applies to PSI, I think PSI has a few
advantages: they are collecting data on net usage (though the monitoring
leaves much to be desired), they explicitly focus on marketing, and on
the margins I think selling rather than freely distributing nets is more
likely to lead to sustained usage.

Among PIH, Stop TB Partnership, and PSI, I don't have concerns of
such large magnitudes.  Among these three, I feel confident that a
donation would meet my goal of saving/improving lives in a
cost-effective manner.  In thinking through how to choose between them,
I'd be weighing off fairly subjective concerns like whether it's
better to focus on the most cost-effective interventions (which would
favor PSI or Stop TB) or whether the more wide-ranging treatments PIH
offers is actually a better way to improve lives.

That leaves Village Reach, which I'm kind of torn about.  Their
model is simple and logical, and their commitment to monitoring,
evaluation, and reflection seems fantastic.  My intuition is also that a
small, focused organization like Village Reach is more likely to be able
to effectively use the results of monitoring and evaluation to make
necessary adjustments than a larger organization would be.  The
methodology in their monitoring is about as sound as we're likely to
see, and the data looks great.  But… there are really large risks,
larger than some of the risks of the charities I see as second tier.  I
think you nailed them in your summary- it's a young charity, trying
to massively expand in scope, that hasn't proven it can realize a
substantial part of the benefit of its activities yet (via the transfer
to governments).  That's a really scary combo.  Since there are
alternatives that I don't think have risks of these magnitudes,
I'd grudgingly put Village Reach in the second tier.  A couple of
things might change my mind.  It would be great to get an assessment of
how well the pilot program went from a project management perspective
(did it stay on budget, were timelines met, etc.).  If it went well,
I'd be less concerned about the expansion in scope.  Some recovery
in the data from the pilot program would be nice too.






--- In givewell@yahoogroups.com, Holden Karnofsky <Holden@...> wrote:
>
> See http://blog.givewell.net/?p=396
>

Thanks for sending these comments. Here are some initial thoughts; we'll
plan to send more later.

There's a big difference in the confidence I have in our "top-rated" vs
merely"recommended" charities, so I disagree with some of the conclusions
you reach below. I wanted to discuss two points in particular:

   1. "Among PIH, Stop TB Partnership, and PSI, I don't have concerns of
   such large magnitudes.  Among these three, I feel confident that a donation
   would meet my goal of saving/improving lives in a cost-effective manner."
   2. "[VillageReach] is a young charity, trying to massively expand in
   scope, that hasn't proven it can realize a substantial part of the benefit
   of its activities yet (via the transfer to governments).  That's a really
   scary combo.  Since there are alternatives that I don't think have risks of
   these magnitudes, I'd grudgingly put Village Reach in the second tier."

*PIH and PSI vs Stop TB

*I think there are strong reasons to support both PSI and PIH, but at the
same time, both have significant weaknesses that would lead me to support
Stop TB before either of them.

*PIH:* PIH has a common-sense model, but I think there's a big risk that the
effect visible to a donor (a fully-functioning health facility in a location
where one did not previously exist) is largely the result *of shifting
resources* *from one location in a country to another.* For example, because
they don't train doctors, for PIH to staff its Rwandan facility with Rwandan
doctors, it relies on relocating doctors from one location in Rwanda to
another. To the extent that PIH is doing that, we think there's reason to
significantly discount the overall impact their programs have.  We discussed
this issue in our review at
http://givewell.net/pih#Possiblenegativeoffsettingimpact

Even though we this issue leads me to significantly discount PIH's *apparent
* impact, I think they are still have some impact by (a) providing trained
doctors with the facilities they need to provide top-notch healthcare. In
addition, in some of their facilities, many of the clinicians are
developed-world doctors who travel abroad to staff part/much of the clinic.

*PSI: *PSI does have a stronger commitment to monitoring and evaluation than
almost any other charity, but it's the evidence provided in that
documentation provides a mixed case for the *impact *that PSI's programs
have. PSI supports bednet provision, and monitors bednet use, but it's
unclear that PSI's activities have increased the number of people that are
sleeping under nets. And, bednets are a relatively easy case because
ITN-distribution and promotion is a program with extremely strong evidence
behind it (see givewell.net/node/329. Condom promotion and distribution is
much trickier. If you can get people to consistently use condoms, you'll
likely reduce HIV/AIDS transmission but there's no "proven" approach for
accomplishing that (for more see givewell.net/node/375). PSI's approach
seems as good as any, but their relatively inconsistent monitoring and
evaluation that relies solely on self-reported accounts of behavior is
somewhat questionable. It's far less compelling to me than either Stop TB's
data on completed TB treatments and patient outcomes or VillageReach's data
on children vaccinated.

I don't think Stop TB has these same kinds of weakness, and therefore, among
these three, I'd support them.

*VillageReach*

I think the evidence that VillageReach provides for the impact of their
program is unmatched among any charity I've seen. VillageReach came to Cabo
Delgado; they reorganized and supplemented the vaccine-delivery system; they
measured (a) changes in drug availability in clinics and (b) changes in
children immunized, a life-saving intervention. They compared Cabo Delgado's
success to that of a nearby district. On all measures, VillageReach's
programs appears a success. No other charity I've looked at can offer a case
for impact as compeling as that.

I am not terribly concerned about the question of whether VillageReach can
successfully pass off its activities to the government, because we evaluated
them mostly under the assumption that they can't do so *at all*, and even
with this assumption still consider them to be as proven and cost-effective
as any of the other charities we've seen (see the VillageReach review for
details) (see http://givewell.net/node/370#Whatdoyougetforyourdollar for
details).

I'd guess that the risk of VillageReach scaling up is somewhat low, though
there is clearly some risk. VillageReach is currently seeking $750,000 for
2009 which would lead to (approximately) a scale up of 2-3x the size of
their current projects. That seems like an appropriate increase given the
strong success of their current project.

-Elie

On Mon, Jul 6, 2009 at 12:06 AM, jonbehar <jonbehar@...> wrote:

>
>
> After reading through your charity reviews, I put together some initial
> thoughts on your rankings.  Here's how I'd think about ranking the
> charities, coming from the perspective of a donor:
>
> I want my donation to save/improve lives in a cost effective way.  To have
> confidence that it will, I want to see empirical data and strong intuitive
> logic to support a given program/charity.  I know there's so much
> measurement error around any metrics and data we see that that I'm mostly
> concerned about big downside risk, i.e. that there's a material risk my
> money is simply wasted; it's just not worth taking that risk when there's
> other charities available that don't have it.
>
> With that in mind, some of the charities have enough of that downside risk
> to warrant being in the second tier.  It's certainly possible that further
> research could assuage these concerns.
>
> Global Fund:  There are enough red flags that it's tough to get excited
> about the Global Fund.  There's the lack of clarity about where the marginal
> dollar goes, the reliance on procedure-less independent auditors, no
> indication of that ineffective programs (of which there are undoubtedly
> many) are being shut down, and the significant resources dedicated to
> relatively cost-ineffective programs (anti retroviral treatment.)
>
> GAVI: Without a good explanation for the outside grants and a list of
> unfunded projects, I see no reason to risk a donation to a charity that
> might can't provide a good reason why they need the funds.
>
> AMF: The lack of utilization data is precisely the type of thing I'm
> worried about.  Given the other alternatives, I don't see any reason to
> incur the risk that the nets aren't being used or maintained.  While this
> concern also applies to PSI, I think PSI has a few advantages: they are
> collecting data on net usage (though the monitoring leaves much to be
> desired), they explicitly focus on marketing, and on the margins I think
> selling rather than freely distributing nets is more likely to lead to
> sustained usage.
>
> Among PIH, Stop TB Partnership, and PSI, I don't have concerns of such
> large magnitudes.  Among these three, I feel confident that a donation would
> meet my goal of saving/improving lives in a cost-effective manner.  In
> thinking through how to choose between them, I'd be weighing off fairly
> subjective concerns like whether it's better to focus on the most
> cost-effective interventions (which would favor PSI or Stop TB) or whether
> the more wide-ranging treatments PIH offers is actually a better way to
> improve lives.
>
> That leaves Village Reach, which I'm kind of torn about.  Their model is
> simple and logical, and their commitment to monitoring, evaluation, and
> reflection seems fantastic.  My intuition is also that a small, focused
> organization like Village Reach is more likely to be able to effectively use
> the results of monitoring and evaluation to make necessary adjustments than
> a larger organization would be.  The methodology in their monitoring is
> about as sound as we're likely to see, and the data looks great.  But… there
> are really large risks, larger than some of the risks of the charities I see
> as second tier.  I think you nailed them in your summary- it's a young
> charity, trying to massively expand in scope, that hasn't proven it can
> realize a substantial part of the benefit of its activities yet (via the
> transfer to governments).  That's a really scary combo.  Since there are
> alternatives that I don't think have risks of these magnitudes, I'd
> grudgingly put Village Reach in the second tier.  A couple of things might
> change my mind.  It would be great to get an assessment of how well the
> pilot program went from a project management perspective (did it stay on
> budget, were timelines met, etc.).  If it went well, I'd be less concerned
> about the expansion in scope.  Some recovery in the data from the pilot
> program would be nice too.
>
>
>
>
>
>
> --- In givewell@yahoogroups.com, Holden Karnofsky <Holden@...> wrote:
> >
> > See http://blog.givewell.net/?p=396
> >
>  
>

I talked with representatives of two charities (AMREF and IDE) yesterday and
thought it was worth sharing notes on our conversation.

*AMREF* (African Medical and Research Foundation) http://www.amref.org/

We gave AMREF one star because of (a) its rare publication of a
comprehensive summary report of projects, budgets, and expected outcomes for
all of its projects and (b) its online database of reports from completed
projects.

*The bottom line for a donor*

Itseems to me that unrestricted funds are essentially used to support
AMREF's entire portfolio of activities and therefore, the right way to
assess the impact of a donation is to evaluate the entirety of AMREF's
activities.

It also seems like funding AMREF may be functionally equivalent, at least
partly, to funding the large government donors (e.g., USAID, CIDA) because
AMREF is largely acting on the bilaterals' priorities and individual
donations support those activities.

*Basic structure*

AMREF is a large (~$70-80m budget) international organization. We think its
structure is likely reasonably representative of the structure of other
"household name" charities like CARE, Save the Children, World Vision, etc.
It runs many, highly diverse projects (around 100).

AMREF has 12 offices around the world (10 in Europe, 1 in Canada and 1 in
the US.) These offices raise funds from donors in their country to support
AMREF activities in Africa. I spoke with a representative from AMREF USA.

*How are individual, unrestricted donations used?

*
AMREF raises a significant portion -- the representative I spoke with wasn't
sure but guessed that it's more than half -- of their funds by responding to
RFPs put out by major government funders (e.g., CDC, USAID, CIDA). In
responding to these RFPs, we think that AMREF is essentially acting as a
contractor to implement the donor's program. Funding given for these
proposals is restricted to the specific, narrowly-defined project for which
they were given.

AMREF raises approximately 20-25% of its funds as unrestricted
donations. *AMREF
uses unrestricted donations to (a) run pilot projects for which they
subsequently seek government-donor funding to scale up, (b) responding to
short-term crises (e.g., disaster relief), (c) to supplement funding for the
"restricted" projects mentioned above.* The representative I spoke with gave
several examples of why (c) might be necessary:

   1. AMREF received a grant to implement a health program in Kenya but,
   after receiving the grant, prices for fuel, food, etc. in Kenya rose
   significantly.  AMREF's grant was in U.S. dollars and the donor wouldn't
   increase the amount of the grant; therefore, AMREF needed to supplement
   funding for that project with unrestricted funds so that they could complete
   the project.
   2. AMREF received a grant to train x number of people in Kenya. The
   Kenyan government then decided that AMREF needed to train 4x as many people.
   (I'm not entirely clear on why or how this happened.) To fullfill the Kenyan
   government's requirements, AMREF needed to use unrestricted funds to allow
   them to train more people.

*What monitoring and evaluation is available?*

Based on what's currently available online, it's not possible to fully
evaluate AMREF's activities. The Annual Program Reports provide an overview
of all projects and list publications that evaluate those projects, but
those documents are often not online.

The online database presents some AMREF documents that they've decided to
share. The representative I spoke with didn't know how they chose which
documents to share, but did say that the online database is not meant to be
a comprehensive repository of all evaluation documents.

*IDE* (International Development Enterprises) http://www.ideorg.org/

IDE focuses on economic empowerment in the developing world. We are going to
grant $250,000 to developing-world economic empowerment and we're talking to
organizations to learn more about what the do and how they measure impact.

IDE develops and markets technologies for use by farmers; trains farmers in
improved techniques; and helps farmers access markets for their crops.

IDE tries to identify "market failures" in agriculture in a specific region.
For example, they'll talk to farmers in a region about what they do and
problems they face, and learn that the lack of proper irrigation is a major
obstacle for farmers. IDE has developed several products that can fill this
gap: a drip irrigation kit or a foot-operated pump. Then, IDE finds a local
manufacturer to produce and sell the pumps. The products are sold at market
prices to farmers by the manufacturer.

It seems to me that if we saw that farmers continue to buy IDE's products
over time that would be relatively strong evidence that the farmers value
the products and IDE is probably having an impact.

The IDE representative I spoke with says that they track farmers carefully
for three years and said she could share some of that monitoring and
evaluation with us.

A couple things to add.

For me, the distinction between *** and ** is a very important one.  Basically,
I feel that for a *** charity, we have reasonable confidence in the full set
of its activities, and feel that the cost-effectiveness estimate provided is
a reasonable (if very rough) approximation to its overall impact.  By
contrast, for a ** charity there is a crucial piece of the puzzle missing –
missing data on a highly questionable link in the chain, questions about how
representative the data we have is, etc. – and while we feel the charity is
a much better bet than lower-rated charities, and is likely doing a
substantial amount of good, we don't have a good sense for how often its
activities are going as hoped.

To me, the two charities Jon mentioned as being in his "top tier" are in the
** tier:

*PSI.  *PSI stands above non-recommended charities because it is
systematically asking the questions we feel need to be asked to give
confidence in their activities, and a significant amount of data appears to
be actually collected and available.  That said:

1.      We've only seen a sample of their data, and while we don't feel the
sample was "cherry-picked," we do feel that better-run projects may be more
likely to get their data in to the central office.  So we're concerned about
representativeness.

2.      The data do not point strongly to impact.  *Changes* in reported
behavior do not particularly suggest impact; the high *levels* of reported
condom/ITN use, combined with PSI's role as a dominant supplier, make it
seem likely that PSI is getting these materials to people who are using
them.  It may simply be substituting for for-profit suppliers, though a
limited set of studies (see
http://givewell.net/node/329#Freenetsvssellingnetsforafee) suggests that
subsidized/free distribution has benefits.

*PIH.  *Elie has addressed a couple of the concerns with PIH.  I would add
that

1.      PIH does appear to conduct a variety of programs whose impact would
be harder to assess than that of direct medical care (microloans, school
scholarships, population-based health initiatives).  In the Rwanda budget
(the only budget we have that can give a sense for the relative size of
these programs), these programs appear to consume about 7% of the funds.  We
aren't sure whether we're under-allocating administrative expenses to these
programs, whether they've grown since we last looked, whether they're larger
at other locations, etc.

2.      We have next to no actual data on health outcomes; as our report
states, we're basing our recommendation on the feeling that their model has
a lower burden of proof and a high profile.

*Our top-rated charities.  *
With Stop TB, because of the consistency of its programming and its
auditing, we can see a summary of how things are going in every country, and
a sample of the details that go into this summary data.  With VillageReach,
we are looking at a charity that has had one pilot project we feel is
successful and is looking to scale up the same model to more areas at a pace
we feel is reasonable (it is not looking to drastically expand its funding
or diversifying its activities).  The case for these charities isn't
airtight or close to it, but in both cases I feel I can look across the
complete set of the organization's activities, know what information is
available on the biggest questions, and feel that the organization as a
whole is a good bet.  With the ** charities, there are huge advantages over
"typical" charities and reason to believe that they're having positive
impact in many cases, but the "missing pieces" are qualitatively bigger and
the sense of what you get for a donation to the organization as a whole is
much weaker.

On Mon, Jul 6, 2009 at 6:30 PM, Elie Hassenfeld <ehassenfeld@...>wrote:

>
>
> Thanks for sending these comments. Here are some initial thoughts; we'll
> plan to send more later.
>
> There's a big difference in the confidence I have in our "top-rated" vs
> merely"recommended" charities, so I disagree with some of the conclusions
> you reach below. I wanted to discuss two points in particular:
>
>    1. "Among PIH, Stop TB Partnership, and PSI, I don't have concerns of
>    such large magnitudes.  Among these three, I feel confident that a donation
>    would meet my goal of saving/improving lives in a cost-effective manner."
>    2. "[VillageReach] is a young charity, trying to massively expand in
>    scope, that hasn't proven it can realize a substantial part of the benefit
>    of its activities yet (via the transfer to governments).  That's a really
>    scary combo.  Since there are alternatives that I don't think have risks of
>    these magnitudes, I'd grudgingly put Village Reach in the second tier."
>
> *PIH and PSI vs Stop TB
>
> *I think there are strong reasons to support both PSI and PIH, but at the
> same time, both have significant weaknesses that would lead me to support
> Stop TB before either of them.
>
> *PIH:* PIH has a common-sense model, but I think there's a big risk that
> the effect visible to a donor (a fully-functioning health facility in a
> location where one did not previously exist) is largely the result *of
> shifting resources* *from one location in a country to another.* For
> example, because they don't train doctors, for PIH to staff its Rwandan
> facility with Rwandan doctors, it relies on relocating doctors from one
> location in Rwanda to another. To the extent that PIH is doing that, we
> think there's reason to significantly discount the overall impact their
> programs have.  We discussed this issue in our review at
> http://givewell.net/pih#Possiblenegativeoffsettingimpact
>
> Even though we this issue leads me to significantly discount PIH's *
> apparent* impact, I think they are still have some impact by (a) providing
> trained doctors with the facilities they need to provide top-notch
> healthcare. In addition, in some of their facilities, many of the clinicians
> are developed-world doctors who travel abroad to staff part/much of the
> clinic.
>
> *PSI: *PSI does have a stronger commitment to monitoring and evaluation
> than almost any other charity, but it's the evidence provided in that
> documentation provides a mixed case for the *impact *that PSI's programs
> have. PSI supports bednet provision, and monitors bednet use, but it's
> unclear that PSI's activities have increased the number of people that are
> sleeping under nets. And, bednets are a relatively easy case because
> ITN-distribution and promotion is a program with extremely strong evidence
> behind it (see givewell.net/node/329. Condom promotion and distribution is
> much trickier. If you can get people to consistently use condoms, you'll
> likely reduce HIV/AIDS transmission but there's no "proven" approach for
> accomplishing that (for more see givewell.net/node/375). PSI's approach
> seems as good as any, but their relatively inconsistent monitoring and
> evaluation that relies solely on self-reported accounts of behavior is
> somewhat questionable. It's far less compelling to me than either Stop TB's
> data on completed TB treatments and patient outcomes or VillageReach's data
> on children vaccinated.
>
> I don't think Stop TB has these same kinds of weakness, and therefore,
> among these three, I'd support them.
>
> *VillageReach*
>
> I think the evidence that VillageReach provides for the impact of their
> program is unmatched among any charity I've seen. VillageReach came to Cabo
> Delgado; they reorganized and supplemented the vaccine-delivery system; they
> measured (a) changes in drug availability in clinics and (b) changes in
> children immunized, a life-saving intervention. They compared Cabo Delgado's
> success to that of a nearby district. On all measures, VillageReach's
> programs appears a success. No other charity I've looked at can offer a case
> for impact as compeling as that.
>
> I am not terribly concerned about the question of whether VillageReach can
> successfully pass off its activities to the government, because we evaluated
> them mostly under the assumption that they can't do so *at all*, and even
> with this assumption still consider them to be as proven and cost-effective
> as any of the other charities we've seen (see the VillageReach review for
> details) (see http://givewell.net/node/370#Whatdoyougetforyourdollar for
> details).
>
> I'd guess that the risk of VillageReach scaling up is somewhat low, though
> there is clearly some risk. VillageReach is currently seeking $750,000 for
> 2009 which would lead to (approximately) a scale up of 2-3x the size of
> their current projects. That seems like an appropriate increase given the
> strong success of their current project.
>
> -Elie
>
> On Mon, Jul 6, 2009 at 12:06 AM, jonbehar <jonbehar@...> wrote:
>
>>
>>
>> After reading through your charity reviews, I put together some initial
>> thoughts on your rankings.  Here's how I'd think about ranking the
>> charities, coming from the perspective of a donor:
>>
>> I want my donation to save/improve lives in a cost effective way.  To have
>> confidence that it will, I want to see empirical data and strong intuitive
>> logic to support a given program/charity.  I know there's so much
>> measurement error around any metrics and data we see that that I'm mostly
>> concerned about big downside risk, i.e. that there's a material risk my
>> money is simply wasted; it's just not worth taking that risk when there's
>> other charities available that don't have it.
>>
>> With that in mind, some of the charities have enough of that downside risk
>> to warrant being in the second tier.  It's certainly possible that further
>> research could assuage these concerns.
>>
>> Global Fund:  There are enough red flags that it's tough to get excited
>> about the Global Fund.  There's the lack of clarity about where the marginal
>> dollar goes, the reliance on procedure-less independent auditors, no
>> indication of that ineffective programs (of which there are undoubtedly
>> many) are being shut down, and the significant resources dedicated to
>> relatively cost-ineffective programs (anti retroviral treatment.)
>>
>> GAVI: Without a good explanation for the outside grants and a list of
>> unfunded projects, I see no reason to risk a donation to a charity that
>> might can't provide a good reason why they need the funds.
>>
>> AMF: The lack of utilization data is precisely the type of thing I'm
>> worried about.  Given the other alternatives, I don't see any reason to
>> incur the risk that the nets aren't being used or maintained.  While this
>> concern also applies to PSI, I think PSI has a few advantages: they are
>> collecting data on net usage (though the monitoring leaves much to be
>> desired), they explicitly focus on marketing, and on the margins I think
>> selling rather than freely distributing nets is more likely to lead to
>> sustained usage.
>>
>> Among PIH, Stop TB Partnership, and PSI, I don't have concerns of such
>> large magnitudes.  Among these three, I feel confident that a donation would
>> meet my goal of saving/improving lives in a cost-effective manner.  In
>> thinking through how to choose between them, I'd be weighing off fairly
>> subjective concerns like whether it's better to focus on the most
>> cost-effective interventions (which would favor PSI or Stop TB) or whether
>> the more wide-ranging treatments PIH offers is actually a better way to
>> improve lives.
>>
>> That leaves Village Reach, which I'm kind of torn about.  Their model is
>> simple and logical, and their commitment to monitoring, evaluation, and
>> reflection seems fantastic.  My intuition is also that a small, focused
>> organization like Village Reach is more likely to be able to effectively use
>> the results of monitoring and evaluation to make necessary adjustments than
>> a larger organization would be.  The methodology in their monitoring is
>> about as sound as we're likely to see, and the data looks great.  But… there
>> are really large risks, larger than some of the risks of the charities I see
>> as second tier.  I think you nailed them in your summary- it's a young
>> charity, trying to massively expand in scope, that hasn't proven it can
>> realize a substantial part of the benefit of its activities yet (via the
>> transfer to governments).  That's a really scary combo.  Since there are
>> alternatives that I don't think have risks of these magnitudes, I'd
>> grudgingly put Village Reach in the second tier.  A couple of things might
>> change my mind.  It would be great to get an assessment of how well the
>> pilot program went from a project management perspective (did it stay on
>> budget, were timelines met, etc.).  If it went well, I'd be less concerned
>> about the expansion in scope.  Some recovery in the data from the pilot
>> program would be nice too.
>>
>>
>>
>>
>>
>>
>> --- In givewell@yahoogroups.com, Holden Karnofsky <Holden@...> wrote:
>> >
>> > See http://blog.givewell.net/?p=396
>> >
>>
>
>  
>

I think you've done a good job clarifying the advantages of Stop TB
relative to PIH and PSI.  I'm now in agreement that Stop TB is a
notch above.  However, I still have some questions about Village Reach.



The evidence that Village Reach had a major impact in its pilot program
in Cabo Delgado is pretty compelling.  So, I think the key questions are
how closely the proposed expanded program resembles the pilot and how
likely it is that the expanded program can be executed.  Specific
questions I have along those line are:



·         In the broader program, who will be responsible for
distributing the vaccines to the hospital?  The effectiveness of the
pilot program suffered when field coordinators were no longer
responsible for this delivery (see pages 20-21 of

http://www.villagereach.org/PDF%20Documents/VillageReach%20Evaluation%20\
of%20the%20Project%20to%20Support%20PAV%20-%20Executive%20Summary%20and%\
20Report.pdf
<http://www.villagereach.org/PDF%20Documents/VillageReach%20Evaluation%2\
0of%20the%20Project%20to%20Support%20PAV%20-%20Executive%20Summary%20and\
%20Report.pdf> )

·         What is the extent of Village Reach's scaling up?  Elie
put out an estimate of 2-3x current activities.  How was this arrived
at?  Scanning Village Reach's past expenditures, it looks like they
totaled ~4mm over the life of pilot program.  My understanding is that
the bulk of that was spent on the pilot itself.  Yet the expansion of
the program is expected to cost 3.5mm (including government funds).  It
doesn't make sense to me that the expanded program would cost less
than the pilot (though the pilot ran for longer than the expanded
program is due to run, so the relative annual costs make a little more
sense).  Do you guys have a sense of why the budgeted cost of the
expanded program seems so low relative to the pilot?  Will the expanded
program have the same high degree of monitoring that the pilot had?

·         Was the pilot program roughly on budget?  If not, were
overruns diagnosed such that we can have confidence that the budget for
the broader program is reasonable?

·         How secure is the Mozambique government's commitment to
fund ~1/3 of the cost of the expanded program?





--- In givewell@yahoogroups.com, Holden Karnofsky <Holden@...> wrote:
>
> A couple things to add.
>
> For me, the distinction between *** and ** is a very important one.
Basically,
> I feel that for a *** charity, we have reasonable confidence in the
full set
> of its activities, and feel that the cost-effectiveness estimate
provided is
> a reasonable (if very rough) approximation to its overall impact. By
> contrast, for a ** charity there is a crucial piece of the puzzle
missing –
> missing data on a highly questionable link in the chain, questions
about how
> representative the data we have is, etc. – and while we feel the
charity is
> a much better bet than lower-rated charities, and is likely doing a
> substantial amount of good, we don't have a good sense for how often
its
> activities are going as hoped.
>
> To me, the two charities Jon mentioned as being in his "top tier" are
in the
> ** tier:
>
> *PSI. *PSI stands above non-recommended charities because it is
> systematically asking the questions we feel need to be asked to give
> confidence in their activities, and a significant amount of data
appears to
> be actually collected and available. That said:
>
> 1. We've only seen a sample of their data, and while we don't feel the
> sample was "cherry-picked," we do feel that better-run projects may be
more
> likely to get their data in to the central office. So we're concerned
about
> representativeness.
>
> 2. The data do not point strongly to impact. *Changes* in reported
> behavior do not particularly suggest impact; the high *levels* of
reported
> condom/ITN use, combined with PSI's role as a dominant supplier, make
it
> seem likely that PSI is getting these materials to people who are
using
> them. It may simply be substituting for for-profit suppliers, though a
> limited set of studies (see
> http://givewell.net/node/329#Freenetsvssellingnetsforafee) suggests
that
> subsidized/free distribution has benefits.
>
> *PIH. *Elie has addressed a couple of the concerns with PIH. I would
add
> that
>
> 1. PIH does appear to conduct a variety of programs whose impact would
> be harder to assess than that of direct medical care (microloans,
school
> scholarships, population-based health initiatives). In the Rwanda
budget
> (the only budget we have that can give a sense for the relative size
of
> these programs), these programs appear to consume about 7% of the
funds. We
> aren't sure whether we're under-allocating administrative expenses to
these
> programs, whether they've grown since we last looked, whether they're
larger
> at other locations, etc.
>
> 2. We have next to no actual data on health outcomes; as our report
> states, we're basing our recommendation on the feeling that their
model has
> a lower burden of proof and a high profile.
>
> *Our top-rated charities. *
> With Stop TB, because of the consistency of its programming and its
> auditing, we can see a summary of how things are going in every
country, and
> a sample of the details that go into this summary data. With
VillageReach,
> we are looking at a charity that has had one pilot project we feel is
> successful and is looking to scale up the same model to more areas at
a pace
> we feel is reasonable (it is not looking to drastically expand its
funding
> or diversifying its activities). The case for these charities isn't
> airtight or close to it, but in both cases I feel I can look across
the
> complete set of the organization's activities, know what information
is
> available on the biggest questions, and feel that the organization as
a
> whole is a good bet. With the ** charities, there are huge advantages
over
> "typical" charities and reason to believe that they're having positive
> impact in many cases, but the "missing pieces" are qualitatively
bigger and
> the sense of what you get for a donation to the organization as a
whole is
> much weaker.
>
> On Mon, Jul 6, 2009 at 6:30 PM, Elie Hassenfeld ehassenfeld@...wrote:
>
> >
> >
> > Thanks for sending these comments. Here are some initial thoughts;
we'll
> > plan to send more later.
> >
> > There's a big difference in the confidence I have in our "top-rated"
vs
> > merely"recommended" charities, so I disagree with some of the
conclusions
> > you reach below. I wanted to discuss two points in particular:
> >
> > 1. "Among PIH, Stop TB Partnership, and PSI, I don't have concerns
of
> > such large magnitudes. Among these three, I feel confident that a
donation
> > would meet my goal of saving/improving lives in a cost-effective
manner."
> > 2. "[VillageReach] is a young charity, trying to massively expand in
> > scope, that hasn't proven it can realize a substantial part of the
benefit
> > of its activities yet (via the transfer to governments). That's a
really
> > scary combo. Since there are alternatives that I don't think have
risks of
> > these magnitudes, I'd grudgingly put Village Reach in the second
tier."
> >
> > *PIH and PSI vs Stop TB
> >
> > *I think there are strong reasons to support both PSI and PIH, but
at the
> > same time, both have significant weaknesses that would lead me to
support
> > Stop TB before either of them.
> >
> > *PIH:* PIH has a common-sense model, but I think there's a big risk
that
> > the effect visible to a donor (a fully-functioning health facility
in a
> > location where one did not previously exist) is largely the result
*of
> > shifting resources* *from one location in a country to another.* For
> > example, because they don't train doctors, for PIH to staff its
Rwandan
> > facility with Rwandan doctors, it relies on relocating doctors from
one
> > location in Rwanda to another. To the extent that PIH is doing that,
we
> > think there's reason to significantly discount the overall impact
their
> > programs have. We discussed this issue in our review at
> > http://givewell.net/pih#Possiblenegativeoffsettingimpact
> >
> > Even though we this issue leads me to significantly discount PIH's *
> > apparent* impact, I think they are still have some impact by (a)
providing
> > trained doctors with the facilities they need to provide top-notch
> > healthcare. In addition, in some of their facilities, many of the
clinicians
> > are developed-world doctors who travel abroad to staff part/much of
the
> > clinic.
> >
> > *PSI: *PSI does have a stronger commitment to monitoring and
evaluation
> > than almost any other charity, but it's the evidence provided in
that
> > documentation provides a mixed case for the *impact *that PSI's
programs
> > have. PSI supports bednet provision, and monitors bednet use, but
it's
> > unclear that PSI's activities have increased the number of people
that are
> > sleeping under nets. And, bednets are a relatively easy case because
> > ITN-distribution and promotion is a program with extremely strong
evidence
> > behind it (see givewell.net/node/329. Condom promotion and
distribution is
> > much trickier. If you can get people to consistently use condoms,
you'll
> > likely reduce HIV/AIDS transmission but there's no "proven" approach
for
> > accomplishing that (for more see givewell.net/node/375). PSI's
approach
> > seems as good as any, but their relatively inconsistent monitoring
and
> > evaluation that relies solely on self-reported accounts of behavior
is
> > somewhat questionable. It's far less compelling to me than either
Stop TB's
> > data on completed TB treatments and patient outcomes or
VillageReach's data
> > on children vaccinated.
> >
> > I don't think Stop TB has these same kinds of weakness, and
therefore,
> > among these three, I'd support them.
> >
> > *VillageReach*
> >
> > I think the evidence that VillageReach provides for the impact of
their
> > program is unmatched among any charity I've seen. VillageReach came
to Cabo
> > Delgado; they reorganized and supplemented the vaccine-delivery
system; they
> > measured (a) changes in drug availability in clinics and (b) changes
in
> > children immunized, a life-saving intervention. They compared Cabo
Delgado's
> > success to that of a nearby district. On all measures,
VillageReach's
> > programs appears a success. No other charity I've looked at can
offer a case
> > for impact as compeling as that.
> >
> > I am not terribly concerned about the question of whether
VillageReach can
> > successfully pass off its activities to the government, because we
evaluated
> > them mostly under the assumption that they can't do so *at all*, and
even
> > with this assumption still consider them to be as proven and
cost-effective
> > as any of the other charities we've seen (see the VillageReach
review for
> > details) (see http://givewell.net/node/370#Whatdoyougetforyourdollar
for
> > details).
> >
> > I'd guess that the risk of VillageReach scaling up is somewhat low,
though
> > there is clearly some risk. VillageReach is currently seeking
$750,000 for
> > 2009 which would lead to (approximately) a scale up of 2-3x the size
of
> > their current projects. That seems like an appropriate increase
given the
> > strong success of their current project.
> >
> > -Elie
> >
> > On Mon, Jul 6, 2009 at 12:06 AM, jonbehar jonbehar@... wrote:
> >
> >>
> >>
> >> After reading through your charity reviews, I put together some
initial
> >> thoughts on your rankings. Here's how I'd think about ranking the
> >> charities, coming from the perspective of a donor:
> >>
> >> I want my donation to save/improve lives in a cost effective way.
To have
> >> confidence that it will, I want to see empirical data and strong
intuitive
> >> logic to support a given program/charity. I know there's so much
> >> measurement error around any metrics and data we see that that I'm
mostly
> >> concerned about big downside risk, i.e. that there's a material
risk my
> >> money is simply wasted; it's just not worth taking that risk when
there's
> >> other charities available that don't have it.
> >>
> >> With that in mind, some of the charities have enough of that
downside risk
> >> to warrant being in the second tier. It's certainly possible that
further
> >> research could assuage these concerns.
> >>
> >> Global Fund: There are enough red flags that it's tough to get
excited
> >> about the Global Fund. There's the lack of clarity about where the
marginal
> >> dollar goes, the reliance on procedure-less independent auditors,
no
> >> indication of that ineffective programs (of which there are
undoubtedly
> >> many) are being shut down, and the significant resources dedicated
to
> >> relatively cost-ineffective programs (anti retroviral treatment.)
> >>
> >> GAVI: Without a good explanation for the outside grants and a list
of
> >> unfunded projects, I see no reason to risk a donation to a charity
that
> >> might can't provide a good reason why they need the funds.
> >>
> >> AMF: The lack of utilization data is precisely the type of thing
I'm
> >> worried about. Given the other alternatives, I don't see any reason
to
> >> incur the risk that the nets aren't being used or maintained. While
this
> >> concern also applies to PSI, I think PSI has a few advantages: they
are
> >> collecting data on net usage (though the monitoring leaves much to
be
> >> desired), they explicitly focus on marketing, and on the margins I
think
> >> selling rather than freely distributing nets is more likely to lead
to
> >> sustained usage.
> >>
> >> Among PIH, Stop TB Partnership, and PSI, I don't have concerns of
such
> >> large magnitudes. Among these three, I feel confident that a
donation would
> >> meet my goal of saving/improving lives in a cost-effective manner.
In
> >> thinking through how to choose between them, I'd be weighing off
fairly
> >> subjective concerns like whether it's better to focus on the most
> >> cost-effective interventions (which would favor PSI or Stop TB) or
whether
> >> the more wide-ranging treatments PIH offers is actually a better
way to
> >> improve lives.
> >>
> >> That leaves Village Reach, which I'm kind of torn about. Their
model is
> >> simple and logical, and their commitment to monitoring, evaluation,
and
> >> reflection seems fantastic. My intuition is also that a small,
focused
> >> organization like Village Reach is more likely to be able to
effectively use
> >> the results of monitoring and evaluation to make necessary
adjustments than
> >> a larger organization would be. The methodology in their monitoring
is
> >> about as sound as we're likely to see, and the data looks great.
But… there
> >> are really large risks, larger than some of the risks of the
charities I see
> >> as second tier. I think you nailed them in your summary- it's a
young
> >> charity, trying to massively expand in scope, that hasn't proven it
can
> >> realize a substantial part of the benefit of its activities yet
(via the
> >> transfer to governments). That's a really scary combo. Since there
are
> >> alternatives that I don't think have risks of these magnitudes, I'd
> >> grudgingly put Village Reach in the second tier. A couple of things
might
> >> change my mind. It would be great to get an assessment of how well
the
> >> pilot program went from a project management perspective (did it
stay on
> >> budget, were timelines met, etc.). If it went well, I'd be less
concerned
> >> about the expansion in scope. Some recovery in the data from the
pilot
> >> program would be nice too.
> >>
> >>
> >>
> >>
> >>
> >>
> >> --- In givewell@yahoogroups.com, Holden Karnofsky Holden@ wrote:
> >> >
> >> > See http://blog.givewell.net/?p=396
> >> >
> >>
> >
> >
> >
>

It would be great to have some kind of social-networking system which could
harness the power of reputation.

Chris Blattman is a blogging development professor with oodles of field
experience, so when he suggests a charity   (
http://feedproxy.google.com/~r/chrisblattman/~3/J2hdIcvi_Hc/just-in-case-you-really-want-to-do-some.html)
   I'm putting my hand in my pocket straight away, and I pretty rarely
give to charity. But it would be great to have some kind of interactive
forum in which individuals who might be passing through the area could
upload their own comments; this would increase the accountability of
recommendations - people would be putting that reputation on the line.




2009/5/22 Holden Karnofsky <Holden@...>

>
>
> Thanks for the feedback.
>
> We're leaning toward an incremental approach.  First step would be to
> identify the charities where a visit would be most worth doing.  At that
> point we would invite anyone who cared to to submit photos, video, and
> comments, but we wouldn't sit back and wait for it – we would do visits
> ourselves and hopefully come back with a better sense of what we're looking
> for and a better sense of who else could help add to our knowledge base (and
> how).  From there we might more actively push others (locals?  Students?  Aid
> workers?  Donors?  No need to decide this yet) to participate.  If we were
> getting loads of useful data, we would then think about how to
> process/display it more systematically.
>
> This would all start post 7/1 (except perhaps the "making a list" part), so
> it would be unlikely to become a major info source in the short term.
>
>
> On Thu, May 21, 2009 at 8:44 PM, Phil Steinmeyer <psteinmeyer@...>wrote:
>
>>
>>
>>  In general, I like Holden's idea.
>>
>> Key questions are:
>>
>> 1) To what extent can "did it happen" be readily verified on the ground?
>> i.e. In what percentage of cases would this be important?
>> 2) How difficult are such visits?  Effort, time, cost...  How cooperative
>> would charities be?
>>
>> Finding a local to report, more cheaply, SEEMS (at first blush) like a
>> good idea, but on further reflection, there are some problems:
>>
>> Is our goal to find someone with a donor mindset and experience to report
>> on the charity activities?  If so, the likelihood that we would find many
>> donors with ready access to rural Africa or Asia or Latin America seems low.
>>
>> We might have better luck finding either Western NGO workers or local
>> residents close to the action.
>>
>> But there are problems with each of these.  I suspect that in a given
>> area, the community of Western NGO workers is generally small and somewhat
>> tight, which would interfere with the ability/willingness of an NGO worker
>> to be brutally honest in reporting on the activities of others.
>>
>> As for locals, they would probably be much harder to find.  If charities
>> are working in areas where folks earn a dollar or two a day, how many of
>> those locals are going to have internet access, English fluency, and be
>> reachable by someone in, say, New York?  Even if we did find such folks, I
>> foresee possible conflicts and simply communication gaps in trying to get
>> the kind of information that I think Holden is talking about.
>>
>> That said, finding locals would certainly be nice, but I wouldn't want to
>> see TOO much effort put in that direction if it early efforts didn't seem
>> fruitful.
>>
>> ===
>>
>> A few years ago, when I was first ramping up my charitable giving, I had
>> vague thoughts of traveling to Africa or other poor areas to get a feel for
>> charity activities and the general dynamics of poor nations from the ground
>> level.  I have not followed through with it, and haven't really thought
>> about it recently.  Still, I like the concept, and would at least consider
>> doing something like this personally in the right scenario.
>>
>>
>>
>  
>

I think these are good questions for VillageReach, and more generally, I
think you're right to express skepticism about giving to a charity based on
a single pilot project.

I also concede that we are much lighter on details than we'd like to be
about VillageReach's future projects.  We understand that they will be
taking the same basic approach, and planning the same level of rigorous
monitoring and evaluation, but we are still waiting on details about their
projected expenses as well as about the specifics of the regions they'll be
expanding into.  (Note that we are still communicating with both
VillageReach and Stop TB, asking for more information about a variety of
things.)

Still, I would personally be at least as confident giving to VillageReach as
to StopTB, and Elie feels the same way.  I'll try to explain why:

*VillageReach's small size carries additional benefits as well as additional
risks.  *It's true that with Stop TB, we have much more "sample size" and
the case is stronger that it has a pattern of success, rather than a single
success.  On the other hand, Stop TB is so huge and working in so many
different places that there's a limit to how confident I can be, even with
the exceptional amount of information it provides.  With VillageReach, I
feel that I will be able to track and stay up to date on every project they
carry out (for the foreseeable future), and furthermore that every project
they carry out will have the direct involvement of people who were
instrumental in the first success.

I believe that uncertainties come up both with extrapolating from a single
project to future success, and with extrapolating from audits to a general
picture of the impact of a large bureaucracy.

*The "riskiness" of VillageReach cuts both ways as well.  *VillageReach is
still largely trying to establish an approach to improving health systems;
the approach may turn out not to work in new settings, but if it does work
repeatedly, it may eventually influence the work of other charities and
other governments, attract large amounts of government aid, etc.

Generally, we stay away from "innovative" or "pilot" programs because we
feel that individual donors (including us) are not well positioned to
understand their likelihood of success; but having actual past results from
a past project puts VillageReach in a different category.  I recognize a
substantial risk of failure for VillageReach's expansion, but feel confident
that the risk is worth funding (I am more confident that this is a "good
bet" than I am that PSI or PIH is a "good bet").
*Bottom line:* for any charity, we can make a long list of things that still
might go wrong (and for both of the charities in question, we're still
working on getting more info).  But both VillageReach and Stop TB have
provided fairly compelling, systematic, empirical answers to the biggest
questions that jump to mind about what they're trying to do (and, what
they're trying to do is very cost-effective in changing lives if
successful).  That puts them in the same basic category to me, even though
the specifics of their advantages and disadvantages are very different, and
individual donors will probably differ quite a bit on whether they are more
comfortable with an already large and established charity or with a small
but promising charity.

On Sun, Jul 12, 2009 at 11:00 PM, jonbehar <jonbehar@...> wrote:

>
>
>
>
> I think you've done a good job clarifying the advantages of Stop TB
> relative to PIH and PSI.  I'm now in agreement that Stop TB is a notch
> above.  However, I still have some questions about Village Reach.
>
>
>
> The evidence that Village Reach had a major impact in its pilot program in
> Cabo Delgado is pretty compelling.  So, I think the key questions are how
> closely the proposed expanded program resembles the pilot and how likely it
> is that the expanded program can be executed.  Specific questions I have
> along those line are:
>
>
>
> ·         In the broader program, who will be responsible for distributing
> the vaccines to the hospital?  The effectiveness of the pilot program
> suffered when field coordinators were no longer responsible for this
> delivery (see pages 20-21 of
>
>
> http://www.villagereach.org/PDF%20Documents/VillageReach%20Evaluation%20of%20the%20Project%20to%20Support%20PAV%20-%20Executive%20Summary%20and%20Report.pdf)
>
>
> ·         What is the extent of Village Reach's scaling up?  Elie put out
> an estimate of 2-3x current activities.  How was this arrived at?  Scanning
> Village Reach's past expenditures, it looks like they totaled ~4mm over the
> life of pilot program.  My understanding is that the bulk of that was
> spent on the pilot itself.  Yet the expansion of the program is expected
> to cost 3.5mm (including government funds).  It doesn't make sense to me
> that the expanded program would cost less than the pilot (though the pilot
> ran for longer than the expanded program is due to run, so the relative
> annual costs make a little more sense).  Do you guys have a sense of why
> the budgeted cost of the expanded program seems so low relative to the
> pilot?  Will the expanded program have the same high degree of monitoring
> that the pilot had?
>
> ·         Was the pilot program roughly on budget?  If not, were overruns
> diagnosed such that we can have confidence that the budget for the broader
> program is reasonable?
>
> ·         How secure is the Mozambique government's commitment to fund
> ~1/3 of the cost of the expanded program?
>
>
>
>
> --- In givewell@yahoogroups.com, Holden Karnofsky <Holden@...> wrote:
> >
> > A couple things to add.
> >
> > For me, the distinction between *** and ** is a very important one.
> Basically,
> > I feel that for a *** charity, we have reasonable confidence in the full
> set
> > of its activities, and feel that the cost-effectiveness estimate provided
> is
> > a reasonable (if very rough) approximation to its overall impact. By
> > contrast, for a ** charity there is a crucial piece of the puzzle missing
> –
> > missing data on a highly questionable link in the chain, questions about
> how
> > representative the data we have is, etc. – and while we feel the charity
> is
> > a much better bet than lower-rated charities, and is likely doing a
> > substantial amount of good, we don't have a good sense for how often its
> > activities are going as hoped.
> >
> > To me, the two charities Jon mentioned as being in his "top tier" are in
> the
> > ** tier:
> >
> > *PSI. *PSI stands above non-recommended charities because it is
> > systematically asking the questions we feel need to be asked to give
> > confidence in their activities, and a significant amount of data appears
> to
> > be actually collected and available. That said:
> >
> > 1. We've only seen a sample of their data, and while we don't feel the
> > sample was "cherry-picked," we do feel that better-run projects may be
> more
> > likely to get their data in to the central office. So we're concerned
> about
> > representativeness.
> >
> > 2. The data do not point strongly to impact. *Changes* in reported
> > behavior do not particularly suggest impact; the high *levels* of
> reported
> > condom/ITN use, combined with PSI's role as a dominant supplier, make it
> > seem likely that PSI is getting these materials to people who are using
> > them. It may simply be substituting for for-profit suppliers, though a
> > limited set of studies (see
> > http://givewell.net/node/329#Freenetsvssellingnetsforafee) suggests that
> > subsidized/free distribution has benefits.
> >
> > *PIH. *Elie has addressed a couple of the concerns with PIH. I would add
> > that
> >
> > 1. PIH does appear to conduct a variety of programs whose impact would
> > be harder to assess than that of direct medical care (microloans, school
> > scholarships, population-based health initiatives). In the Rwanda budget
> > (the only budget we have that can give a sense for the relative size of
> > these programs), these programs appear to consume about 7% of the funds.
> We
> > aren't sure whether we're under-allocating administrative expenses to
> these
> > programs, whether they've grown since we last looked, whether they're
> larger
> > at other locations, etc.
> >
> > 2. We have next to no actual data on health outcomes; as our report
> > states, we're basing our recommendation on the feeling that their model
> has
> > a lower burden of proof and a high profile.
> >
> > *Our top-rated charities. *
> > With Stop TB, because of the consistency of its programming and its
> > auditing, we can see a summary of how things are going in every country,
> and
> > a sample of the details that go into this summary data. With
> VillageReach,
> > we are looking at a charity that has had one pilot project we feel is
> > successful and is looking to scale up the same model to more areas at a
> pace
> > we feel is reasonable (it is not looking to drastically expand its
> funding
> > or diversifying its activities). The case for these charities isn't
> > airtight or close to it, but in both cases I feel I can look across the
> > complete set of the organization's activities, know what information is
> > available on the biggest questions, and feel that the organization as a
> > whole is a good bet. With the ** charities, there are huge advantages
> over
> > "typical" charities and reason to believe that they're having positive
> > impact in many cases, but the "missing pieces" are qualitatively bigger
> and
> > the sense of what you get for a donation to the organization as a whole
> is
> > much weaker.
> >
> > On Mon, Jul 6, 2009 at 6:30 PM, Elie Hassenfeld ehassenfeld@...wrote:
> >
> > >
> > >
> > > Thanks for sending these comments. Here are some initial thoughts;
> we'll
> > > plan to send more later.
> > >
> > > There's a big difference in the confidence I have in our "top-rated" vs
> > > merely"recommended" charities, so I disagree with some of the
> conclusions
> > > you reach below. I wanted to discuss two points in particular:
> > >
> > > 1. "Among PIH, Stop TB Partnership, and PSI, I don't have concerns of
> > > such large magnitudes. Among these three, I feel confident that a
> donation
> > > would meet my goal of saving/improving lives in a cost-effective
> manner."
> > > 2. "[VillageReach] is a young charity, trying to massively expand in
>
> > > scope, that hasn't proven it can realize a substantial part of the
> benefit
> > > of its activities yet (via the transfer to governments). That's a
> really
> > > scary combo. Since there are alternatives that I don't think have risks
> of
> > > these magnitudes, I'd grudgingly put Village Reach in the second tier."
> > >
> > > *PIH and PSI vs Stop TB
> > >
> > > *I think there are strong reasons to support both PSI and PIH, but at
> the
> > > same time, both have significant weaknesses that would lead me to
> support
> > > Stop TB before either of them.
> > >
> > > *PIH:* PIH has a common-sense model, but I think there's a big risk
> that
> > > the effect visible to a donor (a fully-functioning health facility in a
> > > location where one did not previously exist) is largely the result *of
> > > shifting resources* *from one location in a country to another.* For
> > > example, because they don't train doctors, for PIH to staff its Rwandan
> > > facility with Rwandan doctors, it relies on relocating doctors from one
> > > location in Rwanda to another. To the extent that PIH is doing that, we
> > > think there's reason to significantly discount the overall impact their
> > > programs have. We discussed this issue in our review at
> > > http://givewell.net/pih#Possiblenegativeoffsettingimpact
> > >
> > > Even though we this issue leads me to significantly discount PIH's *
> > > apparent* impact, I think they are still have some impact by (a)
> providing
> > > trained doctors with the facilities they need to provide top-notch
> > > healthcare. In addition, in some of their facilities, many of the
> clinicians
> > > are developed-world doctors who travel abroad to staff part/much of the
> > > clinic.
> > >
> > > *PSI: *PSI does have a stronger commitment to monitoring and evaluation
> > > than almost any other charity, but it's the evidence provided in that
> > > documentation provides a mixed case for the *impact *that PSI's
> programs
> > > have. PSI supports bednet provision, and monitors bednet use, but it's
> > > unclear that PSI's activities have increased the number of people that
> are
> > > sleeping under nets. And, bednets are a relatively easy case because
> > > ITN-distribution and promotion is a program with extremely strong
> evidence
> > > behind it (see givewell.net/node/329. Condom promotion and
> distribution is
> > > much trickier. If you can get people to consistently use condoms,
> you'll
> > > likely reduce HIV/AIDS transmission but there's no "proven" approach
> for
> > > accomplishing that (for more see givewell.net/node/375). PSI's
> approach
> > > seems as good as any, but their relatively inconsistent monitoring and
> > > evaluation that relies solely on self-reported accounts of behavior is
> > > somewhat questionable. It's far less compelling to me than either Stop
> TB's
> > > data on completed TB treatments and patient outcomes or VillageReach's
> data
> > > on children vaccinated.
> > >
> > > I don't think Stop TB has these same kinds of weakness, and therefore,
> > > among these three, I'd support them.
> > >
> > > *VillageReach*
> > >
> > > I think the evidence that VillageReach provides for the impact of their
> > > program is unmatched among any charity I've seen. VillageReach came to
> Cabo
> > > Delgado; they reorganized and supplemented the vaccine-delivery system;
> they
> > > measured (a) changes in drug availability in clinics and (b) changes in
> > > children immunized, a life-saving intervention. They compared Cabo
> Delgado's
> > > success to that of a nearby district. On all measures, VillageReach's
> > > programs appears a success. No other charity I've looked at can offer a
> case
> > > for impact as compeling as that.
> > >
> > > I am not terribly concerned about the question of whether VillageReach
> can
> > > successfully pass off its activities to the government, because we
> evaluated
> > > them mostly under the assumption that they can't do so *at all*, and
> even
> > > with this assumption still consider them to be as proven and
> cost-effective
> > > as any of the other charities we've seen (see the VillageReach review
> for
> > > details) (see http://givewell.net/node/370#Whatdoyougetforyourdollarfor
> > > details).
> > >
> > > I'd guess that the risk of VillageReach scaling up is somewhat low,
> though
> > > there is clearly some risk. VillageReach is currently seeking $750,000
> for
> > > 2009 which would lead to (approximately) a scale up of 2-3x the size of
> > > their current projects. That seems like an appropriate increase given
> the
> > > strong success of their current project.
> > >
> > > -Elie
> > >
> > > On Mon, Jul 6, 2009 at 12:06 AM, jonbehar jonbehar@... wrote:
> > >
> > >>
> > >>
> > >> After reading through your charity reviews, I put together some
> initial
> > >> thoughts on your rankings. Here's how I'd think about ranking the
> > >> charities, coming from the perspective of a donor:
> > >>
> > >> I want my donation to save/improve lives in a cost effective way. To
> have
> > >> confidence that it will, I want to see empirical data and strong
> intuitive
> > >> logic to support a given program/charity. I know there's so much
> > >> measurement error around any metrics and data we see that that I'm
> mostly
> > >> concerned about big downside risk, i.e. that there's a material risk
> my
> > >> money is simply wasted; it's just not worth taking that risk when
> there's
> > >> other charities available that don't have it.
> > >>
> > >> With that in mind, some of the charities have enough of that downside
> risk
> > >> to warrant being in the second tier. It's certainly possible that
> further
> > >> research could assuage these concerns.
> > >>
> > >> Global Fund: There are enough red flags that it's tough to get excited
> > >> about the Global Fund. There's the lack of clarity about where the
> marginal
> > >> dollar goes, the reliance on procedure-less independent auditors, no
> > >> indication of that ineffective programs (of which there are
> undoubtedly
> > >> many) are being shut down, and the significant resources dedicated to
> > >> relatively cost-ineffective programs (anti retroviral treatment.)
> > >>
> > >> GAVI: Without a good explanation for the outside grants and a list of
> > >> unfunded projects, I see no reason to risk a donation to a charity
> that
> > >> might can't provide a good reason why they need the funds.
> > >>
> > >> AMF: The lack of utilization data is precisely the type of thing I'm
> > >> worried about. Given the other alternatives, I don't see any reason to
> > >> incur the risk that the nets aren't being used or maintained. While
> this
> > >> concern also applies to PSI, I think PSI has a few advantages: they
> are
> > >> collecting data on net usage (though the monitoring leaves much to be
> > >> desired), they explicitly focus on marketing, and on the margins I
> think
> > >> selling rather than freely distributing nets is more likely to lead to
> > >> sustained usage.
> > >>
> > >> Among PIH, Stop TB Partnership, and PSI, I don't have concerns of such
> > >> large magnitudes. Among these three, I feel confident that a donation
> would
> > >> meet my goal of saving/improving lives in a cost-effective manner. In
> > >> thinking through how to choose between them, I'd be weighing off
> fairly
> > >> subjective concerns like whether it's better to focus on the most
> > >> cost-effective interventions (which would favor PSI or Stop TB) or
> whether
> > >> the more wide-ranging treatments PIH offers is actually a better way
> to
> > >> improve lives.
> > >>
> > >> That leaves Village Reach, which I'm kind of torn about. Their model
> is
> > >> simple and logical, and their commitment to monitoring, evaluation,
> and
> > >> reflection seems fantastic. My intuition is also that a small, focused
> > >> organization like Village Reach is more likely to be able to
> effectively use
> > >> the results of monitoring and evaluation to make necessary adjustments
> than
> > >> a larger organization would be. The methodology in their monitoring is
> > >> about as sound as we're likely to see, and the data looks great. But…
> there
> > >> are really large risks, larger than some of the risks of the charities
> I see
> > >> as second tier. I think you nailed them in your summary- it's a young
> > >> charity, trying to massively expand in scope, that hasn't proven it
> can
> > >> realize a substantial part of the benefit of its activities yet (via
> the
> > >> transfer to governments). That's a really scary combo. Since there are
> > >> alternatives that I don't think have risks of these magnitudes, I'd
> > >> grudgingly put Village Reach in the second tier. A couple of things
> might
> > >> change my mind. It would be great to get an assessment of how well the
> > >> pilot program went from a project management perspective (did it stay
> on
> > >> budget, were timelines met, etc.). If it went well, I'd be less
> concerned
> > >> about the expansion in scope. Some recovery in the data from the pilot
> > >> program would be nice too.
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> --- In givewell@yahoogroups.com, Holden Karnofsky Holden@ wrote:
> > >> >
> > >> > See http://blog.givewell.net/?p=396
> > >> >
> > >>
> > >
> > >
> > >
> >
>
>
>  
>

I have mixed feelings about this idea.
On one hand, I definitely want to know which charities the most experienced
people recommend.

On the other hand, I feel it's important that people be clear about *why* they
support some charities as opposed to others.  Otherwise, relying on their
opinions could lead to an unhealthy dynamic in which the amount raised by a
charity has to do with "who they know" rather than with what they're
accomplishing.  No matter how much experience a person has, when I don't see
them explicitly putting forth their case, I don't know how much of the
recommendation has to do with a careful consideration of impact vs. other
factors like family/personal connections.  (I'm also unable to gauge how
intelligently they're considering impact, what information they're factoring
in, what unspoken judgment calls and personal values are factored in and how
they accord with my own, etc.)

In my opinion, Prof. Blattman's recommendation isn't explicitly answering
the questions I feel need to be answered - not explicitly putting forth the
case that I would need to have confidence - and so it largely comes down to
trust.  Many people, including Lee, might trust Prof. Blattman's conclusion,
but I don't know enough about him to do so (the fact that he has experience,
alone, is not sufficient for me).

Bottom line - I think that knowing where the experienced people stand is
very valuable, but that they should be encouraged to be clear about *why* they
make the recommendations they do, and not just about *whom* they're
recommending.


On Wed, Jul 15, 2009 at 8:23 AM, Lee Crawfurd (MoFEP) <
lee.crawfurd@...> wrote:

>
>
> It would be great to have some kind of social-networking system which could
> harness the power of reputation.
>
> Chris Blattman is a blogging development professor with oodles of field
> experience, so when he suggests a charity   (
> http://feedproxy.google.com/~r/chrisblattman/~3/J2hdIcvi_Hc/just-in-case-you-really-want-to-do-some.html)    I'm putting my hand in my pocket straight away, and I pretty rarely
> give to charity. But it would be great to have some kind of interactive
> forum in which individuals who might be passing through the area could
> upload their own comments; this would increase the accountability of
> recommendations - people would be putting that reputation on the line.
>
>
>
>
> 2009/5/22 Holden Karnofsky <Holden@...>
>
>>
>>
>> Thanks for the feedback.
>>
>> We're leaning toward an incremental approach.  First step would be to
>> identify the charities where a visit would be most worth doing.  At that
>> point we would invite anyone who cared to to submit photos, video, and
>> comments, but we wouldn't sit back and wait for it – we would do visits
>> ourselves and hopefully come back with a better sense of what we're looking
>> for and a better sense of who else could help add to our knowledge base (and
>> how).  From there we might more actively push others (locals?  Students?
>> Aid workers?  Donors?  No need to decide this yet) to participate.  If we
>> were getting loads of useful data, we would then think about how to
>> process/display it more systematically.
>>
>> This would all start post 7/1 (except perhaps the "making a list" part),
>> so it would be unlikely to become a major info source in the short term.
>>
>>
>> On Thu, May 21, 2009 at 8:44 PM, Phil Steinmeyer <psteinmeyer@...
>> > wrote:
>>
>>>
>>>
>>>  In general, I like Holden's idea.
>>>
>>> Key questions are:
>>>
>>> 1) To what extent can "did it happen" be readily verified on the ground?
>>> i.e. In what percentage of cases would this be important?
>>> 2) How difficult are such visits?  Effort, time, cost...  How cooperative
>>> would charities be?
>>>
>>> Finding a local to report, more cheaply, SEEMS (at first blush) like a
>>> good idea, but on further reflection, there are some problems:
>>>
>>> Is our goal to find someone with a donor mindset and experience to report
>>> on the charity activities?  If so, the likelihood that we would find many
>>> donors with ready access to rural Africa or Asia or Latin America seems low.
>>>
>>> We might have better luck finding either Western NGO workers or local
>>> residents close to the action.
>>>
>>> But there are problems with each of these.  I suspect that in a given
>>> area, the community of Western NGO workers is generally small and somewhat
>>> tight, which would interfere with the ability/willingness of an NGO worker
>>> to be brutally honest in reporting on the activities of others.
>>>
>>> As for locals, they would probably be much harder to find.  If charities
>>> are working in areas where folks earn a dollar or two a day, how many of
>>> those locals are going to have internet access, English fluency, and be
>>> reachable by someone in, say, New York?  Even if we did find such folks, I
>>> foresee possible conflicts and simply communication gaps in trying to get
>>> the kind of information that I think Holden is talking about.
>>>
>>> That said, finding locals would certainly be nice, but I wouldn't want to
>>> see TOO much effort put in that direction if it early efforts didn't seem
>>> fruitful.
>>>
>>> ===
>>>
>>> A few years ago, when I was first ramping up my charitable giving, I had
>>> vague thoughts of traveling to Africa or other poor areas to get a feel for
>>> charity activities and the general dynamics of poor nations from the ground
>>> level.  I have not followed through with it, and haven't really thought
>>> about it recently.  Still, I like the concept, and would at least consider
>>> doing something like this personally in the right scenario.
>>>
>>>
>>>
>>
>  
>

I am the newest member of the GiveWell team. Since I started at the
beginning of July, I have been working on expanding the background
information that GiveWell offers within its most recent report on
international aid. We always welcome feedback on our work, and I would
especially appreciate it if you would take a few minutes to look at what
I've been working on and send me your thoughts.

*The reports below aren't public yet. If you'd like to read them, email us
at info@... and we'll send you login information.*

Here's a quick summary of what I've found with links to more information*:

Program: Antiretroviral therapy for the prevention of mother-to-child
transmission of HIV*
In Africa, it is estimated that without antiretroviral therapy (ART) 25-35%
of HIV positive mothers pass the virus to their infants. Clinical trails
have shown that a single dose of antiretroviral drugs given to each mother
(during labor) and newborn (immediately after birth) is a cost-effective way
to lower transmission rates.

Unlike ART to treat HIV/AIDS, ART to prevent mother-to-child transmission of
HIV is a much shorter regimen and therefore, significantly more
cost-effective. We would be excited to find a charity primarily implementing
this program but haven't yet found any.*  *

See the full report at http://www.givewell.net/node/452.

*Cause: Reducing maternal mortality in developing countries*
The World Health Organization (WHO) estimates that in 2005 over 500,000
women died from pregnancy- and birth-related causes. We reviewed recent
literature reviews of interventions to reduce maternal mortality including*
*training traditional birth attendants, providing skilled birth attendants,
expanding antenatal care, community mobilibization, and distribution of
clean delivery kits, but we have not identified an intervention whose
effectiveness at reducing maternal mortality is strongly supported by the
available evidence. Success stories rely on broad, systemic improvements to
the provision of health care.

See the full report at http://givewell.net/node/454.*

Overview: Standard of living in the developing world*
Donors often ask us, "I understand that saving a life in Africa is far
cheaper than educating a child in the U.S., but if you save someone's life
in Africa, what type of life does he/she lead? If you save someone from
malaria, does he or she just die from tuberculosis? Does this person have
the opportunity to live a happy life or are they significantly harmed by all
the problems in Africa?" We started to answer some of this with this page on
life expectancies in Sub-Saharan Africa (http://givewell.net/node/98) and
this page goes into the issue further.

It shows that income and self-reported life satisfaction are fairly strongly
correlated, incomes are low and erratic for a large proportion of the
population, and mortality is high under 5 and over 60 but considerably less
so in between.

See the full report at http://givewell.net/node/458.

I look forward to hearing what you think.

Best,
 Natalie Stone
Research Analyst
www.givewell.net

Some of you might have read about this:
  http://www.philanthropyaction.com/articles/interview_ipa_project_director_nathanael_goldberg_talks_about_the_impacts_o


- sam
"Joy comes not to him who seeks it for himself, but to him who seeks
it for other people."

Our summary of the recent rigorous studies: http://blog.givewell.net/?p=408

On Wed, Oct 28, 2009 at 5:17 PM, Wai-Kwong Sam Lee <orionlee@...>wrote:

>
>
> Some of you might have read about this:
>
> http://www.philanthropyaction.com/articles/interview_ipa_project_director_nathanael_goldberg_talks_about_the_impacts_o
>
> - sam
> "Joy comes not to him who seeks it for himself, but to him who seeks
> it for other people."
>  
>

Hello all,

We haven't been using the research email list lately.  We have been blogging
quite frequently.

One of the main things we're working on - and trying to work toward with the
blog - is presenting the basic material of our research report in clearer
and more accessible ways.

*We'd really like your feedback* on the blog, as much as you are able to
give.  We'd like to know what you find interesting, informative and
compelling and what you find irrelevant/boring/simply something that doesn't
motivate you to read.  (And we won't be surprised if a lot of what we write
falls into the latter category - we know it tends to be dense. Please be
honest!)

I've just installed a "rate this post" feature.  The first way you can help
is simply by rating posts (old ones or new ones).  Then, if you have the
time to shoot us an email (info@...) explaining your rating, we'd
appreciate it.

The people on this list are many of the people most interested in our
research, so we really want your thoughts.

Best,
Holden

I love the blog every time I remember to read it, but I rarely remember to do so. And I've pretty much given up on RSS, so that's not a solution for me either.

But this email finally reminded me to take the 30 seconds to subscribe to the blog via email, so that it will now come to me. If anyone else is interested in doing so, just go here (and click the "Get the GiveWell Blog delivered by email" link):

http://feeds.feedburner.com/givewell/rss2/

________________________________________
From: givewell@yahoogroups.com [mailto:givewell@yahoogroups.com] On Behalf Of Holden Karnofsky
Sent: Monday, November 09, 2009 7:54 AM
To: givewell@yahoogroups.com
Subject: [givewell] Email list and blog

  

Hello all,

We haven't been using the research email list lately.  We have been blogging quite frequently.

One of the main things we're working on - and trying to work toward with the blog - is presenting the basic material of our research report in clearer and more accessible ways.

We'd really like your feedback on the blog, as much as you are able to give.  We'd like to know what you find interesting, informative and compelling and what you find irrelevant/boring/simply something that doesn't motivate you to read.  (And we won't be surprised if a lot of what we write falls into the latter category - we know it tends to be dense. Please be honest!)

I've just installed a "rate this post" feature.  The first way you can help is simply by rating posts (old ones or new ones).  Then, if you have the time to shoot us an email (info@...) explaining your rating, we'd appreciate it.

The people on this list are many of the people most interested in our research, so we really want your thoughts.

Best,
Holden

Our primary research focus right now is reviewing organizations to make
$250,000 in grants to economic empowerment organizations in Sub-Saharan
Africa. We're going to make the grant by the end of 2009. Context:
http://blog.givewell.net/?p=401

This email outlines the process we've followed so far and updates where we
currently stand.

As always, we would appreciate any questions or comments on our approach or
preliminary conclusions.

*Finding applicants*
*
*
**We cast the net very wide. We considered any organization that we had
already considered for our "main" international report. We also tried to
identify "major" international aid charities by reviewing Charity
Navigator-listed charities as well as any international development (as
identified by their tax form) charity that had at least $1 million in
previous year's revenues. Of those, we invited any organization that
appeared to potentially work in Sub-Saharan Africa and have an economic
empowerment program. We invited 152 organizations to apply. 44 did.

*Reviewing applicants*
*
*
**Our application is online: http://www.givewell.net/node/456

Ultimately, we looked for applicants that could potentially demonstrate one
of the following (as we stated at
http://www.givewell.net/node/455#Whatarethecriteriabywhichthisgrantwillbeawarded
):

   1. A past impact on client incomes, relative to what likely would have
   happened in the absence of the program -- we looked for a relatively
   rigorous impact study. We received some impact studies but none that meet
   what we consider a reasonable standard.
   2. That they are transferring wealth to people with low standards of
   living -- we looked for a strong demonstration that clients are very poor
   *and *evidence that those poor ultimately receive the intended funds. We
   have some promising leads here.
   3. That they are creating value in low-income areas by starting programs
   that eventually become self-sustaining -- we found little evidence of any
   kind in this category.

*Top Contenders*

These are the main organizations we're focusing on now. There are also a
couple other organizations that we need more information from before we can
really say where they stand.

*Organizations transferring wealth*

Both organizations below focus a significant portion of their activities on
directly transferring cash to the very poor. In both cases, we have some
concerns about whether there is adequate auditing/monitoring in place to
ensure that the intended recipients receive the funds. We also wonder -
depending on the size of the grant and the portion of people, all living in
a small geographic area who receive it - what unintended consequences there
could be on prices or crime if everyone in a village receives a lot of extra
cash all at once.

Finally, one of the other main factors we consider here is what we call the
"cash ratio": the amount a charity needs to spend to transfer cash. Those
rough numbers are listed next to the organization's name:

   - Women for Women International: 15-20%
   - Village Enterprise Fund: 35-40%

*Microfinance institutions (MFIs)*

We've written a lot about microfinance recently. See the page on our website
for a summary:
http://www.givewell.net/international/economic-empowerment/microfinance

Of the ones we invited, the only ones that could provide some information on
(a) dropout rates or (b) standard of living for clients were the large U.S.
networks listed below. However, we've found it near impossible to get a
handle on what value these networks are adding and whether they're actually
increasing access to financial services or just providing some
hard-to-evaluate technical assistance to local microfinance institutions.
(More on this at http://blog.givewell.net/?p=464)

Also, to a large degree,  the US-based headquarters seems somewhat
disconnected from the actual partner microfinance institutions that might
have much of the data we're looking for. This has made followup and
evaluation difficult. (What we're looking for:
http://blog.givewell.net/?p=447)

The contenders:

   - ACCION International
   - Opportunity International
   - Grameen Foundation

*BRAC*

BRAC is in a category all its own. It's one of the most well-respected
international aid charities. Academics speak highly of it and a couple *
other* charities we've reviewed partner with BRAC in implementation.

BRAC is a huge organization that runs all sorts of programs. It's about 35%
microfinance, 10% wealth transfer and 55% other programs.

We like BRAC largely because they really appear to be a "learning
organization." They have a separate site (www.bracresearch.org) devoted to
evaluating their programs and publishing what they learn. From what
we've carefully reviewed, their programs are very promising -- BRAC's wealth
transfer program is the most carefully monitored we've seen.

Unlike the US networks, BRAC starts and owns all the
microfinance institutions in its network.

*Other approaches*

While we've identified some strong contenders, we're not satisfied with our
options, so we continue to try to identify additional organizations that we
might grant. We've used two approaches.

   1. We met with another international funder that shares much of our
   philosophy about giving. They pointed us to some of their grantees who we
   consider promising leads and offered to try to persuade those organizations
   to share their information with us.
   2. Much of the problem we've had with MFIs has been (a) lack of
   understanding about whether donations would be used to increase access to
   financial services or for something else and (b) data on program
   implementation. We've contacted one actual MFI (Small Enterprise Foundation
   in South Africa) and plan to contact more. So far, this appears to be a
   promising path.

Hi,

I don't know them in details, but at high level it sounds promising,
obtaining local buy-in with money in the skin, fee-based access, and
vetted by Ashoka (in their Changemakers competition)
  http://action.globalwaterchallenge.org/page/content/naandi

On their website, they post a paper on (a partial or work-in-progress)
evaluation of their water program:
  http://www.naandi.org/strategy_papers/PDfs/OBApproaches21_IndiaWater.pdf


- sam
"Joy comes not to him who seeks it for himself, but to him who seeks
it for other people."

Hi Sam,

Thanks for sending this. We've added this to our list of charities to
consider and will review it when we have time. (If anyone is particularly
interested in seeing that list, we could share it with you individually, so
let us know.)

In the meantime, I'd be interested if you (or others on this list) have a
sense of how this organization answers the primary questions we lay out for
water charities at
http://www.givewell.net/international/health/water#Whatprogramsshouldyousupport

-Elie

On Wed, Dec 2, 2009 at 11:06 PM, Wai-Kwong Sam Lee <orionlee@...>wrote:

>
>
> Hi,
>
> I don't know them in details, but at high level it sounds promising,
> obtaining local buy-in with money in the skin, fee-based access, and
> vetted by Ashoka (in their Changemakers competition)
> http://action.globalwaterchallenge.org/page/content/naandi
>
> On their website, they post a paper on (a partial or work-in-progress)
> evaluation of their water program:
> http://www.naandi.org/strategy_papers/PDfs/OBApproaches21_IndiaWater.pdf
>
> - sam
> "Joy comes not to him who seeks it for himself, but to him who seeks
> it for other people."
>  
>

Earlier this year a donor gave us $250,000 to give to the best charity (or
split between the best charities) we could find that was working on economic
empowerment in sub-Saharan Africa.  A few of the charities that applied for
the grant were microfinance charities--charities working to increase access
to loans and saving services for the poor (more on microfinance at
http://www.givewell.net/international/economic-empowerment/microfinance).
These microfinance charities were big U.S. organizations who, instead of
directly serving poor people, were often providing services to the
organizations on the ground. We have found it hard to tell what the concrete
impact would be of additional donations to these  "technical assistance"
 organizations in terms of the operations of local banks actually providing
loans. (For more on this see http://blog.givewell.net/?p=464)

We decided to look for microfinance institutions that we could check out and
potentially give to directly. We seek MFIs that don't just have strong
financial performance, but that have evidence relevant to the questions at
http://blog.givewell.net/?p=447, i.e., whether they're improving the lives
of - or at least providing satisfactory service to - the poor.

So far we have contacted 6 microfinance institutions (MFIs). In looking for
MFIs, our most important principles were that the organizations be located
in sub-Saharan Africa and accepting donations. Here's how we found these 6:

   - Mix Market (http://www.mixmarket.org/), an online database of
   microfinance data, honored 5 African microfinance institutions (MFIs) with
   "Social Performance Reporting Awards" for 2009 (full list:
   http://www.themix.org/sites/default/files/2009%20SP%20Reporting%20Awards.xls).
   We are looking for MFIs that (a) are providing high quality financial
   services to the poor and (b) can provide evidence of this such as surveys
   that show that their clients are poor, data on whether clients pay back
   loans and take out new loans, and policies and procedures that protect
   clients against harassment. (For more on our evaluation criteria see
   http://blog.givewell.net/?p=447). We thought a list of organizations that
   report on their own social performance was a good place to start. 5 of the
   MFIs that received this award are in sub-Saharan Africa. 3 of these accepted
   donations; 2 did not. We contacted all of 3 that did: Small Enterprise
   Foundation, ID-Ghana, and Microloan Foundation.
   - We also looked at all of the organizations listed on Mix Market and
   narrowed the list down by which ones were located in sub-Saharan Africa and
   received a 5-diamond rating by Mix Market for transparency of information
   (rating system explained at http://www.mixmarket.org/faq/diamond-rankings).
   In order to limit this list, we contacted the 3 MFIs from this list that
   took in over $200,000 in donations in 2008, and had a Mix Market profile in
   English. The $200,000 cut off was somewhat arbitrary, but was based on the
   idea that since we are potentially giving a large grant, we are looking for
   organizations that are more likely to be able to use a large donation
   productively.
   - We also looked at Kiva's 10 largest partners (listed at
   http://blog.givewell.net/?p=419). 3 are located in sub-Saharan Africa.
   LAPO in Nigeria did not accept donations in 2008 and Sinapi Aba Trust in
   Ghana explicitly said it used donations for programs other than
   microfinance. The third was a bank run by BRAC, an organization which we are
   already considering for the grant.

There are over 1,084 microfinance institutions listed on Mix Market. 333 of
these are in sub-Saharan Africa. We have contacted 6 based on the above
selection process:

   - The Small Enterprise Foundation in South Africa
   - Initiative Development in Ghana
   - Microloan Foundation in Malawi
   - Micro Enterprise Development Network in Uganda
   - PAPME in Benin
   - Kondo Jigima in Mali

The first three have emailed us back and we have spoken to them by phone. We
have not yet heard from the last three, who we have contacted more recently.
If we do not hear from them soon, we may contact additional MFIs. We will
likely contact MFIs that received slightly lower transparency ratings from
Mix Market and/or which received less than $200,000 in donations in 2008.

Best,
Natalie

Hello,

My name is Jonah Sinick and I'm a new member of the GiveWell Mailing List.
I'm presently a math graduate student at University of Illinois at Urbana
Champaign. I had been vaguely aware of GiveWell since 2007 but was reminded
of GiveWell by a friend in September 2009 and decided to donate based on
GiveWell's recommendations as well as make contact with Holden and Elie.
Since then I've been following more closely. I would like to do something
with my life that has a positive effect on society (construed in a broad
sense). It will become evident in this message that I have yet to develop a
clear sense of which causes and charities I find the most compelling, but
this past year I directed 95% of my charitable contributions to
VillageReach.

To move onto the main subject of this message:

I periodically hear claims of the type "The rate at which we're using
Earth's resources is such that if it keeps up, we will have no resources
left in 50 years." One such example is this UK Guardian article
http://www.guardian.co.uk/uk/2002/jul/07/research.waste . My reaction to
such claims has generally been to forget about them soon after hearing them
because (a) they may be exaggerated or taken out of context for
sensationalism and (b) I've found it unclear what there is that I can do
personally to make a difference if such claims are in fact true. I suspect
that many people have similar reactions.

Just two weeks ago I came across another such claim, this time from the
legendary mathematician named Mikhail Gromov on page 30 of the linked pdf
http://www.ems-ph.org/journals/newsletter/pdf/2009-09-73.pdf. The relevant
excerpt is

---------------------

* If you try to look into the future, 50 or 100 years from now...
*
50 and 100 is very different. We know more or less about the next 50 years.
We shall continue in the way we go. But 50 years from now, the Earth will
run out of the basic resources and we cannot predict what will happen after
that. We will run out of water, air, soil, rare metals, not to mention oil.
Everything will essentially come to an end within 50 years. What will happen
after that? I am scared. It may be okay if we find solutions but if we don't
then everything may come to an end very quickly!

Mathematics may help to solve the problem but if we are not successful,
there will not be any mathematics left, I am afraid!
*
Are you pessimistic?*

I don't know. It depends on what we do. if we continue to move blindly into
the future, there will be a disaster within 100 years and it will start to
be very critical in 50 years already. Well, 50 is just an estimate. It may
be 40 or it may be 70 but the problem will definitely come. If we are ready
for the problems and manage to solve them, it will be fantastic. I think
there is potential to solve them but this potential should be used and this
potential is education. It will not be solved by God. People must have ideas
and they must prepare now. In two generations people must be educated.
Teachers must be educated now, and then the teachers will educate a new
generation. Then there will be sufficiently many people to face the
difficulties. I am sure this will give a result. If not, it will be a
disaster. It is an exponential process. If we run along an exponential
process, it will explode. That is a very simple computation. For example,
there will be no soil. Soil is being exhausted everywhere in the world. It
is not being said often enough. Not to mention water. It is not an
insurmountable problem but requires solutions on a scale we have never faced
before, both socially and intellectually.

---------------------

Gromov's suggestion is that there is an impending crisis situation which we
will be able to work through if and only if we are prepared for it.

To the extent that Gromov is right, I find sustainability a much more
compelling cause than international health care. If we totally run out of
resources in 50 years then we'll be forced to revert to a preagricultural
state in which we will have no health care, no means of communication or
travel over long distances, no clean water, no heating, no electricity, etc.
Not only will quality of life in the developed world plunge (perhaps
permanently), but there will no longer be any hope for chance in the
developing world.

So the question for me is the extent to which Gromov is right. I have no
background in environmental science and so do not have the skills to make an
independent judgment of the severity of the environmental situation, nor the
timescale on which resource depletion will occur, nor how plausible it is
that we can do something about the situation even if we try. I have tried
looking around online for information and have been frustrated by the fact
that many of the claims that people make seem to be in conflict, and I can't
tell which sources are reliable.

Note that there are many environmental issues (global warming, acid rain,
increased presence of toxins in the environment, and depletion of soil,
water, fish, rare metals, coal, and oil to name a few) - part of what I'm
wondering is whether there's some consensus among the knowledgeable about
which of these are of greatest concern and why.

The question that I would really like an answer to is how much I should be
focused on the environment and why. But I don't expect that any of you have
immediate answers to this one. So I'll end this message with three more
specific questions:

(1) Does anyone know of good websites or books for learning about what is
known about various environmental problems and what ideas there are for how
we might cope? I'm aware that there are many websites and books that address
such things, but what I'm looking for are sources that are reliable,
analytical, and big picture in bent.

(2) Does anyone have an understanding of whether or not saving the lives of
people in the developing world causes indirect environmental damage? What I
have in mind in writing this is that on the face of it, I would guess that
saving lives in the developing world would increase the rate at which the
planet's natural resources are being depleted. At the same time I'm aware
that this is a complicated issue - that improving health in the developing
world may actually reduce population growth, that maybe people in the
developing world are using only a negligible fraction of natural resources
anyway, etc.

(3) Is GiveWell considering systematically researching charities that are
working toward halting environmental problems? Presumably it would be harder
for such charities to demonstrate effectiveness than it is for charities
that work to improve health in the developing world, but (depending on the
severity of environmental concerns!) the cause may be sufficiently important
to warrant investigation even so.

Jonah

Some thoughts:

1) I agree with Gromov in that predicting, roughly, the state of the world in 50 years is far easier than predicting it in 100 years.

2) I disagree with the notion (I'm not sure if Gromov specifically said this or if it's your inference or maybe even my bad read of your message) that it is highly likely that there will come a point in time roughly 50 years from now where we will simultaneously run out of most major resources and experience rapid societal collapse.  That's not to say that we won't experience resource shortages in the future or that such shortages won't have serious negative impacts.

3) More broadly, there are a variety of really pessimistic scenarios you can come up with for the world over the next century or so - resource depletion, environmental havoc, nuclear or biological warfare, and so on.  Because uncertainty rises as we move our prediction horizon further out, there is more room for extremely negative (as well as extremely positive) predictions.  

It's hard to strike a balance between being concerned about the very long term and yet still being grounded enough to not be flailing about randomly in the present based on very uncertain predictions of the long term future.

4) I would suggest that you do some reading on economics.  At least part of the issues of your immediate concern (resource usage and depletion) revolve around economic issues.

5) I would also suggest that in this, as in similar issues, you read a variety of opinions on the subject, ranging from alarmist to the opposite side ("all is well") along with more middle-of-the road positions.  It may be, after reading these things, that you remain very concerned, but at least you will be familiar with the arguments and counter-arguments, as expressed by their respective adherents (instead of straw-man type arguments put in the mouths of side A by those who actually support side B).

6) To the best of my knowledge, GiveWell has not been focused on environmental issues to date.  There has been some discussion of looking at these issues, but it is problematic, because the issues don't (IMO) readily lend themselves to the kind of analysis that GiveWell has done so far.  

(Disclosure - I'm not a GiveWell employee, but have been an advisor/mentor to them for a while.)

My name is Uri Weg and I used to do some work for Give Well.

My background is in Chemical Engineering, with a focus on Environmental Technologies. While at school, I had the privilege of taking a class on Atmospheric Chemistry at the NASA Goddard Institute. We studied the chemical reactions that contribute to climate modeling.

Climate modeling is incredibly complex and predicting anything is really tough. One variable can change by a seeming inconsequential degree, but due to the complexity of the system, it can change outcomes drastically. What is clear is that comparing while comparing historical ppm (part per million) of CO2 in the atmosphere, things are at a new peak. The CO2 level data is incredibly accurate due to ice cores. Generally speaking, temperature data is too limited a data set to be conclusive. 

In terms of a doomsday scenario, it is entirely possible we are already there. It is also possible that it is a 100+ years away. Simply too hard to say with certainty.

My personal point of view is that environmental challenges will be conquered by business. Once we internalize the externality of carbon (hopefully by setting a global price), the market will solve the issue. Water will also become more market-ized.

At least that's my wish.

If you have any more questions on environmental issues, please free to contact me. 

Uri
uri.weg@...




--- In givewell@yahoogroups.com, "psteinx" <psteinmeyer@...> wrote:
>
> (Disclosure - I'm not a GiveWell employee, but have been an advisor/mentor to them for a while.)
>

Phil - More than anything else, I'm bemoaning the difficulty that I've had
finding (*) an unbiased and systematic summary and analysis of what is known
about various environmental problems and the relative strengths and
weaknesses of potential solutions(*).

Your suggestion of reading a variety of opinions on the subject is fine in
the abstract, but there's a serious issue of the opinions that I've
encountered having little contact with one another. It seems like to figure
out what's actually going on, one has to spend a lot of time digging into
referenced sources, possibly making personal contact with the authors of
articles, etc. This is time that most people don't have. I'm reminded of a
quote from Barack Obama's "The Audacity of Hope":

"A typical story might begin: “The White House today reported that despite
the latest round of tax cuts, the deficit is projected to be cut in half by
the year 2010.” This lead will then be followed by a quote from a liberal
analyst attacking the White House numbers and a conservative analyst
defending the White House numbers. Is one analyst more credible than the
other? Is there an independent analyst somewhere who might walk us through
the numbers? Who knows?"

In my preceding message I had asked whether GiveWell is considering
investigating environmental charities. This question is perhaps premature.
For the moment, what I would really like to see is an organization
comparable to GiveWell in integrity and analytical power devoted to
producing (*) above. Perhaps such an organization already exists - please
let me know if you have recommendations. I presume that it would be to much
to hope for GiveWell itself to tackle this project.

Uri - Thanks for your post. I will be in touch.

Jonah, I understand and sympathize with your concern.  On complex issues (including future environmental scenarios and, I would imagine, resource-scarcity), it is difficult for an interested but not deeply involved reader to form a reliable opinion.  So much of what you read has already been filtered by the author.  i.e. An author might quote 3 experts all in agreement on an issue, but in fact, those 3 experts may be outliers and consensus expert opinion may lie in a different direction.  

When a topic is politically charged, as many environmental issues are, things become trickier.

The problem is, even if there is say, 1 really good and neutral website out there amidst, say, 49 other websites that are slanted or otherwise not very trustworthy, it is not necessarily easy to find that 1 website or determine that it is trustworthy.

Finding a good information source amidst many noisy and low quality sources may be helpful, but it is not an easy task in and of itself.

Sorry I can't be more specifically helpful.  I guess I'm just sort of warning you that your task may be more difficult than you think.

Hello all,

As noted, we haven't yet done any work in this area.  Here are some
preliminary thoughts:

1.      We do hope to research these causes.  The approach we take will have
to be different in many ways from the kind of work we've done so far, but we
think we can add a lot of value by (a) getting a basic picture of the range
of scholarly opinion and the major points of consensus and disagreement; (b)
examining charities' activities in light of this picture.

2.      I don't have much to say at this point about how promising
environmental causes are.  My gut instinct, considering everything I've
heard and seen, is that international aid is a more promising area for an
individual donor (which is different from saying that it's a more important
area).  But I expect to learn a lot and possibly change my mind as we look
into the issues more.

3.      I think it's important not to put too much trust in any single
person's view based simply on credentials.  That includes both Mikhail
Gromov and Uri, among others.

4.      I agree with Jonah and Phil re: what kind of resource would be
helpful (a systematic summary and analysis of what is known and what the
range of opinion), but I don't as of now have such a resource that I have
investigated enough to really stand behind.  The resource that I most
commonly see pointed to as a large-scale attempt to summarize the state of
knowledge is http://www.ipcc.ch/ .

5.      My impression (though we have yet to vet the research itself) is
that there is a fairly strong consensus in the development economics
community that reducing infant mortality can be expected to slow, not
accelerate, population growth.  More broadly, in my limited experience with
the arguments on environmental issues, I don't recall anyone bringing up the
idea of deliberately keeping the developing world sick/poor as a
high-priority way to avert environmental disaster.

On Tue, Dec 8, 2009 at 2:59 PM, Phil Steinmeyer <psteinmeyer@...>wrote:

>
>
> Jonah, I understand and sympathize with your concern.  On complex issues
> (including future environmental scenarios and, I would
> imagine, resource-scarcity), it is difficult for an interested but not
> deeply involved reader to form a reliable opinion.  So much of what you read
> has already been filtered by the author.  i.e. An author might quote 3
> experts all in agreement on an issue, but in fact, those 3 experts may be
> outliers and consensus expert opinion may lie in a different direction.
>
> When a topic is politically charged, as many environmental issues are,
> things become trickier.
>
> The problem is, even if there is say, 1 really good and neutral website out
> there amidst, say, 49 other websites that are slanted or otherwise not very
> trustworthy, it is not necessarily easy to find that 1 website or determine
> that it is trustworthy.
>
> Finding a good information source amidst many noisy and low quality sources
> may be helpful, but it is not an easy task in and of itself.
>
> Sorry I can't be more specifically helpful.  I guess I'm just sort of
> warning you that your task may be more difficult than you think.
>
>  
>