A bit of background for the debate. There's been a lot of conversation in Oxford about cause selection and potential flow-on effects of our actions, and GiveWell has also written various blog posts in [?] history about those issues. Paul Christiano, who is also on this call, wrote a long essay about flow-on effects and cause prioritization suggesting that it had the potential to be a fruitful cause.

So yeah: we'll just go around and introduce everyone.

Another distinction which I think maybe just has to do with how I read Paul's essay or maybe I just didn't read it right, but I think it's worthwhile ... Paul's essay, a lot of it, kind of reads to me like what we call an argument for strategic cause selection, which is this idea that you should be looking at lots of different very broad areas which a philanthropist might go into, and try to decide, in some broad sense, which one has the most value. That's something I don't see a lot of groups doing, it's something that we're trying to do, it's something that you could argue, that I would argue, is valuable, I think Paul argues is valuable. Then there's this specific aspect of that which is how do you evaluate the causes? There's lots of different axes and lots of different criteria by which you could evaluate a cause, and so the places where I think we would be more likely to have a disagreement would be how much of that analysis should be focused on questions like what is the long run impact of economic growth versus the kind of questions we're tackling in our shallow overviews.

So those are distinctions I would want to make at the outset. If you'd like I can kind of talk a little bit about the philosophy of what we're doing now in Labs and what questions we think come first and which would come second, in case that clarifies anything? Or maybe we could start, alternatively, with you guys telling us what you wish you saw more of from us.

In terms of what we're actually doing, process wise, the shallow investigations are what they sound like. They're very quick investigations of a cause, they're usually just a couple phone calls. What we're trying to figure out is according to just the conventional wisdom of the experts in the field, what is the problem, how tractable is it, what are some things that we could do, and then who else works on the problem.

Then we also have, and this is all kind of a spectrum, I mean none of these terms are really rigid, but it's helpful to think about medium depth investigations. These are asking basically the same questions as a shallow depth investigation, but instead of talking to like two or three people and reading two or three papers we're talking to like 30 people and taking months on it. The goal there is to get a kind of representative view of what people think and not be as much at risk of only talking to a couple people. However, there are other things we're doing that are kind of cross cutting and are kind of different from those. So one thing we're doing is, I am trying to understand just the basics of philanthropic involvement in fundamental scientific research and in political advocacy. Those are two, you might call them super causes. They're big worlds that we know very little about, and we want to know more about how philanthropy should attack those areas. And so it's like to look at any cause within basic science, to look at aging research, AI research, biomedical research, we'll want to feel more literate, in general, about the world of philanthropy's involvement in scientific research than we are currently.

Similarly with politics. Whether we want to do factory farming or labor mobility or whatever, just have a sense of the history of philanthropy's involvement in politics and how it has gone well and how it hasn't, what works and what doesn't, and what the basic tools are and what the basic techniques are that have been used in the past successes I think it really important. That's a lot of what I've been focusing on.

Finally, there's a couple other cross cutting initiatives that are really targeted at being good philanthropists on the whole. There's the history of philanthropy project which is trying to find everything we can about what philanthropy's done well and poorly in the past so we can think intelligently about what we're likely to be able to do well and what we're not. Then there's this more present oriented investigation which is getting to know the major foundations which exist today, understanding what they fund, what they work on, how they think. We've been doing cofunding, and by the way Good Ventures is very involved in all this work that I'm describing on labs and has particularly been involved in getting to know other foundations and in doing cofunding projects with them, which is basically one tool of learning about that and how they work.

So that is our process at this moment. It's a very high level description and it will be off in many of the details, but that's the basic framework we're operating in. I'm going to pause there, and maybe let's start of with you guys asking any questions you might have about that and get to whether there are things in that process that we ought to trade for other things.

It's a bigger time investment in a broader set of things and so the percentage of what I'm doing that I understand is probably smaller but the time I'm investing is larger but that's because I'm trying to understand something bigger. That's on the political advocacy front.

On the scientific research front I've had probably five conversations so far with major funders trying to understand how they see the scientific research world but I've also done things like, Dario Amodei is like the informal scientific advisor to us, I've just spent a lot of time with him, just trying to absorb his kind of intuitive knowledge of how the scientific research field works and on his advice I read half of a biology textbook, I went to a meeting on cancer research, and I've just generally been trying to absorb and immerse a lot of the language and a lot of the approaches of philanthropy's involvement in scientific research. The next thing we're likely to do on that front is really do a push to find better scientific advisors. One thing we've picked up is that in this case the scientific advisors are much more central to the process of investigating anything in scientific research than they are in other causes. Obviously you always want to work with experts but I think it works differently.

In both cases it's a bigger effort, it's more reading, it's more talking, more effort to find people who are broad which can be hard compared to people who know one issue, and there's a kind of undirected immersion component of it, which is that we're trying to figure out what the basic approaches are and what the basic tools are, and it's not always trying to answer the same three questions. Hopefully that kind of clarifies it?

The big issue for me, and the reason I would say we're doing all those things I just described instead of researching questions like "what is the impact of infant mortality on outcomes several decades down the line" has to do with the return on investment of investigative hours. This is a concept we've covered in one or two of our recent blog posts where I kind of feel like if you're going to spend an hour investigating a question, the importance of the question should be a major factor in which question you choose but also the tractability of the question to further investigation, and that's where I'm just kind of not sold on these questions that I agree are very important. Let me just lay out how I feel about what would happen if we went two different paths.

On the path we're on I think we're learning at a pretty good clip. The politics and science stuff is the hardest to really have a progress meter for but it's not going to be long now before we have a pretty good framework for thinking about those things and can break down both politics and science into smaller causes and can take the shallow plus medium approach to them. And then within the shallow and medium approach I think the learning is really good. We're getting a good sense of which things are crowded and which aren't, which I think in some ways is my favorite question to ask right now because although it's far from the only important question it's a very answerable question. It clearly matters a great deal which areas are kind of crowded with other philanthropists and which are kind of opportunities waiting to be picked, so I think that's very valuable.

I also think as we're doing this we're also gaining a more nuanced understanding of just about any cause we investigate. When we do a shallow we kind of go from knowing how much you would know after reading a New Yorker article on the cause which is just an incredibly superficial, to at least having talked to a couple experts and having a little bit better more realistic picture of what people are doing, what the debates are, and what there's still room for a new philanthropist to do. And then when we do a medium I think we get a fairly sophisticated view, and the result of that is that if we were later to start looking at long term effects of this cause or that cause or that cause we have a better sense of what it is we're trying to analyze.

I guess I draw an analogy a little bit to the charity cost effectiveness where the cost analysis is just much easier to do once you know you're analyzing AMF. You know what countries they work in, you know how much they're paying for nets, you know how they're distributing the nets, you know what the concerns are, what they're not. To do cost effectiveness estimates for bednets or deworming in the abstract vs to do them for a particular charity, they are different endeavors and the more of the subtleties you understand I think the better job you'll do, and I think a similar analogy holds for these kind of causes. We're gaining a fuller picture of what they are, and it's much easier to analyze how something affects the far future or how cost effective it is when you have a pretty developed picture of what it is and when you're not going on a naive or cartoonish "I heard about this once" description of what something is, which is what I feel like we'd be doing without these shallow and medium depth investigations.

So that's the path we're on. In the future I could definitely see us, if we figure out a good way, I could see us putting a lot of effort into these questions. But to think about the long term impact of scientific research it seems pretty important for me to know what kind of scientific research we're talking about and where in the scientific research we're more likely to get involved.

Then if we go the other path, the thing I'm afraid of is that we'll take on these very important questions but we won't really get anywhere. We'll kind of take these back-of-the-envelope calculations, they'll say something but we won't really trust them, and so I won't really feel comfortable ... I think it's very unlikely that I would feel comfortable deprioritizing a cause that's intuitively promising and not very crowded because a back-of-the-envelope calc about what's going to happen thousands of years from now told me that another cause dominates it. I just have trouble believing that I would trust the back-of-the-envelope calc that much to make me discard the idea of investigating the cause and wouldn't believe that investigating the causes further would change my picture of the back-of-the-envelope. That's based on my past experience with cast-effectiveness estimates. I have kind of a sense for how much those can change when you get into the details, they can change a great deal, and so I like to get those details and know what I'm comparing before we do it.

Those are the two paths I'm looking at. One of them I feel like we're going to pick up real information, we're going to make real progress, and it will affect in a positive way the way we do these long term analyses and cost effectiveness calcs later, if and when we do them. Whereas, if we go the other direction, I'm just afraid we'll spin our wheels a lot, come up with stuff that's basically guesswork, and then I won't want to use it to prioritize cause, and so it won't actually change the direction of the other stuff at all. The classic thing here is, we're interested on all fronts. I'm interested in factory farming, I'm interested in scientific research, and I'm interested in bed nets, and to compare those three things I'm having trouble imagining that there's a comparison I can do that will convince me that one of them wins with how little I know about all three of them right now.

The part I don't agree with is the conclusion that we can kind of close the book based on that analysis. Part of my thinking here is that when you look at, and this is a lot of the area where I end up disagreeing with a lot of the way that other people who are concerned about the far future frame and analyze things, is that when I look at ... Let's look at "astronomical waste" for a second, the idea that we have a potentially unbelievably bright future in the future of humanity and can colonize other planets and things like this. When I think about how we got here, it just seems like we got here by doing a lot of things that weren't really explicitly aimed at getting here, that no one could have predicted would get us here. There was a lot of stuff people did that was just kind of like "I made this thing that can make us more productive," and that's good; it's good to be more productive, and it helps people. The sum of a ton of those things put us in a position where now it doesn't look crazy that we'll someday colonize the stars, and without that kind of thought I'm not sure we ever would have gotten there. So when I look at what I believe are the most impactful people and the most impactful actions in human history, even if you want to restrict it to things that are impactful by improving the prospects for the far future a thousand years out, a lot of them just look like people who tried to be helpful or even just tried to help themselves. It was more because they did something incredibly well that they got some unexpected result. They did something really novel, really creative, really well and they got some totally unexpected result, and that is what has accounted for a lot of the progress.

So when I think about that stuff a lot of what I keep coming back to on this factory farming stuff is the possibility of unknown unknown flow-through effects. There's all the things I can tell you about how helping people with bednets might speed the pace of development, which might improve economic productivity, and all that stuff. But there's also just this big block of things where if you just help people and you do it really well, or you just do things that are heuristically good and you do them really well, things that we can't think of yet will happen that will be good. I think there's been real good created by people in entertainment and people analyzing sports and just all kinds of stuff. It doesn't mean that I think that's the most good and it doesn't mean ... we're not having people do that, but if you're looking at two causes and one of them is really dominant in terms of how big an opportunity there is, in terms of what you'd actually be able to do in terms of how big a difference you'd be able to make, I think then there arises a strong possibility that your flow-through effects are big in a way you just can't see. So with factory farming I think there is some case for flow-through effects but I think most of the case is just stuff I haven't thought of yet. I know that may sound like a somewhat crazy view, but that is definitely my view at this time. I'm just not, especially with how little I know about the whole factory farming landscape, just not ready to say that I can conclusively determine that there's basically no flow-through effects here and so I'm not going to look into it further.

But we might instead say that looking at individual people's lives happiness does not go linearly with dollars of economic output. Maybe your impact on science and technology by way of funding research into pharmaceuticals but your happiness grows more like the logarithm of your income. One measure would be Quality Adjusted Life Years, another we might use would be log(income) or an approximation of log(income), we could look at total income, we could look at population, we could look at per-capita income, or a more complicated [?] of the distribution. A lot of the improvements we've seen in the world over time have been correlated with ... if you look between countries, say at the frequency at which wars happen, or the quality of political institutions, these seem naively to be very good things, have some nice flow-through effects, and they tend to go more with the per-capita income of the country or average(log(income)). You might even look at features like, what fraction of the worlds economic output, total capacity, is distributed and how unequally it is distributed. If you look at, say, India and Pakistan, which are very large countries which have lots of economic capability because of their size, they both have nuclear weapons, they're able to afford nuclear weapons because of their size, but there's a relatively high amount of instability and poverty there. You might think that one of the factors affecting the global stability of the world is do you have a large amount of military potential relative to the amount of people, so we should really focus on improving the quality of institutions in countries that are large and relatively influential. This would be a case not to go for world GDP but to bring up per-capita income in poor countries. There's a sort of trade off in that you want countries to become more stable, more peaceful, generally better places to live. We think that we tend to improve the situation for the whole, and then there's a bit of a tradeoff in that capabilities for destruction go up at the same time. And that's something you might look at if you're comparing say migration and [?]. So it seems like there's fairly a large variety, at least a dozen, of plausible kind of metrics that you could use. Ordinary goodness, things that look like they have a lot of flow through effects. A lot of interventions that seem to be very promising, that seem to be wow this is kicking one particular metric off the scale, often the way it works is that it's simultaneously shifting other metrics in the other direction and that's part of why the intervention hasn't been widely adopted and endorsed.

When I think right now about what our priorities should be, there's this information that's having actionable effects on what we're exploring further and what we're not, and When I think right now about what our priorities should be, there's this information that's having actionable effects on what we're exploring further and what we're not, and then there's that investigation that I think would serve a valuable clarificatory role but wouldn't be as likely to affect our process or our priorities. That's why in terms of just pure efficiency and speed it seems better to go in the other order and to save that analysis for the stage where we're actually recommending things and we're going around talking about them and arguing about them, that's when we should start filling in that picture. Or maybe prior to actually making the recommendations, but maybe as a last step before doing so.

I do think you're absolutely right that if the thing GiveWell were maximizing was the quality of the analysis on our top three charities and how to choose between them I think you're absolutely right that we should be doing that sort of thing, because you're right, cash, deworming, and malaria control are all very different things, and some of the differences have to do with mortality vs earnings, but we have made an organizational decision that that's not our primary goal at this time. If we tried to always do everything we could possibly do to make those kind of traditional recommendations as good as they could possibly we could spend another 20 years working full time on that and pretty likely not even change the order of our recommendations because that's kind of been the pattern. The outside view so far is that we did a ton of work last year and it didn't shift us anywhere. Meanwhile, we wouldn't be exploring these other causes. We wouldn't be making progress on understanding the options of scientific research, and politics, and global catastrophic risk prevention and mitigation. When we think about our priorities, I would say that we're trying to uphold about the level of quality we have right now on our traditional product, but it's not a priority for us to substantially improve that quality. We're taking the resources that could go to to improve that quality and those are really going to GiveWell Labs.

I think our analysis on our top three charities is very good, I think it's very helpful, I think it's better than anything else out these, but it's not close to as good as it theoretically could be. Rather than invest in making it that good we're choosing to take those resources and invest them in GiveWell Labs instead. And on the GiveWell Labs front that's where I feel that these questions ... and I do, I think I've made pretty clear, but I think there's a reason that we've avoided these questions, and we have avoided them, because to me they come last because they're not going to affect your workflow, they're not going to change your priorities. They may clarify your conclusions, they may affect your ultimate giving decisions, but they're the kind of thing that for any given process I think you want do to that analysis last. Again, if the priority were making GiveWell 1 as good it possibly could be perhaps that's what we'd be working on now, but since the priority is GiveWell labs they come later in the process.

But again, like I said, that's why I don't ... there was a lot to that analysis beyond back of the envelope calculations that really convinced me. That's why I think when we start talking about what has more effect on the human race 1000 years from now, a small amount of marginal counterfactual progress in scientific research or a large amount of counterfactual progress in reducing factory farming, I mean I just don't believe I'm going to arrive at a firm view on that. I certainly have an intuition now, and I certainly expect to revise that intuition in light of new information, but to get to the level of robustness we need in order to say this thing is off the agenda I think is really unlikely.

And just one other thing I wanted to comment on. I think you can distinguish between doing this type of analysis is enough to robustly make me think that cause A is better than cause B, and saying that doing this kind of long term consideration analysis is enough to have one consideration that strongly speaks in favor of cause A over cause B. I think it might be valuable because of the second thing and not because of the first thing.

Another thing that's worth acknowledging here is that I've thought about all these things and I've talked about them a lot with the people that care the most about them, and I've looked into them a little bit and I've kind of seen what the arguments out there look like, so that is a lot different than when I used to be into US direct aid, which was when I worked at a hedge fund, I didn't really know anything. So given the level of knowledge that I have, not only of what the facts are but of what facts are available and what analysis can be done. It's worth keeping in mind that that's non-trivial. I think there's a really strong possibility, I think it's more likely than not, that my views right now on things like science vs factory farming are just not going to shift significantly. I hope that's wrong, I hope we get really good information to help me think about stuff, I think there will be things that change our mind a little bit, but it's very possible that my views are not going to change very much from here. Unless new information comes out, like new things happen in the world, but further analysis of the facts that already exist I don't know. I think there will be subtle changes, but I think it's quite likely that there's as much more movement happening ... and other people on staff disagree with me about that, and it's certainly not a commitment, I'm certainly hoping to learn a lot more and to gain the kind of information that could make me more confident in things and could help me shift my views.

Given that, I think a very likely ... you ask when in the process we think is the right time. I think a likely sequence of events is that we do this process I've described and then we arrive at lists of priority causes. And then at that point we're kind of recommending to major philanthropists or to ourselves via whatever pool of money we can raise from individuals that we should be hiring experts to really get deep into these causes, and then at that point would probably be the earliest point at which we would start doing this other analysis. It's like the recommendations are already out there and we largely feel like we've covered the landscape we can cover in terms of shallow investigations and drawn what we can from it out of what causes look most promising, and then we've got other people really investigating the causes in depth. It think at that point at the earliest I think you'd want to start saying how can we refine our views of what causes we think are most important based on considerations around flow through effects, long term effects, etc.

And then there's much more vague observations, which is just looking at all the different dimensions on which life has improved since the industrial revolution and just hand-waving and saying, well clearly scientific innovation has been a major driver and a part of them. I mean I buy into that, and I've read a fair amount on that topic, but it feels like I'm reaching the point of where ... it will take a lot of work to learn a little more about that in terms of moving priors.

And I don't mean to say that any of these questions are just worthless to work on. It think for a lot of people and for me at one point learning about the history of how much and in how many dimensions human lives have improved over time is very important and really changes the way you think about different causes relative to each other, but it just happens that you reach the point where it would take a lot of work to learn a little more, and I feel like we're at that point. And that point can come fairly early for these really big broad questions.