Some people seem to think that our procedure for approving grants is roughly “YOLO #sendit.”
Nick Beckstead, “Some clarifications on the Future Fund’s approach to grantmaking“
1. Recap
This is Part 4 in my series on epistemics: practices that shape knowledge, belief and opinion within a community. In this series, I focus on areas where community epistemics could be productively improved.
Part 1 introduced the series and briefly discussed the role of funding, publication practices, expertise and deference within the effective altruist ecosystem.
Part 2 discussed the role of examples within discourse by effective altruists, focusing on the cases of Aum Shinrikyo and the Biological Weapons Convention.
Part 3 looked at the role of peer review within the effective altruism movement.
The next two posts in this series will discuss the role of cost-effectiveness analysis in guiding decisionmaking. Today’s post (Part 4) chronicles the declining importance of cost-effectiveness analysis in many corners of the effective altruism movement. The next post in this series (Part 5) will discuss the importance of cost-effectiveness analysis.
2. Early emphasis on cost-effectiveness analysis
Early effective altruists stressed the importance of conducting rigorous cost-effectiveness analyses to guide giving. Because charities differ dramatically in their cost-effectiveness, and because cost-effectiveness is not always apparent from a surface reading, it is important to use detailed cost-effectiveness analyses to determine which charities are in fact most cost-effective.
Early effective altruists did not merely support the use of cost-effectiveness analysis. They were instrumental in the popularization of rigorous cost-effectiveness analysis, and published some of the best available cost-effectiveness analyses. In The life you can save (second edition), Peter Singer writes:
One of the biggest changes in philanthropy and the international development community that has taken place since I first wrote this book is the increased focus of independent organizations on measuring the impact of particular interventions to help people in extreme poverty, and in assessing the effectiveness of the organizations providing the most successful interventions. GiveWell … was the pioneer here, setting new standards for rigorous evaluation of the work of charities … GiveWell’s strict standards mean that if you go to their website and select one of their top-ranked charities, you can be confident that people in extreme poverty will benefit from your donation, and benefit in a manner that is highly cost-effective.
Effective altruists were so wedded to rigorous cost-effectiveness analysis that many early criticisms of effective altruism suggested the group had, if anything, gone too far in its singleminded pursuit of rigorous cost-effectiveness analysis. Here is Brian Berkey, summarizing one theme from these early critiques:
It is sometimes suggested that effective altruists’ emphasis on relying on evidence about effectiveness and probability of success leads to a bias in favour of efforts the effects of which are easily measurable and quantifiable. To some extent, effective altruists invite this charge, since a significant proportion of their public outreach efforts highlight, often exclusively, their support for organizations whose programmes have been tested for effectiveness in randomized control trials (RCTs), and performed well. … Effective altruists’ efforts here have, however, contributed to the impression that they believe that nearly all of our efforts to, for example, combat global poverty should run through organizations whose programmes have been shown by RCTs to produce significant results.
While effective altruists were not, perhaps, quite so fanatical in their adherence to cost-effectiveness analysis as critics suggested, they did retain substantial allegiance to the practice.
Many short-termist organizations continue to practice rigorous cost-effectiveness analysis, viewing it as an essential guide to effective giving. For example, GiveWell’s website states directly:
Our cost-effectiveness analyses are the single most important input into our charity recommendations. We view cost-effectiveness analyses as valuable for helping us identify large differences in the cost-effectiveness of grants we’re considering for funding and to encourage staff to think through relevant issues related to charities’ work.
This is admirable. However, the growth of the effective altruism movement and the spread of longtermism have led, in some circles, to a dramatic about-face in the emphasis on cost-effectiveness analysis. Let’s look at how cost-effectiveness analysis has fared in other corners of the effective altruism movement.
3. Longtermism and cost-effectiveness analysis
In longtermist circles, cost-effectiveness analysis quickly lost its vaunted place. One reason for this is that conducting reliable cost-effectiveness analyses for longtermist projects is much harder than assessing the cost-effectiveness of short-termist projects.
Another reason for this shift is that many longtermists simply thought there was little need for cost-effectiveness analysis. Because they held that anything at all which could be done to improve the long-term future was, in expectation, orders of magnitude better than competing projects, they thought that almost any reasonable longtermist expenditure was worth funding, if only funds were available. With available funding in the tens of billions of dollars, the funds were plentiful and they were spent extravagantly. As one EA Forum reader commented last year:
I have noticed people (especially Bay Area longtermists) acting like almost anything that saves time or is at all connected to longtermism is a good use of money. As a result, money gets wasted because cheaper ways of creating the same impact are missed. For example, one time an EA offered to pay $140 of EA money (I think) for me for two long Uber rides so that we could meet up, since there wasn’t a fast public transport link. The conversation turned out to be a 30-minute data-gathering task with set questions that worked fine when we did it on Zoom instead.
Another user said much the same, again placing the blame for wasteful spending on a reduced tendency to emphasize cost-effectiveness within the movement.
I was at an EA party this year where there was definitely an overspend of hundreds of pounds of EA money on food which was mostly wasted. As someone who was there, at the time, this was very clearly avoidable. … I think this happened because the flow of money into EA has made the obligations to optimise cost-efficiency and to think counterfactually seem a lot weaker to many EAs.
Part 6 of my series on billionaire philanthropy spoke at length about some other ways in which easy money drove wasteful spending. But I would like to now emphasize something I forgot to emphasize before: the message that cost-effectiveness had assumed a dramatically reduced importance was communicated even by leading effective altruists. For example, here is what Will MacAskill had to say about cost-effectiveness last year:
Frugality is now comparatively less valuable, and saving time and boosting productivity in order to make more progress on the most pressing problems is comparatively more valuable. Creating projects that are maximally cost-effective is now comparatively less valuable; creating projects that are highly scalable with respect to funding, and can thereby create greater total impact even at lower cost-effectiveness, is comparatively more valuable.
Certainly, not everyone expressed this view. One reply to MacAskill has it that:
I think this framing is wrong, or at best unhelpful because we shouldn’t avoid prioritizing cost-effectiveness. When you stop prioritizing cost-effectiveness, it stops being effective altruism. Resources are still finite. The effectiveness of solutions to dire problems still differs dramatically. And we have only scratched the surface of understanding which solutions are gold and which are duds. I think it’s cost-effectiveness all the way down.
But this was by no means a universal view, or perhaps even the majority view.
In any case, we should not rest content with anecdotes. Let’s look at a concrete place where there is good evidence for a diminished role of cost-effectiveness analysis within effective altruism, and where longtermists look to be among the chief culprits.
4. Case study: the FTX Future Fund
The FTX Future Fund, led by leading effective altruists such as Nick Beckstead and Will MacAskill, pledged over a hundred million dollars in grants, almost exclusively to longtermist causes.
Ordinarily, an organization handing out that kind of money proceeds carefully, selecting the most deserving projects on the basis of rigorous vetting, designed to ensure that only the best projects are funded. The process at the FTX Future Fund appears to have been rather less rigorous than this.
(Some readers may complain that the FTX Future Fund is a special case, quite unrepresentative of the movement at large. While I agree that there are some ways in which the Future Fund was unusual, leadership and a substantial share of decisionmaking power within the Future Fund rested with leading effective altruists who exerted, and continue to exert sizable influence over the direction of funding within the effective altruism movement. The fraud perpetrated by FTX was, I hope, exceptional, but that doesn’t get the Future Fund off the hook for explaining its decisionmaking, or those involved off the hook for explaining their actions there.).
A number of commentators have suggested that the fast-paced spending at the Future Fund fell rather short of careful vetting. One asked how the fund could possibly make well-informed decisions with a small staff, promising a review turnaround time of just two weeks, substantially shorter than that at almost all foundations offering grants of this scale.
My two cents about why people may be concerned about the decision-making process without having concrete details: For instance, the initially advertised decision timeline of 2 weeks. While I appreciate the fast pace and the benefits that come with it, a complex system of review and decision-making is almost impossible to achieve at that timeline, especially given the interest in the program.
Another asked what kind of calculations were being made to evaluate the cost-effectiveness of grants. Surely nothing approaching a GiveWell-style cost-effectiveness analysis was being done, but was any form of modeling being done? At the minimum, perhaps grants were evaluated through a quick back-of-the-envelope calculation (BOTEC)? This commentator wrote:
Do you (FTX grantmakers) do or reference a BOTEC for each grant? Would you publish BOTECs you make or comments on the BOTECs you reference? Without this, it seems like EAs would often need to guess and reconstruct your reasoning or make their own models in order to critique a grant, which is much more costly for individuals to do, is much less likely to happen at all, and risks strawmanning or low quality critiques. I think this also gets at the heart of two concerns with “free-spending” EA: we don’t know what impact the EA community is buying with some of our spending, and we don’t have clear arguments to point to to justify particular possibly suspicious expenses to others.
Nick Beckstead clarified that only “a minority of decisions” were made even on the basis of a back-of-the-envelope calculation.
What, precisely, was the decisionmaking process at the Future Fund? To a large extent, we may never know. But here’s what Nick Beckstead said about the process (keep in mind that Beckstead was writing here in public defense of the rigor of the process, so this is if anything a charitable description):
Some people seem to think that our procedure for approving grants is roughly “YOLO #sendit.” This impression isn’t accurate. In reality, before a typical grant goes out it is:
- Recommended by a staff member or regrantor,
- Screened for red flags by a staff member, and then when needed reviewed (in specialized Slack channels created for this purpose) to address legal risks, public communications risks, interference with the work of other EA grantmakers, community health risks, and other potential harms, (we usually find a way to make good grants, but this process often improves them and reduces their risks)
- (If relevant) Reviewed by technical expert(s),
- Endorsed by another staff member, and
- Independently reviewed for final signoff
On this basis, it is hard to be confident in the rigor of the process. The first, second and fifth bullet points describe basic preliminary and final steps that are used by nearly any reputable granting organization at any level. The third and fourth bullet points describe the substance of the grant review process.
In these bullet points, we are told that the grant is (sometimes!) “reviewed by technical expert(s)” and then “endorsed by another staff member”. That is certainly good to hear. Sizable grants made by large grantmakers should be, and almost always are, reviewed by experts (Step 3). And those reviews are then always used by grant administrators to make a funding recommendation (Step 4). What we would like to hear is this: exactly what kind of review (Step 3) and subsequent decisionmaking on the basis of reviews was being done?
Most granting organizations publish extensive details about their review and evaluation processes. These include the composition or selection of review committees (yes, committees); the criteria used to score applications; and the process used to make decisions based on scores. Each process is typically extensive, conducted by expert evaluators trained in the specific organization’s grantmaking process.
To my knowledge, nothing like this has ever been published by the Future Fund. I am somewhat pessimistic that the process, if publicly described, would approach minimum international standards. The process would be highly unlikely to exceed international standards, contrasting sharply with effective altruists’ early emphasis on methodological rigor.
Note also that all of the above applies to grants submitted directly to the Future Fund. In many cases, grants were made through the FTX Regrantor Program, which gave selected individuals six-to-seven-figure pots of money to regrant with comparatively less supervision and procedural oversight. This was not a small program. Beckstead wrote in June 2022 that at least a hundred million dollars had been set aside for the regranting program, and thirty-one million dollars allocated by June 2022.
Here is Beckstead’s description of the structure of the regranting program:
The basic structure is that the regrantors have been given discretionary budgets ranging from $100k to a few million dollars. (A larger number towards the lower end, a smaller number towards the higher end—there is a wide variation in budget sizes.) Regrantors submit grant recommendations to the Future Fund, which we screen primarily for downsides, conflicts of interests, and community effects. We typically review and approve regranting submissions within 72 hours. Grant recommenders have access to a streamlined grant recommendation form from us where we give them some deference, but they don’t have a discretionary budget. (We wanted to try out multiple versions, and in part randomized participation in the different programs.) We compensate regrantors for the volume and quality of their grantmaking, including an element based on whether we fund the projects they seeded ourselves in the future. We also unlock additional discretionary funding when we’re ready to see more of what they’ve been doing.
What, precisely, have we been told about the process? Precious little. If this description is accurate, authority to grant over a hundred million dollars was delegated primarily to regrantors, whose recommendations would be screened within three days, checking “primarily for downsides, conflicts of interest, and community effects.” We are told that some form of retroactive evaluation would be performed, with regrantors rewarded for the quality and quantity of their grants. What kind of evaluation was made? Again, we are not told.
It is almost impossible to emphasize enough how strongly such a process differs from the slow, rigorous, data-driven, quantitative, model-driven and often RCT-backed cost-effectiveness analysis championed by early effective altruists. The apple has fallen very far from the tree.
De-emphasizing rigorous cost-effectiveness analysis has benefits, such as speed and flexibility of grantmaking. But it also has costs. I’ll discuss those costs in the next post in this serious. (My apologies. I had initially intended to discuss those costs today, but I think I’ve gone on for too long already).
5. Conclusion
Today’s post discussed the declining role of cost-effectiveness analysis within the effective altruism movement. We saw that early effective altruists were so strongly committed to rigorous cost-effectiveness analysis that they not only insisted on using cost-effectiveness analyses to guide their own decisionmaking, but also pioneered the spread of cost-effectiveness analysis throughout many areas of philanthropic giving.
However, we also saw that the passage of time and the rise of longtermism brought a declining role for cost-effectiveness analysis within the effective altruism movement. We saw in general terms how the movement became, in the words of its own adherents, less concerned with cost-effectiveness and more free-spending. And we saw how the declining emphasis on cost-effectieness analysis was echoed even by leaders of the movement in their public statements.
We also looked at a concrete instance, the FTX Future Fund, in which the declining role of cost-effectiveness analysis can be clearly seen. We saw that the Future Fund donated, by many accounts, over a hundred million dollars on the basis of vetting procedures that not merely failed to be field-leading, but quite plausibly fell beneath minimal international standards. We also saw that the Future Fund set aside at least a hundred million dollars for its regranting program, which incorporated even less systematic vetting than the standard granting calls at the Future Fund did.
The decline of cost-effectiveness analysis is a notable change within the effective altruism movement, and it is worth asking whether this change was for good or for ill. The next post in this series will highlight some costs associated with the declining role of cost-effectiveness analysis within the effective altruism movement.
Leave a Reply