Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Learning from our mistakes: how HLI plans to improve, published by PeterBrietbart on September 1, 2023 on The Effective Altruism Forum.Hi folks, in this post we'd like to describe our views as the Chair (Peter) and Director (Michael) of HLI in light of the recent conversations around HLI's work. The purpose of this post is to reflect on HLI's work and its role within the EA community in response to community member feedback, highlight what we're doing about it, and engage in further constructive dialogue on how HLI can improve moving forward.HLI hasn't always got things right. Indeed, we think there have been some noteworthy errors (quick note: our goal here isn't to delve into details but to highlight broad lessons learnt, so this isn't an exhaustive list):Most importantly, we were overconfident and defensive in communication, particularly around our 2022 giving season post.We described our recommendation for StrongMinds using language that was too strong: "We're now in a position to confidently recommend StrongMinds as the most effective way we know of to help other people with your money". We agree with feedback that this level of confidence was and is not commensurate with the strength of the evidence and the depth of our analysis.The post's original title was "Don't give well, give WELLBYs". Though this was intended in a playful manner, it was tone-deaf, and we apologise.We made mistakes in our analysis.We made a data entry error. In our meta-analysis, we recorded that Kemp et al. (2009) found a positive effect, but in fact it was a negative effect. This correction reduced our estimated 'spillover effect' for psychotherapy (the effect that someone receiving an intervention had on other people) from 53% to 38% and therefore reduced the total cost-effectiveness estimate from 9.5x cash transfers to 7.5x.We did not include standard diagnostic tests of publication bias. If we had done this, we would have decreased our confidence in the quality of the literature on psychotherapy that we were using.After receiving feedback about necessary corrections to our cost-effectiveness estimates for psychotherapy and StrongMinds, we failed to update our materials on our website in a timely manner.As a community, EA prides itself on its commitment to epistemic rigour, and we're both grateful and glad that folks will speak up to maintain high standards. We have heard these constructive critiques, and we are making changes in response.We'd like to give a short outline of what HLI is doing next and has done in order to improve its epistemic health and comms processes.We've added an "Our Blunders" page on the HLI website, which lists the errors and missteps we mentioned above. The goal of this page is to be transparent about our mistakes, and to keep us accountable to making improvements.We've added the following text to the places in our website where we discuss StrongMinds:"Our current estimation for StrongMinds is that a donation of $1,000 produces 62 WELLBYs (or 7.5 times GiveDirectly cash transfers). See our changelog. However, we have been working on an update to our analysis since July 2023 and expect to be ready by the end of 2023. This will include using new data and improving our methods. We expect our cost-effectiveness estimate will decrease by about 25% or more - although this is a prediction we are very uncertain about as the analysis is yet to be done. While we expect the cost-effectiveness of StrongMinds will decrease, we think it is unlikely that the cost-effectiveness will be lower than GiveDirectly. Donors may want to wait to make funding decisions until the updated report is finished."We have added more/higher quality controls to our work:Since the initial StrongMinds report, we've added Samuel Dupret (researcher) and Dr Ryan Dwyer (senior research...