All of delton137's Comments + Replies

Responses to Toby Ord's 'critique' of progress studies?

Answer by delton137Apr 13, 20252

I think Leopold Aschenbrenner's argument here is interesting to consider: https://worksinprogress.co/issue/securing-posterity/?ref=forourposterity.com

(full paper https://globalprioritiesinstitute.org/wp-content/uploads/Philip-Trammell-and-Leopold-Aschenbrenner-Existential-Risk-and-Growth.pdf)

Regarding discount rate, 0% discount is pretty common in EA circles, I think, although I think many recognize it should be at least a bit above 0% to account for epistemic uncertainty about how long humans will continue to exist.

AMA: Mark Khurana, author of The Trajectory of Discovery, creator of Untold Health

delton1372y1

That makes sense, thank you.

"studies suggest that the true rate of fraud among published studies lies somewhere between 0.01% and 0.4%". Even 0.4% seems drastically too low - perhaps 10 times too low. I'd be curious to see the source for this claim. An analysis by Elizabeth Bik and others found problematic image duplication in 3.8% of studies. Some of that may have been accidental, but I suspect most were intentional fraud. If ~3.8% percent of papers have this one specific type of fraud, that suggests an even larger percentage contain fraud in genera... (read more)

AMA: Mark Khurana, author of The Trajectory of Discovery, creator of Untold Health

delton1372y2

Why are there so many medical studies using sloppy research methods, and how big a problem do you think this is?

I noticed this when trying to figure out how common Long Covid is - most of the studies being reported in the media, at least early on, did not have a control group. On the basis of these studies, the media was saying that Long Covid affects 30, 50, or even 60% of people who get Covid.

Many of the studies also use methods which suffer from responder bias like surveying online support groups. Studies which track cohorts over time and h... (read more)

1M Khurana2yDefinitely an enormous issue! I’m not as familiar with the Long-COVID data, but the issue applies to a lot of other fields/areas. I argue in the book that one of the most prevalent issues is linked to how researchers are incentivized to publish quickly and often. This means that studies will often tend to be under-powered (i.e., too few participants to show the ‘right’ level of statistical certainty) because it’s time-consuming to include more patients and because it’s often more expensive. The result is, as you point out, that we’re inundated with studies that don’t show much of an effect size, or at least not enough to conclude anything meaningful. Ultimately, this wastes research resources on a systems level, because one big study would have sufficed for fewer resources overall. But because it’s a publishing game, researchers aren’t incentivized to collaborate as much as we’d like from a progress perspective. In the book, I call this ‘artificial progress’, where we think we’ve learnt something new about the world (through the publishing of these studies), but ultimately we’re just misleading ourselves and need to use even more resources to clarify studies that should have been clear from the outset. One could argue that it should ‘cost’ more for authors to submit under-powered studies to journals, since journals often accept their research despite the methodological flaws, and therefore authors aren’t penalized for this type of behavior. The journals might also prioritize interesting results over the study size being adequate – meaning that too many of these articles get published. Authors and journals ultimately both ‘win’ from this behavior. I think this issue of sloppy research methods is probably MUCH more prevalent than we think, but I haven’t been able to find reliable sources. In the book I talk about research misconduct and fraud, where some “studies suggest that the true rate of fraud among published studies lies somewhere between 0.01% and 0.4%.” I’d