Why are there so many medical studies using sloppy research methods, and how big a problem do you think this is?
I noticed this when trying to figure out how common Long Covid is—most of the studies being reported in the media, at least early on, did not have a control group. On the basis of these studies, the media was saying that Long Covid affects 30, 50, or even 60% of people who get Covid.
Many of the studies also use methods which suffer from responder bias like surveying online support groups. Studies which track cohorts over time and have a good control group find more modest figures like 10-15% of patients experiencing greater than expected symptoms at 3 months. However nearly all of these are retrospective studies which as I understand it are not as good as prospective studies. More recently a study came out which does what should be done all along—it compares outcomes of Covid patients with patients who got symptomatic non-covid upper respiratory infection. They found more symptoms in the control group than the Covid group at 3 months. This calls into question whether Long Covid is actually an actual phenomena in its own right or just another iteration of post-viral illness / post-viral chronic fatigue syndrome (see Vinay Prasad’s video).
I wonder, if low quality studies can be so misleading, is it worth doing them at all? It seems to me we should be pooling resources to do more high quality studies rather than many low quality ones.
That makes sense, thank you.
“studies suggest that the true rate of fraud among published studies lies somewhere between 0.01% and 0.4%”. Even 0.4% seems drastically too low—perhaps 10 times too low. I’d be curious to see the source for this claim. An analysis by Elizabeth Bik and others found problematic image duplication in 3.8% of studies. Some of that may have been accidental, but I suspect most were intentional fraud. If ~3.8% percent of papers have this one specific type of fraud, that suggests an even larger percentage contain fraud in general. It’s extremely hard to know, though. I doubt it’s over 10% but I could easily see it being 5%, which is obviously still a massive problem.