Large datasets make spurious correlations almost certain — how do we stay honest?

There's a statistical guarantee that should make every data scientist uncomfortable: if you test enough hypotheses on a dataset, you will find statistically significant correlations that are entirely spurious. With a p-value threshold of 0.05 and 100 independent tests, you expect 5 false positives just from chance. Most large analyses run far more than 100 tests, often implicitly through feature engineering and model selection decisions that aren't formally counted as tests.

Tyler Vigen's Spurious Correlations is a useful provocation: per-capita cheese consumption correlates with deaths by bedsheet tangling at r = 0.947. These are obviously ridiculous. But the same statistical process that produces these also produces subtler false positives that look exactly like real findings and survive peer review.

The replication crisis in psychology and medicine is partly a story about this. Underpowered studies, flexible analysis pipelines, and publication bias conspire to fill the literature with findings that don't hold up. Pre-registration and registered reports are institutional responses, but adoption is uneven.

What I find useful: distinguishing exploration from confirmation before you start, being explicit about how many things you tested, and treating any single analysis as hypothesis-generating rather than hypothesis-confirming.

What tools or practices have you found genuinely useful for staying honest when working with large datasets?