hubby is studying for a uscg captain's license. one of the exam sections requires recognizing what you see at night, based on boat lights. what kind of vessel it is, what condition it is in, and what side of the vehicle you're looking at based on red, white and green dots in the dark. total madness!
![Me: It is also generally very strange to me that you feel the need to preface it with "Not if the goal were to produce a significant result and report it as a confirmatory finding." What does it matter what the goal is if you were always going to do this to some end? You have no control over how people use what you give them. So this moral directive seems misplaced unless you follow through.
Claude: [Weighing whether the refusal argument actually holds up.]
You're right that intent was the wrong axis. I can't verify anyone's goal and can't govern what happens downstream, so a refusal conditioned on purpose is mostly a gesture. Worse, the transparent demonstration I offered would surface the same "winning" specification, which anyone could lift out and report on its own.
The only line I can actually hold is on the artifact I produce. A multiverse analysis or a simulation of error-rate inflation hands back the whole search, with the selected result embedded in the distribution it came from. What I wouldn't produce is the selected result stripped of that context and written up as if it were the planned test, because that object is misleading on its face regardless of who uses it or why. Past that point you're correct that I have no leverage, any more than a methods textbook explaining optional stopping does.
So the honest version of my answer would have been shorter. I'll run analyses that keep the search visible, I won't write up a cherry-picked result as confirmatory, and I don't pretend that second commitment prevents misuse of the first.](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:yha64k6gbnmndrc2u6bxpmzo/bafkreicyjreqx4obsf5iqltsrx7rrf2safdyl7ztaxhnmxbhigssv2qf7i)
![Me: Can you define p-hacking?
Claude: [Defining p-hacking in precise, concise terms.]
P-hacking is the practice of making data collection or analysis decisions contingent on the observed results, so that a reported p-value falls below a significance threshold (typically .05). Common forms include optional stopping, selectively dropping outliers or conditions, trying multiple covariate sets or outcome measures, and reporting only the specifications that reach significance.
The statistical problem is that a p-value's error-rate interpretation assumes a procedure fixed in advance. When choices depend on the data, the actual sampling distribution of the test statistic differs from the nominal one, so the reported p-value overstates the evidence and the realized Type I error rate can far exceed α. Simmons, Nelson, and Simonsohn (2011) showed this under the label "researcher degrees of freedom," and the term "p-hacking" itself was popularized by Simonsohn, Nelson, and Simmons (2014) in their p-curve work.
Gelman and Loken's "garden of forking paths" extends the point, since inflation can occur even without deliberate search. A researcher who runs only one analysis, but would have chosen a different one had the data looked different, faces the same problem of an undefined reference distribution.](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:yha64k6gbnmndrc2u6bxpmzo/bafkreiatjw7mkqwcwhnl4zm7kkzgzyildx3z4vs46ufstepe4qdkvkn3dm)
