iiinova — where your digital self keeps evolving

000

We only offer CPU compute. Should be sufficient for most of social science but we are working on allowing GPU compute within reason. If you have multi-day model fits, it will take that long. We may kill some processes if it takes more than a day, though.

100

Expectation management. This is a beta. Things may not work. The beta is free. We run on limited compute capacity and tokens available for this beta. You may need to be patient with the reproduction. We won’t keep your uploads for longer than the beta. We don’t train anything.

You also get a free round of AI feedback. Since we know how the sausage is made, we think our AI feedback can offer different insights compared to previous models of AI “review”. It's also tuned to be very critical. Example of my (outdated) preprint: rigor.me/examples/hct...

100

We have done this already for the big sets of papers in the Nature reproducibility papers (Miske et al. & Brodeur et al.). We evaluate more than 160 papers and more than 50,000 claims. You can find the results here: rigor.me/examples (also, paper soon)

Third, this can be helpful for meta science: Automating reproducibility reports can help us make more informed decisions about which findings to put trust in. Again: not replacing human judgement but supporting it. If you want to use rigor.me, text me.

100

Second, peer review is tough and time consuming. While some may look at the code to make sure it runs and produces values reported in papers is a big task. If we can make such reports available, this could speed things up. This is not to replace review but to support human it.

200

Why do this? Three reasons: First, we think this can be helpful for scientists that want to make sure their stuff reproduces cleanly before submitting to a journal/conference. If it does not, we will tell you why.

100

How does it work? You provide a paper, data and code and we execute everything (but see expectation management in last tweet in thread). In the end, we compare our independent execution to what is stated in the paper and decide whether there is a match.

After having processed almost 800 papers, we now launch an open beta. Now you can test how reproducible your papers are. Its simple. You provide us with a text (published or preprint), data and code and we check what we can validate. Ping me if you want to work with us.