iiinova — where your digital self keeps evolving

And since we all prefer to read free and openly on the internet - here's a preprint: osf.io/preprints/ps... 5/5

osf.io

OSF

010

If the variance in true scores (latent factor scores) varies across comparisons, even if you are measuring equally precise, the reliability coefficients will differ anyways. If you're not just interested in this relative measurement precision, then Boot-Err might be the better tool for you. 4/5

Why wouldn't we want to look at reliability coefficients directly? Score reliability is a relative parameter, informing you which fraction of variance in your data is attributable to differences in true scores (in latent factor scores for those people). 3/5

100

We propose an alternative way to assess in how far measurement precision differs across sites: Just model the error variance explicitly (sheepishly named Boot-Err). Using bootstrapping & appropriate transformations (variance-stabilizing etc.), we can inspect random measurement error directly. 2/5

100

Maria Bruntsch @mbruntsch.bsky.social · 12/07/2024

Interested in comparing measurement precision across a bunch of data-sets/administration sites? Tempted to just throw a meta-analysis at reliability coefficients (also known as Reliability Generalization)? Maybe don't, read here on what to do instead: doi.org/10.1037/met0000740 #metasci 1/5

doi.org

APA PsycNet

173

Reposted by Lukas Beinhauer

My weekend starts with great news! Our stage 1 registered report received an in-principle-acceptance by @pci-regreports.bsky.social The IPA and manuscript can be found here: rr.peercommunityin.org/articles/rec... Looking forward to start working on the things we planned!

rr.peercommunityin.org

Integrative meta-analysis of anxiety-related traits and fear pro...

1103

Reposted by Lukas Beinhauer

Rogue HCI Guy @roguehci.bsky.social · 10/07/2024

Neuer Job gefällig?

132

Reposted by Lukas Beinhauer

Ian Hussey @ianhussey.mmmdata.io · 07/07/2024

Now accepted at AMPPS: It's not just p-values that get hacked. We present evidence that the distribution of published Cronbach’s alpha values is also distorted in some way due to hacking and/or bias. @malte.the100.ci @ruben.the100.ci @taymalsalti.bsky.social osf.io/preprints/ps...

13918

Reposted by Lukas Beinhauer

meta-rep.bsky.social @meta-rep.bsky.social · 24/04/2024

This year we host in October the META-REP 2024 an international conference focusing on meta-science and replicability in the cognitive, behavioral, and social sciences. The deadline for submissions is April 30. Contribute to vital discussions on meta-science and replicability. #metarep #openscience

conference2024.meta-rep.uni-muenchen.de

META-REP Conference 2024 - LMU Munich

01112

Lukas Beinhauer @lbnhr.bsky.social · 03/04/2024

Preprint updated because actually submitted - we discuss why differences in reliability might not actually inform you about differences in measuring quality - and what to do instead! Get the full story at osf.io/preprints/ps...

osf.io

OSF

000

Reposted by Lukas Beinhauer

Taym @taymalsalti.bsky.social · 23/02/2024

hello yes remember this? we have a new and improved version, check it out! osf.io/preprints/ps...

285

Reposted by Lukas Beinhauer

Framework for Open and Reproducible Research Training @forrt.bsky.social · 15/02/2024

We love data from replication studies. Together with 146 researchers, we aggregated and coded replication findings. You can browse them in the @forrt.bsky.social #ReplicationDatabase: metaanalyses.shinyapps.io/replicationd... (1/3)

13122

I wasn't aware of this, this is outrageous, what has Wiesbaden ever done to them?

Probably late to the party, but MS Word changed the default font to something most of us never heard of before?

100

I see, the magic lies in the "prompt engineering" - I guess my prompt was too specific/distracting, so it started spewing information on what a replication is instead. That looks very promising, thanks for the follow-up!

210

@forrt.bsky.social Ideally, it would carry information on each conceptual replication on what kind of intervention and what kind of outcome measure was used. We're currently not looking at specific effects, but trying to understand how different interventions of the same effect may impact results.

100

metaanalyses.shinyapps.io

Just to clarify - I didn't mean to throw shade! Greatly appreciate everything you're doing! Is this the database you are referring to? metaanalyses.shinyapps.io/replicationd... Because this is great stuff, it just doesn't happen to carry the niche information I am currently looking for.

Oh really, I just remember some controversies surrounding audits from that period - thanks for the heads-up

Ah yes, that's great! I saw it used for direct replications in the past, but they did keep track of some individual replications as well! Thanks! :)

010

Tried it now! Seems like it's closer to scite.ai and other tools, which make use of AI to answer research questions, right? At least I haven't been able to make it spew out the meta-analyses that I am looking for (not even the ones I already know of).

Not like I'm sad about it - I'm happy there's not more replications of "Feeling the future" than we already have!

020

Lukas Beinhauer @lbnhr.bsky.social · 10/01/2024

I wasn't even awareof this specific FORRT-project! Looks like a great resource. Unfortunately, I was hoping for something with more replications for a single effect, something that qualifies for e.g. a meta-analysis - it seems like that isn't what was attempted to be gathered here.

320

Similarly, if anyone is aware of a literature search tool that allows for selection based on data being openly available, that would be great as well! Like a Googe Scholar, EBSCOhost etc. where I can select for the presence of open data

100

Lukas Beinhauer @lbnhr.bsky.social · 10/01/2024

Anything that goes in similar directions also helps! I'm aware of typical ManyLabs and Registered Replication Report stuff, but systematical variation in either outcome measure (psychometric scale etc.) or the treatment, manipulation, intervention etc. has been coming up short unfortunately

100

Lukas Beinhauer @lbnhr.bsky.social · 10/01/2024

I'm searching for open replication data! Conceptual replications employing either: - same treatment/intervention but different outcome measure or: - different treatment (dose)/intervention but identical outcome measure (or both) Any pointers appreciated! #openscience #metasci

333

Lukas Beinhauer @lbnhr.bsky.social · 21/12/2023

Ah that's very interesting! Do you happen to have some introductory reading for measurement error in epidemiology in mind?

000

There's a "crying face behind smug mask"-meme about CTT and true scores in this, but I'm too lazy to actually put effort into it right now

010

Thanks - the "estimand" sounds like a good approach I hadn't considered yet! :) Not sure if I'm seeing the issue lying in CTT or the vagueness of using "true" to mean "free of sampling error" or "underlying population value" - but I guess that historically, one must have come before the other.

100

In order to properly distinguish the two types of "true-ness", is there another adjective I could use for the first case, the population mu (greek \mu)? (3/3) #stats

210

In classical test theory, we use "true score" to distinguish between an observed score, that contains error, and the score that we should have gotten if no measurement error was involved. (2/3)