Sign in

Matt Makel

@mattmakel.bsky.social
977 followers 272 following 212 posts

educational researcher focusing on academic talent (identification and development) and open science research practices (implementation and assessment). Now in Canada @UCalgary

PostsRepliesMedia
Reposted by Matt Makel
Nick Brown @steamtraen.eu · 27/09/2026
Current mood...
Padmé meme.

Anakin: "I can't share my data yet as it may contain other interesting stuff"
Padmé: "But you're going to write those analyses up as exploratory, right?"
0143
Reposted by Matt Makel
moin syed @syeducation.bsky.social · 22/09/2026
New paper! Despite qualitative researchers claiming that generalizability is not a goal, their published work does indeed contain generalizable claims. We document this and provide recommendations for how all researchers can better calibrate their generalizability claims. doi.org/10.1177/2515...
article header for, "Qualitative Researchers Can and Do Generalize: Generalizability Claims in Qualitative Psychological Research". Metascience!
619258
Reposted by Matt Makel
AERA @aeraedresearch.bsky.social · 09/09/2026
Reminder! The Graduate Student Research-in-Progress Roundtable Series Call for Submissions is open. The deadline to submit to the call is October 15! Full eligibility and submission information here: www.aera.net/Education-Re...
011
Reposted by Matt Makel
Amanda Kay Montoya @akmontoya.bsky.social · 08/09/2026
Curious about Zenodo as an OSF Project alternative? Here's a short video to get you started: File Uploads, Communities, and generating links for Anonymous Peer Review. youtu.be/jW84PxuJTMw
youtu.be
Getting Started with Zenodo (transitioning from OSF Projects)
YouTube video by AMANDA MONTOYA
67440
Reposted by Matt Makel
Olúfẹ́mi O. Táíwò @olufemiotaiwo.bsky.social · 08/09/2026
if it took you 5 seconds of contemplation to think of an alternative you should take at least another 5-10 to at least consider the possibility that it at least crossed the minds of 1 or 2 of the people who dedicate their lives and research careers to dealing with the problems we post about on here
282379299
Reposted by Matt Makel
Cas Goos @cas-goos.bsky.social · 03/09/2026
I am proud to announce that my article "Measurement reliability, construct validity, and transparent reporting in original and replication psychological research" has just been published in PLoS One. You can find the published article here: journals.plos.org/plosone/arti...
journals.plos.org
Measurement reliability, construct validity, and transparent reporting in original and replication psychological research
Published (replication) studies that use measures to assess psychological phenomena need to transparently report information on measurement procedures, validity, and reliability to allow verification ...
0113
Matt Makel @mattmakel.bsky.social · 03/09/2026
Could be a nice treat for yourself after you accomplish something big!
100
Reposted by Matt Makel
Randy Ellis @randalljellis.bsky.social · 02/09/2026
Here's my conversation with @bethclarke.bsky.social, incoming Junior Professor of Psychological Metascience at the Leibniz-Institut für Psychologie. www.youtube.com/watch?v=2Hak... We discussed direct replications, how study limitations are ignored, and p-curving social and personality psychology.
youtube.com
Direct replications, P-curves, and Acknowledging Limitations | Beth Clarke
YouTube video by Metascience Matters
1199
Matt Makel @mattmakel.bsky.social · 31/08/2026
Always consult a librarian is a lesson I have 100% learned. I just wish I had learned it sooner.
142
Matt Makel @mattmakel.bsky.social · 31/08/2026
I think it is totally fair and reasonable for different people to have differing opinions/definitions of "What counts as a replication". Rakow's new paper certainly is not the final word on replication rates. But I think it is an important update that merits attention.
010
Matt Makel @mattmakel.bsky.social · 31/08/2026
Don’t forget to read the new paper kclpure.kcl.ac.uk/portal/en/pu... [non-paywall version] 22/22
kclpure.kcl.ac.uk
Estimating the prevalence of replication research is challenging, not least because even a thorough database search can be shallow.
190
Matt Makel @mattmakel.bsky.social · 31/08/2026
That it took over a decade for this to come to light raises several questions. How can we improve the identification of such errors? How can we better platform those who observe a finding that does not sit well with them? How can we increase the rate and speed of correction? 21/
160
Matt Makel @mattmakel.bsky.social · 31/08/2026
Now that I know about the mistake, I am writing to the editors of the journals where I published in an effort to correct the record. My hope is that we can do so quickly. 20/
180
Matt Makel @mattmakel.bsky.social · 31/08/2026
I will be thinking about this for the rest of my career. I apologize to my co-authors and all readers of this work. I honestly thought we were searching the full text. I should have caught the error. 19/
170
Matt Makel @mattmakel.bsky.social · 31/08/2026
As I wrote recently, “Long term behavior change upon acquisition of new information is useful. It's called learning.” I hope I never become someone who stops learning. I believe the field is stronger when we update our beliefs when we acquire new information. journals.sagepub.com/doi/10.1177/... 18/
journals.sagepub.com
Sage Journals: Discover world-class research
Subscription and open access journals from Sage, the world's leading independent academic publisher.
160
Matt Makel @mattmakel.bsky.social · 31/08/2026
In other words, exactly what we want to happen in scientific research (detection of errors to improve our understanding of the world) happened. …It just took much longer than I’d like. 17/
240
Matt Makel @mattmakel.bsky.social · 31/08/2026
Because of this fallibility, replication (particularly independent replication) is hugely valuable for helping us observe and understand the world around us as well as consume research findings. 16/
140
Matt Makel @mattmakel.bsky.social · 31/08/2026
Perhaps ironically, these new results give me a deeply personal re-invigorated belief in the value of replication of research and the fallibility of well-meaning researchers and a flawed academic research publication system. 15/
170
Matt Makel @mattmakel.bsky.social · 31/08/2026
I am reminded of the famous invisible gorilla research in social psychology. The new results reveal that replications have been hidden in plain sight. We simply had to look in the right way to be aware that authors have been conducting and reporting replications. www.youtube.com/watch?v=vJG6... 14/
youtube.com
selective attention test
YouTube video by Daniel Simons
180
Matt Makel @mattmakel.bsky.social · 31/08/2026
Loss of confidence in research results is not rare. Public acknowledgement of loss of confidence is rare. journals.sagepub.com/doi/10.1177/... 13/
journals.sagepub.com
Putting the Self in Self-Correction: Findings From the Loss-of-Confidence Project - Julia M. Rohrer, Warren Tierney, Eric L. Uhlmann, Lisa M. DeBruine, Tom Heyman, Benedict Jones, Stefan C. Schmukle, ...
Science is often perceived to be a self-correcting enterprise. In principle, the assessment of scientific claims is supposed to proceed in a cumulative fashion,...
191
Matt Makel @mattmakel.bsky.social · 31/08/2026
Science self-corrects when scientists take the effort to do so. It’s not automatic. And it’s not easy. Kudos and thank you to Dr. Rakow for doing it here. If I have a frustration with the new paper, it is that it didn’t come out about 14 years sooner. 12/
1100
Matt Makel @mattmakel.bsky.social · 31/08/2026
Rakow deserves rich credit for helping reveal this error in framing. I thank him for this work. I owe him a sincere debt of gratitude for helping bring to light a substantive and meaningful inference change. I hope everyone interested in replication rate research reads this new paper. 11/
2341
Matt Makel @mattmakel.bsky.social · 31/08/2026
This new paper should meaningfully change beliefs and expectations about prior replication rates. How future work is done—and what we should expect from it—should be informed by these new results. 10/
160
Matt Makel @mattmakel.bsky.social · 31/08/2026
Rakow: “My investigation suggests that many researchers are not shy about conducting and reporting replication. Plausibly, at least 20% of psychology articles include author-identified replication; and replication could be considered to be a mainstream practice in experimental psychology.” 9/
1101
Matt Makel @mattmakel.bsky.social · 31/08/2026
Rakow’s new paper found that “around 90% of articles that discuss replication and over 80% of articles with author-identified replications are missed by searching databases for replicat* because databases hold few details per article”. 8/
1145
Matt Makel @mattmakel.bsky.social · 31/08/2026
I would have bet money that many of the articles we coded included discussion of replication in text and not just in title/abstract/keywords. Of course, relying on the (obviously) faulty memory of individual researchers is an astoundingly terrible foundation for a scholarly field. 7/
181
Matt Makel @mattmakel.bsky.social · 31/08/2026
In other words, the results we reported reflect the rate at which researchers were highlighting replications and not conducting replications. That is a meaningful difference. 6/
1150
Matt Makel @mattmakel.bsky.social · 31/08/2026
Database searches did NOT search “all text” as I believed. It did not have that capability. Instead, it searched only Titles, Abstracts, and Keywords of papers. 5/
2170
Matt Makel @mattmakel.bsky.social · 31/08/2026
a new paper by Tim Rakow of King’s College London reveals that prior work that relied on database searches to report replication rates underestimated the number of replications that have been conducted. journals.sagepub.com/doi/abs/10.1... 4/
journals.sagepub.com
Sage Journals: Discover world-class research
Subscription and open access journals from Sage, the world's leading independent academic publisher.
12911
Matt Makel @mattmakel.bsky.social · 31/08/2026
In 2012-2016, we used Web of Science to estimate a “replication rate” of how often researchers in various fields attempted to check (or conceptually build on) prior results. In general, we found that replications were quite rare (often < 1%). 3/
1100
Matt Makel @mattmakel.bsky.social · 31/08/2026
I started looking at replication rates because I had tried to replicate research that I believed to be true. Soon, I learned that many were unable to replicate prior results. This led me to want to know how many others tried to replicate prior findings and how their efforts turned out 2/
1100
Matt Makel @mattmakel.bsky.social · 31/08/2026
I made a mistake. And because of that mistake, I no longer have confidence that results that I reported in several research papers are an accurate reflection of replication rates. 1/
19430
Reposted by Matt Makel
Prof Dynarski @dynarski.bsky.social · 27/08/2026
Test-optional policies reduce college admissions of high-achieving, disadvantaged students Under test-optional policies, too few of these applicants submit test scores; they more than double their admission probability (from 2.9% to 7.2 %) by reporting scores. drive.google.com/file/d/10mW5...
35221
Reposted by Matt Makel
Dr Ellie Murray, ScD @epiellie.bsky.social · 27/08/2026
Love this! We need more and better descriptive studies. For those whose descriptive psych studies are somewhat epidemiologic in nature, might I recommend this paper on how to do good descriptive work👇🏼 pubmed.ncbi.nlm.nih.gov/35774001/
pubmed.ncbi.nlm.nih.gov
A Framework for Descriptive Epidemiology - PubMed
In this paper, we propose a framework for thinking through the design and conduct of descriptive epidemiologic studies. A well-defined descriptive question aims to quantify and characterize some feature of the health of a population and must clearly state: 1) the target population, characterized by …
0102
Reposted by Matt Makel
Beth Clarke @bethclarke.bsky.social · 21/08/2026
I’m hiring a PhD student to work with me on metascience in psychology at @zpid.bsky.social! The position is very open - could cover a wide range of topics/themes 🤸🏻‍♀️✨ If you have questions, please reach out! Applications are open until October 1st. leibniz-psychology.onlyfy.jobs/en/job/fyjwt...
leibniz-psychology.onlyfy.jobs
PhD in Psychological Meta-Science (m/f/d)
Become part of the open science movement The Leibniz Institute for Psychology Information (ZPID) in Trier (Germany) is an internationally recognized research-based psychology service center located in...
36157
Reposted by Matt Makel
Mary Dudziak @marydudziak.bsky.social · 18/08/2026
Beginning my research seminar, as always, with Lynn Hunt's essay How Writing Leads to Thinking (and Not the Other Way Around). Should be mandatory reading in light of the impact of AI. www.historians.org/perspectives...
Writing is stressful. Sitting in my computer chair my neck and shoulder muscles almost immediately tense up as I dig around in my brain for the best phrase or even any coherent string of words, whether I am writing an essay like this one, a book chapter, a letter of recommendation, or an email message to a friend. Writing is time-consuming. It’s a great way to pass the time on a long airplane flight because you lose track of the passage of time altogether. It’s even better, from that point of view exclusively, than watching an episode of Mad Men on your laptop. Writing means many different things to me but one thing it is not: writing is not the transcription of thoughts already consciously present in my mind. Writing is a magical and mysterious process that makes it possible to think differently.
9479153
Reposted by Matt Makel
Society for the Improvement of Psychological Science (SIPS) @improvingpsych.org · 15/08/2026
🥳Today, PsyArXiv – the community preprint server for psychology – turns 10! 🥳 Thank you to everyone who contributed to the server's growth, including volunteers who work behind the scenes to keep it running, preprint authors, and readers!
06917
Matt Makel @mattmakel.bsky.social · 14/08/2026
Are you in... Beach jail?
040
Reposted by Matt Makel
vincentholst.bsky.social @vincentholst.bsky.social · 13/08/2026
In 2023, @nature.com published 'Papers and patents are becoming less disruptive over time', receiving world-wide media attention. Our Matters Arising, published after a 32 month delay (more on that soon), shows that the reported decline can largely be attributed to dataset artefacts. 🧵
The average CD_5 index per year for Web of Science. The original Park et al. decline (top curve) becomes essentially flat (bottom curve) when removing papers with CD_5=1. Those papers largely correspond to dataset artefacts.
9352173
Reposted by Matt Makel
simine vazire @simine.com · 12/08/2026
It’s amazing what Brian and OSF have done for science. I can imagine how hard the folks at COS must’ve worked to avoid this outcome. I’m extremely grateful to them for creating the reality we live in now. The fact that we’ll miss these services so acutely is a testament to how much they shaped us.
09324
Reposted by Matt Makel
Katie Corker @katiecorker.bsky.social · 12/08/2026
For those looking for alternatives to OSF, I recommend two resources comparing features of other Generalist Repositories. Resources are from GREI, an NIH funded initiative. Comparison tool to help you pick which one you like best: doi.org/10.5281/zeno... Decision tree:  doi.org/10.5281/zeno...
doi.org
Generalist Repository Comparison Chart
The Generalist Repository Comparison Chart and FAIRsharing Collection (https://fairsharing.org/collection/GeneralRepositoryComparison) was originally an outcome of the NIH Workshop on the Role of Gene...
03523
Reposted by Matt Makel
Amanda Kay Montoya @akmontoya.bsky.social · 11/08/2026
Here's the list put together by OSF: help.osf.io/article/735-.... Zenodo is my top preference currently. Can host any kind of research output, and gives everything a DOI, and free. Hosted by CERN and funded by many European government groups.
help.osf.io
Where should I share my Data, Materials, Code now?
This Article Is Licensed Under CCO For Maximum Reuse. The following is a Table of Contents that links directly to specific sections within the guide: Table of R
131
Reposted by Matt Makel
Brian Nosek @briannosek.bsky.social · 11/08/2026
Over the next several months, we will phase out some popular OSF features. These changes do not affect current public content, which remains safe and accessible. We have prepared support material to help users adapt. I am very sorry for the disruption to your work. Please read this post for details.
cos.io
OSF Changes | Center for Open Science
We are preparing substantial changes to OSF that will reduce its functionality, focus OSF on its unique strengths, and move toward an integrated model with complementary services. As part of that shif...
10120113
Reposted by Matt Makel
Joshua Goodman @joshua-goodman.com · 03/08/2026
I'm quoted in this morning's @bostonglobe.com editorial about why middle school is likely the right age to begin tracking math classes. By middle grades, the range of math skills in a given cohort is too large to be taught effectively in a single class. www.bostonglobe.com/2026/08/03/o...
2133
Reposted by Matt Makel
Matthew A Kraft @matthewakraft.com · 24/07/2026
Take 1 minute to read this intuitive and incisive review of our study on the portability of teacher effectiveness by @nctq.bsky.social's Sarah Almy. Beautiful policy writing in my opinion. www.nctq.org/research-ins...
nctq.org
Does great teaching travel from school to school? It’s complicated. - National Council on Teacher Quality
Do great teachers perform as well after switching schools? New research on a $20,000 transfer incentive program shows the answer is complicated—and prep matters.
052
Reposted by Matt Makel
Anne Scheel @annemscheel.bsky.social · 23/07/2026
New preprint w/ Leo Tiokhin & @lakens.bsky.social! Registered Reports (RRs) are great for science b/c publication is guaranteed before results are known, reducing publication bias & QRPs. This feature supposedly also makes them great for *scientists*. But is that really true? 1/ tinyurl.com/4sy993rr
tinyurl.com
Career incentives make Registered Reports unattractive under realistic academic conditions
Registered Reports are a publication format designed to reduce publication bias by guaranteeing publication before results are known. This guarantee is considered attractive in the prevailing ‘publish...
15934
Reposted by Matt Makel
Paul Bruno @paul-bruno.com · 22/07/2026
"I'd like to see state and federal policymakers place a stronger emphasis on making student growth and improvement measures publicly available and giving them the kind of attention we've historically given measures like student 'proficiency'" www.newsweek.com/map-reveals-...
newsweek.com
Map reveals states with best and worst public schools
A new ranking lists New York as the state with the best public schools, while Arizona ranks at the bottom.
251
Reposted by Matt Makel
bergelsonlab @bergelsonlab.bsky.social · 17/07/2026
within every [academic] there are 2 [reasons we agree to review] 1) sounds interesting & we wanna know about it...bc we are huge nerds 2) sounds problematic & we wanna make sure someone is sufficiently critical...bc we are huge nerds (ok ok & sometimes we wanna do the editor a solid) #AcademicSky
0213
Reposted by Matt Makel
Arvind Narayanan @randomwalker.bsky.social · 15/07/2026
I had the honor of giving a keynote at the International Conference on Machine Learning last week. I addressed the widespread anxiety about how we should adapt as AI capabilities increase. I was thrilled by the talk’s reception, so I have made my slides available www.cs.princeton.edu/~arvindn/tal...
cs.princeton.edu
What will be left for us to work on?
ICML 2026 invited keynote — slides and edited transcript, presented click-by-click as delivered. Arvind Narayanan, Princeton University.
614048
Reposted by Matt Makel
Crystal Steltenpohl @cnsyoung.com · 06/07/2026
New article from @mattmakel.bsky.social, Melanie Meyer, and Mary Pei about qualitative replication.
doi.org
Replication is possible, can be relevant, and is useful in qualitative research
In 2022, we argued that replication is relevant to qualitative research. In this manuscript, we update and clarify that idea to make three more specific claims about replication in qualitative rese...
011