iiinova — where your digital self keeps evolving

And I, too, would think I was beating a dead horse if it weren't for the fact that top journals keep publishing simple lag-1 CLPMs with no controls (meta-analytic CLPMs are still CLPMs): www.apa.org/pubs/journal...

apa.org

060

Reposting this thread about another recent preprint that discusses some of the reasons why: bsky.app/profile/rich...

160

Rates of significant effects are also arguably too high with the RI-CLPM. Individual effects are about half as likely to be significant with the RI-CLPM as with the CLPM, but you still get at least one significant lagged effect in 61% of models.

We already know that lagged effects in CLPMs are likely to be upwardly biased, but just how easy is it to find significant effects? Way too easy. I tested CLPMS in 100 randomly selected pairs of correlated variables and found significant effects in 98 of them. New preprint: osf.io/preprints/ps...

osf.io

OSF

44619

Anyway, we welcome any thoughts or suggestions you have about the paper! [15/15]

030

Also: state variance is not just measurement error, so using latent variables with the CLPM or RI-CLPM won't always fix this (though it helps). Even with multiple-item Big Five scores modeled using latent traits, the same pattern emerged. [14/x]

Patterns of stability over increasingly long lags for the Big Five personality traits. The slope of these stability coefficients is very shallow.

Our analyses also suggest that once you account for state variance, stabilities of variables included in these panel studies are extremely high: Median 1-year stability = .90. This suggests that there is often very little change occurring that could be accounted for by a lagged effect [13/x].

Our analyses suggest that the CLPM and RI-CLPM would be the most appropriate model for just 4% of variables we examined; the other 96% were divided equally between ARTS and STARTS [12/x]

What type of model can account for these patterns? Models that include a state component (like the STARTS or even the ARTS, which drops the stable trait). You can reproduce actual patterns of stability even without assuming the existence of a stable trait (this plot is for the ARTS) [11/x]

Implied and actual stability coefficients for a 10-year ARTS model. The implied and actual stability coefficients are very similar.

It turns out it's very common. Here are the plots of stability over increasingly long lags for about 400 variables from a large panel study. For anything less than almost perfect short-term stability, stability coefficients should reach an asymptote long before 22 years; very few do [10/x]

Actual stability coefficients across increasingly long lags for over 400 variables. Most stabilities decline slowly without ever reaching an asymptote.

One pattern it can't handle is when short-term stability is moderate, medium-term stability is just slightly lower than long-term stability, and there is no clear asymptote with increasingly long lags (as is true with life satisfaction and health). How commons is this pattern? [9/x]

Why doesn't the RI-CLPM just estimate a lower asymptote (corresponding to less stable trait variance)? It turns out that the RI-CLPM is quite limited in the types of patterns of stability with which it is compatible (won't go into details here, but paper does). [8/x]

This gets even worse with more waves of data. In our experience, this is a pretty common pattern: If you have more than 4 or 5 waves, the RI-CLPM overestimates long-term stability (and often underestimates medium-term stability). This misfit can also lead to bias in estimates [7/x]

Implied and actual stability coefficients for a 10-wave RI-CLPM. Evidence of misspecification emerges at longer lags.

The RI-CLPM does better, but even here, there is a hint of misspecification (see how stability coefficients continue to decline even after the RI-CLPM predicts an asymptote) [6/x]

Implied and actual stability coefficients for the RI-CLPM. These values are close to one another with a hint of misspecification at long lags.

Here's a plot of actual stability coefficients and those implied by a fitted CLPM with life satisfaction and health. It's really bad...CLPM dramatically underestimates stability. This misspecification introduces bias, which can be severe [5/x]

Implied and actual stability coefficients over five years for life satisfaction and health. The implied stability coefficients are much lower than the actual stability coefficients.

There are already lots of good papers comparing these models from a causal inference perspective, but we focus more on typical patterns of stability in real data and how well they match with the patterns of stability implied by these models. Why is this useful? Consider the CLPM. [4/x]

TL/DR: More reasons why the standard lag-1 CLPM is really bad, but also some important concerns about models that assume the existence of a stable trait (like the RI-CLPM). Our analyses suggest that we should be paying more attention to state variance when modeling these effects [3/x]

We try to do three new things. 1. Focus explicitly on model misfit and its implications for bias. 2. Conduct simulations showing how specific forms of misfit affect bias. 3. Analyze the longitudinal structure of over 400 variables to see what type of misfit is likely for specific models. [2/x]

Rich Lucas @richlucas.bsky.social · 12/08/2025

Interested in models used to estimate lagged effects in panel data? We (@rebiweidmann.bsky.social, Hyewon Yang) have a new paper looking at patterns of stability and their implications for bias and model choice: osf.io/preprints/ps... [1/x]

osf.io

OSF

12512

If you're looking for a good quarto extension to do APA-style documents, this one does it! Seems to have everything papaja had for Rmarkdown and more. Really useful and easy to implement (including great export to docx): wjschne.github.io/apaquarto/

wjschne.github.io

Introduction to apaquarto – APA Style Documents with apaquarto

1246

Rich Lucas @richlucas.bsky.social · 12/02/2025

Will get right on that (checks to see whether @syeducation.bsky.social has something I can use as a template that I can pass on to @jnfrltackett.bsky.social )

140

Reposted by Rich Lucas

mark brandt @mjbsp.bsky.social · 03/11/2024

The problems w/ two-wave cross-lagged models are becoming widely known. Some proposed a "triangulation method" (2nd screenshot) to identify spurious effects in such designs We show that this test doesn't work in plausible situations. It shouldn't be used. journals.sagepub.com/doi/10.1177/...

Abstract: The cross-lagged panel model (CLPM) is an analytic technique used to examine the reciprocal causal effects of two or more variables assessed on two or more occasions. Although widely used, the CLPM has been criticized for relying on implausible assumptions, the violation of which can often lead to biased estimates of causal effects. Recently, a triangulation method has been proposed to identify spurious effects in simple CLPM analyses
(e.g., Sorjonen & Melin, 2024b). We use simulations and a discussion of the formulas underlying regression coefficients to show that this method does not provide a valid indicator of spuriousness. This method identifies true causal effects as spurious in realistic situations and should not be used to diagnose whether a causal effect estimated from the CLPM is spurious or not. There are clear reasons to doubt causal estimates from the CLPM, but the results of the triangulation method do not add information about whether such estimates are spurious.

43314

Rich Lucas @richlucas.bsky.social · 23/04/2024

Me, trying to rerun analyses from a years-old paper for a request from a meta-analyist

160

Rich Lucas @richlucas.bsky.social · 23/02/2024

Collabra: Psychology is looking for new associate editors for the Social Psychology section. If you're interested, please apply! rug.eu.qualtrics.com/jfe/form/SV_...

12439

Reposted by Rich Lucas

Michael D. Krämer @mdkraemer.bsky.social · 12/02/2024

My first Skeet! Happy to announce that our paper on life events and life satisfaction is now published at the European Journal of Personality (w/ @dingdingpeng.the100.ci, @richlucas.bsky.social, and @drichter77.bsky.social)! journals.sagepub.com/doi/10.1177/... (preprint: osf.io/preprints/ps... ).

1152

Rich Lucas @richlucas.bsky.social · 30/01/2024

Oh no; that is so sad. Gert was so warm and welcoming when I started working with the SOEP data, and supportive and encouraging ever since. So much of the way that personality psychologists work now (with lots of analysis of panel data) is due to Gert's early support of psych measures in the SOEP.

030

Reposted by Rich Lucas

Farid Anvari @faridanvari.bsky.social · 11/12/2023

Research suggested a response styles bias in people's repeated self-reports (e.g. daily diary studies, EMA). We tested this and found evidence against a broad bias. Open access: journals.sagepub.com/doi/10.1177/... with @richlucas.bsky.social, Elise Kalokerinos, Noelle Rensing, and Iris Schneider.

22810

The second thing is that for the second time in a row (see below), our post-publication critique was received really well by the original authors. Could this be a new norm? We have one more coming soon, so hopefully it is :) bsky.app/profile/rich...

030

They were also able to present some of this work at their undergrad research fair, and after preparing that presentation, our undergrads now have a really good understanding of some of the problems with difference scores

This was a great opportunity for our undergrad assistants to see that just because something is published, that doesn't mean there aren't questions left and that we can address those questions with new analyses and data

This was a fun paper for a few reasons. First, like most of the projects involving replication or reanalysis, this arose out of a lab meeting with our undergrad research assistants. They were interested in the topic, we discussed strengths and weaknesses. We reanalyzed and attempted to replicate

1103

The paper's under review now, but comments and suggestions are welcome! /end

010

For variables that social scientists study, CLPM (short dash) usually overestimates (in model-implied correlations) how quickly stability coefs decline with longer lags (as seen here; solid = actual). This leads to bias in the lagged effects. Other models (like RICLPM -- long-dash) do better /10

Figure showing implied stability coefficients from the cross-lagged panel model and random-intercept cross-lagged model compared to actual stability coefficients

Finally, this is another example of a case where using the standard lag-1 cross-lagged panel model (which was used in the original paper) is probably not appropriate (see: journals.sagepub.com/doi/10.1177/...) /9

Hopefully this means that the code is now in much better shape to be run by others; but if anyone wants to try it, the code (and instructions for running it) is here: github.com/rlucas11/gso.... Feel free to create an issue on github if something doesn't work. /8

This meant that I had to write the code (which is quite complex given the nature of the project), test it on incomplete data that didn't have all the variables, and then send it to @dingdingpeng.bsky.social to run. We went back and forth fixing errors. /7

Regarding writing reproducible code, this project had a challenging twist: I couldn't access the version of the data that the original authors analyzed (it uses the SOEP, which, for privacy reasons, has different version for EU and non-EU users). /6

I also want to thank the original authors for making their own code available, for thoughtful comments on our reanalysis, and for being really great to work with throughout this process. /5

More importantly, given the strong reactions that sometime arise when really problematic papers are subjected to post-publication critique, it might be helpful to normalize post-publication critique by doing it regularly for published papers that vary in quality (including some good ones) /4

131

In short, we test variations on three modeling choices, and none of the effects is robust across these different models (figure shows variability in effects). That doesn't mean the original authors' choices were wrong, just that some caution is warranted when interpreting the results /3

Figure showing how estimates for lagged effect of personality on religion vary across different model specifications

Regarding post-publication critique: I've been asked why we chose this paper to reproduce/critique. It's a good paper with a lot of strengths, but there are (as is usually true) some alternative choices that could have been made. Turns out these alternative choices lead to different conclusions /2