I am a Research Scientist at the Humane and Sustainable Food Lab at Stanford and a nonresident fellow at the Kahneman-Treisman Center at Princeton. By trade, I am a meta-analyst.
No meaningful relationship! (see code below.) However, big caveat here that we had to guess on some of the samples because many studies do not report how many subjects or meals were treated (e.g. they report how many restaurants or days were assigned to treatment and control but didn't count how many people participated)
> summary(lm(d ~ total_sample, data = dat))
Call:
lm(formula = d ~ total_sample, data = dat)
Residuals:
Min 1Q Median 3Q Max
-0.59897 -0.13702 -0.01868 0.12322 0.75767
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0.06330835 0.02664964 2.376 0.0193 *
total_sample -0.00002876 0.00004690 -0.613 0.5410
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Residual standard error: 0.2474 on 110 degrees of freedom
Multiple R-squared: 0.003407, Adjusted R-squared: -0.005653
F-statistic: 0.376 on 1 and 110 DF, p-value: 0.541
Delay indicates the number of days that have elapsed between the beginning of treatment and the final outcome measure. How outcomes are measured varies from study to study, so in some cases it's a 24 hour food recall X number of days after treatment is administered (the last part of it), in others it's a continuous outcome measurement in a cafeteria (the entire period of delay).
This I can say more about!
The median delay, in days, is 14, and the mean is 52 (we have a few studies with long delays, the longest is 3 years (Jalil et al. 2023).
So I'd say, think "about 2 weeks on average with some lengthy outliers". Also there's basically no relationship between delay and effect size.
to replicate in R (from the root directory of our project):
> source('./scripts/libraries.R')
> source('./scripts/load-data.R')
> summary(dat$delay)
Min. 1st Qu. Median Mean 3rd Qu. Max.
4.00 11.50 14.00 52.05 60.00 1095.00
> source('./functions/sum-lm.R') # this is a little function we wrote that puts summary(lm()) into a dplyr-friendly pipe
> dat |> sum_lm(y = d, x = delay)
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0.05312 0.02552 2.08181 0.03968
delay 0.00005 0.00019 0.23166 0.81723
Thank you David! We will post any updates to https://doi.org/10.31219/osf.io/q6xyr
The paper is currently under submission at a journal and we likely won’t modify it until we get some feedback.
Definitely! When I went vegan, I prompted someone I know to look up how dairy cows are treated (not well), and they changed their diet quite a bit in light of that. So I have seen downstream effects personally. Caveat that I am annoying and prone to evangelize.
And if i were going to promote one definitely-not-scalable intervention to one very-hard-to-reach-population, I would take a bunch of die-hard meat eaters to Han Dynasty on the upper west side of Manhattan and order 1) DanDan noodles without pork 2) pea leaves with garlic 3) cumin tofu 4) kung pao tofu and 5) eggplant in garlic sauce for the table, and then just be "like hello is this not delicious??" every 30 seconds 😃
My implicit knowledge on the topic of knowledge production (rather than of veganuary) is that rosy results like the one you are citing often do not stand up to scrutiny. Maya raised one very salient objection to a gap between the headline interpretation and the data of a past iteration of this work: https://forum.effectivealtruism.org/posts/vg3rxwcu7una8nSpr/veganuary-s-impact-has-been-huge-here-are-the-stats-to-prove?commentId=32xKWjRjgDc4cyaDj
I believe that if I dig into it, I’ll find other, similar issues. Another way to phrase this: I have pessimistic beliefs about nonstatistical sources of uncertainty and/or bias whose magnitude is itself a hard estimation problem. Sorry for such a meta answer…