Why the analysis plan comes first
analysis plan, pre-registration, survey design, research integrity, forking paths, surveyframe
TL;DR: If you decide how to analyse a survey after you have seen the responses, you will tend to find the answer you were hoping for, even with the best intentions. surveyframe asks you to write the analysis plan into the instrument before you collect anything, and then it runs that plan as declared. This post explains why that order is worth the small extra effort, for readers who run surveys.
The usual order, and the problem with it
Here is how most survey work goes. You write the questions, send the survey out, wait for responses and then open the results and start looking. You try one comparison, then another. You drop the two respondents who clearly rushed. You split by region because the overall number looked flat. Somewhere in there you find something that reaches significance, and that becomes the finding you report.
Each step in that story is a reasonable choice a careful person might make. The problem is that many reasonable choices were available, and you made them after seeing which ones pointed somewhere interesting. When the number of paths is large, at least one of them will look like a result by chance alone. Choosing the path after seeing the data quietly stacks the odds in favour of finding something, real or otherwise.
This is the everyday way survey results are produced, and it helps explain why so many survey findings fail to replicate.
The order surveyframe uses instead
surveyframe changes the order. The analysis plan is part of the instrument you build, and you build the instrument before you collect data. You state which questions form which scale, how they are scored, which groups you will compare and which test answers each question. That plan is written down before any response exists.
When the data arrives, surveyframe runs the plan you declared. The comparisons you committed to are the comparisons you get. You can explore further, and the pre-declared result stays on its own, clearly separate from anything you went looking for afterwards.
Exploration stays open to you. What changes is that the plan fixed in advance holds its shape, because it was fixed before you could see what you found.
Why this matters outside academia
This reads like a concern for journals and peer review, yet the same trap sits inside ordinary commercial survey work.
Suppose you run a brand survey and want to know whether a campaign shifted how people feel. If you decide what counts as a shift after seeing the numbers, you can almost always describe some slice of the data as a win. That reads well in a deck. The next campaign, judged the same loose way, will look like a win too, and you lose the chance to learn which campaigns worked. A plan fixed in advance lets a real effect and a lucky slice look different from each other, and that is worth money as well as methodological tidiness.
The integrity angle
A stronger version of this idea is where surveyframe is heading. If the instrument and its analysis plan are fixed before data collection, you can make that fact verifiable, so that anyone reading the results can confirm the plan stayed as declared. The methodology behind that is set out in a companion preprint, Social Science Research 6.0: A Proof-of-Integrity Framework for Tamper-Evident Survey Instruments. Read it if you want to prove, with evidence, that you called your shot in advance.
Where the methods go next
Declaring a plan in advance is more demanding with a small sample, the situation most applied researchers work in. Large-sample rules of thumb fall away, and every analytic choice carries more weight. surveyframe 0.4 added small-sample corrections for that case, and the reasoning is set out in the companion textbook, Quantitative Analysis with Small Samples, which is free to read.
Try it
surveyframe is on CRAN. You can install it and build an instrument in one line:
install.packages("surveyframe")The latest release notes are in surveyframe 0.4.2 on CRAN, and the full documentation is at mohammedalisharafuddin.github.io/surveyframe.