Why the analysis plan comes first
analysis plan, pre-registration, survey design, research integrity, forking paths, surveyframe
TL;DR: If you decide how to analyse a survey after you have seen the responses, you will tend to find the answer you were hoping for, even when you are being honest. surveyframe asks you to write the analysis plan into the instrument before you collect anything, and then it runs that plan as declared. This post explains why that order is worth the small extra effort, for a reader who is running surveys rather than writing code.
The usual order, and the problem with it
Here is how most survey work goes. You write the questions, send the survey out, wait for responses, and then open the results and start looking. You try one comparison, then another. You drop the two respondents who clearly rushed. You split by region because the overall number looked flat. Somewhere in there you find something that reaches significance, and that becomes the finding you report.
Nothing in that story is dishonest. Every single step is a reasonable choice a careful person might make. The problem is that there were many reasonable choices available, and you made them after seeing which ones pointed somewhere interesting. When the number of paths you could have taken is large, at least one of them will look like a result by chance alone. Pick your path after seeing the data and you have quietly stacked the odds in favour of finding something, whether or not anything is really there.
This is not a rare edge case. It is the default way survey results are produced, and it is why so many survey findings do not hold up when someone tries them again.
The order surveyframe uses instead
surveyframe changes the order. The analysis plan is part of the instrument you build, and you build the instrument before you collect data. You state which questions form which scale, how they are scored, which groups you will compare, and which test answers each question. That plan is written down before any response exists.
When the data arrives, surveyframe runs the plan you declared. It does not offer you a menu of tests to try until one works. The comparisons you committed to are the comparisons you get. If you want to explore further, you can, but the pre-declared result stands on its own and is clearly separate from anything you went looking for afterwards.
The point is not that exploration is forbidden. The point is that the plan you fixed in advance cannot be quietly bent to fit what you found, because it was fixed before you could see what you found.
Why this matters if you are not an academic
It is easy to read the above as a concern for journals and peer review. It is not. The same trap sits inside ordinary commercial survey work.
Suppose you run a brand survey and want to know whether a campaign shifted how people feel. If you decide what counts as a shift after seeing the numbers, you will almost always be able to describe some slice of the data as a win. That reads well in a deck. It also means the next campaign, judged the same loose way, will look like a win too, and you will never learn which of your campaigns actually worked. A plan fixed in advance is what lets a real effect and a lucky slice look different from each other. That is worth money, not just methodological tidiness.
The integrity angle
There is a stronger version of this idea, and it is where surveyframe is heading. If the instrument and its analysis plan are fixed before data collection, you can make that fact verifiable, so that anyone reading the results can confirm the plan was not changed after the fact. The methodology behind that is set out in a companion preprint, Social Science Research 6.0: A Proof-of-Integrity Framework for Tamper-Evident Survey Instruments. It is worth reading if you care about being able to prove, rather than merely assert, that you called your shot in advance.
Where the methods go next
Declaring a plan in advance is more demanding when your sample is small, which is the situation most applied researchers are actually in. You cannot lean on large-sample rules of thumb, and every analytic choice carries more weight. The next version of surveyframe leans into that case, and the reasoning is set out in the companion textbook, Quantitative Analysis with Small Samples, which is free to read.
Try it
surveyframe is on CRAN. You can install it and build an instrument in one line:
install.packages("surveyframe")The release notes for the current version are in surveyframe 0.3.3 is on CRAN, and the full documentation is at mohammedalisharafuddin.github.io/surveyframe.