Approach
Every analysis at Norlowe Research goes through the same sequence, from a question written down at the start to results checked before release. The aim is work that another analyst could rerun and arrive at the same numbers.
The question and the data
Each project starts with a question stated in writing and a list of the data needed to answer it. The release or version of each source is recorded at the outset.
A plan written before analysis
Before any results are examined, a written analysis plan sets out the population, the definitions, the outcomes and the planned comparisons.
Later changes are dated and explained. Analyses added after the results were seen are labeled as exploratory wherever they appear.
Code from source files to tables
All data handling is done in code, from the original downloaded files to the final tables and figures. Numbers are not typed in or edited by hand, so the whole analysis can be rerun from the beginning.
Checks before release
Where a data publisher reports its own figures, results are compared with them. Key numbers are recomputed by a second route, and automated tests cover the steps where mistakes are most likely.
Limits and corrections
Written results say what the data cannot show. If an error is found after release, it is corrected, and the correction is dated and described.
Records kept with each analysis
For every analysis, the following are kept together:
- the written plan, with any dated amendments
- the release or version of every data source
- the code, from the original files to the final tables
- the checks that were run and what they showed