Varnasr · Sep 8, 2026 · Sep 8, 2026
@@ -28,7 +28,7 @@ jobs:
       - name: Install dependencies
         run: |
           python -m pip install --upgrade pip
-          pip install numpy pandas pyreadstat networkx statsmodels scipy
+          pip install numpy pandas pyreadstat networkx statsmodels scipy openpyxl
 
       # No survey data ships or can. Each folder builds seeded synthetic files
       # carrying the structure and nobody's data.
@@ -49,6 +49,20 @@ jobs:
       - name: Network effects
         run: python -m network_effects_sni.test_network_effects
 
+      - name: Data validation
+        run: python data_validation/test_data_validation.py
+
+      - name: Label variables
+        run: python label_variables/test_label_variables.py
+
+      - name: Survey to codebook
+        run: python survey_to_codebook/test_survey_to_codebook.py
+
+      # Runs the package end to end in a temporary copy, corrupts a value, and
+      # checks that verification fails. The R cross-check runs in the R job.
+      - name: Replication
+        run: python replication/test_replication.py
+
   r:
     name: R
     runs-on: ubuntu-latest
@@ -86,3 +100,11 @@ jobs:
       - name: Sample data
         working-directory: tests
         run: Rscript test_dummy_population.R
+
+      # The replication package fits the same model in both languages and
+      # run_all.py fails if the point estimates differ at six places.
+      - name: Replication, Python against R
+        run: |
+          pip install statsmodels scipy
+          Rscript replication/run_regression.R
+          python replication/run_all.py
@@ -45,6 +45,12 @@ Rscript test_load_dhs.R     # 33: the same ones plus the survey design object
 cd ../plfs-india && python make_fixture.py --outdir fixtures
 python test_load_plfs.py    # 28 checks
 Rscript test_load_plfs.R    # the same 28
+
+python -m network_effects_sni.test_network_effects       # 17 tests
+python data_validation/test_data_validation.py           # 18
+python label_variables/test_label_variables.py           # 8
+python survey_to_codebook/test_survey_to_codebook.py     # 9
+python replication/test_replication.py                   # 4; Rscript replication/run_regression.R for the cross-check
 ```
 
 Stata has no free runtime, so the `.do` files are the only untested ones. Each
@@ -67,6 +73,41 @@ plainly rather than implying otherwise.
   holds the figures DHS published. Reproducing them is the only cheap proof a
   pipeline is right end to end.
 
+## The 2026-09-08 rebuild of the long tail
+
+An audit found twenty-one of thirty folders with fewer than six files and nine
+with no code, several of them a README and one image presented on the landing
+page as a module. What was done, so nobody re-inflates it:
+
+- **Seven tool showcases consolidated into `tool_notes/`** (Excalidraw, Kumu,
+  Observable, RawGraphs, Flourish, Power BI, Miro). Each is now a page of
+  judgement, when to use it and what goes wrong, plus its one worked file. They
+  are one landing-page row, not seven. Do not promote them back to modules;
+  the tools live on their own sites and a repository cannot hold a template
+  for them.
+- **`foundation_tools/` is gone.** It held a second, different implementation
+  of `data_validation`, `label_variables`, `replication` and
+  `survey_to_codebook`, in five-line functions that overlapped with the
+  top-level folders. The unique pieces (two `.sps` companions) were moved up
+  and the rest deleted.
+- **Four folders rebuilt as tested Python packages**, each with a plain-assert
+  suite in CI: `data_validation` (18 tests), `label_variables` (8),
+  `survey_to_codebook` (9), `replication` (4, plus an R cross-check in the R
+  job). The three share one data dictionary format (`variable,label,values,
+  min,max,allowed,required`), so a codebook generated from an XLSForm can be
+  applied as labels and used as validation rules without retyping.
+- **Five folders built out as written content**: `annotated_research` (five
+  illustrative briefs under one annotation scheme; invented programmes, no
+  real citations, said so on every page), `eval_docs`, `writing_guides`,
+  `learning_layers`, `KM_tools`.
+
+Two things caught while doing it. pandas 3 gives text columns a `str` dtype,
+so a blank check keyed on `dtype == object` silently skips every text column;
+CI here installs unpinned pandas, so it would have shipped. And the bundled
+`survey_to_codebook/input/survey.xlsx` named a choice list `hobby` that the
+form referenced as `hobbies`; the old notebook printed an empty list without
+comment, the new tool reported it on first run, and the sample is fixed.
+
 ## network_effects_sni
 
 Rewritten 2026-09-08 from four scripts that could not run as a chain: one wrote
@@ -185,8 +226,10 @@ older version of this table said InsightStack used `minima` and EquityStack was
 unpublished; both were true once and neither is now.
 
 The six calculators are the largest design surface in the stack family and the
-obvious place to start. Beyond the stacks: SignalStack, Experiments,
-openstacks.dev and the ImpactMojo properties.
+obvious place to start. Beyond the stacks: Experiments, openstacks.dev and the
+ImpactMojo properties. SignalStack is dead and ViewStack, BridgeStack and
+RootStack are archived; the live family is InsightStack, FieldStack,
+EquityStack and PolicyStack under OpenStacks-for-Change.
 
 One constraint that catches people: `Experiments` serves under a strict Content
 Security Policy allowlisting specific CDNs, so a design pulling fonts or scripts

@@ -1,17 +1,19 @@
-# 🧰 KM Tools
+# KM_tools
 
-This folder contains real, ready-to-use knowledge management tools that support better file organization, documentation workflows, and internal communication.
+Conventions for organising a project's files so that a location says what a
+file is and a newcomer can find last year's endline without asking.
 
----
+| File | What it is |
+|---|---|
+| `folder_structure_template.md` | One structure, numbered in workflow order, with the rules that make it hold: raw data never edited, every output generated by a script |
+| `readme_template.md` | The two-page README a project folder starts with: people, where things are, codes, how to run the analysis |
+| `file_naming_conventions.md` | `project_doctype_scope_version` and why |
+| `tagging_structure_guidance.md` | Tag axes for a document store: theme, geography, stage, type, audience |
+| `documentation_flowchart.pdf` | The path from draft to final for a brief, deck or report |
 
-## 📄 Included Files
-
-- `file_naming_conventions.md` — A lightweight standard to organize files by purpose, version, and geography
-- `tagging_structure_guidance.md` — A guidance note on tagging files, folders, and KM entries in systems like Dropbox, Notion, or Baserow
-- `documentation_flowchart.pdf` — A sample documentation flow from draft creation to final output (brief, deck, or report)
-
----
-
-These tools are especially helpful when you're setting up shared drives, internal documentation processes, or collaborative writing flows with field teams.
-
-All tools are under MIT — adapt as needed!
+Two rules do most of the work. Raw data is never edited; anything wrong with
+it is fixed by a script that writes somewhere else. And every table and
+figure in a report was produced by a script that is still in the folder. A
+project that keeps those two can be handed over, replicated
+(`../replication/`), and audited. One that does not has a set of files that
+only one person understands.
@@ -2,19 +2,19 @@
 
 Effective file naming helps improve searchability, version control, and team coordination.
 
-### 📂 Recommended Structure
+### Recommended Structure
 
 ```
 [project/code]_[doc_type]_[location/scope]_[version/date].[ext]
 ```
 
-### ✅ Examples
+### Examples
 
 - `ihcrf_blog_khunti_v01.docx`
 - `gain_report_foodculture2024_final.pdf`
 - `mohfw_melframework_ncds_draft_2025-04-01.xlsx`
 
-### 🔁 Tips
+### Tips
 
 - Use lowercase and underscores for clarity
 - Include dates in `YYYY-MM-DD` format for sorting

@@ -0,0 +1,60 @@
+# Folder structure for a programme or research project
+
+One structure, used from the first week, so that a file's location says what
+it is and a new team member can find last year's endline without asking.
+
+```
+project_code/
+  00_admin/            contracts, budgets, MoUs, approvals
+  01_design/           proposal, theory of change, MEL framework, logframe
+  02_instruments/      questionnaires by version, XLSForms, codebooks, consent forms
+  03_data/
+    raw/               exactly as received; never edited; read-only where the system allows
+    clean/             produced from raw by a script in 05_code; can be deleted and regenerated
+    external/          published data used alongside (census, NFHS, admin data), with source and date
+  04_field/            training materials, field manuals, daily reports, back-check results
+  05_code/             every script that turns raw into clean and clean into outputs
+  06_outputs/
+    tables/            generated, not hand-edited
+    figures/
+    reports/           drafts and finals, versioned by name (see file_naming_conventions.md)
+  07_dissemination/    briefs, decks, press, published versions
+  08_archive/          anything superseded, with a note saying by what
+  README.md            what this project is, who runs it, where things are, what the codes mean
+```
+
+## The rules that make it work
+
+**Raw data is never edited.** Anything wrong with it is fixed by a script in
+`05_code/` that writes to `03_data/clean/`. The fix is then visible,
+repeatable, and reversible. A hand-edited raw file is a raw file nobody can
+trust and a change nobody can find.
+
+**Everything in `06_outputs/` was generated by something in `05_code/`.** If a
+table was made by hand in a spreadsheet, it lives in `08_archive/` with a
+note, and a scripted version replaces it. This is the rule most often broken
+and the one that makes a replication package (`../../replication/`) possible
+at all.
+
+**`03_data/external/` records where each file came from and when.** A census
+table with no source and no date is a number with no provenance.
+
+**The README is written in the first week and updated when the structure
+changes.** It says what the project code means, who to ask, and where the
+things are that are not obvious. `readme_template.md` is a starting point.
+
+**Numbered top-level folders sort in workflow order**, which is the order a
+newcomer needs them in.
+
+## What goes where, when in doubt
+
+| It is | It goes in |
+|---|---|
+| A questionnaire | `02_instruments/`, with a version in the file name |
+| The CSV the platform exported | `03_data/raw/`, untouched |
+| The script that cleaned it | `05_code/` |
+| The cleaned file | `03_data/clean/`, regenerable |
+| A chart | `06_outputs/figures/`, from a script |
+| The report the chart is in | `06_outputs/reports/` |
+| The two-page brief | `07_dissemination/` |
+| Last year's version of the questionnaire | `02_instruments/`, older version number; or `08_archive/` if it will not be used again |
@@ -0,0 +1,50 @@
+# [Project code]: [one-line description]
+
+*Copy this file to the root of a new project folder and fill it in. Delete the
+italic instructions. Keep it under two pages; it is the page a new team member
+reads first.*
+
+## What this is
+
+*Two or three sentences. What the programme or study does, for whom, where,
+and over what period.*
+
+## People
+
+| Role | Name | Contact |
+|---|---|---|
+| Programme lead | | |
+| MEL lead | | |
+| Data manager | | |
+| Field coordinator | | |
+
+## Where things are
+
+*The things that are not obvious from the folder structure
+(`folder_structure_template.md`).*
+
+- Raw data: `03_data/raw/`, received on [date] from [source]
+- The cleaning script: `05_code/01_clean.py`; run it with `[command]`
+- The current questionnaire: `02_instruments/[file]`, version [n]
+- The MEL framework: `01_design/[file]`
+- Approvals: `00_admin/[file]`
+
+## Codes and conventions
+
+- Project code: `[code]`, used as the prefix in every file name
+  (`file_naming_conventions.md`)
+- Site identifiers: *[what the id looks like, and where the list is]*
+- Missing value codes: *[e.g. -999 refused, -998 do not know]*
+- Rounds: *[baseline = 2024-07, endline = 2025-03]*
+
+## How to run the analysis
+
+*The commands, in order, that go from raw data to the tables in the report.
+If there is a `run_all` script, name it.*
+
+## Status
+
+*What is done, what is in progress, what is next. Dated. Update when it
+changes.*
+
+Last updated: [date] by [initials]
@@ -4,7 +4,7 @@ For internal drives, Notion, Baserow, or Dropbox-style KM systems, use structure
 
 ---
 
-## 🏷 Recommended Tag Axes
+## Recommended Tag Axes
 
 - **Thematic**: health, gender, education, foodculture, WEE, climate
 - **Geographic**: india, assam, jharkhand, district-specific
@@ -14,7 +14,7 @@ For internal drives, Notion, Baserow, or Dropbox-style KM systems, use structure
 
 ---
 
-## 📌 Sample Tags
+## Sample Tags
 
 ```
 ["health", "gender", "madhya_pradesh", "learning", "brief", "donor"]
@@ -23,7 +23,7 @@ For internal drives, Notion, Baserow, or Dropbox-style KM systems, use structure
 
 ---
 
-## 📁 Suggested Usage
+## Suggested Usage
 
 - As labels in Notion databases
 - In metadata columns in Excel/Baserow

@@ -30,36 +30,33 @@ This is the **knowledge systems layer** of [OpenStacks for Change](https://opens
 |-----------|-------------|----------|--------|
 | `data_starters/` | Loaders for real public-use survey microdata, one folder per dataset. DHS across South Asia, and PLFS | Stata, R, Python | Ready |
 | `calculators/` | District-level calculators for health, education, finance, environment, and population | HTML, JavaScript | Ready |
-| `data_validation/` | Data cleaning and validation workflows with intentionally messy test data | Python, R, Stata | Ready |
+| `data_validation/` | Rule-driven validation from a data dictionary: duplicates, required, ranges, allowed values, types, name style, column set, cross-file ids; one issue row per problem, by identifier | Python (18 tests); Stata, R, SPSS companions |
 | `stata_snippets/` | Reusable Stata code: data management, graphs, regression, impact evaluation, surveys | Stata | Ready |
 | `spss_scripts/` | Survey analysis syntax: cleaning, regression, missing data, frequencies | SPSS | Ready |
 | `network_effects_sni/` | Peer effects estimation, centrality analysis, diffusion modelling for SHG networks | Python | Ready |
-| `replication/` | Replication template with parallel Python and R implementations | Python, R | Ready |
-| `label_variables/` | Variable labelling and codebook generation | Stata | Ready |
-| `survey_to_codebook/` | Survey instrument to codebook conversion | Python | Ready |
+| `replication/` | A replication package that verifies itself: one entry point, a recorded result, a Python-against-R cross-check, and a test that the verification can fail | Python, R (4 tests) |
+| `label_variables/` | Variable and value labels from a dictionary, written into `.dta` and `.sav` where they survive; round-trips both ways | Python (8 tests); Stata, R, SPSS companions |
+| `survey_to_codebook/` | XLSForm to Markdown codebook and to a label dictionary; resolves groups, repeats and choice lists; reports form defects | Python (9 tests) |
 | `econometrics/` | Causal inference: DiD, PSM, IV/2SLS, RDD, sensitivity analysis — with Python, R, and sample data | Python, R | Ready |
 
 ### Visual and Interactive Tools
 
 | Directory | What It Does | Tool |
 |-----------|-------------|------|
 | `vensim/` | 8 system dynamics models (health, agriculture, climate, migration, education) | Vensim |
-| `observable_notebooks/` | Interactive district vulnerability dashboard | Observable |
-| `excalidraw_frameworks/` | MEL framework and ecosystem map diagrams | Excalidraw |
-| `flourish_charts/` | Data visualisation templates | Flourish |
-| `miro/` | Participatory analysis board templates | Miro |
-| `kumu_maps/` | Network and systems mapping | Kumu |
-| `powerbi_reports/` | Report templates | Power BI |
+| `tool_notes/` | Seven visual and collaboration tools (Excalidraw, Kumu, Observable, RawGraphs, Flourish, Power BI, Miro): when to use each, when not, what goes wrong, one worked file apiece | Notes |
 | `excel_visuals/` | Excel visualisation templates | Excel |
 
 ### Knowledge and Documentation
 
 | Directory | What It Contains |
 |-----------|-----------------|
 | `learning_library/` | Curated PDFs across 10 categories: programming, data science, research methods, AI tools, MLE resources |
-| `writing_guides/` | Style guides and writing templates |
-| `annotated_research/` | Annotated bibliography and research notes |
-| `learning_layers/` | Learning design frameworks |
+| `writing_guides/` | Theory of change, results chain, evaluation report structure, policy brief, writing about uncertainty; the verb is decided by the design | Markdown |
+| `eval_docs/` | Logframe template, indicator reference sheet, MEL framework outline, indicator guidance, with a completed example | Markdown, CSV, XLSX |
+| `KM_tools/` | Folder structure, project README template, file naming, tagging | Markdown |
+| `annotated_research/` | Five research briefs marked up with one eight-tag annotation scheme, weakest design to strongest; illustrative, for teaching critical reading | Markdown |
+| `learning_layers/` | Embedding learning in the MEL cycle as scheduled decisions at three tempos | Markdown, PDF |
 | `taguette_coding/` | Qualitative coding workflows |
 | `visual_ethnography_descript/` | Ethnographic documentation tools |
 | `latex/` | LaTeX templates for research reports |
@@ -87,9 +84,9 @@ InsightStack is one of several stacks in the [OpenStacks](https://openstacks.dev
 | Stack | Focus |
 |-------|-------|
 | **InsightStack** (this repo) | MEL tools, calculators, documentation |
-| [FieldStack](https://github.com/Varnasr/FieldStack) | R notebooks for fieldwork and evaluation |
-| [EquityStack](https://github.com/Varnasr/EquityStack) | Python workflows for development data |
-| [SignalStack](https://github.com/Varnasr/SignalStack) | Research Rundown newsletter archive |
+| [FieldStack](https://github.com/Varnasr/FieldStack) | Field operations and survey analysis in R |
+| [EquityStack](https://github.com/Varnasr/EquityStack) | Distributional analysis and survey estimation in Python |
+| [PolicyStack](https://github.com/Varnasr/PolicyStack) | 15 flagship schemes, four years of budget data, performance indicators; superseded for new work by [PolicyDhara](https://github.com/Varnasr/PolicyDhara) |
 
 ## Contributing