diff --git a/ISSUE_TEMPLATE/repo-readiness-checklist.md b/ISSUE_TEMPLATE/repo-readiness-checklist.md
index 6ff4ecb..671c6b1 100644
--- a/ISSUE_TEMPLATE/repo-readiness-checklist.md
+++ b/ISSUE_TEMPLATE/repo-readiness-checklist.md
@@ -7,7 +7,7 @@ assignees: ''
---
-# ๐๏ธ Repo Hygiene Checklist
+# Repo Hygiene Checklist
- [ ] LICENSE.md present (MIT)
- [ ] README.md complete (badges, overview, folders)
diff --git a/KM_tools/README.md b/KM_tools/README.md
index 1e2e8f9..6423fc3 100644
--- a/KM_tools/README.md
+++ b/KM_tools/README.md
@@ -1,19 +1,14 @@
# KM_tools
-Conventions for organising a project's files so that a location says what a
-file is and a newcomer can find last year's endline without asking.
+Conventions for organising a project's files.
| File | What it is |
-|---|---|
-| `folder_structure_template.md` | One structure, numbered in workflow order, with the rules that make it hold: raw data never edited, every output generated by a script |
-| `readme_template.md` | The two-page README a project folder starts with: people, where things are, codes, how to run the analysis |
-| `file_naming_conventions.md` | `project_doctype_scope_version` and why |
+| --- | --- |
+| `folder_structure_template.md` | One structure, numbered in workflow order |
+| `readme_template.md` | A two-page README for a project folder: people, where things are, codes, how to run the analysis |
+| `file_naming_conventions.md` | `project_doctype_scope_version` |
| `tagging_structure_guidance.md` | Tag axes for a document store: theme, geography, stage, type, audience |
| `documentation_flowchart.pdf` | The path from draft to final for a brief, deck or report |
-Two rules do most of the work. Raw data is never edited; anything wrong with
-it is fixed by a script that writes somewhere else. And every table and
-figure in a report was produced by a script that is still in the folder. A
-project that keeps those two can be handed over, replicated
-(`../replication/`), and audited. One that does not has a set of files that
-only one person understands.
+Two rules: raw data is never edited, and every table and figure in a report
+comes from a script that is still in the folder.
diff --git a/README.md b/README.md
index 5f23798..4170a28 100644
--- a/README.md
+++ b/README.md
@@ -1,117 +1,92 @@
# InsightStack
-**MEL tools, calculators, and research documentation for development work.**
+MEL tools, calculators, research documentation and loaders for survey
+microdata, in Stata, Python, R and SPSS. Part of
+[OpenStacks](https://openstacks.dev). Status: Stable, per the family
+[maintenance policy](https://github.com/Varnasr/OpenStacks-for-Change/blob/main/MAINTENANCE.md).
+DOI: [10.5281/zenodo.15245182](https://doi.org/10.5281/zenodo.15245182).
+
+Site: [varnasr.github.io/InsightStack](https://varnasr.github.io/InsightStack/).
+
+## Code and data
+
+| Directory | What it does | Language |
+| --- | --- | --- |
+| `data_starters/` | Loaders for DHS (28 surveys, seven South Asian countries) and PLFS (India) that open the files as distributed, with the weight and scale rules applied. Tested on synthetic fixtures; no data ships | Stata, R, Python |
+| `calculators/` | Six browser calculators for district planning: population projection, school needs, banking access, district health indicators, block-level estimates, environmental fragility | HTML, JavaScript |
+| `econometrics/` | Difference-in-differences, propensity score matching, instrumental variables, regression discontinuity, sensitivity analysis, with sample data | Python, R |
+| `stata_snippets/` | 44 do-files: data management, descriptives, regression, impact evaluation, graphs, survey settings | Stata |
+| `spss_scripts/` | 20 files: cleaning, recoding, tabulation, regression, Excel export | SPSS |
+| `network_effects_sni/` | Centrality, peer association with leave-one-out means, and threshold diffusion for self-help group data. 17 tests | Python |
+| `data_validation/` | Checks from a data dictionary: duplicates, required fields, ranges, allowed values, types, column set, cross-file ids. One row per problem. 18 tests | Python; Stata, R, SPSS companions |
+| `label_variables/` | Variable and value labels from a dictionary, written into `.dta` and `.sav`. 8 tests | Python; Stata, R, SPSS companions |
+| `survey_to_codebook/` | An XLSForm to a Markdown codebook and a label dictionary. 9 tests | Python |
+| `replication/` | A replication package with one entry point, a recorded result and an R cross-check. 4 tests | Python, R |
+| `vensim/` | Eight system dynamics models | Vensim |
+
+## Documents and notes
+
+| Directory | What it holds |
+| --- | --- |
+| `tool_notes/` | Nine notes on visual and document tools (Excalidraw, Kumu, Observable, RawGraphs, Flourish, Power BI, Miro, Excel, LaTeX), each with one example file |
+| `writing_guides/` | Theory of change, results chain, evaluation report structure, policy brief, writing about uncertainty |
+| `eval_docs/` | Logframe, indicator reference sheet and MEL framework templates, with a completed example |
+| `annotated_research/` | Five illustrative research briefs annotated with one eight-tag scheme |
+| `learning_layers/` | Learning in the MEL cycle as scheduled decisions |
+| `KM_tools/` | Folder structure, README template, file naming, tagging |
+| `taguette_coding/` | One focus group coded in Taguette, with the method |
+| `visual_ethnography_descript/` | Recorded interviews: consent, transcription, clips, quotes |
+| `learning_library/` | PDFs and guides in five categories |
+
+## Tests
-[](https://openstacks.dev)
-[](LICENSE)
-[](https://doi.org/10.5281/zenodo.15245182)
-[](https://github.com/Varnasr/OpenStacks-for-Change/blob/main/MAINTENANCE.md)
-
-> In development work, we talk about knowledge โ but rarely structure it. InsightStack organises the tools, templates, and workflows that make research, evaluation, and program design actually work.
-
-> **Status: Stable.** This repository works and is correct, but it is not under active
-> development. Bug reports are welcome and issues stay open; new features are unlikely,
-> and replies are measured in weeks rather than days. Dependencies are pinned deliberately
-> so that a clone still runs years from now. See the [maintenance policy](https://github.com/Varnasr/OpenStacks-for-Change/blob/main/MAINTENANCE.md).
-
----
-
-## What This Is
-
-InsightStack is a collection of practical scripts, templates, and tools for **monitoring, evaluation, and learning (MEL)** work in the development sector. It covers data validation, survey analysis, system dynamics modelling, network analysis, qualitative coding, and visual storytelling.
-
-This is the **knowledge systems layer** of [OpenStacks for Change](https://openstacks.dev) โ an open ecosystem of tools for public interest research and evaluation.
-
-## What's Inside
-
-### Analysis Tools
-
-| Directory | What It Does | Language | Status |
-|-----------|-------------|----------|--------|
-| `data_starters/` | Loaders for real public-use survey microdata, one folder per dataset. DHS across South Asia, and PLFS | Stata, R, Python | Ready |
-| `calculators/` | District-level calculators for health, education, finance, environment, and population | HTML, JavaScript | Ready |
-| `data_validation/` | Rule-driven validation from a data dictionary: duplicates, required, ranges, allowed values, types, name style, column set, cross-file ids; one issue row per problem, by identifier | Python (18 tests); Stata, R, SPSS companions |
-| `stata_snippets/` | Reusable Stata code: data management, graphs, regression, impact evaluation, surveys | Stata | Ready |
-| `spss_scripts/` | Survey analysis syntax: cleaning, regression, missing data, frequencies | SPSS | Ready |
-| `network_effects_sni/` | Peer effects estimation, centrality analysis, diffusion modelling for SHG networks | Python | Ready |
-| `replication/` | A replication package that verifies itself: one entry point, a recorded result, a Python-against-R cross-check, and a test that the verification can fail | Python, R (4 tests) |
-| `label_variables/` | Variable and value labels from a dictionary, written into `.dta` and `.sav` where they survive; round-trips both ways | Python (8 tests); Stata, R, SPSS companions |
-| `survey_to_codebook/` | XLSForm to Markdown codebook and to a label dictionary; resolves groups, repeats and choice lists; reports form defects | Python (9 tests) |
-| `econometrics/` | Causal inference: DiD, PSM, IV/2SLS, RDD, sensitivity analysis โ with Python, R, and sample data | Python, R | Ready |
-
-### Visual and Interactive Tools
-
-| Directory | What It Does | Tool |
-|-----------|-------------|------|
-| `vensim/` | 8 system dynamics models (health, agriculture, climate, migration, education) | Vensim |
-| `tool_notes/` | Nine tool notes (Excalidraw, Kumu, Observable, RawGraphs, Flourish, Power BI, Miro, Excel, LaTeX): when each is worth the setup, when it is not, what goes wrong, with one worked file each | Notes |
-
-### Knowledge and Documentation
-
-| Directory | What It Contains |
-|-----------|-----------------|
-| `learning_library/` | Curated PDFs across 10 categories: programming, data science, research methods, AI tools, MLE resources |
-| `writing_guides/` | Theory of change, results chain, evaluation report structure, policy brief, writing about uncertainty; the verb is decided by the design | Markdown |
-| `eval_docs/` | Logframe template, indicator reference sheet, MEL framework outline, indicator guidance, with a completed example | Markdown, CSV, XLSX |
-| `KM_tools/` | Folder structure, project README template, file naming, tagging | Markdown |
-| `annotated_research/` | Five research briefs marked up with one eight-tag annotation scheme, weakest design to strongest; illustrative, for teaching critical reading | Markdown |
-| `learning_layers/` | Embedding learning in the MEL cycle as scheduled decisions at three tempos | Markdown, PDF |
-| `taguette_coding/` | One focus group coded start to finish in Taguette, with the method: frame first, double-code two transcripts, count carefully | Qualitative |
-| `visual_ethnography_descript/` | From recording to clip to quote: three consents, quotes checked against audio, clips cut with context visible | Qualitative |
-| `workflows/` | End-to-end analysis workflow guides |
-
-## Getting Started
-
-1. **Browse by need** โ Each directory is self-contained. Pick the tool that matches your task.
-2. **Check the sample data** โ Use `sample_data/` and the test data in `data_validation/` to try scripts.
-3. **Read the workflows** โ `workflows/` ties multiple tools together into end-to-end processes.
-
-### Prerequisites
-
-Different tools require different software:
-- **Stata 15+** for Stata scripts and snippets
-- **Python 3.8+** with pandas, statsmodels, networkx for Python scripts
-- **R 4.0+** with tidyverse for R scripts
-- **SPSS** for survey analysis syntax
-- **Observable**, **Flourish**, **Miro**, **Kumu** accounts for visual tools (free tiers available)
-
-## How It Connects
+```
+cd data_starters/dhs-south-asia && python make_fixture.py --outdir fixtures
+python test_load_dhs.py && Rscript test_load_dhs.R
+cd ../plfs-india && python make_fixture.py --outdir fixtures
+python test_load_plfs.py && Rscript test_load_plfs.R
+python -m network_effects_sni.test_network_effects
+python data_validation/test_data_validation.py
+python label_variables/test_label_variables.py
+python survey_to_codebook/test_survey_to_codebook.py
+python replication/test_replication.py
+```
-InsightStack is one of several stacks in the [OpenStacks](https://openstacks.dev) ecosystem:
+All of these run in CI on every pull request. The Stata files are not
+tested, since Stata has no free runtime; each carries a fixture check at the
+bottom.
-| Stack | Focus |
-|-------|-------|
-| **InsightStack** (this repo) | MEL tools, calculators, research documentation, and loaders for real survey microdata. Stata, Python, R, SPSS |
-| [FieldStack](https://github.com/Varnasr/FieldStack) | Field operations while a survey is being collected, and survey analysis after. R |
-| [EquityStack](https://github.com/Varnasr/EquityStack) | Equity from a development economics perspective: distributional analysis, the concentration index, and design-based survey estimation. Python |
+## Requirements
-[openstacks.dev](https://openstacks.dev) is the index for all of it. [SignalStack](https://github.com/Varnasr/SignalStack) is the companion archive for the [Research Rundown](https://varna.substack.com) newsletter, alongside the stacks rather than one of them. [PolicyStack](https://github.com/Varnasr/PolicyStack) is superseded by [PolicyDhara](https://github.com/Varnasr/PolicyDhara). RootStack, BridgeStack and ViewStack are archived.
+Stata 15 or later for the do-files. Python 3.8 or later with pandas,
+statsmodels and networkx. R 4.0 or later with tidyverse, survey and haven.
+SPSS for the syntax files.
-## Contributing
+## The family
-Contributions welcome โ especially from practitioners who use these tools in real fieldwork. See [contributing guidelines](https://github.com/Varnasr/.github/blob/main/CONTRIBUTING.md) for guidelines.
+| Repository | What it is for | Language |
+| --- | --- | --- |
+| **InsightStack** (this repository) | MEL tools, calculators, research documentation, loaders for survey microdata | Stata, Python, R, SPSS |
+| [FieldStack](https://github.com/Varnasr/FieldStack) | Field operations while a survey is in the field; sampling and weighted estimation after | R |
+| [EquityStack](https://github.com/Varnasr/EquityStack) | Inequality measurement and design-based survey estimation | Python |
-High-impact areas:
-- **Econometrics** โ causal inference implementations (DiD, PSM, IV, RDD) in Python, R, or Stata
-- **Calculators** โ new district-level planning tools
-- **Stata/SPSS scripts** โ analysis templates for common MEL tasks
-- **Sample data** โ synthetic datasets for testing
+[openstacks.dev](https://openstacks.dev) is the index.
+[SignalStack](https://github.com/Varnasr/SignalStack) is the companion archive
+for the [Research Rundown](https://varna.substack.com) newsletter, beside the
+stacks rather than one of them.
+[PolicyStack](https://github.com/Varnasr/PolicyStack) is superseded by
+[PolicyDhara](https://github.com/Varnasr/PolicyDhara). RootStack, BridgeStack
+and ViewStack are archived.
-## Citation
+## Citation and license
```bibtex
@software{insightstack,
author = {Sri Raman, Varna},
- title = {InsightStack: MEL Tools for Development Work},
+ title = {InsightStack: MEL tools, calculators and survey data loaders},
url = {https://github.com/Varnasr/InsightStack},
doi = {10.5281/zenodo.15245182}
}
```
-## License
-
-MIT โ free to use, modify, and share. See [LICENSE](LICENSE).
-
----
-
-Part of [OpenStacks for Change](https://openstacks.dev). Created by [Varna Sri Raman](https://on-web.link/varna).
+MIT. See [LICENSE](LICENSE).
diff --git a/annotated_research/README.md b/annotated_research/README.md
index bca92f4..d508fdf 100644
--- a/annotated_research/README.md
+++ b/annotated_research/README.md
@@ -1,38 +1,25 @@
# annotated_research
-Five research briefs marked up to show where they are strong and where they
-are wrong, with one annotation scheme applied to all of them.
-
-The point is the scheme. Eight tags, each a question to put to any research
-note that lands on your desk: what is it really asking, what is the
-comparison, who is in the data, what was measured, is the arithmetic right for
-the design, does it name its real weaknesses, does the conclusion's verb match
-the design, and who consented. `annotation_scheme.md` defines them and says
-how to teach with them.
+Five research briefs marked up with one annotation scheme. Eight tags, each
+a question to put to a research note: what is it asking, what is the
+comparison, who is in the data, what was measured, is the arithmetic right
+for the design, does it name its weaknesses, does the conclusion's verb match
+the design, who consented. `annotation_scheme.md` defines them.
| Brief | Design | Main lesson |
-|---|---|---|
+| --- | --- | --- |
| `briefs/01_school_health_clubs.md` | Before-after, one arm | A year of growing up is inside the estimate |
| `briefs/02_remedial_reading.md` | Difference-in-differences, matched | The baseline table contradicts the identifying assumption |
| `briefs/03_flood_early_warning.md` | Post-event cross-section | The households not interviewed are the outcome |
| `briefs/04_shg_savings.md` | Randomised by village | Differential attrition and unclustered errors on a good design |
-| `briefs/05_cash_transfer_nutrition.md` | Regression discontinuity | A credible local estimate, extended past the population it identifies |
-
-Ordered from weakest design to strongest, so a reader who works through them
-in sequence sees the ceiling on what each design can claim rise, and the
-errors change from fatal to specific.
-
-**All five are illustrative.** The programmes, districts, organisations and
-numbers are invented to make the teaching points. No brief describes a real
-study or cites one. They are written the way such notes are actually written,
-which is why each contains the errors real notes contain.
+| `briefs/05_cash_transfer_nutrition.md` | Regression discontinuity | A credible local estimate extended past the population it identifies |
-The original `annotated_research_brief_mhh.pdf` is the example this folder
-started with and is kept as it was.
+Ordered from weakest design to strongest.
-## Using them
+All five are illustrative. The programmes, districts, organisations and
+numbers are invented. No brief describes or cites a real study. The original
+`annotated_research_brief_mhh.pdf` is kept as it was.
-Hand out a brief without its annotations. Have readers tag it themselves
-using the scheme, then compare. Where two readers tag the same sentence
-differently, the text is ambiguous, and the ambiguity is usually where the
-misleading claim lives.
+To teach with them, hand out a brief without its annotations, have readers
+tag it with the scheme, and compare. Where two readers tag the same sentence
+differently, the text is ambiguous.
diff --git a/calculators/README.md b/calculators/README.md
index c3b5cb0..a46a001 100644
--- a/calculators/README.md
+++ b/calculators/README.md
@@ -1,35 +1,18 @@
-# ๐งฎ Calculators for District and Program Planning
+# Calculators
-This folder contains practical tools for estimating health, education, infrastructure, and population needs at the district or block level โ especially in data-scarce environments.
+Six browser calculators for district and block planning. Each folder holds
+an HTML file that runs locally, a note on the logic, and a README.
-These calculators are designed for use by:
-- Program managers and MEL teams
-- District planners or consultants
-- NGOs and policy support groups
-- Fellows or research trainees
+| Folder | What it calculates |
+| --- | --- |
+| `population_projection/` | Future population at a constant annual growth rate |
+| `education_needs/` | Schools needed from projected school-age population |
+| `district_health_indicators/` | A district estimate from a state NFHS figure and three multipliers |
+| `financial_inclusion/` | Banking access points against the number a population implies |
+| `block_level_development/` | A district value split to blocks by population share |
+| `environmental_change/` | A fragility score from rainfall, forest and water indicators |
-Each subfolder includes:
-- A working HTML calculator
-- A markdown file explaining the logic
-- A usage README
+Each calculator states its assumptions on the page. Where a published
+district figure exists, use that.
----
-
-## ๐ Included Tools
-
-| Folder | What it Calculates |
-|--------|--------------------|
-| `education_needs/` | Projects number of schools needed based on population growth |
-| `district_health_indicators/` | Estimates health indicators by adjusting NFHS data using local context |
-| `financial_inclusion/` | Computes bank access gaps based on population norms |
-| `environmental_change/` | Flags environmental fragility using rainfall, forest, and water indicators |
-| `block_level_development/` | Disaggregates district-level values to blocks using population share |
-| `population_projection/` | Forecasts future population using annual growth %
-
----
-
-Each calculator is **real**, **modular**, and **customizable**. They can be embedded, adapted, or scaled for dashboards, training kits, or planning exercises.
-
-MIT licensed. Attribution appreciated.
-
-Made with โฅ by [Varna Sri Raman](https://on-web.link/varna)
+MIT licence.
diff --git a/calculators/block_level_development/README.md b/calculators/block_level_development/README.md
index 9621ef8..062236b 100644
--- a/calculators/block_level_development/README.md
+++ b/calculators/block_level_development/README.md
@@ -1,10 +1,10 @@
-# ๐งฑ Block-Level Development Estimator
+# Block-Level Development Estimator
This tool helps estimate indicator values at the block level using population-based shares when district-level data is available but block data is missing.
---
-## ๐ฆ Files
+## Files
- `block_level_development_calculator.html`: Interactive calculator
- `block_level_development_logic.md`: Explanation of the estimation method
@@ -12,7 +12,7 @@ This tool helps estimate indicator values at the block level using population-ba
---
-## ๐ง How It Works
+## How It Works
Enter:
- Total population of the district
@@ -23,4 +23,4 @@ The tool will estimate the block-level equivalent using a simple proportional sh
---
-๐ Assumes uniform distribution across blocks unless better data is available.
+ Assumes uniform distribution across blocks unless better data is available.
diff --git a/calculators/block_level_development/block_level_development_logic.md b/calculators/block_level_development/block_level_development_logic.md
index b8afe93..727020f 100644
--- a/calculators/block_level_development/block_level_development_logic.md
+++ b/calculators/block_level_development/block_level_development_logic.md
@@ -1,10 +1,10 @@
-# ๐งฑ Block-Level Development Estimation โ Logic
+# Block-Level Development Estimation โ Logic
This tool helps estimate block-level indicator values by disaggregating district-level data using population or area shares.
---
-## ๐ข Inputs
+## Inputs
- Total District Population
- Block Population
@@ -12,7 +12,7 @@ This tool helps estimate block-level indicator values by disaggregating district
---
-## ๐ Disaggregation Logic
+## Disaggregation Logic
### Method 1: Population-Based Disaggregation
@@ -32,7 +32,7 @@ Block Estimate = District Value ร Block Area Share
---
-## ๐ง Example
+## Example
District has 600,000 population and 30 PHCs. A block with 75,000 population:
diff --git a/calculators/district_health_indicators/README.md b/calculators/district_health_indicators/README.md
index fb90aed..859b496 100644
--- a/calculators/district_health_indicators/README.md
+++ b/calculators/district_health_indicators/README.md
@@ -1,10 +1,10 @@
-# ๐ฉบ District Health Indicator Estimator
+# District Health Indicator Estimator
This tool helps estimate district-level health indicators using state NFHS data and contextual adjustments.
---
-## ๐ฆ Files
+## Files
- `health_indicators_calculator.html`: Calculator to adjust state NFHS values using contextual multipliers
- `health_indicators_logic.md`: Markdown explanation of how the projection logic works
@@ -12,20 +12,20 @@ This tool helps estimate district-level health indicators using state NFHS data
---
-## ๐ง How It Works
+## How It Works
You enter:
- State-level value (e.g., % of institutional deliveries from NFHS)
- Contextual inputs:
- - Health infrastructure (low/average/high)
- - Road access (poor/average/good)
- - Tribal/SC/ST population density (high/medium/low)
+ - Health infrastructure (low/average/high)
+ - Road access (poor/average/good)
+ - Tribal/SC/ST population density (high/medium/low)
The calculator multiplies the state value by three factors to provide a district-specific estimate.
---
-๐ Example: Chitrakoot, Uttar Pradesh
+ Example: Chitrakoot, Uttar Pradesh
Assume state value is 72.4% for institutional deliveries, with:
- Low infra (0.9)
- Average roads (1.0)
diff --git a/calculators/district_health_indicators/health_indicators_logic.md b/calculators/district_health_indicators/health_indicators_logic.md
index 201373b..320b463 100644
--- a/calculators/district_health_indicators/health_indicators_logic.md
+++ b/calculators/district_health_indicators/health_indicators_logic.md
@@ -1,20 +1,20 @@
-# ๐ฉบ District Health Indicator Projection โ General Logic
+# District Health Indicator Projection โ General Logic
This tool helps estimate district-level health indicators (e.g., immunization, maternal health) when only state-level NFHS data is available. It uses adjustment factors based on known district context.
---
-## ๐ข Inputs
+## Inputs
1. **State-level NFHS value** (e.g., institutional deliveries = 72.4%)
2. **Contextual Adjustments**:
- - Health infrastructure (low / average / high)
- - Road access (poor / average / good)
- - Demographics (e.g. high tribal/SC/ST %)
+ - Health infrastructure (low / average / high)
+ - Road access (poor / average / good)
+ - Demographics (e.g. high tribal/SC/ST %)
---
-## ๐ How the Adjustment Works
+## How the Adjustment Works
We multiply the state value by an adjustment factor based on local context. For example:
@@ -24,17 +24,17 @@ District Estimate = State Value ร Adjustment Factor
Each context input adjusts the factor slightly up or down:
-| Factor | Low | Avg | High |
+| Factor | Low | Avg | High |
|-----------------------|---------|---------|---------|
-| Health Infra | 0.9 | 1.0 | 1.1 |
-| Road Access | 0.95 | 1.0 | 1.05 |
-| Tribal/SC/ST Density | 0.9 | 1.0 | 1.1 |
+| Health Infra | 0.9 | 1.0 | 1.1 |
+| Road Access | 0.95 | 1.0 | 1.05 |
+| Tribal/SC/ST Density | 0.9 | 1.0 | 1.1 |
The final adjustment is the product of all applicable factors.
---
-## ๐ง Example
+## Example
NFHS State Value (Institutional Deliveries): 72.4%
Khunti has low infra (0.9), avg roads (1.0), and high tribal % (0.9):
diff --git a/calculators/education_needs/README.md b/calculators/education_needs/README.md
index da79704..edfe15a 100644
--- a/calculators/education_needs/README.md
+++ b/calculators/education_needs/README.md
@@ -1,10 +1,10 @@
-# ๐งฎ Education Needs Calculator
+# Education Needs Calculator
This tool estimates the number of schools needed in a district over a given period, based on growth in the school-age population (6โ14 years).
---
-## ๐ฆ Files
+## Files
- `education_calculator.html`: Lightweight calculator (no login, no libraries needed)
- `education_logic.md`: Full logic breakdown with formulas and generalization steps
@@ -12,7 +12,7 @@ This tool estimates the number of schools needed in a district over a given peri
---
-## ๐ง How It Works
+## How It Works
You enter:
- Current school-age population
@@ -25,4 +25,4 @@ The calculator projects population, computes schools needed, and shows the infra
---
-๐ Example district used: *Chitrakoot, Uttar Pradesh*
+ Example district used: *Chitrakoot, Uttar Pradesh*
diff --git a/calculators/education_needs/education_logic.md b/calculators/education_needs/education_logic.md
index 5a7319f..fcedbde 100644
--- a/calculators/education_needs/education_logic.md
+++ b/calculators/education_needs/education_logic.md
@@ -1,54 +1,54 @@
-# ๐งฎ Education Needs Projection Calculator โ General Logic
+# Education Needs Projection Calculator โ General Logic
This tool helps districts or planners estimate how many schools are needed over a future period based on population growth and existing infrastructure.
---
-## ๐ข Inputs Required
+## Inputs Required
1. **Current school-age population** (ages 6โ14)
2. **Annual population growth rate** (e.g., 1.85%)
3. **Number of years to project** (e.g., 3 years from now)
4. **Current number of schools** by type:
- - Primary
- - Middle
- - Secondary/Sr. Secondary
+ - Primary
+ - Middle
+ - Secondary/Sr. Secondary
5. **Norms**: max students per school (e.g., 150 per primary school)
---
-## ๐ Steps Used in Projection
+## Steps Used in Projection
1. **Project population over N years**:
- ```
- Year 1: P1 = P0 ร (1 + g)
- Year 2: P2 = P1 ร (1 + g)
- ...
- Year N: PN = PN-1 ร (1 + g)
- ```
+ ```
+ Year 1: P1 = P0 ร (1 + g)
+ Year 2: P2 = P1 ร (1 + g)
+ ...
+ Year N: PN = PN-1 ร (1 + g)
+ ```
- Where:
- - P0 = current school-age population
- - g = annual growth rate (in decimal)
+ Where:
+ - P0 = current school-age population
+ - g = annual growth rate (in decimal)
2. **Estimate schools needed**:
- ```
- Schools Needed = Projected Population / Norm per School
- ```
+ ```
+ Schools Needed = Projected Population / Norm per School
+ ```
- Apply separately for each school type if needed.
+ Apply separately for each school type if needed.
3. **Calculate school gap**:
- ```
- Gap = Schools Needed - Current Number of Schools
- ```
+ ```
+ Gap = Schools Needed - Current Number of Schools
+ ```
---
-## ๐ง Example
+## Example
If you have 92,000 children aged 6โ14 today, with a 1.85% growth rate, and want to project 3 years out:
diff --git a/calculators/environmental_change/README.md b/calculators/environmental_change/README.md
index 2ba3635..c66b4c0 100644
--- a/calculators/environmental_change/README.md
+++ b/calculators/environmental_change/README.md
@@ -1,10 +1,10 @@
-# ๐ฑ Environmental Change Risk Calculator
+# Environmental Change Risk Calculator
This tool estimates environmental fragility for a district using ecological indicators like forest cover, rainfall, and temperature change.
---
-## ๐ฆ Files
+## Files
- `environmental_change_calculator.html`: Interactive calculator
- `environmental_change_logic.md`: Explanation of indicators and scoring
@@ -12,7 +12,7 @@ This tool estimates environmental fragility for a district using ecological indi
---
-## ๐ง How It Works
+## How It Works
Enter values for:
- Forest cover (%)
@@ -24,4 +24,4 @@ The calculator gives a composite score and risk category based on predefined thr
---
-๐ Example based on Bundelkhand-style exposure
+ Example based on Bundelkhand-style exposure
diff --git a/calculators/environmental_change/environmental_change_logic.md b/calculators/environmental_change/environmental_change_logic.md
index 0853ef0..793e110 100644
--- a/calculators/environmental_change/environmental_change_logic.md
+++ b/calculators/environmental_change/environmental_change_logic.md
@@ -1,10 +1,10 @@
-# ๐ฑ Environmental Change Risk Estimation โ Logic
+# Environmental Change Risk Estimation โ Logic
This tool estimates climate fragility and environmental risk levels for a district using key ecological indicators.
---
-## ๐ข Inputs
+## Inputs
- Forest Cover (% of total area)
- Households with Drinking Water Nearby (%)
@@ -13,16 +13,16 @@ This tool estimates climate fragility and environmental risk levels for a distri
---
-## ๐ Scoring Logic
+## Scoring Logic
Each variable is scored between 0โ1 and weighted to compute a composite risk score.
-| Indicator | Weight | Risk Thresholds |
+| Indicator | Weight | Risk Thresholds |
|--------------------------|--------|------------------|
-| Forest Cover (%) | 0.3 | <25% = high risk |
-| Drinking Water (%) | 0.3 | <60% = high risk |
-| Rainfall Deviation (mm) | 0.2 | >100mm = high |
-| Temperature Change (ยฐC) | 0.2 | >1.5ยฐC = high |
+| Forest Cover (%) | 0.3 | <25% = high risk |
+| Drinking Water (%) | 0.3 | <60% = high risk |
+| Rainfall Deviation (mm) | 0.2 | >100mm = high |
+| Temperature Change (ยฐC) | 0.2 | >1.5ยฐC = high |
Risk categories:
@@ -32,7 +32,7 @@ Risk categories:
---
-## ๐ง Example
+## Example
Forest cover = 21%, Water access = 55%, Rainfall dev = 120mm, Temp rise = 1.8ยฐC
diff --git a/calculators/financial_inclusion/README.md b/calculators/financial_inclusion/README.md
index 04da9ff..f637b30 100644
--- a/calculators/financial_inclusion/README.md
+++ b/calculators/financial_inclusion/README.md
@@ -1,10 +1,10 @@
-# ๐ฆ Financial Inclusion Calculator
+# Financial Inclusion Calculator
This tool estimates how many access points (banks, ATMs, Bank Mitras) are needed in a district to meet population norms for financial access.
---
-## ๐ฆ Files
+## Files
- `financial_inclusion_calculator.html`: Calculator for estimating service gaps
- `financial_inclusion_logic.md`: Explanation of how the tool works
@@ -12,7 +12,7 @@ This tool estimates how many access points (banks, ATMs, Bank Mitras) are needed
---
-## ๐ง How It Works
+## How It Works
You enter:
- Total population
@@ -23,4 +23,4 @@ The calculator compares required vs existing access points and shows whether the
---
-๐ Example District: Any in India (default data inspired by Chitrakoot, UP)
+ Example District: Any in India (default data inspired by Chitrakoot, UP)
diff --git a/calculators/financial_inclusion/financial_inclusion_logic.md b/calculators/financial_inclusion/financial_inclusion_logic.md
index 566bd68..3b05382 100644
--- a/calculators/financial_inclusion/financial_inclusion_logic.md
+++ b/calculators/financial_inclusion/financial_inclusion_logic.md
@@ -1,22 +1,22 @@
-# ๐ฆ Financial Inclusion Projection Logic
+# Financial Inclusion Projection Logic
This tool helps estimate how many banking access points (banks, ATMs, CSPs) are needed in a district based on population and service norms.
---
-## ๐ข Inputs
+## Inputs
- Current district population
- Norm: people per access point (e.g., 2,000 people per bank point)
- Current access points:
- - Commercial Banks
- - Rural Banks
- - ATMs
- - Bank Mitras / CSPs
+ - Commercial Banks
+ - Rural Banks
+ - ATMs
+ - Bank Mitras / CSPs
---
-## ๐ Formula
+## Formula
```
Needed Access Points = Population / Norm
@@ -26,7 +26,7 @@ Gap = Needed - Current
---
-## ๐ง Example
+## Example
For a district with:
- Population: 675,000
diff --git a/calculators/population_projection/README.md b/calculators/population_projection/README.md
index f727829..ffe3dbe 100644
--- a/calculators/population_projection/README.md
+++ b/calculators/population_projection/README.md
@@ -1,10 +1,10 @@
-# ๐ Population Projection Calculator
+# Population Projection Calculator
This tool projects future population based on a compound growth rate model.
---
-## ๐ฆ Files
+## Files
- `population_projection_calculator.html`: Simple calculator for projecting population over time
- `population_projection_logic.md`: Explanation of the formula and logic
@@ -12,7 +12,7 @@ This tool projects future population based on a compound growth rate model.
---
-## ๐ง How It Works
+## How It Works
You enter:
- Base population (current year)
@@ -23,4 +23,4 @@ The tool applies the compound growth formula and returns the projected value.
---
-๐ Used in demographic planning, forecasting infrastructure demand, or service delivery estimation.
+ Used in demographic planning, forecasting infrastructure demand, or service delivery estimation.
diff --git a/calculators/population_projection/population_projection_logic.md b/calculators/population_projection/population_projection_logic.md
index 6ced547..ba3fdd3 100644
--- a/calculators/population_projection/population_projection_logic.md
+++ b/calculators/population_projection/population_projection_logic.md
@@ -1,10 +1,10 @@
-# ๐ Population Projection Calculator โ Logic
+# Population Projection Calculator โ Logic
This tool helps project district or block population over time using a compound annual growth formula.
---
-## ๐ข Inputs
+## Inputs
- Current population (base year)
- Annual growth rate (%)
@@ -12,7 +12,7 @@ This tool helps project district or block population over time using a compound
---
-## ๐ Formula Used
+## Formula Used
```
Population in Year N = Base Population ร (1 + Growth Rate)^N
@@ -24,7 +24,7 @@ Where:
---
-## ๐ง Example
+## Example
- Current population = 520,000
- Growth rate = 1.85%
diff --git a/data_starters/README.md b/data_starters/README.md
index 307f593..1ea85f4 100644
--- a/data_starters/README.md
+++ b/data_starters/README.md
@@ -1,43 +1,28 @@
# Data starters
-Starter code for real public-use survey microdata, one folder per dataset, in
-Stata, R and Python.
+Loaders for real public-use survey microdata, one folder per dataset, in
+Stata, R and Python. Each opens the files as they are distributed and
+applies the weight and scale rules that the documentation states.
-The rest of InsightStack teaches technique against data that stands in for
-something. `stata_snippets/` imports a `health_survey.csv` and generates age
-bands; that is a good way to learn `import delimited` and no help at all to
-someone holding an actual survey file. This section is the other half: code that
-knows about a specific named dataset, opens the files as they actually arrive,
-and handles the things that produce a wrong answer rather than an error.
+| Folder | Covers | Languages |
+| --- | --- | --- |
+| [`dhs-south-asia/`](dhs-south-asia/) | 28 DHS surveys: India (NFHS 1 to 5), Bangladesh, Nepal, Pakistan, Maldives, Afghanistan, Sri Lanka | Stata, R, Python |
+| [`plfs-india/`](plfs-india/) | Periodic Labour Force Survey unit-level data, fixed-width text read from the round's own layout | Stata, R, Python |
## What a folder contains
-| | |
-|---|---|
-| `README.md` | Which files to download and from where, what goes wrong silently, and what is not comparable across rounds |
-| `load_*.do`, `load_*.R`, `load_*.py` | The same loader in three languages, with the same guards |
-| `variables.csv` | Canonical name to the dataset's variable, with scale factors and caveats. Read by the loaders at runtime, so documentation and behaviour cannot drift apart |
-| `make_fixture.py` | Builds synthetic files with the structure but no real data, since none of these datasets may be redistributed |
-| `test_*.py` | Checks against those fixtures, one per failure mode |
-
-No folder ships data. Every dataset here needs a registration you hold
-personally, and the READMEs say which.
-
-## Folders
-
-| Folder | Covers | Languages |
-|---|---|---|
-| [`dhs-south-asia/`](dhs-south-asia/) | 28 DHS surveys: India (NFHS 1 to 5), Bangladesh, Nepal, Pakistan, Maldives, Afghanistan, Sri Lanka | Stata, R, Python |
-| [`plfs-india/`](plfs-india/) | Periodic Labour Force Survey unit-level data: fixed-width text read from the round's own layout | Stata, R, Python |
+| File | What it is |
+| --- | --- |
+| `README.md` | Which files to download and from where, what goes wrong silently, what is not comparable across rounds |
+| `load_*.do`, `load_*.R`, `load_*.py` | The same loader in three languages |
+| `variables.csv` | Canonical name to the dataset's variable, with scale factors and caveats, read at runtime |
+| `make_fixture.py` | Builds synthetic files with the structure and no real data |
+| `test_*.py`, `test_*.R` | Checks against the fixtures |
-## Adding one
+No folder ships data. DHS and MoSPI both prohibit redistribution; each README
+says what to download.
-Copy the shape of `dhs-south-asia/`. A folder earns its place when its loader
-opens the real files and its README names at least one error the data makes easy
-and quiet. A folder that is a README and a stub is worse than no folder, because
-it looks like coverage.
+## Adding a folder
-Nothing is listed here before it is built. Under the
-[maintenance policy](https://github.com/Varnasr/OpenStacks-for-Change/blob/main/MAINTENANCE.md)
-a roadmap entry is a promise to a reader, so datasets appear in the table above
-when they work and not before.
+Copy the shape of `dhs-south-asia/`. A folder is listed when its loader opens
+the real files and its tests pass, not before.
diff --git a/data_starters/dhs-south-asia/README.md b/data_starters/dhs-south-asia/README.md
index d437348..111ce37 100644
--- a/data_starters/dhs-south-asia/README.md
+++ b/data_starters/dhs-south-asia/README.md
@@ -1,24 +1,18 @@
# DHS recode files, South Asia
Starter code in Stata, R and Python for the Demographic and Health Surveys of
-India, Bangladesh, Nepal, Pakistan, Maldives, Afghanistan and Sri Lanka. Twenty
-eight surveys, one loader, because the recode structure is the same in all of
+India, Bangladesh, Nepal, Pakistan, Maldives, Afghanistan and Sri Lanka. Twenty-eight surveys, one loader: the recode structure is the same in all of
them.
This folder does not contain data and cannot. The DHS Program licence lets you
-use the files, not pass them on, so everyone downloads their own copy. What is
-here is the part that takes a week to work out and five minutes to get wrong.
+use the files, not pass them on, so everyone downloads their own copy.
-## What this is for
+## What the loaders do
-You have `IAIR7EFL.DTA` on your desktop. It holds 724,115 women and roughly five
-thousand variables named `v106`, `v190`, `hw70`. Opening it fills your memory,
-the codebook is 800 pages, and the first four numbers you compute will be wrong
-in ways that do not announce themselves.
-
-These scripts read only the columns you name, attach the right weight and design
-for the recode you are holding, apply the documented scale factors, and stop with
-an explanation when something is off. They are a starting point, not an analysis.
+They read only the columns you name from a recode such as `IAIR7EFL.DTA`
+(724,115 women, about five thousand variables), attach the weight and design
+variables for that recode, apply the documented scale factors, and stop with an
+explanation when something is off.
## Getting the files
@@ -89,8 +83,7 @@ python load_dhs.py IAKR7EFL.DTA --vars v190 hw70 hw71 b5 --anthro --out stunting
## What goes wrong silently
-Every item here produces a plausible number rather than an error. That is why the
-scripts handle them rather than leaving them to a checklist.
+Each of these produces a plausible number rather than an error.
**Weights are stored with six implied decimals.** `v005` averages about a million
and has to be divided by 1,000,000. Leaving it out does not change a weighted
@@ -108,9 +101,7 @@ back to `v023` in older rounds.
**Anthropometry is stored times 100, with flags above it.** `hw70` of `-162` means
a height-for-age z-score of -1.62. Values of 9996 and up are flags and missing.
Divide before you drop them and every flagged child becomes a plausible 99.96,
-then a plausible 0.9996, and your stunting rate falls. This bug appeared in this
-folder's own first draft and the test suite caught it, which is the argument for
-the test suite.
+then a plausible 0.9996, and your stunting rate falls.
**The anthropometry columns are named differently in different recodes.** `hw70`
to `hw73` in the children's recode, `hc70` to `hc73` in the household member
@@ -141,10 +132,7 @@ NFHS-5 child stunting: 35.5 percent nationally, 46.1 in the poorest wealth
quintile falling to 22.9 in the richest, with breakdowns by residence and
mother's education. NFHS-4's national figure of 38.4 is there for comparison.
-Reproducing a published table is the only cheap check that a pipeline is right
-end to end. Each of the silent errors above produces a number that looks
-reasonable on its own and visibly wrong beside the survey agency's own report.
-Two worked examples do exactly this, one in R and one in Python:
+Two worked examples reproduce the table, one in R and one in Python:
- [FieldStack](https://github.com/Varnasr/FieldStack) `survey_tools/dhs_stunting.R`
- [EquityStack](https://github.com/Varnasr/EquityStack) `survey_estimation/dhs_stunting.py`
@@ -239,10 +227,10 @@ python test_load_dhs.py # 30 checks
Rscript test_load_dhs.R # 33: the same ones, plus the design object
```
-The R suite mirrors the Python one check for check and adds three for the survey
-design object, so both languages are held to the same standard, and both were run: verified on Python 3 with pandas and
-pyreadstat, and on R 4.3.3 with survey 4.2.1 and haven 2.5.4. Where both read the
-same fixture they return the same numbers.
+The R suite mirrors the Python one and adds three checks for the survey
+design object. Verified on Python 3 with pandas and pyreadstat, and on R 4.3.3
+with survey 4.2.1 and haven 2.5.4; both return the same numbers from the same
+fixture.
Stata has no free runtime, so `load_dhs.do` is the one file here that has not been
executed. It carries the same logic and the same guards, and the fixture check at
@@ -258,5 +246,4 @@ data.
[dhsprogram.com/publications](https://dhsprogram.com/publications/).
- Anthropometry plausibility bounds: WHO Child Growth Standards, 2006.
-Where this folder and your survey's own codebook disagree, the codebook is right.
-Rounds vary more than any summary of them does.
+Where this folder and your survey's codebook disagree, the codebook is right.
diff --git a/data_starters/plfs-india/README.md b/data_starters/plfs-india/README.md
index f73cc5e..15e971a 100644
--- a/data_starters/plfs-india/README.md
+++ b/data_starters/plfs-india/README.md
@@ -4,8 +4,7 @@ Starter code in Stata, R and Python for the Periodic Labour Force Survey
unit-level data published by MoSPI.
This folder does not contain data and cannot. You register at
-[microdata.gov.in](https://microdata.gov.in/) and download your own copy. What is
-here is the part that is tedious to work out and easy to get wrong.
+[microdata.gov.in](https://microdata.gov.in/) and download your own copy.
## What you are dealing with
@@ -27,8 +26,7 @@ complaining.
## The weight is not MLTS divided by 100
-This is the one to get right, and it is the reason this folder exists. MoSPI's
-README says, verbatim:
+MoSPI's README says, verbatim:
> For generating sub-sample wise estimate for the Calendar Year, weight may be
> applied as follows:
@@ -84,10 +82,9 @@ MoSPI gives the common primary key as:
> Second Stage Stratum No. = 38(1)
> Sample Household No. = 39(2)
-with the sub-sample code at byte 27. Note that the parenthetical for Quarter
-does not agree with the `11(2)` notation used for the rest, which is one more
-reason to take positions from the layout file rather than from any prose,
-including this page.
+with the sub-sample code at byte 27. The parenthetical for Quarter does not
+agree with the `11(2)` notation used for the rest; take positions from the
+layout file, not from prose.
Build the key by zero-padding each part to its declared width. Concatenating the
parts unpadded collapses distinct households onto one key: FSU 1234 with
@@ -210,9 +207,9 @@ Both suites were run and both pass, and they return identical numbers from the
same fixture: 240 households, 1,099 persons, mean `MLTS` of 1217.72 and a
correct sum of weights of 2190.4 against the naive rule's 2922.5.
-The strongest check available without real data is the last one. Household size
-is written into one file and the members into another, so the two agree only if
-the byte positions, the zero padding and the merge are all correct at once.
+Household size is written into one file and the members into another, so the
+last check passes only if the byte positions, the zero padding and the merge
+are all correct.
Stata has no free runtime, so `load_plfs.do` is the one file here that has not
been executed. The fixture check at the bottom of it takes about a minute and
@@ -230,4 +227,4 @@ names the number to look for.
included in the download for each round. Retrieved 2026-09-08.
Where this page and the documents shipped with your round disagree, your round
-is right. Positions, and occasionally field names, change.
+is right.
diff --git a/data_validation/README.md b/data_validation/README.md
index 007316b..42e6746 100644
--- a/data_validation/README.md
+++ b/data_validation/README.md
@@ -1,6 +1,7 @@
# data_validation
-A list of what is wrong with a file, by identifier, before it reaches a finding.
+Checks a data file against a data dictionary and lists every problem by
+identifier.
```python
from data_validation import validate, Rules
@@ -9,63 +10,47 @@ rules = Rules.from_csv("sample_data/data_dictionary.csv", id="id", names="snake_
report = validate(df, rules)
report.issues # one row per problem: id, variable, value, check, message
report.summary() # counts by check and severity
-report.ok # True only when nothing fired
+report.ok # True when nothing fired
```
```
python data_validation/test_data_validation.py # 18 tests, 40 checks
```
-## What it checks
+## Checks
| Check | Fires when |
-|---|---|
+| --- | --- |
| `id_blank`, `id_duplicate` | The identifier is missing or repeated |
-| `required_blank` | A variable listed as required is blank, NA, `.`, or a sentinel code |
-| `out_of_range` | A numeric value falls outside a declared `(min, max)`; either end may be open |
-| `not_allowed` | A value is not in the declared list, compared as trimmed strings |
-| `type_numeric`, `type_integer`, `type_binary`, `type_date` | A value cannot be read as the declared type, reported per value |
-| `name_style` | A column name breaks `snake_case` or Stata's 32-character rule |
+| `required_blank` | A required variable is blank, NA, `.`, or a sentinel code |
+| `out_of_range` | A numeric value is outside the declared `(min, max)`; either end may be open |
+| `not_allowed` | A value is not in the declared list |
+| `type_numeric`, `type_integer`, `type_binary`, `type_date` | A value cannot be read as the declared type |
+| `name_style` | A column name breaks `snake_case` or Stata's 32-character limit |
| `column_missing`, `column_unexpected` | The file's columns differ from the dictionary's |
-| `orphan_child`, `childless_parent` | `check_cross_file_ids`: a roster row whose household is not in the household file, and the reverse |
+| `orphan_child`, `childless_parent` | `check_cross_file_ids`: a roster row without a household, or the reverse |
-## Rules come from the dictionary
+## The dictionary
-`Rules.from_csv` reads the same dictionary that `label_variables/` uses. Add
-`type`, `min`, `max`, `allowed` (pipe-separated) and `required` columns and the
-rules are the dictionary; leave them out and only the name and column-set
-checks run. One file describes the data for validation and for labelling, so
-the two cannot drift.
+`Rules.from_csv` reads the dictionary that `label_variables/` also uses,
+with `type`, `min`, `max`, `allowed` (pipe-separated) and `required`
+columns. Without them, only the name and column-set checks run.
-## Three decisions that shape the output
+## Behaviour
-**Missing is not wrong, unless the variable is required.** A blank in a
-skip-pattern question is correct data. Only variables in `required` are flagged
-for blanks; every other check ignores NA. The opposite policy, which the
-previous version of this folder had, produces hundreds of false flags and
-teaches people to ignore the report.
+Only variables listed in `required` are flagged for blanks. A blank in a
+skip-pattern question is correct data.
-**Sentinel codes are missing.** -999, -99 and the rest arrive from ODK and
-Stata as numbers. A range check that does not know that reports every refusal
-as an age of -999. `Rules.sentinels` lists them and defaults to the common set;
-pass `sentinels=[]` if the codes are meaningful in your instrument.
+Sentinel codes (-999, -99 and the rest) count as missing. `Rules.sentinels`
+lists them; pass `sentinels=[]` if the codes mean something in your
+instrument.
-**Everything is reported by identifier, never by row number.** A row number
-changes when the file is re-sorted; an id is what you phone the field team
-with. The one exception is a blank id, which has nothing to be reported by,
-and the message says so.
+Problems are reported by identifier, not row number. A blank identifier is
+the one exception, and the message says so.
## Companions
-`companions/validate.do` and `companions/validate.R` run the same duplicate,
-required, range and allowed checks against the same dictionary, for a team
-that works in Stata or R. They are not tested in CI (Stata has no free runtime)
-and they do fewer things than the Python; the Python is the reference.
-
-## What this replaced
-
-Three scripts that printed `df[~df.gender.isin(["M", "F"])]` and stopped, in
-three languages, plus a second folder under `foundation_tools/` with a
-different set of five-line functions that overlapped with the first. Neither
-reported an id, neither knew about sentinels, and neither could be pointed at a
-new file without editing the source.
+`companions/validate.do` and `companions/validate.R` run the duplicate,
+required, range and allowed checks against the same dictionary in Stata and
+R. They are not tested in CI and do less than the Python, which is the
+reference.
diff --git a/docs/data_flow.md b/docs/data_flow.md
index 979c1b3..d8091a9 100644
--- a/docs/data_flow.md
+++ b/docs/data_flow.md
@@ -2,34 +2,34 @@
This file explains the typical flow of data through the InsightStack toolkit.
-## ๐ Typical Workflow:
+## Typical Workflow:
1. **Raw Data Entry**
- - Source: External surveys, XLSForms, CSV, STATA/SPSS
- - Tools: `survey_to_codebook/`, `data_validation/`
+ - Source: External surveys, XLSForms, CSV, STATA/SPSS
+ - Tools: `survey_to_codebook/`, `data_validation/`
2. **Data Validation**
- - Scripts check for:
- - Missing values
- - Out-of-range entries
- - Duplicates
- - Folder: `data_validation/`
+ - Scripts check for:
+ - Missing values
+ - Out-of-range entries
+ - Duplicates
+ - Folder: `data_validation/`
3. **Labeling Variables**
- - Apply human-readable labels to variables
- - Folder: `label_variables/`
+ - Apply human-readable labels to variables
+ - Folder: `label_variables/`
4. **Exploration & Analysis**
- - Can now run:
- - Summary statistics
- - Regressions
- - Visualizations
- - Folder: `replication/` or use external tools
+ - Can now run:
+ - Summary statistics
+ - Regressions
+ - Visualizations
+ - Folder: `replication/` or use external tools
5. **Documentation**
- - Generate Markdown codebooks from XLSForms
- - Folder: `survey_to_codebook/`
+ - Generate Markdown codebooks from XLSForms
+ - Folder: `survey_to_codebook/`
6. **Archival & Replication**
- - Structure analysis for reuse or publication
- - Folder: `replication/`
\ No newline at end of file
+ - Structure analysis for reuse or publication
+ - Folder: `replication/`
\ No newline at end of file
diff --git a/eval_docs/README.md b/eval_docs/README.md
index 3906858..4b530f6 100644
--- a/eval_docs/README.md
+++ b/eval_docs/README.md
@@ -1,34 +1,26 @@
# eval_docs
-Templates and guidance for the three documents an evaluation is built on: the
-logframe, the indicator reference sheet, and the MEL framework.
+Templates for the logframe, the indicator reference sheet and the MEL
+framework, with a completed example of each.
| File | What it is |
-|---|---|
-| `logframe_template.md` | The four-row table, how to fill it in (outcome first), and the errors that recur |
+| --- | --- |
+| `logframe_template.md` | The four-row table, how to fill it in outcome first, and common errors |
| `logframe_template.csv` | The same table as a spreadsheet |
| `logframe_mhh_program.xlsx` | A completed example: school health clubs for menstrual health |
-| `indicator_reference_sheet.md` | One sheet per indicator, with the three fields that matter most |
+| `indicator_reference_sheet.md` | One sheet per indicator |
| `indicator_reference_sheet.csv` | The fields as columns, for a whole indicator set |
-| `indicator_guidance.md` | Five tests for an indicator, the types and what each is for, and the errors |
+| `indicator_guidance.md` | Five tests for an indicator, the types, and common errors |
| `mel_framework_outline.md` | Nine sections, with the question a reviewer asks of each |
| `mel_framework_mhh_program.pdf` | A completed example, same programme |
-The completed examples describe a hypothetical programme. The templates are
-what to reuse.
+The completed examples describe a hypothetical programme.
-## The one thing to read if you read one
+The assumptions column of the logframe, between outputs and outcomes, states
+the programme's theory in one sentence. If it cannot be written, there is no
+theory yet.
-The assumptions column of the logframe, and specifically the assumption
-between outputs and outcomes. That cell is the programme's theory in one
-sentence: "trained peer leaders will run sessions, girls will attend, and
-attending changes what girls do". If it cannot be written, there is no
-theory yet, and no indicator will supply one.
-
-## Related
-
-`../writing_guides/` for the theory of change note and the results chain that
-the logframe summarises. `../annotated_research/` for five worked examples of
-what each evaluation design in section 4 of the framework can and cannot
-claim. FieldStack `survey_tools/` for the sample size calculation section 5
-needs, and `field_ops/` for the data quality checks.
+Related: `../writing_guides/` for the theory of change note and results
+chain; `../annotated_research/` for what each evaluation design can claim;
+FieldStack `survey_tools/` for sample size and `field_ops/` for data quality
+checks.
diff --git a/index.html b/index.html
index 70cb3f9..49d9f52 100644
--- a/index.html
+++ b/index.html
@@ -27,9 +27,9 @@
InsightStack
- Tools, templates and thinking for knowledge work in development. Calculators that run in
- a browser, econometrics you can actually execute, and loaders that open survey microdata
- in the state it arrives in. Part of OpenStacks.
+ Calculators that run in a browser, causal-inference code in Python and R, loaders for DHS and
+ PLFS microdata, and templates for the documents an evaluation produces. Part of
+ OpenStacks.
6Calculators
2Dataset loaders
@@ -43,7 +43,7 @@
InsightStack
Calculators
-
Six tools ยท run in the browser ยท nothing is sent anywhere
+
Six tools ยท run in the browser ยท nothing is sent to a server
- Each one states its assumption on the page. Treat the output as an estimate to
- think with. Where a published district figure exists, cite that figure instead.
+ Each calculator states its assumptions on the page. Where a published district figure
+ exists, use that instead.
@@ -97,21 +97,20 @@ Data starters
28 surveys ยท 7 countries
DHS, South Asia
One loader for every DHS in the region, from India's NFHS-1 to NFHS-5 through Bangladesh,
- Nepal, Pakistan, Maldives, Afghanistan and Sri Lanka. Gets right the three
- things that quietly ruin a stunting rate: the weight scaling, the design variables,
- and the anthropometry flags.
+ Nepal, Pakistan, Maldives, Afghanistan and Sri Lanka. Applies the weight scaling, the design
+ variables and the anthropometry flags.
Read the guide →
Fixed-width ยท layout-driven
PLFS, India
- Opens MoSPI's unit-level files using the layout that shipped with your round, and
- applies the combined-estimate weight rule, which is not simply MLTS divided by 100.
+ Opens MoSPI's fixed-width files using the layout shipped with the round, and applies the
+ combined-estimate weight rule.
Read the guide →
- No data ships, and none can: DHS and MoSPI both forbid passing the files on. You
- download your own copy, and each guide names exactly which files to fetch.
+ No data ships. DHS and MoSPI both prohibit redistribution; each guide names the files
+ to download.
@@ -122,11 +121,11 @@ Code and tooling
EconometricsPython, R
- Did the programme work, and can you defend the answer? Difference-in-differences, matching, instrumental variables and regression discontinuity, each with data to practise on.
+ Difference-in-differences, matching, instrumental variables, regression discontinuity and sensitivity analysis, with sample data.
Stata snippets44 do-files
- The Stata you end up rewriting on every project: merges, labels, graphs, regressions, survey settings.
+ Data management, merges, labels, graphs, regressions, survey settings.
SPSS scripts20 files
@@ -134,43 +133,43 @@ Code and tooling
Network effectsPython
- Who influences whom in a self-help group, how far a practice spreads, and why the obvious peer-effects regression cannot answer the question it looks like it answers.
+ Centrality, peer association with leave-one-out means, and threshold diffusion for self-help group data.
ReplicationPython, R
- A package that checks itself: one command from raw data to results, a recorded answer, R against Python, and a test that the check can fail.
+ One command from raw data to results, a recorded answer, and an R cross-check.
Data validationPy, R, Stata, SPSS
- Catch what is wrong with a file, by household id, before it reaches a finding. Rules come from the data dictionary; blanks in skip-pattern questions are not flagged; sentinel codes count as missing.
+ Checks from a data dictionary, reported by household id: duplicates, required fields, ranges, allowed values, types, column set.
Label variablesPy, R, Stata, SPSS
- Put the labels from the dictionary into a Stata or SPSS file, where they survive. A CSV loses them on every round trip.
+ Variable and value labels from a dictionary, written into Stata and SPSS files.
Survey to codebookPython
- Turn an XLSForm into a codebook a reviewer can read top to bottom, and into the label dictionary for its own export. Reports the choice list the form names and does not have.
+ An XLSForm to a Markdown codebook and a label dictionary. Reports form defects such as a missing choice list.
System dynamicsVensim
- Where an intervention pushes back on itself: eight system models across health, agriculture, climate, migration and education.
+ Eight system models across health, agriculture, climate, migration and education.
Tool notes9 tools
- Excalidraw, Kumu, Observable, RawGraphs, Flourish, Power BI and Miro: when each is worth the setup, when it is not, and what goes wrong, with one worked file apiece.
+ Excalidraw, Kumu, Observable, RawGraphs, Flourish, Power BI, Miro, Excel and LaTeX: when to use each, when not to, and one example file apiece.
Qualitative codingTaguette
- One focus group coded start to finish, and the method: decide the frame first, code two transcripts twice, count carefully. Eleven of fourteen groups is a sentence; 78 per cent of respondents is a survey result a focus group cannot give.
+ One focus group coded start to finish in Taguette, with the method.
Recorded fieldworkDescript
- From recording to clip to quote, and what is decided at each step. Three separate consents, every quote checked against the audio, and a clip cut with its context visible to whoever approves it.
+ Recorded interviews: consent, transcription, clips and quotes.
Annotated research5 briefs
- Five research briefs marked up with the same eight questions, weakest design to strongest. Hand one out unmarked, have readers tag it, compare.
+ Five illustrative research briefs annotated with one eight-tag scheme, weakest design to strongest.
Learning libraryReference
@@ -178,22 +177,22 @@ Code and tooling
Learning layersPractice
- A MEL system that changes nothing has no learning in the calendar. Three layers at three tempos, each a decision with a date, a person and the evidence in front of them.
+ Learning in the MEL cycle as scheduled decisions, at three tempos.
Writing guides5 guides
- Theory of change, results chain, the evaluation report, the policy brief, and how to write uncertainty into the same sentence as the number. The verb is decided by the design.
+ Theory of change, results chain, evaluation report, policy brief, writing about uncertainty.
Evaluation documentsTemplates
- Logframe, indicator reference sheet and MEL framework: the template, how to fill it in outcome-first, and the errors that recur. The assumptions column is the programme's theory.
+ Logframe, indicator reference sheet and MEL framework templates, with a completed example.
Knowledge managementConventions
- One folder structure and a README template, so a file's location says what it is. Raw data is never edited; every table came from a script still in the folder.
+ A folder structure, a README template, file naming and tagging conventions.
- These open on GitHub, where the code lives.
+ Links open on GitHub.
@@ -202,12 +201,11 @@ Code and tooling
Status
Stable
- This repository works and is correct, but is not under active development. Bug
- reports are welcome and issues stay open; new features are unlikely, and replies are measured
- in weeks rather than days. See the
+
The code works and is not under active development. Bug reports are welcome; new
+ features are unlikely, and replies take weeks. See the
maintenance policy.
- Citable: 10.5281/zenodo.15245182.
- The companion repositories are
+
DOI 10.5281/zenodo.15245182.
+ Companion repositories:
FieldStack for R and
EquityStack for Python.
@@ -221,7 +219,7 @@ Status
Changelog
MIT Licence
- Varna Sri Raman. Built for open knowledge, and licensed so you can take it.
+ Varna Sri Raman. MIT licence.