Skip to content

Plain-language pass on the reader-facing copy #24

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Varnasr merged 1 commit into from
Sep 8, 2026
Merged

Plain-language pass on the reader-facing copy #24

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion ISSUE_TEMPLATE/repo-readiness-checklist.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ assignees: ''

---

# 🗂️ Repo Hygiene Checklist
# Repo Hygiene Checklist

- [ ] LICENSE.md present (MIT)
- [ ] README.md complete (badges, overview, folders)
Expand Down
19 changes: 7 additions & 12 deletions KM_tools/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,19 +1,14 @@
# KM_tools

Conventions for organising a project's files so that a location says what a
file is and a newcomer can find last year's endline without asking.
Conventions for organising a project's files.

| File | What it is |
|---|---|
| `folder_structure_template.md` | One structure, numbered in workflow order, with the rules that make it hold: raw data never edited, every output generated by a script |
| `readme_template.md` | The two-page README a project folder starts with: people, where things are, codes, how to run the analysis |
| `file_naming_conventions.md` | `project_doctype_scope_version` and why |
| --- | --- |
| `folder_structure_template.md` | One structure, numbered in workflow order |
| `readme_template.md` | A two-page README for a project folder: people, where things are, codes, how to run the analysis |
| `file_naming_conventions.md` | `project_doctype_scope_version` |
| `tagging_structure_guidance.md` | Tag axes for a document store: theme, geography, stage, type, audience |
| `documentation_flowchart.pdf` | The path from draft to final for a brief, deck or report |

Two rules do most of the work. Raw data is never edited; anything wrong with
it is fixed by a script that writes somewhere else. And every table and
figure in a report was produced by a script that is still in the folder. A
project that keeps those two can be handed over, replicated
(`../replication/`), and audited. One that does not has a set of files that
only one person understands.
Two rules: raw data is never edited, and every table and figure in a report
comes from a script that is still in the folder.
171 changes: 73 additions & 98 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,117 +1,92 @@
# InsightStack

**MEL tools, calculators, and research documentation for development work.**
MEL tools, calculators, research documentation and loaders for survey
microdata, in Stata, Python, R and SPSS. Part of
[OpenStacks](https://openstacks.dev). Status: Stable, per the family
[maintenance policy](https://github.com/Varnasr/OpenStacks-for-Change/blob/main/MAINTENANCE.md).
DOI: [10.5281/zenodo.15245182](https://doi.org/10.5281/zenodo.15245182).

Site: [varnasr.github.io/InsightStack](https://varnasr.github.io/InsightStack/).

## Code and data

| Directory | What it does | Language |
| --- | --- | --- |
| `data_starters/` | Loaders for DHS (28 surveys, seven South Asian countries) and PLFS (India) that open the files as distributed, with the weight and scale rules applied. Tested on synthetic fixtures; no data ships | Stata, R, Python |
| `calculators/` | Six browser calculators for district planning: population projection, school needs, banking access, district health indicators, block-level estimates, environmental fragility | HTML, JavaScript |
| `econometrics/` | Difference-in-differences, propensity score matching, instrumental variables, regression discontinuity, sensitivity analysis, with sample data | Python, R |
| `stata_snippets/` | 44 do-files: data management, descriptives, regression, impact evaluation, graphs, survey settings | Stata |
| `spss_scripts/` | 20 files: cleaning, recoding, tabulation, regression, Excel export | SPSS |
| `network_effects_sni/` | Centrality, peer association with leave-one-out means, and threshold diffusion for self-help group data. 17 tests | Python |
| `data_validation/` | Checks from a data dictionary: duplicates, required fields, ranges, allowed values, types, column set, cross-file ids. One row per problem. 18 tests | Python; Stata, R, SPSS companions |
| `label_variables/` | Variable and value labels from a dictionary, written into `.dta` and `.sav`. 8 tests | Python; Stata, R, SPSS companions |
| `survey_to_codebook/` | An XLSForm to a Markdown codebook and a label dictionary. 9 tests | Python |
| `replication/` | A replication package with one entry point, a recorded result and an R cross-check. 4 tests | Python, R |
| `vensim/` | Eight system dynamics models | Vensim |

## Documents and notes

| Directory | What it holds |
| --- | --- |
| `tool_notes/` | Nine notes on visual and document tools (Excalidraw, Kumu, Observable, RawGraphs, Flourish, Power BI, Miro, Excel, LaTeX), each with one example file |
| `writing_guides/` | Theory of change, results chain, evaluation report structure, policy brief, writing about uncertainty |
| `eval_docs/` | Logframe, indicator reference sheet and MEL framework templates, with a completed example |
| `annotated_research/` | Five illustrative research briefs annotated with one eight-tag scheme |
| `learning_layers/` | Learning in the MEL cycle as scheduled decisions |
| `KM_tools/` | Folder structure, README template, file naming, tagging |
| `taguette_coding/` | One focus group coded in Taguette, with the method |
| `visual_ethnography_descript/` | Recorded interviews: consent, transcription, clips, quotes |
| `learning_library/` | PDFs and guides in five categories |

## Tests

[![Part of OpenStacks](https://img.shields.io/badge/Part%20of-OpenStacks-blue)](https://openstacks.dev)
[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)
[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.15245182.svg)](https://doi.org/10.5281/zenodo.15245182)
[![Status: Stable](https://img.shields.io/badge/Status-Stable-0969da?style=flat-square)](https://github.com/Varnasr/OpenStacks-for-Change/blob/main/MAINTENANCE.md)

> In development work, we talk about knowledge — but rarely structure it. InsightStack organises the tools, templates, and workflows that make research, evaluation, and program design actually work.

> **Status: Stable.** This repository works and is correct, but it is not under active
> development. Bug reports are welcome and issues stay open; new features are unlikely,
> and replies are measured in weeks rather than days. Dependencies are pinned deliberately
> so that a clone still runs years from now. See the [maintenance policy](https://github.com/Varnasr/OpenStacks-for-Change/blob/main/MAINTENANCE.md).

---

## What This Is

InsightStack is a collection of practical scripts, templates, and tools for **monitoring, evaluation, and learning (MEL)** work in the development sector. It covers data validation, survey analysis, system dynamics modelling, network analysis, qualitative coding, and visual storytelling.

This is the **knowledge systems layer** of [OpenStacks for Change](https://openstacks.dev) — an open ecosystem of tools for public interest research and evaluation.

## What's Inside

### Analysis Tools

| Directory | What It Does | Language | Status |
|-----------|-------------|----------|--------|
| `data_starters/` | Loaders for real public-use survey microdata, one folder per dataset. DHS across South Asia, and PLFS | Stata, R, Python | Ready |
| `calculators/` | District-level calculators for health, education, finance, environment, and population | HTML, JavaScript | Ready |
| `data_validation/` | Rule-driven validation from a data dictionary: duplicates, required, ranges, allowed values, types, name style, column set, cross-file ids; one issue row per problem, by identifier | Python (18 tests); Stata, R, SPSS companions |
| `stata_snippets/` | Reusable Stata code: data management, graphs, regression, impact evaluation, surveys | Stata | Ready |
| `spss_scripts/` | Survey analysis syntax: cleaning, regression, missing data, frequencies | SPSS | Ready |
| `network_effects_sni/` | Peer effects estimation, centrality analysis, diffusion modelling for SHG networks | Python | Ready |
| `replication/` | A replication package that verifies itself: one entry point, a recorded result, a Python-against-R cross-check, and a test that the verification can fail | Python, R (4 tests) |
| `label_variables/` | Variable and value labels from a dictionary, written into `.dta` and `.sav` where they survive; round-trips both ways | Python (8 tests); Stata, R, SPSS companions |
| `survey_to_codebook/` | XLSForm to Markdown codebook and to a label dictionary; resolves groups, repeats and choice lists; reports form defects | Python (9 tests) |
| `econometrics/` | Causal inference: DiD, PSM, IV/2SLS, RDD, sensitivity analysis — with Python, R, and sample data | Python, R | Ready |

### Visual and Interactive Tools

| Directory | What It Does | Tool |
|-----------|-------------|------|
| `vensim/` | 8 system dynamics models (health, agriculture, climate, migration, education) | Vensim |
| `tool_notes/` | Nine tool notes (Excalidraw, Kumu, Observable, RawGraphs, Flourish, Power BI, Miro, Excel, LaTeX): when each is worth the setup, when it is not, what goes wrong, with one worked file each | Notes |

### Knowledge and Documentation

| Directory | What It Contains |
|-----------|-----------------|
| `learning_library/` | Curated PDFs across 10 categories: programming, data science, research methods, AI tools, MLE resources |
| `writing_guides/` | Theory of change, results chain, evaluation report structure, policy brief, writing about uncertainty; the verb is decided by the design | Markdown |
| `eval_docs/` | Logframe template, indicator reference sheet, MEL framework outline, indicator guidance, with a completed example | Markdown, CSV, XLSX |
| `KM_tools/` | Folder structure, project README template, file naming, tagging | Markdown |
| `annotated_research/` | Five research briefs marked up with one eight-tag annotation scheme, weakest design to strongest; illustrative, for teaching critical reading | Markdown |
| `learning_layers/` | Embedding learning in the MEL cycle as scheduled decisions at three tempos | Markdown, PDF |
| `taguette_coding/` | One focus group coded start to finish in Taguette, with the method: frame first, double-code two transcripts, count carefully | Qualitative |
| `visual_ethnography_descript/` | From recording to clip to quote: three consents, quotes checked against audio, clips cut with context visible | Qualitative |
| `workflows/` | End-to-end analysis workflow guides |

## Getting Started

1. **Browse by need** — Each directory is self-contained. Pick the tool that matches your task.
2. **Check the sample data** — Use `sample_data/` and the test data in `data_validation/` to try scripts.
3. **Read the workflows** — `workflows/` ties multiple tools together into end-to-end processes.

### Prerequisites

Different tools require different software:
- **Stata 15+** for Stata scripts and snippets
- **Python 3.8+** with pandas, statsmodels, networkx for Python scripts
- **R 4.0+** with tidyverse for R scripts
- **SPSS** for survey analysis syntax
- **Observable**, **Flourish**, **Miro**, **Kumu** accounts for visual tools (free tiers available)

## How It Connects
```
cd data_starters/dhs-south-asia && python make_fixture.py --outdir fixtures
python test_load_dhs.py && Rscript test_load_dhs.R
cd ../plfs-india && python make_fixture.py --outdir fixtures
python test_load_plfs.py && Rscript test_load_plfs.R
python -m network_effects_sni.test_network_effects
python data_validation/test_data_validation.py
python label_variables/test_label_variables.py
python survey_to_codebook/test_survey_to_codebook.py
python replication/test_replication.py
```

InsightStack is one of several stacks in the [OpenStacks](https://openstacks.dev) ecosystem:
All of these run in CI on every pull request. The Stata files are not
tested, since Stata has no free runtime; each carries a fixture check at the
bottom.

| Stack | Focus |
|-------|-------|
| **InsightStack** (this repo) | MEL tools, calculators, research documentation, and loaders for real survey microdata. Stata, Python, R, SPSS |
| [FieldStack](https://github.com/Varnasr/FieldStack) | Field operations while a survey is being collected, and survey analysis after. R |
| [EquityStack](https://github.com/Varnasr/EquityStack) | Equity from a development economics perspective: distributional analysis, the concentration index, and design-based survey estimation. Python |
## Requirements

[openstacks.dev](https://openstacks.dev) is the index for all of it. [SignalStack](https://github.com/Varnasr/SignalStack) is the companion archive for the [Research Rundown](https://varna.substack.com) newsletter, alongside the stacks rather than one of them. [PolicyStack](https://github.com/Varnasr/PolicyStack) is superseded by [PolicyDhara](https://github.com/Varnasr/PolicyDhara). RootStack, BridgeStack and ViewStack are archived.
Stata 15 or later for the do-files. Python 3.8 or later with pandas,
statsmodels and networkx. R 4.0 or later with tidyverse, survey and haven.
SPSS for the syntax files.

## Contributing
## The family

Contributions welcome — especially from practitioners who use these tools in real fieldwork. See [contributing guidelines](https://github.com/Varnasr/.github/blob/main/CONTRIBUTING.md) for guidelines.
| Repository | What it is for | Language |
| --- | --- | --- |
| **InsightStack** (this repository) | MEL tools, calculators, research documentation, loaders for survey microdata | Stata, Python, R, SPSS |
| [FieldStack](https://github.com/Varnasr/FieldStack) | Field operations while a survey is in the field; sampling and weighted estimation after | R |
| [EquityStack](https://github.com/Varnasr/EquityStack) | Inequality measurement and design-based survey estimation | Python |

High-impact areas:
- **Econometrics** — causal inference implementations (DiD, PSM, IV, RDD) in Python, R, or Stata
- **Calculators** — new district-level planning tools
- **Stata/SPSS scripts** — analysis templates for common MEL tasks
- **Sample data** — synthetic datasets for testing
[openstacks.dev](https://openstacks.dev) is the index.
[SignalStack](https://github.com/Varnasr/SignalStack) is the companion archive
for the [Research Rundown](https://varna.substack.com) newsletter, beside the
stacks rather than one of them.
[PolicyStack](https://github.com/Varnasr/PolicyStack) is superseded by
[PolicyDhara](https://github.com/Varnasr/PolicyDhara). RootStack, BridgeStack
and ViewStack are archived.

## Citation
## Citation and license

```bibtex
@software{insightstack,
author = {Sri Raman, Varna},
title = {InsightStack: MEL Tools for Development Work},
title = {InsightStack: MEL tools, calculators and survey data loaders},
url = {https://github.com/Varnasr/InsightStack},
doi = {10.5281/zenodo.15245182}
}
```

## License

MIT — free to use, modify, and share. See [LICENSE](LICENSE).

---

Part of [OpenStacks for Change](https://openstacks.dev). Created by [Varna Sri Raman](https://on-web.link/varna).
MIT. See [LICENSE](LICENSE).
41 changes: 14 additions & 27 deletions annotated_research/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,38 +1,25 @@
# annotated_research

Five research briefs marked up to show where they are strong and where they
are wrong, with one annotation scheme applied to all of them.

The point is the scheme. Eight tags, each a question to put to any research
note that lands on your desk: what is it really asking, what is the
comparison, who is in the data, what was measured, is the arithmetic right for
the design, does it name its real weaknesses, does the conclusion's verb match
the design, and who consented. `annotation_scheme.md` defines them and says
how to teach with them.
Five research briefs marked up with one annotation scheme. Eight tags, each
a question to put to a research note: what is it asking, what is the
comparison, who is in the data, what was measured, is the arithmetic right
for the design, does it name its weaknesses, does the conclusion's verb match
the design, who consented. `annotation_scheme.md` defines them.

| Brief | Design | Main lesson |
|---|---|---|
| --- | --- | --- |
| `briefs/01_school_health_clubs.md` | Before-after, one arm | A year of growing up is inside the estimate |
| `briefs/02_remedial_reading.md` | Difference-in-differences, matched | The baseline table contradicts the identifying assumption |
| `briefs/03_flood_early_warning.md` | Post-event cross-section | The households not interviewed are the outcome |
| `briefs/04_shg_savings.md` | Randomised by village | Differential attrition and unclustered errors on a good design |
| `briefs/05_cash_transfer_nutrition.md` | Regression discontinuity | A credible local estimate, extended past the population it identifies |

Ordered from weakest design to strongest, so a reader who works through them
in sequence sees the ceiling on what each design can claim rise, and the
errors change from fatal to specific.

**All five are illustrative.** The programmes, districts, organisations and
numbers are invented to make the teaching points. No brief describes a real
study or cites one. They are written the way such notes are actually written,
which is why each contains the errors real notes contain.
| `briefs/05_cash_transfer_nutrition.md` | Regression discontinuity | A credible local estimate extended past the population it identifies |

The original `annotated_research_brief_mhh.pdf` is the example this folder
started with and is kept as it was.
Ordered from weakest design to strongest.

## Using them
All five are illustrative. The programmes, districts, organisations and
numbers are invented. No brief describes or cites a real study. The original
`annotated_research_brief_mhh.pdf` is kept as it was.

Hand out a brief without its annotations. Have readers tag it themselves
using the scheme, then compare. Where two readers tag the same sentence
differently, the text is ambiguous, and the ambiguity is usually where the
misleading claim lives.
To teach with them, hand out a brief without its annotations, have readers
tag it with the scheme, and compare. Where two readers tag the same sentence
differently, the text is ambiguous.
45 changes: 14 additions & 31 deletions calculators/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,35 +1,18 @@
# 🧮 Calculators for District and Program Planning
# Calculators

This folder contains practical tools for estimating health, education, infrastructure, and population needs at the district or block level — especially in data-scarce environments.
Six browser calculators for district and block planning. Each folder holds
an HTML file that runs locally, a note on the logic, and a README.

These calculators are designed for use by:
- Program managers and MEL teams
- District planners or consultants
- NGOs and policy support groups
- Fellows or research trainees
| Folder | What it calculates |
| --- | --- |
| `population_projection/` | Future population at a constant annual growth rate |
| `education_needs/` | Schools needed from projected school-age population |
| `district_health_indicators/` | A district estimate from a state NFHS figure and three multipliers |
| `financial_inclusion/` | Banking access points against the number a population implies |
| `block_level_development/` | A district value split to blocks by population share |
| `environmental_change/` | A fragility score from rainfall, forest and water indicators |

Each subfolder includes:
- A working HTML calculator
- A markdown file explaining the logic
- A usage README
Each calculator states its assumptions on the page. Where a published
district figure exists, use that.

---

## 📁 Included Tools

| Folder | What it Calculates |
|--------|--------------------|
| `education_needs/` | Projects number of schools needed based on population growth |
| `district_health_indicators/` | Estimates health indicators by adjusting NFHS data using local context |
| `financial_inclusion/` | Computes bank access gaps based on population norms |
| `environmental_change/` | Flags environmental fragility using rainfall, forest, and water indicators |
| `block_level_development/` | Disaggregates district-level values to blocks using population share |
| `population_projection/` | Forecasts future population using annual growth %

---

Each calculator is **real**, **modular**, and **customizable**. They can be embedded, adapted, or scaled for dashboards, training kits, or planning exercises.

MIT licensed. Attribution appreciated.

Made with ♥ by [Varna Sri Raman](https://on-web.link/varna)
MIT licence.
Loading
Loading