From c5951d4b02a7c882d6d692ef91e1dd4c36da11aa Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 8 Sep 2026 13:11:23 +0000 Subject: [PATCH] Consolidate the tool showcases, rebuild four packages, write out five folders An audit found twenty-one of thirty folders with fewer than six files and nine with no code, several a README and one image presented on the landing page as a module. This is the long tail, done properly. Seven tool showcases consolidated into tool_notes/. Excalidraw, Kumu, Observable, RawGraphs, Flourish, Power BI and Miro live on their own sites; what a repository can hold about each is a page of judgement, when it is worth the setup and what goes wrong, plus one worked file. That is what each now contains, and the seven are one landing-page row. foundation_tools/ deleted. It carried a second, different implementation of data_validation, label_variables, replication and survey_to_codebook, in five-line functions overlapping with the top-level folders. The two unique .sps companions moved up; the rest went. Four folders rebuilt as tested Python packages, 39 tests in CI. data_validation: rule-driven from a data dictionary; duplicates, required, ranges, allowed values, types, name style, column set, cross-file ids; one issue row per problem, by identifier. Blanks in skip-pattern questions are not flagged and sentinel codes count as missing, because the opposite policy, which the old scripts had, teaches people to ignore the report. label_variables: labels from the dictionary written into .dta and .sav, where they survive; round-trips both ways. The old script attached labels to df.attrs and wrote a CSV on the next line, which discards them. survey_to_codebook: XLSForm to codebook, resolving groups, repeats and choice lists and reporting form defects. The bundled sample form named a list `hobby` that the form referenced as `hobbies`; the old notebook printed an empty list without comment, the new tool reported it on first run, and the sample is fixed. replication: one entry point, a recorded result, a Python-against-R cross-check that agrees to six places, and a test that corrupts a value and confirms verification fails. A verification that cannot fail is not verifying anything. The three packages share one dictionary format, so a codebook generated from an XLSForm can be applied as labels and used as validation rules without retyping. Five folders built out as written content. annotated_research: five illustrative briefs under one eight-tag annotation scheme, weakest design to strongest, invented programmes and no real citations, said so on every page. eval_docs: logframe template, indicator reference sheet, MEL framework outline, indicator guidance. writing_guides: results chain, evaluation report structure, policy brief, writing about uncertainty, around the idea that the verb is decided by the design. learning_layers: learning as scheduled decisions at three tempos. KM_tools: folder structure and a project README template. SignalStack is dead and is gone from the family table; PolicyStack is in it. Emoji stripped from the notes that are kept. One bug caught on the way. pandas 3 gives text columns a str dtype, so a blank check keyed on dtype == object silently skipped every text column. CI installs unpinned pandas and would have shipped it. A test now pins the behaviour on string dtype. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01DMYhECx2Dzbqad6KJxWZ7t --- .github/workflows/tests.yml | 24 +- CLAUDE.md | 47 ++- KM_tools/README.md | 30 +- KM_tools/file_naming_conventions.md | 6 +- KM_tools/folder_structure_template.md | 60 ++++ KM_tools/readme_template.md | 50 +++ KM_tools/tagging_structure_guidance.md | 6 +- README.md | 29 +- annotated_research/README.md | 59 ++-- annotated_research/annotation_scheme.md | 51 +++ .../briefs/01_school_health_clubs.md | 118 +++++++ .../briefs/02_remedial_reading.md | 128 +++++++ .../briefs/03_flood_early_warning.md | 119 +++++++ annotated_research/briefs/04_shg_savings.md | 130 +++++++ .../briefs/05_cash_transfer_nutrition.md | 123 +++++++ data_validation/README.md | 76 +++- data_validation/__init__.py | 28 ++ .../companions}/check_duplicates.sps | 0 data_validation/companions/validate.R | 59 ++++ data_validation/companions/validate.do | 109 ++++++ .../sample_data/data_dictionary.csv | 5 + .../scripts/python/validate_data.py | 17 - data_validation/scripts/r/validate_data.R | 14 - .../scripts/stata/validate_data.do | 11 - data_validation/test_data_validation.py | 208 +++++++++++ data_validation/validate.py | 328 ++++++++++++++++++ eval_docs/README.md | 55 +-- eval_docs/indicator_guidance.md | 68 ++++ eval_docs/indicator_reference_sheet.csv | 1 + eval_docs/indicator_reference_sheet.md | 47 +++ eval_docs/logframe_template.csv | 10 + eval_docs/logframe_template.md | 66 ++++ eval_docs/mel_framework_outline.md | 96 +++++ excalidraw_frameworks/README.md | 11 - flourish_charts/README.md | 10 - foundation_tools/README.md | 57 --- foundation_tools/data_flow.md | 54 --- foundation_tools/data_validation/README.md | 14 - .../data_validation/check_code_labels.sps | 2 - .../data_validation/check_duplicates.do | 3 - .../data_validation/check_duplicates.py | 6 - .../data_validation/scan_value_ranges.py | 9 - .../validate_variable_names.py | 6 - foundation_tools/label_variables/README.md | 10 - .../label_variables/label_variables.do | 5 - .../label_variables/label_variables.py | 10 - foundation_tools/replication/README.md | 25 -- .../replication/replication_analysis.py | 11 - foundation_tools/replication/requirements.txt | 2 - foundation_tools/survey_to_codebook/README.md | 14 - .../survey_to_codebook/output_codebook.md | 1 - .../survey_to_codebook/sample_xlsform.xlsx | Bin 6159 -> 0 bytes .../survey_to_codebook/xlsform_to_codebook.py | 26 -- index.html | 64 ++-- kumu_maps/README.md | 10 - label_variables/README.md | 65 +++- label_variables/__init__.py | 20 ++ label_variables/companions/label.R | 37 ++ label_variables/companions/label.do | 57 +++ .../companions/label.sps | 0 label_variables/input/data_dictionary.csv | 11 +- label_variables/labels.py | 175 ++++++++++ label_variables/scripts/label_data_python.py | 14 - label_variables/scripts/label_data_r.R | 15 - label_variables/scripts/label_data_stata.do | 16 - label_variables/test_label_variables.py | 141 ++++++++ learning_layers/README.md | 43 ++- .../embedding_learning_in_the_mel_cycle.md | 93 +++++ learning_layers/why_cite_insightstack.md | 16 +- miro/README.md | 15 - observable_notebooks/README.md | 11 - open_data_editor/README.md | 10 - powerbi_reports/README.md | 62 ---- replication/.gitignore | 2 + replication/README.md | 74 +++- replication/analysis.py | 79 +++++ replication/expected_results.json | 18 + replication/run_all.py | 64 ++++ replication/run_regression.R | 31 ++ replication/scripts/run_regression.R | 8 - replication/scripts/run_regression.py | 11 - replication/test_replication.py | 88 +++++ replication/verify.py | 68 ++++ survey_to_codebook/README.md | 54 ++- survey_to_codebook/__init__.py | 14 + survey_to_codebook/__main__.py | 3 + .../convert_xlsform_to_codebook.ipynb | 1 - survey_to_codebook/input/survey.xlsx | Bin 6392 -> 5807 bytes survey_to_codebook/output/dictionary.csv | 6 + .../output/generated_codebook.md | 95 ++--- survey_to_codebook/test_survey_to_codebook.py | 174 ++++++++++ survey_to_codebook/xlsform_codebook.py | 234 +++++++++++++ tool_notes/README.md | 39 +++ tool_notes/excalidraw_frameworks/README.md | 33 ++ .../ecosystem_map.excalidraw | 0 .../excalidraw_frameworks}/ecosystem_map.png | Bin .../mel_framework.excalidraw | 0 .../excalidraw_frameworks}/mel_framework.png | Bin tool_notes/flourish_charts/README.md | 32 ++ .../flourish_charts}/embed_example.md | 0 .../flourish_embed_screenshot.png | Bin tool_notes/kumu_maps/README.md | 41 +++ .../kumu_maps}/district_health_network.json | 0 .../kumu_maps}/district_health_network.png | Bin {miro => tool_notes/miro}/HCD_Miro.pdf | Bin .../miro}/Miro HCD Template Example.png | Bin {miro => tool_notes/miro}/Miro Purpose.png | Bin tool_notes/miro/README.md | 36 ++ .../miro}/miro_use_case_india.md | 0 .../miro}/sample_miro_board_links.txt | 0 tool_notes/observable_notebooks/README.md | 37 ++ .../dashboard_screenshot.png | Bin .../district_vulnerability_dashboard.ojs | 0 .../public_health_data.csv | 0 tool_notes/open_data_editor/README.md | 36 ++ .../district_health_gaps.csv | 0 .../open_data_editor}/stacked_bar_chart.png | Bin tool_notes/powerbi_reports/README.md | 41 +++ .../mel_dashboard_screenshot.png | Bin writing_guides/README.md | 32 +- writing_guides/evaluation_report_structure.md | 79 +++++ writing_guides/policy_brief.md | 60 ++++ writing_guides/results_chain.md | 64 ++++ writing_guides/theory_of_change_notes.md | 12 +- writing_guides/theory_of_change_notes.qmd | 2 +- writing_guides/writing_about_uncertainty.md | 68 ++++ 126 files changed, 4198 insertions(+), 767 deletions(-) create mode 100644 KM_tools/folder_structure_template.md create mode 100644 KM_tools/readme_template.md create mode 100644 annotated_research/annotation_scheme.md create mode 100644 annotated_research/briefs/01_school_health_clubs.md create mode 100644 annotated_research/briefs/02_remedial_reading.md create mode 100644 annotated_research/briefs/03_flood_early_warning.md create mode 100644 annotated_research/briefs/04_shg_savings.md create mode 100644 annotated_research/briefs/05_cash_transfer_nutrition.md create mode 100644 data_validation/__init__.py rename {foundation_tools/data_validation => data_validation/companions}/check_duplicates.sps (100%) create mode 100644 data_validation/companions/validate.R create mode 100644 data_validation/companions/validate.do create mode 100644 data_validation/sample_data/data_dictionary.csv delete mode 100644 data_validation/scripts/python/validate_data.py delete mode 100644 data_validation/scripts/r/validate_data.R delete mode 100644 data_validation/scripts/stata/validate_data.do create mode 100644 data_validation/test_data_validation.py create mode 100644 data_validation/validate.py create mode 100644 eval_docs/indicator_guidance.md create mode 100644 eval_docs/indicator_reference_sheet.csv create mode 100644 eval_docs/indicator_reference_sheet.md create mode 100644 eval_docs/logframe_template.csv create mode 100644 eval_docs/logframe_template.md create mode 100644 eval_docs/mel_framework_outline.md delete mode 100644 excalidraw_frameworks/README.md delete mode 100644 flourish_charts/README.md delete mode 100644 foundation_tools/README.md delete mode 100644 foundation_tools/data_flow.md delete mode 100644 foundation_tools/data_validation/README.md delete mode 100644 foundation_tools/data_validation/check_code_labels.sps delete mode 100644 foundation_tools/data_validation/check_duplicates.do delete mode 100644 foundation_tools/data_validation/check_duplicates.py delete mode 100644 foundation_tools/data_validation/scan_value_ranges.py delete mode 100644 foundation_tools/data_validation/validate_variable_names.py delete mode 100644 foundation_tools/label_variables/README.md delete mode 100644 foundation_tools/label_variables/label_variables.do delete mode 100644 foundation_tools/label_variables/label_variables.py delete mode 100644 foundation_tools/replication/README.md delete mode 100644 foundation_tools/replication/replication_analysis.py delete mode 100644 foundation_tools/replication/requirements.txt delete mode 100644 foundation_tools/survey_to_codebook/README.md delete mode 100644 foundation_tools/survey_to_codebook/output_codebook.md delete mode 100644 foundation_tools/survey_to_codebook/sample_xlsform.xlsx delete mode 100644 foundation_tools/survey_to_codebook/xlsform_to_codebook.py delete mode 100644 kumu_maps/README.md create mode 100644 label_variables/__init__.py create mode 100644 label_variables/companions/label.R create mode 100644 label_variables/companions/label.do rename foundation_tools/label_variables/label_variables.sps => label_variables/companions/label.sps (100%) create mode 100644 label_variables/labels.py delete mode 100644 label_variables/scripts/label_data_python.py delete mode 100644 label_variables/scripts/label_data_r.R delete mode 100644 label_variables/scripts/label_data_stata.do create mode 100644 label_variables/test_label_variables.py create mode 100644 learning_layers/embedding_learning_in_the_mel_cycle.md delete mode 100644 miro/README.md delete mode 100644 observable_notebooks/README.md delete mode 100644 open_data_editor/README.md delete mode 100644 powerbi_reports/README.md create mode 100644 replication/.gitignore create mode 100644 replication/analysis.py create mode 100644 replication/expected_results.json create mode 100644 replication/run_all.py create mode 100644 replication/run_regression.R delete mode 100644 replication/scripts/run_regression.R delete mode 100644 replication/scripts/run_regression.py create mode 100644 replication/test_replication.py create mode 100644 replication/verify.py create mode 100644 survey_to_codebook/__init__.py create mode 100644 survey_to_codebook/__main__.py delete mode 100644 survey_to_codebook/convert_xlsform_to_codebook.ipynb create mode 100644 survey_to_codebook/output/dictionary.csv create mode 100644 survey_to_codebook/test_survey_to_codebook.py create mode 100644 survey_to_codebook/xlsform_codebook.py create mode 100644 tool_notes/README.md create mode 100644 tool_notes/excalidraw_frameworks/README.md rename {excalidraw_frameworks => tool_notes/excalidraw_frameworks}/ecosystem_map.excalidraw (100%) rename {excalidraw_frameworks => tool_notes/excalidraw_frameworks}/ecosystem_map.png (100%) rename {excalidraw_frameworks => tool_notes/excalidraw_frameworks}/mel_framework.excalidraw (100%) rename {excalidraw_frameworks => tool_notes/excalidraw_frameworks}/mel_framework.png (100%) create mode 100644 tool_notes/flourish_charts/README.md rename {flourish_charts => tool_notes/flourish_charts}/embed_example.md (100%) rename {flourish_charts => tool_notes/flourish_charts}/flourish_embed_screenshot.png (100%) create mode 100644 tool_notes/kumu_maps/README.md rename {kumu_maps => tool_notes/kumu_maps}/district_health_network.json (100%) rename {kumu_maps => tool_notes/kumu_maps}/district_health_network.png (100%) rename {miro => tool_notes/miro}/HCD_Miro.pdf (100%) rename {miro => tool_notes/miro}/Miro HCD Template Example.png (100%) rename {miro => tool_notes/miro}/Miro Purpose.png (100%) create mode 100644 tool_notes/miro/README.md rename {miro => tool_notes/miro}/miro_use_case_india.md (100%) rename {miro => tool_notes/miro}/sample_miro_board_links.txt (100%) create mode 100644 tool_notes/observable_notebooks/README.md rename {observable_notebooks => tool_notes/observable_notebooks}/dashboard_screenshot.png (100%) rename {observable_notebooks => tool_notes/observable_notebooks}/district_vulnerability_dashboard.ojs (100%) rename {observable_notebooks => tool_notes/observable_notebooks}/public_health_data.csv (100%) create mode 100644 tool_notes/open_data_editor/README.md rename {open_data_editor => tool_notes/open_data_editor}/district_health_gaps.csv (100%) rename {open_data_editor => tool_notes/open_data_editor}/stacked_bar_chart.png (100%) create mode 100644 tool_notes/powerbi_reports/README.md rename {powerbi_reports => tool_notes/powerbi_reports}/mel_dashboard_screenshot.png (100%) create mode 100644 writing_guides/evaluation_report_structure.md create mode 100644 writing_guides/policy_brief.md create mode 100644 writing_guides/results_chain.md create mode 100644 writing_guides/writing_about_uncertainty.md diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index bfad8bc..a59fcf6 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -28,7 +28,7 @@ jobs: - name: Install dependencies run: | python -m pip install --upgrade pip - pip install numpy pandas pyreadstat networkx statsmodels scipy + pip install numpy pandas pyreadstat networkx statsmodels scipy openpyxl # No survey data ships or can. Each folder builds seeded synthetic files # carrying the structure and nobody's data. @@ -49,6 +49,20 @@ jobs: - name: Network effects run: python -m network_effects_sni.test_network_effects + - name: Data validation + run: python data_validation/test_data_validation.py + + - name: Label variables + run: python label_variables/test_label_variables.py + + - name: Survey to codebook + run: python survey_to_codebook/test_survey_to_codebook.py + + # Runs the package end to end in a temporary copy, corrupts a value, and + # checks that verification fails. The R cross-check runs in the R job. + - name: Replication + run: python replication/test_replication.py + r: name: R runs-on: ubuntu-latest @@ -86,3 +100,11 @@ jobs: - name: Sample data working-directory: tests run: Rscript test_dummy_population.R + + # The replication package fits the same model in both languages and + # run_all.py fails if the point estimates differ at six places. + - name: Replication, Python against R + run: | + pip install statsmodels scipy + Rscript replication/run_regression.R + python replication/run_all.py diff --git a/CLAUDE.md b/CLAUDE.md index 78b7e6e..56349b7 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -45,6 +45,12 @@ Rscript test_load_dhs.R # 33: the same ones plus the survey design object cd ../plfs-india && python make_fixture.py --outdir fixtures python test_load_plfs.py # 28 checks Rscript test_load_plfs.R # the same 28 + +python -m network_effects_sni.test_network_effects # 17 tests +python data_validation/test_data_validation.py # 18 +python label_variables/test_label_variables.py # 8 +python survey_to_codebook/test_survey_to_codebook.py # 9 +python replication/test_replication.py # 4; Rscript replication/run_regression.R for the cross-check ``` Stata has no free runtime, so the `.do` files are the only untested ones. Each @@ -67,6 +73,41 @@ plainly rather than implying otherwise. holds the figures DHS published. Reproducing them is the only cheap proof a pipeline is right end to end. +## The 2026-09-08 rebuild of the long tail + +An audit found twenty-one of thirty folders with fewer than six files and nine +with no code, several of them a README and one image presented on the landing +page as a module. What was done, so nobody re-inflates it: + +- **Seven tool showcases consolidated into `tool_notes/`** (Excalidraw, Kumu, + Observable, RawGraphs, Flourish, Power BI, Miro). Each is now a page of + judgement, when to use it and what goes wrong, plus its one worked file. They + are one landing-page row, not seven. Do not promote them back to modules; + the tools live on their own sites and a repository cannot hold a template + for them. +- **`foundation_tools/` is gone.** It held a second, different implementation + of `data_validation`, `label_variables`, `replication` and + `survey_to_codebook`, in five-line functions that overlapped with the + top-level folders. The unique pieces (two `.sps` companions) were moved up + and the rest deleted. +- **Four folders rebuilt as tested Python packages**, each with a plain-assert + suite in CI: `data_validation` (18 tests), `label_variables` (8), + `survey_to_codebook` (9), `replication` (4, plus an R cross-check in the R + job). The three share one data dictionary format (`variable,label,values, + min,max,allowed,required`), so a codebook generated from an XLSForm can be + applied as labels and used as validation rules without retyping. +- **Five folders built out as written content**: `annotated_research` (five + illustrative briefs under one annotation scheme; invented programmes, no + real citations, said so on every page), `eval_docs`, `writing_guides`, + `learning_layers`, `KM_tools`. + +Two things caught while doing it. pandas 3 gives text columns a `str` dtype, +so a blank check keyed on `dtype == object` silently skips every text column; +CI here installs unpinned pandas, so it would have shipped. And the bundled +`survey_to_codebook/input/survey.xlsx` named a choice list `hobby` that the +form referenced as `hobbies`; the old notebook printed an empty list without +comment, the new tool reported it on first run, and the sample is fixed. + ## network_effects_sni Rewritten 2026-09-08 from four scripts that could not run as a chain: one wrote @@ -185,8 +226,10 @@ older version of this table said InsightStack used `minima` and EquityStack was unpublished; both were true once and neither is now. The six calculators are the largest design surface in the stack family and the -obvious place to start. Beyond the stacks: SignalStack, Experiments, -openstacks.dev and the ImpactMojo properties. +obvious place to start. Beyond the stacks: Experiments, openstacks.dev and the +ImpactMojo properties. SignalStack is dead and ViewStack, BridgeStack and +RootStack are archived; the live family is InsightStack, FieldStack, +EquityStack and PolicyStack under OpenStacks-for-Change. One constraint that catches people: `Experiments` serves under a strict Content Security Policy allowlisting specific CDNs, so a design pulling fonts or scripts diff --git a/KM_tools/README.md b/KM_tools/README.md index cbdd706..1e2e8f9 100644 --- a/KM_tools/README.md +++ b/KM_tools/README.md @@ -1,17 +1,19 @@ -# 🧰 KM Tools +# KM_tools -This folder contains real, ready-to-use knowledge management tools that support better file organization, documentation workflows, and internal communication. +Conventions for organising a project's files so that a location says what a +file is and a newcomer can find last year's endline without asking. ---- +| File | What it is | +|---|---| +| `folder_structure_template.md` | One structure, numbered in workflow order, with the rules that make it hold: raw data never edited, every output generated by a script | +| `readme_template.md` | The two-page README a project folder starts with: people, where things are, codes, how to run the analysis | +| `file_naming_conventions.md` | `project_doctype_scope_version` and why | +| `tagging_structure_guidance.md` | Tag axes for a document store: theme, geography, stage, type, audience | +| `documentation_flowchart.pdf` | The path from draft to final for a brief, deck or report | -## 📄 Included Files - -- `file_naming_conventions.md` — A lightweight standard to organize files by purpose, version, and geography -- `tagging_structure_guidance.md` — A guidance note on tagging files, folders, and KM entries in systems like Dropbox, Notion, or Baserow -- `documentation_flowchart.pdf` — A sample documentation flow from draft creation to final output (brief, deck, or report) - ---- - -These tools are especially helpful when you're setting up shared drives, internal documentation processes, or collaborative writing flows with field teams. - -All tools are under MIT — adapt as needed! +Two rules do most of the work. Raw data is never edited; anything wrong with +it is fixed by a script that writes somewhere else. And every table and +figure in a report was produced by a script that is still in the folder. A +project that keeps those two can be handed over, replicated +(`../replication/`), and audited. One that does not has a set of files that +only one person understands. diff --git a/KM_tools/file_naming_conventions.md b/KM_tools/file_naming_conventions.md index 3b35876..d78f81e 100644 --- a/KM_tools/file_naming_conventions.md +++ b/KM_tools/file_naming_conventions.md @@ -2,19 +2,19 @@ Effective file naming helps improve searchability, version control, and team coordination. -### 📂 Recommended Structure +### Recommended Structure ``` [project/code]_[doc_type]_[location/scope]_[version/date].[ext] ``` -### ✅ Examples +### Examples - `ihcrf_blog_khunti_v01.docx` - `gain_report_foodculture2024_final.pdf` - `mohfw_melframework_ncds_draft_2025-04-01.xlsx` -### 🔁 Tips +### Tips - Use lowercase and underscores for clarity - Include dates in `YYYY-MM-DD` format for sorting diff --git a/KM_tools/folder_structure_template.md b/KM_tools/folder_structure_template.md new file mode 100644 index 0000000..64d3be6 --- /dev/null +++ b/KM_tools/folder_structure_template.md @@ -0,0 +1,60 @@ +# Folder structure for a programme or research project + +One structure, used from the first week, so that a file's location says what +it is and a new team member can find last year's endline without asking. + +``` +project_code/ + 00_admin/ contracts, budgets, MoUs, approvals + 01_design/ proposal, theory of change, MEL framework, logframe + 02_instruments/ questionnaires by version, XLSForms, codebooks, consent forms + 03_data/ + raw/ exactly as received; never edited; read-only where the system allows + clean/ produced from raw by a script in 05_code; can be deleted and regenerated + external/ published data used alongside (census, NFHS, admin data), with source and date + 04_field/ training materials, field manuals, daily reports, back-check results + 05_code/ every script that turns raw into clean and clean into outputs + 06_outputs/ + tables/ generated, not hand-edited + figures/ + reports/ drafts and finals, versioned by name (see file_naming_conventions.md) + 07_dissemination/ briefs, decks, press, published versions + 08_archive/ anything superseded, with a note saying by what + README.md what this project is, who runs it, where things are, what the codes mean +``` + +## The rules that make it work + +**Raw data is never edited.** Anything wrong with it is fixed by a script in +`05_code/` that writes to `03_data/clean/`. The fix is then visible, +repeatable, and reversible. A hand-edited raw file is a raw file nobody can +trust and a change nobody can find. + +**Everything in `06_outputs/` was generated by something in `05_code/`.** If a +table was made by hand in a spreadsheet, it lives in `08_archive/` with a +note, and a scripted version replaces it. This is the rule most often broken +and the one that makes a replication package (`../../replication/`) possible +at all. + +**`03_data/external/` records where each file came from and when.** A census +table with no source and no date is a number with no provenance. + +**The README is written in the first week and updated when the structure +changes.** It says what the project code means, who to ask, and where the +things are that are not obvious. `readme_template.md` is a starting point. + +**Numbered top-level folders sort in workflow order**, which is the order a +newcomer needs them in. + +## What goes where, when in doubt + +| It is | It goes in | +|---|---| +| A questionnaire | `02_instruments/`, with a version in the file name | +| The CSV the platform exported | `03_data/raw/`, untouched | +| The script that cleaned it | `05_code/` | +| The cleaned file | `03_data/clean/`, regenerable | +| A chart | `06_outputs/figures/`, from a script | +| The report the chart is in | `06_outputs/reports/` | +| The two-page brief | `07_dissemination/` | +| Last year's version of the questionnaire | `02_instruments/`, older version number; or `08_archive/` if it will not be used again | diff --git a/KM_tools/readme_template.md b/KM_tools/readme_template.md new file mode 100644 index 0000000..a857610 --- /dev/null +++ b/KM_tools/readme_template.md @@ -0,0 +1,50 @@ +# [Project code]: [one-line description] + +*Copy this file to the root of a new project folder and fill it in. Delete the +italic instructions. Keep it under two pages; it is the page a new team member +reads first.* + +## What this is + +*Two or three sentences. What the programme or study does, for whom, where, +and over what period.* + +## People + +| Role | Name | Contact | +|---|---|---| +| Programme lead | | | +| MEL lead | | | +| Data manager | | | +| Field coordinator | | | + +## Where things are + +*The things that are not obvious from the folder structure +(`folder_structure_template.md`).* + +- Raw data: `03_data/raw/`, received on [date] from [source] +- The cleaning script: `05_code/01_clean.py`; run it with `[command]` +- The current questionnaire: `02_instruments/[file]`, version [n] +- The MEL framework: `01_design/[file]` +- Approvals: `00_admin/[file]` + +## Codes and conventions + +- Project code: `[code]`, used as the prefix in every file name + (`file_naming_conventions.md`) +- Site identifiers: *[what the id looks like, and where the list is]* +- Missing value codes: *[e.g. -999 refused, -998 do not know]* +- Rounds: *[baseline = 2024-07, endline = 2025-03]* + +## How to run the analysis + +*The commands, in order, that go from raw data to the tables in the report. +If there is a `run_all` script, name it.* + +## Status + +*What is done, what is in progress, what is next. Dated. Update when it +changes.* + +Last updated: [date] by [initials] diff --git a/KM_tools/tagging_structure_guidance.md b/KM_tools/tagging_structure_guidance.md index 1fe863f..5b5c357 100644 --- a/KM_tools/tagging_structure_guidance.md +++ b/KM_tools/tagging_structure_guidance.md @@ -4,7 +4,7 @@ For internal drives, Notion, Baserow, or Dropbox-style KM systems, use structure --- -## 🏷 Recommended Tag Axes +## Recommended Tag Axes - **Thematic**: health, gender, education, foodculture, WEE, climate - **Geographic**: india, assam, jharkhand, district-specific @@ -14,7 +14,7 @@ For internal drives, Notion, Baserow, or Dropbox-style KM systems, use structure --- -## 📌 Sample Tags +## Sample Tags ``` ["health", "gender", "madhya_pradesh", "learning", "brief", "donor"] @@ -23,7 +23,7 @@ For internal drives, Notion, Baserow, or Dropbox-style KM systems, use structure --- -## 📁 Suggested Usage +## Suggested Usage - As labels in Notion databases - In metadata columns in Excel/Baserow diff --git a/README.md b/README.md index eb6a898..ef54d19 100644 --- a/README.md +++ b/README.md @@ -30,13 +30,13 @@ This is the **knowledge systems layer** of [OpenStacks for Change](https://opens |-----------|-------------|----------|--------| | `data_starters/` | Loaders for real public-use survey microdata, one folder per dataset. DHS across South Asia, and PLFS | Stata, R, Python | Ready | | `calculators/` | District-level calculators for health, education, finance, environment, and population | HTML, JavaScript | Ready | -| `data_validation/` | Data cleaning and validation workflows with intentionally messy test data | Python, R, Stata | Ready | +| `data_validation/` | Rule-driven validation from a data dictionary: duplicates, required, ranges, allowed values, types, name style, column set, cross-file ids; one issue row per problem, by identifier | Python (18 tests); Stata, R, SPSS companions | | `stata_snippets/` | Reusable Stata code: data management, graphs, regression, impact evaluation, surveys | Stata | Ready | | `spss_scripts/` | Survey analysis syntax: cleaning, regression, missing data, frequencies | SPSS | Ready | | `network_effects_sni/` | Peer effects estimation, centrality analysis, diffusion modelling for SHG networks | Python | Ready | -| `replication/` | Replication template with parallel Python and R implementations | Python, R | Ready | -| `label_variables/` | Variable labelling and codebook generation | Stata | Ready | -| `survey_to_codebook/` | Survey instrument to codebook conversion | Python | Ready | +| `replication/` | A replication package that verifies itself: one entry point, a recorded result, a Python-against-R cross-check, and a test that the verification can fail | Python, R (4 tests) | +| `label_variables/` | Variable and value labels from a dictionary, written into `.dta` and `.sav` where they survive; round-trips both ways | Python (8 tests); Stata, R, SPSS companions | +| `survey_to_codebook/` | XLSForm to Markdown codebook and to a label dictionary; resolves groups, repeats and choice lists; reports form defects | Python (9 tests) | | `econometrics/` | Causal inference: DiD, PSM, IV/2SLS, RDD, sensitivity analysis — with Python, R, and sample data | Python, R | Ready | ### Visual and Interactive Tools @@ -44,12 +44,7 @@ This is the **knowledge systems layer** of [OpenStacks for Change](https://opens | Directory | What It Does | Tool | |-----------|-------------|------| | `vensim/` | 8 system dynamics models (health, agriculture, climate, migration, education) | Vensim | -| `observable_notebooks/` | Interactive district vulnerability dashboard | Observable | -| `excalidraw_frameworks/` | MEL framework and ecosystem map diagrams | Excalidraw | -| `flourish_charts/` | Data visualisation templates | Flourish | -| `miro/` | Participatory analysis board templates | Miro | -| `kumu_maps/` | Network and systems mapping | Kumu | -| `powerbi_reports/` | Report templates | Power BI | +| `tool_notes/` | Seven visual and collaboration tools (Excalidraw, Kumu, Observable, RawGraphs, Flourish, Power BI, Miro): when to use each, when not, what goes wrong, one worked file apiece | Notes | | `excel_visuals/` | Excel visualisation templates | Excel | ### Knowledge and Documentation @@ -57,9 +52,11 @@ This is the **knowledge systems layer** of [OpenStacks for Change](https://opens | Directory | What It Contains | |-----------|-----------------| | `learning_library/` | Curated PDFs across 10 categories: programming, data science, research methods, AI tools, MLE resources | -| `writing_guides/` | Style guides and writing templates | -| `annotated_research/` | Annotated bibliography and research notes | -| `learning_layers/` | Learning design frameworks | +| `writing_guides/` | Theory of change, results chain, evaluation report structure, policy brief, writing about uncertainty; the verb is decided by the design | Markdown | +| `eval_docs/` | Logframe template, indicator reference sheet, MEL framework outline, indicator guidance, with a completed example | Markdown, CSV, XLSX | +| `KM_tools/` | Folder structure, project README template, file naming, tagging | Markdown | +| `annotated_research/` | Five research briefs marked up with one eight-tag annotation scheme, weakest design to strongest; illustrative, for teaching critical reading | Markdown | +| `learning_layers/` | Embedding learning in the MEL cycle as scheduled decisions at three tempos | Markdown, PDF | | `taguette_coding/` | Qualitative coding workflows | | `visual_ethnography_descript/` | Ethnographic documentation tools | | `latex/` | LaTeX templates for research reports | @@ -87,9 +84,9 @@ InsightStack is one of several stacks in the [OpenStacks](https://openstacks.dev | Stack | Focus | |-------|-------| | **InsightStack** (this repo) | MEL tools, calculators, documentation | -| [FieldStack](https://github.com/Varnasr/FieldStack) | R notebooks for fieldwork and evaluation | -| [EquityStack](https://github.com/Varnasr/EquityStack) | Python workflows for development data | -| [SignalStack](https://github.com/Varnasr/SignalStack) | Research Rundown newsletter archive | +| [FieldStack](https://github.com/Varnasr/FieldStack) | Field operations and survey analysis in R | +| [EquityStack](https://github.com/Varnasr/EquityStack) | Distributional analysis and survey estimation in Python | +| [PolicyStack](https://github.com/Varnasr/PolicyStack) | 15 flagship schemes, four years of budget data, performance indicators; superseded for new work by [PolicyDhara](https://github.com/Varnasr/PolicyDhara) | ## Contributing diff --git a/annotated_research/README.md b/annotated_research/README.md index 1e42b4c..bca92f4 100644 --- a/annotated_research/README.md +++ b/annotated_research/README.md @@ -1,21 +1,38 @@ -# Annotated Research Briefs - -This folder includes a sample annotated research note to help users understand how research design decisions, methods, and limitations are communicated in development sector outputs. - -## 📄 File Included - -- `annotated_research_brief_mhh.pdf`: A real-style 3-page research note on menstrual health and school attendance in Jharkhand, with inline annotations. - -## 🧠 Why Use This? - -Annotated briefs are useful for: -- Teaching how to critically read and interpret rapid assessments or evaluations -- Training early-career researchers or MEL professionals -- Demonstrating framing, methods, and transparency in reporting - -## 🛠️ Suggestions - -- Use for brown-bag or internal learning sessions -- Create your own versions with comments explaining MEL, qualitative insights, sampling strategies, and more - -Licensed under MIT. Contributions welcome. +# annotated_research + +Five research briefs marked up to show where they are strong and where they +are wrong, with one annotation scheme applied to all of them. + +The point is the scheme. Eight tags, each a question to put to any research +note that lands on your desk: what is it really asking, what is the +comparison, who is in the data, what was measured, is the arithmetic right for +the design, does it name its real weaknesses, does the conclusion's verb match +the design, and who consented. `annotation_scheme.md` defines them and says +how to teach with them. + +| Brief | Design | Main lesson | +|---|---|---| +| `briefs/01_school_health_clubs.md` | Before-after, one arm | A year of growing up is inside the estimate | +| `briefs/02_remedial_reading.md` | Difference-in-differences, matched | The baseline table contradicts the identifying assumption | +| `briefs/03_flood_early_warning.md` | Post-event cross-section | The households not interviewed are the outcome | +| `briefs/04_shg_savings.md` | Randomised by village | Differential attrition and unclustered errors on a good design | +| `briefs/05_cash_transfer_nutrition.md` | Regression discontinuity | A credible local estimate, extended past the population it identifies | + +Ordered from weakest design to strongest, so a reader who works through them +in sequence sees the ceiling on what each design can claim rise, and the +errors change from fatal to specific. + +**All five are illustrative.** The programmes, districts, organisations and +numbers are invented to make the teaching points. No brief describes a real +study or cites one. They are written the way such notes are actually written, +which is why each contains the errors real notes contain. + +The original `annotated_research_brief_mhh.pdf` is the example this folder +started with and is kept as it was. + +## Using them + +Hand out a brief without its annotations. Have readers tag it themselves +using the scheme, then compare. Where two readers tag the same sentence +differently, the text is ambiguous, and the ambiguity is usually where the +misleading claim lives. diff --git a/annotated_research/annotation_scheme.md b/annotated_research/annotation_scheme.md new file mode 100644 index 0000000..c3634c2 --- /dev/null +++ b/annotated_research/annotation_scheme.md @@ -0,0 +1,51 @@ +# The annotation scheme + +Every brief in `briefs/` is marked up with the same eight tags. The tags are +the reusable part: apply them to any research note, evaluation summary or +rapid assessment that lands on your desk, and the places where a document is +strong or weak become visible on one reading. + +An annotation is a blockquote beginning with the tag in bold: + +> **[CLAIM]** The sentence above says "reduced", and the design cannot support +> that verb. "Was lower among" is what the data show. + +## The eight tags + +| Tag | The question it asks of the text | +|---|---| +| **[FRAME]** | What question is this actually answering, and is it the one the title promises? A brief titled "impact of X" that describes a cross-section is answering a different question from the one it names. | +| **[DESIGN]** | What comparison is being made, and what would have happened without the programme? Before-after, with-without, difference-in-differences, randomised: each supports a different strength of claim, and the design decides the ceiling on what the conclusion can say. | +| **[SAMPLE]** | Who is in the data, how were they chosen, and who is missing? Convenience samples, self-selected participants, and dropouts between rounds each bias in a predictable direction. Say which. | +| **[MEASURE]** | What was actually recorded, by whom, and does it mean what the text says it means? Self-reported attendance, a proxy for consumption, an index whose construction is not shown. | +| **[ANALYSIS]** | Is the arithmetic right for the design? Clustered standard errors on a clustered sample, survey weights where the sample was stratified, a control for something that is itself an outcome. | +| **[LIMIT]** | Does the text name its own weaknesses, and are the named ones the real ones? A limitations paragraph that lists "sample size" and omits "no comparison group" is doing PR, not science. | +| **[CLAIM]** | Does the conclusion's verb match what the design can support? *Associated with*, *higher among*, *increased*, *caused*: these are not synonyms. | +| **[ETHICS]** | Consent, who benefited from participating, what happened to the control group, whether any respondent could be identified from what is reported. | + +## How to use it for teaching + +Hand out the brief without the annotations first. Have readers mark it up +with the eight tags themselves, then compare against the annotated version. +The disagreements are the lesson: where two readers tag the same sentence +differently, the text is ambiguous, and that ambiguity is usually where a +misleading claim lives. + +## The briefs + +All five are illustrative. The programmes, districts, organisations and +numbers are invented to make the teaching points, and no brief describes a +real study or cites a real one. Each is written the way such notes are +actually written, which means each contains errors that real notes contain, +and the annotations find them. + +| Brief | Sector | Design | The main lesson | +|---|---|---|---| +| `briefs/01_school_health_clubs.md` | Health and education | Before-after, one arm | A before-after comparison during a school year cannot separate the programme from growing up | +| `briefs/02_remedial_reading.md` | Education | Difference-in-differences, matched schools | The parallel-trends assumption is stated and then contradicted by the baseline table | +| `briefs/03_flood_early_warning.md` | Climate | Cross-section after the event | Survivorship in who could be interviewed, and an outcome measured only where the programme ran | +| `briefs/04_shg_savings.md` | Gender and livelihoods | Randomised at village level, with attrition | A good design weakened by differential dropout, and an analysis that ignores the clustering | +| `briefs/05_cash_transfer_nutrition.md` | Livelihoods and nutrition | Regression discontinuity on an eligibility cutoff | A credible design, and a claim that reaches past the population it identifies | + +The earlier `annotated_research_brief_mhh.pdf` is the original example and is +kept as it was. diff --git a/annotated_research/briefs/01_school_health_clubs.md b/annotated_research/briefs/01_school_health_clubs.md new file mode 100644 index 0000000..3eeb3de --- /dev/null +++ b/annotated_research/briefs/01_school_health_clubs.md @@ -0,0 +1,118 @@ +# Brief 1: School health clubs and menstrual health in a tribal block + +*Illustrative. The programme, the block and every number are invented for +teaching. Annotations in blockquotes use the tags in `../annotation_scheme.md`.* + +--- + +## Summary + +Between July 2024 and March 2025, a school health club programme was +implemented in 42 government upper-primary schools in a tribal block of an +eastern Indian state. The programme trained two peer leaders per school and +ran monthly sessions on menstrual health and hygiene. Attendance among girls in +classes 6 to 8 rose from 71 per cent at baseline to 83 per cent at endline, and +the share of girls reporting use of a sanitary product rose from 38 per cent to +64 per cent. The programme reduced menstruation-related absenteeism and +improved hygiene practice. + +> **[FRAME]** The title says "menstrual health"; the outcomes are attendance +> and product use. Product use is a reasonable proxy for one part of menstrual +> health. Attendance is not a health outcome, and the summary slides between +> the two without saying so. + +> **[CLAIM]** "Reduced" and "improved" are causal verbs. Read on to see whether +> the design earns them. + +## Design and data + +The programme was delivered in all 42 schools in the block. A baseline survey +was administered to all girls present in classes 6 to 8 in July 2024 +(n = 1,914), and an endline survey to all girls present in March 2025 +(n = 1,772). Attendance was taken from school registers for the two months +preceding each survey. + +> **[DESIGN]** Every school got the programme, so there is no comparison group. +> The only comparison available is before against after, in the same schools, +> across one school year. Anything else that changed between July and March is +> in the estimate. Three things always do: the girls are eight months older, +> the monsoon has ended, and the register is being filled in by teachers who +> know the programme is being evaluated. + +> **[SAMPLE]** "All girls present." The endline is 142 girls smaller than the +> baseline. Girls who left school, or were absent on survey day, are absent +> from the endline. If the girls most likely to be absent on any given day are +> also the girls most likely to be absent during menstruation, the endline +> sample is missing exactly the group the programme is about, and the observed +> rise in product use is partly the departure of the girls least likely to +> report it. + +> **[MEASURE]** Attendance "from school registers". In many schools the register +> is completed at the end of the week or the month, from memory, and mid-day +> meal reporting gives teachers a reason to record presence generously. The +> two months before baseline fall in the monsoon and the two months before +> endline do not. A seasonal attendance gap of ten points is ordinary in rural +> schools with no programme at all. + +## Results + +| Indicator | Baseline (Jul 2024) | Endline (Mar 2025) | Change | +|---|---|---|---| +| Attendance, classes 6-8 girls | 71% | 83% | +12 pp | +| Reports using a sanitary product | 38% | 64% | +26 pp | +| Reports missing school during last period | 44% | 27% | -17 pp | +| Can name three hygiene practices | 29% | 78% | +49 pp | + +Differences are significant at the 1 per cent level (chi-square tests). + +> **[ANALYSIS]** A chi-square on pooled girls treats 1,914 and 1,772 as +> independent draws. They are clustered in 42 schools, and the programme was +> delivered at school level. The effective sample for a school-level programme +> is closer to 42 than to 1,900, and a test that ignores that overstates its +> certainty by a wide margin. This is the single most common analytic error in +> programme reports and it is present here. + +> **[MEASURE]** "Reports using a sanitary product" and "reports missing school +> during last period" are self-reports collected by the programme's own field +> staff, eight months after the same staff began teaching the girls what the +> right answer is. Some of the 26-point rise is knowledge of what to say. The +> knowledge indicator, which rose 49 points, tells you how much the girls +> learned about the survey. + +## Limitations + +The study was limited by sample size and by the short duration of the +programme. A longer follow-up would strengthen the findings. + +> **[LIMIT]** Neither of the named limitations is the real one. The sample is +> large. The duration is fine for these outcomes. What is missing is a +> comparison group, and the limitations paragraph does not mention it. A +> limitations section that names the weaknesses least damaging to the +> conclusion is a genre, and this is an example of it. + +## Conclusion + +School health clubs are an effective, low-cost intervention for improving +menstrual health outcomes and reducing absenteeism. Scale-up to the remaining +blocks of the district is recommended. + +> **[CLAIM]** "Effective" claims a cause. The design supports: attendance and +> reported product use were higher at endline than at baseline in schools +> that received the programme. Whether the programme is why cannot be known +> from this study. The honest sentence is weaker and still useful: the +> programme was delivered as designed, girls learned the content, and the +> indicators moved in the right direction, which is enough to justify a +> comparison-group study before scale-up, and not enough to justify scale-up. + +> **[ETHICS]** Peer leaders are girls aged 11 to 13 trained to run sessions for +> their classmates. The brief does not say what consent process applied to +> them, to the girls surveyed, or to their parents; nor whether a girl could +> decline the session without her absence being noted in a register that the +> evaluation then used as an outcome. + +--- + +*What the annotated reader takes away: a before-after study across a school +year, with self-reported outcomes collected by programme staff and a test that +ignores clustering, can show a lot of movement and establish nothing about the +programme. The fix is a design change, not a bigger sample.* diff --git a/annotated_research/briefs/02_remedial_reading.md b/annotated_research/briefs/02_remedial_reading.md new file mode 100644 index 0000000..23e6458 --- /dev/null +++ b/annotated_research/briefs/02_remedial_reading.md @@ -0,0 +1,128 @@ +# Brief 2: Remedial reading camps in matched government primary schools + +*Illustrative. The programme, the district and every number are invented for +teaching. Annotations in blockquotes use the tags in `../annotation_scheme.md`.* + +--- + +## Summary + +A 60-day remedial reading camp was run in 30 government primary schools in a +district of central India, targeting children in classes 3 to 5 who could not +read a class 2 paragraph. Thirty comparison schools were matched on +enrolment, distance to block headquarters and baseline reading level. Using a +difference-in-differences design, the camp raised the share of children able +to read a paragraph by 18 percentage points relative to comparison schools. + +> **[FRAME]** The question is clear and matches the title. The outcome is the +> outcome the programme targets. This is a well-framed brief. + +> **[DESIGN]** Difference-in-differences with matched comparison schools. This +> can support a causal claim *if* treated and comparison schools would have +> moved in parallel without the programme. That assumption is the whole +> argument, and the reader's job is to look for evidence on it. + +## Design and data + +Programme schools were selected by the implementing organisation from +among schools where the head teacher agreed to host a camp. Comparison +schools were then selected from the remaining schools in the same blocks, +matched on total enrolment, distance to block headquarters, and the share of +children reading at paragraph level at baseline. Reading was assessed by +trained assessors using a standard graded tool at baseline (June) and +endline (September). + +> **[SAMPLE]** "Where the head teacher agreed." Programme schools are schools +> with a willing head teacher; comparison schools are, by construction, mostly +> schools without one. A willing head teacher is also the head teacher whose +> school would have improved more anyway. Matching on enrolment, distance and +> baseline reading does nothing about this, because willingness is not one of +> the matched variables and cannot be. This is selection on the thing that +> matters most, and it biases the estimate upward. + +> **[MEASURE]** Independent trained assessors using a graded tool, at both +> rounds, in both arms. This is the right way to measure reading and it is a +> strength of the study. Note it, because the next brief in this set does not +> do it. + +## Baseline balance + +| | Programme (n=30) | Comparison (n=30) | +|---|---|---| +| Enrolment, classes 3-5 | 84 | 81 | +| Distance to block HQ (km) | 11.2 | 11.9 | +| Reading at paragraph level, baseline | 31% | 29% | +| Reading at paragraph level, previous year (from school records) | 22% | 26% | +| Teachers present on visit day | 2.4 | 1.9 | + +> **[DESIGN]** Look at the fourth row. A year before baseline, programme +> schools were three points *behind* comparison schools; at baseline they are +> two points ahead. Programme schools were already improving faster before +> the programme began. That is the parallel-trends assumption failing in the +> table that is supposed to support it. The brief presents this row without +> comment. + +> **[SAMPLE]** Teachers present: 2.4 against 1.9. Half a teacher more per +> school is a large difference in a primary school, and it is the kind of +> difference a willing head teacher produces. It was not matched on, and it +> is a plausible reason the programme schools were improving faster anyway. + +## Results + +| | Programme | Comparison | Difference | +|---|---|---|---| +| Baseline, reading at paragraph level | 31% | 29% | +2 | +| Endline, reading at paragraph level | 58% | 38% | +20 | +| Difference-in-differences | | | **+18 pp** (SE 4.1, clustered by school) | + +> **[ANALYSIS]** Standard errors clustered by school, with 60 clusters. This is +> correct for the design and the brief says so. The arithmetic is right; the +> problem is upstream of it. + +> **[ANALYSIS]** Nine points of the comparison group's improvement (29 to 38) +> happened with no programme, over a monsoon term. That is the counterfactual +> doing its job. Now apply the pre-trend: programme schools were gaining +> roughly five points a year faster than comparison schools before anything +> happened. Over one term that is on the order of two to three points of the +> eighteen. The corrected estimate is still large. It is smaller than the +> headline, and a brief that had done this arithmetic would be more credible, +> not less. + +## Limitations + +The comparison schools were not randomly assigned, and programme schools +were selected in part on head teacher willingness, which may be correlated +with unobserved school quality. Pre-programme trends in reading differed +between the two groups. The estimate should therefore be read as an upper +bound. + +> **[LIMIT]** This is what a limitations section is for. It names the actual +> weakness, says which direction it biases, and tells the reader how to use +> the number. It is also in tension with the summary, which said "raised by +> 18 points" with no such qualification. The person who wrote this paragraph +> and the person who wrote the summary should talk. + +## Conclusion + +Sixty-day reading camps produced a substantial improvement in reading +outcomes relative to comparison schools. Given the selection into the +programme, the true effect is likely smaller than 18 points but remains +educationally meaningful. A randomised evaluation at the next phase would +resolve the remaining uncertainty. + +> **[CLAIM]** "Produced ... relative to comparison schools" is defensible for +> a difference-in-differences, and the sentence after it does the honest +> discounting. This is a brief whose body is better than its summary; the +> lesson is that most readers only read the summary. + +> **[ETHICS]** Children in comparison schools who could not read a paragraph +> were identified by the baseline assessment and then received nothing for a +> term while the evaluation ran. The brief does not say whether they were +> offered the camp afterwards. A design that withholds a remedial programme +> from children known to need it owes them a sentence. + +--- + +*What the annotated reader takes away: a good design does not repair a bad +selection rule. The baseline table contained the evidence against the +identifying assumption, and the brief printed it without reading it.* diff --git a/annotated_research/briefs/03_flood_early_warning.md b/annotated_research/briefs/03_flood_early_warning.md new file mode 100644 index 0000000..44f177e --- /dev/null +++ b/annotated_research/briefs/03_flood_early_warning.md @@ -0,0 +1,119 @@ +# Brief 3: Community flood early warning in a riverine district + +*Illustrative. The programme, the district and every number are invented for +teaching. Annotations in blockquotes use the tags in `../annotation_scheme.md`.* + +--- + +## Summary + +A community-based flood early warning system was established in 60 villages +along a major river in a north-eastern state in 2023, with trained village +volunteers, a river-gauge SMS alert chain and pre-agreed evacuation points. +During the August 2024 flood, a household survey in the programme villages +found that 87 per cent of households received a warning at least six hours +before water entered the village, 91 per cent evacuated livestock, and no +lives were lost. The system is a proven model for flood-prone districts. + +> **[FRAME]** The question a reader wants answered is whether the system +> reduced harm compared with what would have happened without it. The brief +> answers a different question: what happened in the villages that had it. +> Those can look identical on the page and they are not the same question. + +> **[CLAIM]** "Proven model." Hold that phrase against the design below. + +## Design and data + +A rapid household survey was conducted in the 60 programme villages three +weeks after the flood peak. Enumerators visited each village and interviewed +the heads of households present at the time of the visit, targeting 15 +households per village (n = 842 completed interviews). The survey asked when +the household received a warning, from whom, what actions it took, and what +losses it suffered. + +> **[DESIGN]** Programme villages only. There is no non-programme village in +> the data, so nothing in the results can be compared to anything. "87 per +> cent received a warning six hours before" is meaningful only next to the +> figure for villages without the system, and that figure was not collected. +> The state's own gauge-based SMS alerts go to every village with a +> functioning mobile network; some of the 87 per cent is the state system, +> not the programme. + +> **[SAMPLE]** "Households present at the time of the visit", three weeks after +> a major flood. The households not present are the ones still displaced, and +> the ones still displaced are disproportionately the ones who did not +> evacuate in time, lost their house, or lost the livestock they would have +> had to leave. The survey interviewed the households the flood treated +> best. This is survivorship in its most literal form, and every outcome in +> the table is affected by it in the same direction. + +> **[SAMPLE]** 842 of a targeted 900 is 94 per cent completion, which sounds +> good and is the wrong denominator. The right denominator is the number of +> households in those 60 villages before the flood, and the brief does not +> report it. + +> **[MEASURE]** "No lives were lost" is a claim about the population, sourced +> from a sample of the households that stayed. Deaths are counted by the +> district administration, and the brief should cite that count rather than +> infer zero from its own survey. + +## Results + +| Indicator | Programme villages (n=842) | +|---|---| +| Received a warning at least 6 hours before inundation | 87% | +| Warning received from the village volunteer | 64% | +| Warning received from state SMS alert | 58% | +| Evacuated livestock before inundation | 91% | +| Moved to the pre-agreed evacuation point | 73% | +| Reported loss of stored grain | 22% | +| Reported death or injury in the household | 0% | + +> **[MEASURE]** "Received a warning from the village volunteer" and "from state +> SMS" sum to 122 per cent, so most households received both. The programme's +> distinctive contribution is the 29 per cent (87 minus 58) who got a warning +> only through the volunteer chain. That number is in the table and the brief +> does not compute it. + +> **[ANALYSIS]** Percentages with no uncertainty, from a sample clustered in 60 +> villages, of a population of unknown size. A confidence interval is not +> optional here and it would be wide, but the deeper problem is that no +> interval fixes a sample that excludes the worst-affected. + +## Limitations + +The survey was conducted rapidly under post-disaster conditions and could not +reach every household. Recall of the exact timing of warnings may be +imprecise. + +> **[LIMIT]** "Could not reach every household" is true and understated: the +> households not reached are the study's missing outcome. "Recall may be +> imprecise" is the least important thing wrong with a recall of whether +> your house flooded three weeks ago. + +## Conclusion + +The community early warning system delivered timely warnings to nearly nine +in ten households and enabled protective action, with no loss of life. The +model is recommended for replication across the state's flood-prone blocks. + +> **[CLAIM]** The design supports: among households that stayed in programme +> villages, most reported receiving a warning, most from more than one source, +> and most reported protective action. It does not support "delivered", +> "enabled", "no loss of life" or "recommended for replication". A comparison +> with non-programme villages on the same river, from the district's own +> loss records rather than a post-flood survey, would support a claim, and +> could be done from existing data at almost no cost. + +> **[ETHICS]** Interviewing flood-affected households three weeks after the +> event, about their losses, by an organisation that is simultaneously +> distributing relief in those villages, raises the question of whether a +> household could decline. The brief does not address it. The 94 per cent +> completion rate is consistent with households who did not feel they could. + +--- + +*What the annotated reader takes away: a survey of who is left after a +disaster, in the places that had the programme, measures the programme's best +case by construction. The comparison that would answer the question is in the +district administration's records and was not requested.* diff --git a/annotated_research/briefs/04_shg_savings.md b/annotated_research/briefs/04_shg_savings.md new file mode 100644 index 0000000..87a2267 --- /dev/null +++ b/annotated_research/briefs/04_shg_savings.md @@ -0,0 +1,130 @@ +# Brief 4: Financial literacy for self-help groups, randomised by village + +*Illustrative. The programme, the district and every number are invented for +teaching. Annotations in blockquotes use the tags in `../annotation_scheme.md`.* + +--- + +## Summary + +A six-module financial literacy curriculum was delivered to women's +self-help groups in 80 villages randomly selected from 160 in two blocks of a +southern state, with the other 80 serving as controls. Twelve months after +the programme, treated women reported monthly savings 31 per cent higher than +control women (₹412 against ₹314) and were 14 percentage points more likely +to hold an individual bank account. The curriculum caused a substantial and +significant increase in savings behaviour. + +> **[FRAME]** Question, outcome and title agree. Savings and account +> ownership are what the curriculum is for. + +> **[DESIGN]** Randomised at village level, with a control arm. This is the +> design that can support a causal verb, and the brief is entitled to use +> one, subject to what follows. + +## Design and data + +The 160 villages were stratified by block and by the number of active SHGs, +and 80 were assigned to treatment by lottery in a public draw. All SHG +members in all 160 villages were listed at baseline (n = 4,120 women) and a +random sample of 15 per village was surveyed (n = 2,400). The endline survey +attempted to re-interview the same women (n = 1,986 completed). + +> **[SAMPLE]** Public lottery, stratified, baseline listing before assignment. +> This is done properly, and it is rarer than it should be. + +> **[SAMPLE]** Endline completed 1,986 of 2,400: 17 per cent attrition +> overall. The number that matters is attrition *by arm*, and it is in the +> next table. + +## Baseline balance and attrition + +| | Treatment | Control | Difference | +|---|---|---|---| +| Villages | 80 | 80 | | +| Women surveyed at baseline | 1,200 | 1,200 | | +| Monthly savings, baseline (₹) | 288 | 291 | -3 (p=0.84) | +| Holds individual bank account | 41% | 43% | -2 (p=0.51) | +| Years of schooling | 5.1 | 5.0 | +0.1 (p=0.77) | +| **Re-interviewed at endline** | **1,062 (89%)** | **924 (77%)** | **+12 pp (p<0.01)** | + +> **[SAMPLE]** Balance at baseline is good, as randomisation should produce. +> Attrition is not balanced: 89 per cent of treated women were found again +> and 77 per cent of control women. Twelve points of differential attrition in +> a savings study is serious. The women who leave an SHG, migrate for work, or +> stop attending are plausibly the women with the least savings. If the +> control arm lost more of them, the control arm's endline mean is inflated +> and the estimated effect is *understated*; if the treatment arm's extra +> retention is the programme keeping low-savers engaged, the treatment mean +> is *deflated*. Either way the 31 per cent is not the number a full sample +> would give, and the brief has to say which way it thinks the bias runs. + +> **[MEASURE]** "Reported monthly savings." Self-reported, to an enumerator +> from the organisation that ran the training, twelve months after that +> training told the women that saving is what good members do. This is the +> same problem as Brief 1 and it is not fixed by randomisation: both arms are +> asked the same question, but only one arm has been taught the desirable +> answer. Passbook data from the SHG's own records would measure the same +> thing without the demand effect, and the brief does not say why it was not +> used. + +## Results + +| Outcome | Treatment | Control | Difference | p | +|---|---|---|---|---| +| Monthly savings (₹) | 412 | 314 | +98 (+31%) | <0.001 | +| Holds individual bank account | 61% | 47% | +14 pp | <0.001 | +| Took a loan from the SHG in last year | 38% | 36% | +2 pp | 0.42 | +| Loan used for income-generating activity | 52% | 49% | +3 pp | 0.55 | + +Standard errors are heteroskedasticity-robust. Estimates are from OLS with +block fixed effects. + +> **[ANALYSIS]** "Heteroskedasticity-robust" is not the same as clustered. +> Treatment was assigned to villages; women in the same village share the +> assignment and share everything else about the village. The standard errors +> must be clustered at the village, the level of randomisation, with 160 +> clusters. Robust-but-unclustered errors on a village-randomised design are +> too small, often by a factor of two or three. The p-values in this table +> are wrong and the direction is known: every one of them is too small. + +> **[ANALYSIS]** Nothing is said about how the 414 missing women were handled. +> The honest options are: report the effect under bounding assumptions (Lee +> bounds are the standard), or reweight on baseline characteristics and say +> so. Reporting the complete-case difference as if attrition were random, +> when the table shows it was not, is the choice the brief made. + +## Limitations + +Savings were self-reported. Attrition was higher in the control group, which +may bias the estimates. The study covered two blocks and may not generalise. + +> **[LIMIT]** All three named limitations are real, and the middle one is the +> one that matters. "May bias" is doing a lot of work: the brief has the +> baseline characteristics of the attritors and could say in which direction. +> Not saying is a choice. + +## Conclusion + +Financial literacy training delivered through SHGs causes a large and +statistically significant increase in savings and formal account ownership, +and should be integrated into the state's SHG programme. + +> **[CLAIM]** "Causes" is earned by the design and undermined by the analysis. +> With village-clustered errors the savings result is probably still +> significant and the account result probably is; with a bounding treatment +> of attrition the savings result may not be. "Large" depends on whether the +> ₹98 survives both corrections. The brief should have run them; a reader +> cannot. + +> **[ETHICS]** Control villages received nothing for twelve months and the +> brief does not say whether they received the curriculum afterwards. In a +> randomised study where the treatment is training rather than a scarce good, +> delivering it to controls after endline costs little and is the norm. + +--- + +*What the annotated reader takes away: randomisation buys the right to a +causal verb and does not pay for the analysis. Differential attrition and +unclustered errors on a clustered design can each turn a real finding into an +unreliable number, and here both are present.* diff --git a/annotated_research/briefs/05_cash_transfer_nutrition.md b/annotated_research/briefs/05_cash_transfer_nutrition.md new file mode 100644 index 0000000..a65b3b8 --- /dev/null +++ b/annotated_research/briefs/05_cash_transfer_nutrition.md @@ -0,0 +1,123 @@ +# Brief 5: Maternity cash transfer and child weight, at an eligibility cutoff + +*Illustrative. The programme, the state and every number are invented for +teaching. Annotations in blockquotes use the tags in `../annotation_scheme.md`.* + +--- + +## Summary + +A state maternity benefit paid ₹6,000 in three instalments to women in +households scoring below 32 on a 100-point deprivation index. Using +administrative data on 41,000 births and the anthropometric records from the +first-year immunisation visit, a regression discontinuity design at the +cutoff finds that the transfer raised weight-for-age at twelve months by +0.14 standard deviations among children of eligible mothers. The transfer +improves child nutrition and the eligibility threshold should be raised. + +> **[FRAME]** Question and outcome agree. Weight-for-age at twelve months is +> an appropriate, objectively measured nutrition outcome for a transfer paid +> during pregnancy and infancy. + +> **[DESIGN]** Regression discontinuity at an administrative cutoff. Where the +> score cannot be manipulated and nothing else changes at 32, households just +> below and just above are as good as randomly assigned, and the design +> supports a causal claim *for households near the cutoff*. That last clause +> is the one the summary's final sentence forgets. + +## Design and data + +The deprivation index is computed by the state from household census data +collected in 2021 and is not visible to households. Eligibility is +determined by the score at the time of the birth registration. Birth +records for 2023 were linked to the immunisation register, which records +weight at the 9-to-12-month visit, by the mother's identifier. Of 41,200 +births, 36,850 (89 per cent) were linked to a weight record. The analysis +uses births to mothers with index scores between 20 and 44, a bandwidth of +12 points either side of the cutoff (n = 14,300), with local linear +regression and triangular weights. + +> **[DESIGN]** Score computed from prior census data, invisible to households, +> fixed before the birth. That closes the usual manipulation channel: a +> household cannot choose to sit just below 32. The brief should show a +> density plot of the score around the cutoff, which is the standard +> evidence that nobody bunched, and it does not. The claim is plausible; the +> figure is missing. + +> **[SAMPLE]** 89 per cent linked to a weight record, and the question is +> whether linkage differs across the cutoff. Eligible mothers receive the +> third instalment on completing immunisation, so they have a reason to +> attend the visit at which weight is recorded; ineligible mothers do not. +> If attendance jumps at the cutoff, so does the probability of having an +> outcome at all, and the children with a weight record just above 32 are a +> different selection from those just below. This is the study's main +> threat and it is checkable from the same data. + +> **[MEASURE]** Weight recorded by an auxiliary nurse-midwife at a routine +> visit, on whatever scale the sub-centre has, with the child's age from the +> birth record. Objective, not self-reported, and measured identically on +> both sides of the cutoff. Measurement error here adds noise and does not +> bias the discontinuity. This is the right kind of outcome. + +## Results + +| | Estimate | SE | Bandwidth | +|---|---|---|---| +| Weight-for-age z at 12 months, at the cutoff | +0.14 | 0.05 | 12 | +| Same, bandwidth 8 | +0.16 | 0.07 | 8 | +| Same, bandwidth 16 | +0.12 | 0.04 | 16 | +| Probability of a linked weight record, at the cutoff | +0.06 | 0.01 | 12 | +| Mother's age at birth, at the cutoff (placebo) | +0.1 | 0.2 | 12 | +| Birth order, at the cutoff (placebo) | -0.02 | 0.04 | 12 | + +> **[ANALYSIS]** Three bandwidths, two placebo outcomes that do not jump, and +> the estimate is stable across bandwidths. This is how a regression +> discontinuity should be reported, and it is done here. + +> **[ANALYSIS]** The fourth row is the important one and the brief reports it +> without discussing it. The probability of having an outcome jumps six +> points at the cutoff. The children who newly appear in the data on the +> eligible side are children whose mothers attended because of the +> transfer's incentive; if those are healthier-than-average children of +> more-engaged mothers, part of the 0.14 is composition. The standard +> response is to bound the effect under assumptions about the six per cent, +> and it is not done. + +## Limitations + +The estimate is local to households near the cutoff and may not apply to +households far below it. The data cover one year of births. + +> **[LIMIT]** The first sentence is the right limitation, stated correctly. +> The second is fine. The missing one is the differential linkage the +> brief's own table shows. + +## Conclusion + +The transfer improves infant nutrition. Raising the eligibility threshold +from 32 to 40 would extend the benefit to an additional 120,000 births per +year at a cost of ₹72 crore. + +> **[CLAIM]** "Improves infant nutrition" is supported for children of +> mothers scoring near 32, subject to the linkage question. The +> recommendation then takes a local estimate and applies it to households +> scoring 32 to 40, who are less deprived than the households the estimate +> is about. A cash transfer's effect on nutrition is plausibly *smaller* for +> less deprived households, so the extension would likely buy less than 0.14 +> per child. That is not an argument against extending it; it is an argument +> that the cost per unit of effect in the recommendation is understated, and +> the brief's own limitations section already said so. + +> **[ETHICS]** Administrative data linked on the mother's identifier, at +> household level, with a deprivation score. The brief should state who +> approved the linkage, whether the analysis dataset was de-identified, and +> where it is held. Individual-level linked welfare and health records are +> the most sensitive data a state holds about a household. + +--- + +*What the annotated reader takes away: this is the strongest design in the +set, reported mostly well, and it still has one uncorrected threat visible in +its own table and one recommendation that reaches past what the design +identifies. Good studies are the ones where the remaining problems are +specific.* diff --git a/data_validation/README.md b/data_validation/README.md index f4bd2c7..007316b 100644 --- a/data_validation/README.md +++ b/data_validation/README.md @@ -1,15 +1,71 @@ +# data_validation -# Data Validation Folder +A list of what is wrong with a file, by identifier, before it reaches a finding. -This folder includes cross-tool scripts to validate field data across: +```python +from data_validation import validate, Rules -- Python (`validate_data.py`) -- R (`validate_data.R`) -- Stata (`validate_data.do`) +rules = Rules.from_csv("sample_data/data_dictionary.csv", id="id", names="snake_case") +report = validate(df, rules) +report.issues # one row per problem: id, variable, value, check, message +report.summary() # counts by check and severity +report.ok # True only when nothing fired +``` -Each script checks: -- Missing values -- Duplicate IDs -- Invalid gender codes +``` +python data_validation/test_data_validation.py # 18 tests, 40 checks +``` -Sample data is provided in `sample_data/sample_data.csv`. +## What it checks + +| Check | Fires when | +|---|---| +| `id_blank`, `id_duplicate` | The identifier is missing or repeated | +| `required_blank` | A variable listed as required is blank, NA, `.`, or a sentinel code | +| `out_of_range` | A numeric value falls outside a declared `(min, max)`; either end may be open | +| `not_allowed` | A value is not in the declared list, compared as trimmed strings | +| `type_numeric`, `type_integer`, `type_binary`, `type_date` | A value cannot be read as the declared type, reported per value | +| `name_style` | A column name breaks `snake_case` or Stata's 32-character rule | +| `column_missing`, `column_unexpected` | The file's columns differ from the dictionary's | +| `orphan_child`, `childless_parent` | `check_cross_file_ids`: a roster row whose household is not in the household file, and the reverse | + +## Rules come from the dictionary + +`Rules.from_csv` reads the same dictionary that `label_variables/` uses. Add +`type`, `min`, `max`, `allowed` (pipe-separated) and `required` columns and the +rules are the dictionary; leave them out and only the name and column-set +checks run. One file describes the data for validation and for labelling, so +the two cannot drift. + +## Three decisions that shape the output + +**Missing is not wrong, unless the variable is required.** A blank in a +skip-pattern question is correct data. Only variables in `required` are flagged +for blanks; every other check ignores NA. The opposite policy, which the +previous version of this folder had, produces hundreds of false flags and +teaches people to ignore the report. + +**Sentinel codes are missing.** -999, -99 and the rest arrive from ODK and +Stata as numbers. A range check that does not know that reports every refusal +as an age of -999. `Rules.sentinels` lists them and defaults to the common set; +pass `sentinels=[]` if the codes are meaningful in your instrument. + +**Everything is reported by identifier, never by row number.** A row number +changes when the file is re-sorted; an id is what you phone the field team +with. The one exception is a blank id, which has nothing to be reported by, +and the message says so. + +## Companions + +`companions/validate.do` and `companions/validate.R` run the same duplicate, +required, range and allowed checks against the same dictionary, for a team +that works in Stata or R. They are not tested in CI (Stata has no free runtime) +and they do fewer things than the Python; the Python is the reference. + +## What this replaced + +Three scripts that printed `df[~df.gender.isin(["M", "F"])]` and stopped, in +three languages, plus a second folder under `foundation_tools/` with a +different set of five-line functions that overlapped with the first. Neither +reported an id, neither knew about sentinels, and neither could be pointed at a +new file without editing the source. diff --git a/data_validation/__init__.py b/data_validation/__init__.py new file mode 100644 index 0000000..387722c --- /dev/null +++ b/data_validation/__init__.py @@ -0,0 +1,28 @@ +"""Data validation: a list of things wrong with a file, before it reaches a finding. + + from data_validation import validate, Rules + + report = validate(df, Rules( + id="hh_id", + required=["consent", "hh_size"], + ranges={"age": (0, 110), "hh_size": (1, 30)}, + allowed={"gender": ["M", "F", "O"]}, + types={"income": "numeric", "date": "date"}, + names="snake_case", + )) + report.issues # one row per problem: id, variable, value, check, message + report.summary() # counts by check + report.ok # True only when nothing fired + +Everything here returns rows a person can act on, not a boolean. A validation +that prints "3 problems" has told you nothing you can fix; one that lists the +three ids and what is wrong with each has. + +Nothing corrects anything. The version this replaced printed +`df[~df.gender.isin(["M", "F"])]` and stopped, which is a validation only in +the sense that a smoke alarm is a fire brigade. +""" + +from .validate import Rules, Report, validate, check_cross_file_ids + +__all__ = ["Rules", "Report", "validate", "check_cross_file_ids"] diff --git a/foundation_tools/data_validation/check_duplicates.sps b/data_validation/companions/check_duplicates.sps similarity index 100% rename from foundation_tools/data_validation/check_duplicates.sps rename to data_validation/companions/check_duplicates.sps diff --git a/data_validation/companions/validate.R b/data_validation/companions/validate.R new file mode 100644 index 0000000..378f633 --- /dev/null +++ b/data_validation/companions/validate.R @@ -0,0 +1,59 @@ +# Validate a data frame against a data dictionary: duplicates, required, ranges, +# allowed values. R companion to data_validation/validate.py, which is the +# reference and does more. Base R, no packages. +# +# source("data_validation/companions/validate.R") +# issues <- validate_against_dictionary(df, "data_validation/sample_data/data_dictionary.csv", id = "id") + +validate_against_dictionary <- function(df, dictionary_path, id, + sentinels = c(-999, -998, -99, -88, -9)) { + dict <- read.csv(dictionary_path, stringsAsFactors = FALSE, colClasses = "character") + dict[is.na(dict)] <- "" + if (!id %in% names(df)) stop("no id column ", id, call. = FALSE) + + issue <- function(check, ids, variable, value, message) { + data.frame(check = check, id = as.character(ids), variable = variable, + value = as.character(value), message = message, stringsAsFactors = FALSE) + } + is_blank <- function(x) { + if (is.numeric(x)) return(is.na(x) | x %in% sentinels) + t <- trimws(as.character(x)) + is.na(x) | t %in% c("", ".", "NA", "N/A", "n/a") | suppressWarnings(as.numeric(t)) %in% sentinels + } + out <- list() + + ids <- df[[id]] + bl <- is_blank(ids) + if (any(bl)) out[[length(out) + 1]] <- issue("id_blank", paste("row", which(bl)), id, NA, "identifier is blank") + dup <- duplicated(ids[!bl]) | duplicated(ids[!bl], fromLast = TRUE) + if (any(dup)) { + tab <- table(ids[!bl][dup]) + out[[length(out) + 1]] <- issue("id_duplicate", names(tab), id, as.integer(tab), "identifier appears more than once") + } + + for (i in seq_len(nrow(dict))) { + v <- trimws(dict$variable[i]); if (!nzchar(v)) next + if (!v %in% names(df)) { + out[[length(out) + 1]] <- issue("column_missing", "(file)", v, NA, "in the dictionary, absent from the file"); next + } + x <- df[[v]]; blank <- is_blank(x) + if ("required" %in% names(dict) && tolower(dict$required[i]) %in% c("yes", "y", "true", "1") && any(blank)) { + out[[length(out) + 1]] <- issue("required_blank", ids[blank], v, NA, paste(v, "is required but blank")) + } + lo <- if ("min" %in% names(dict)) dict$min[i] else ""; hi <- if ("max" %in% names(dict)) dict$max[i] else "" + if (nzchar(lo) || nzchar(hi)) { + num <- suppressWarnings(as.numeric(as.character(x))); num[blank] <- NA + bad <- !is.na(num) & ((nzchar(lo) & num < as.numeric(lo)) | (nzchar(hi) & num > as.numeric(hi))) + if (any(bad)) out[[length(out) + 1]] <- issue("out_of_range", ids[bad], v, num[bad], + sprintf("%s outside [%s, %s]", v, ifelse(nzchar(lo), lo, "-inf"), ifelse(nzchar(hi), hi, "inf"))) + } + if ("allowed" %in% names(dict) && nzchar(dict$allowed[i])) { + opts <- trimws(strsplit(dict$allowed[i], "|", fixed = TRUE)[[1]]) + vals <- trimws(as.character(x)); bad <- !blank & !vals %in% opts + if (any(bad)) out[[length(out) + 1]] <- issue("not_allowed", ids[bad], v, vals[bad], paste(v, "not one of", paste(opts, collapse = ", "))) + } + } + if (!length(out)) return(data.frame(check = character(), id = character(), variable = character(), + value = character(), message = character(), stringsAsFactors = FALSE)) + do.call(rbind, out) +} diff --git a/data_validation/companions/validate.do b/data_validation/companions/validate.do new file mode 100644 index 0000000..7245bad --- /dev/null +++ b/data_validation/companions/validate.do @@ -0,0 +1,109 @@ +* Validate a dataset against a data dictionary: duplicates, required, ranges, +* allowed values. Stata companion to data_validation/validate.py, which is the +* reference and does more (types, name style, column set, cross-file ids). +* +* Usage, from the repository root: +* do data_validation/companions/validate.do +* after setting the two locals below. + +local dictionary "data_validation/sample_data/data_dictionary.csv" +local datafile "data_validation/sample_data/sample_data.csv" +local idvar "id" + +* Sentinel codes that mean missing, so a range check does not report them. +local sentinels -999 -998 -99 -88 -9 + +* --- read the dictionary into locals ------------------------------------------ +preserve +import delimited using "`dictionary'", clear varnames(1) stringcols(_all) +local n = _N +forvalues i = 1/`n' { + local v`i' = variable[`i'] + local req`i' = lower(trim(required[`i'])) + local min`i' = trim(min[`i']) + local max`i' = trim(max[`i']) + local all`i' = trim(allowed[`i']) +} +restore + +import delimited using "`datafile'", clear varnames(1) + +* --- identifier --------------------------------------------------------------- +display as text _n "== identifier" +capture confirm string variable `idvar' +if _rc == 0 { + count if missing(`idvar') | trim(`idvar') == "" +} +else { + count if missing(`idvar') +} +display as result " blank ids: " r(N) +duplicates report `idvar' +duplicates tag `idvar', gen(_dup) +list `idvar' if _dup > 0, noobs +drop _dup + +* --- per variable ------------------------------------------------------------- +forvalues i = 1/`n' { + local v "`v`i''" + capture confirm variable `v' + if _rc { + display as error " `v': listed in the dictionary but absent from the file" + continue + } + capture confirm numeric variable `v' + local isnum = (_rc == 0) + + * treat sentinels as missing for the numeric checks + if `isnum' { + tempvar clean + gen double `clean' = `v' + foreach s of local sentinels { + replace `clean' = . if `clean' == `s' + } + } + + if inlist("`req`i''", "yes", "y", "true", "1") { + if `isnum' count if missing(`clean') + else count if missing(`v') | trim(`v') == "" + if r(N) > 0 { + display as error " `v': required but blank in " r(N) " row(s)" + if `isnum' list `idvar' if missing(`clean'), noobs + else list `idvar' if missing(`v') | trim(`v') == "", noobs + } + } + + if `isnum' & ("`min`i''" != "" | "`max`i''" != "") { + local cond "" + if "`min`i''" != "" local cond "`clean' < `min`i''" + if "`max`i''" != "" { + if "`cond'" != "" local cond "`cond' | " + local cond "`cond'`clean' > `max`i''" + } + count if (`cond') & !missing(`clean') + if r(N) > 0 { + display as error " `v': " r(N) " value(s) outside [`min`i'', `max`i'']" + list `idvar' `v' if (`cond') & !missing(`clean'), noobs + } + } + + if "`all`i''" != "" { + local opts : subinstr local all`i' "|" `"" ""'", all + if `isnum' { + count if !inlist(string(`v'), "`opts'") & !missing(`clean') + if r(N) > 0 { + display as error " `v': " r(N) " value(s) not in {`all`i''}" + list `idvar' `v' if !inlist(string(`v'), "`opts'") & !missing(`clean'), noobs + } + } + else { + count if !inlist(trim(`v'), "`opts'") & trim(`v') != "" + if r(N) > 0 { + display as error " `v': " r(N) " value(s) not in {`all`i''}" + list `idvar' `v' if !inlist(trim(`v'), "`opts'") & trim(`v') != "", noobs + } + } + } + capture drop `clean' +} +display as text _n "done. Nothing above was changed; fix in the source, not here." diff --git a/data_validation/sample_data/data_dictionary.csv b/data_validation/sample_data/data_dictionary.csv new file mode 100644 index 0000000..578d653 --- /dev/null +++ b/data_validation/sample_data/data_dictionary.csv @@ -0,0 +1,5 @@ +variable,label,type,min,max,allowed,required +id,Unique ID of respondent,text,,,,yes +age,Age in completed years,integer,0,110,,yes +gender,Gender of respondent,text,,,M|F|O,yes +income,Monthly income in INR,numeric,0,,,no diff --git a/data_validation/scripts/python/validate_data.py b/data_validation/scripts/python/validate_data.py deleted file mode 100644 index 3e77b84..0000000 --- a/data_validation/scripts/python/validate_data.py +++ /dev/null @@ -1,17 +0,0 @@ - -import pandas as pd - -# Load data -df = pd.read_csv('../sample_data/sample_data.csv') - -# Check missing values -print("Missing values:") -print(df.isnull().sum()) - -# Check duplicates by ID -print("\nDuplicate rows (by ID):") -print(df[df.duplicated('id')]) - -# Check invalid gender values -print("\nInvalid gender values (allowed: M, F):") -print(df[~df['gender'].isin(['M', 'F'])]) diff --git a/data_validation/scripts/r/validate_data.R b/data_validation/scripts/r/validate_data.R deleted file mode 100644 index 535b43c..0000000 --- a/data_validation/scripts/r/validate_data.R +++ /dev/null @@ -1,14 +0,0 @@ - -df <- read.csv('../sample_data/sample_data.csv') - -# Missing values -cat("Missing values:\n") -print(colSums(is.na(df))) - -# Duplicates -cat("\nDuplicate IDs:\n") -print(df[duplicated(df$id), ]) - -# Invalid gender -cat("\nInvalid gender values:\n") -print(subset(df, !(gender %in% c('M', 'F')))) diff --git a/data_validation/scripts/stata/validate_data.do b/data_validation/scripts/stata/validate_data.do deleted file mode 100644 index 4c27d3b..0000000 --- a/data_validation/scripts/stata/validate_data.do +++ /dev/null @@ -1,11 +0,0 @@ - -use "../sample_data/sample_data.dta", clear - -* Missing values -misstable summarize - -* Duplicate check -duplicates report id - -* Invalid gender check -list if gender != "M" & gender != "F" diff --git a/data_validation/test_data_validation.py b/data_validation/test_data_validation.py new file mode 100644 index 0000000..9f31694 --- /dev/null +++ b/data_validation/test_data_validation.py @@ -0,0 +1,208 @@ +"""Tests for data_validation. Run: python data_validation/test_data_validation.py + +Plain asserts and a main(), matching the other suites here. Each rule is tested +in both directions: that it fires on the case it exists for, and that it stays +quiet on the legitimate case that resembles it, because a validator that flags +every skip-pattern blank is one the team stops reading. +""" + +import sys +from pathlib import Path + +import numpy as np +import pandas as pd + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) +from data_validation import Rules, check_cross_file_ids, validate # noqa: E402 + +HERE = Path(__file__).resolve().parent +PASSED = 0 + + +def check(cond, label): + global PASSED + if not cond: + raise AssertionError(label) + PASSED += 1 + + +def clean(): + return pd.DataFrame({ + "id": ["H1", "H2", "H3", "H4"], + "age": [34, 41, 29, 55], + "gender": ["M", "F", "F", "M"], + "income": [1000.0, 2500.5, 800.0, 12000.0], + "consent": ["yes", "yes", "yes", "yes"], + "followup": ["a", None, None, None], # skip pattern, mostly blank + }) + + +RULES = Rules(id="id", required=["consent"], ranges={"age": (0, 110), "income": (0, None)}, + allowed={"gender": ["M", "F", "O"]}, types={"age": "integer", "income": "numeric"}, + names="snake_case") + + +def test_clean_file_has_no_issues(): + r = validate(clean(), RULES) + check(r.ok, "a clean file passes") + check(len(r.issues) == 0, "and has no issue rows") + + +def test_skip_pattern_blanks_are_not_flagged(): + r = validate(clean(), RULES) + check("followup" not in r.issues["variable"].tolist(), "unlisted blanks are fine") + + +def test_blank_required_is_flagged_for_empty_string_and_na_alike(): + df = clean(); df.loc[1, "consent"] = ""; df.loc[2, "consent"] = None + r = validate(df, RULES) + got = set(r.issues.loc[r.issues.check == "required_blank", "id"]) + check(got == {"H2", "H3"}, "both the empty string and the NA are reported by id") + + +def test_blank_detection_works_on_pandas_string_dtype(): + """pandas 3 gives text columns a `str` dtype, not object. The first + version keyed on object and silently skipped every blank in every text + column. This pins the fix on whichever pandas is installed.""" + df = clean() + df["consent"] = pd.Series(["yes", "", "yes", "yes"], dtype="string") + r = validate(df, RULES) + check("H2" in r.issues.loc[r.issues.check == "required_blank", "id"].tolist(), + "a blank in a string-dtype column is caught") + + +def test_duplicate_ids_report_the_id_and_the_count(): + df = clean(); df.loc[3, "id"] = "H2" + r = validate(df, RULES) + row = r.issues[r.issues.check == "id_duplicate"] + check(len(row) == 1 and row.iloc[0]["id"] == "H2", "one row per duplicated id") + check(row.iloc[0]["value"] == "2", "carrying the count") + + +def test_blank_id_is_reported_by_row_number_and_says_so(): + df = clean(); df.loc[2, "id"] = "" + r = validate(df, RULES) + row = r.issues[r.issues.check == "id_blank"].iloc[0] + check(row["id"] == "row 2", "row number, since there is no id to use") + check("row number" in row["message"], "and the message says why") + + +def test_sentinel_codes_count_as_missing_not_as_out_of_range(): + df = clean(); df.loc[0, "age"] = -999 + r = validate(df, RULES) + check("out_of_range" not in r.issues["check"].tolist(), "-999 is missing, not a bad age") + strict = Rules(id="id", ranges={"age": (0, 110)}, sentinels=[]) + r2 = validate(df, strict) + check("out_of_range" in r2.issues["check"].tolist(), "unless sentinels are switched off") + + +def test_out_of_range_reports_the_value_and_ignores_blanks(): + df = clean(); df.loc[1, "age"] = 210; df.loc[2, "age"] = np.nan + r = validate(df, RULES) + rows = r.issues[r.issues.check == "out_of_range"] + check(rows["id"].tolist() == ["H2"], "only the real violation") + check(rows.iloc[0]["value"] == "210.0", "with the value") + + +def test_open_ended_range(): + df = clean(); df.loc[0, "income"] = -5 + r = validate(df, RULES) + check("H1" in r.issues.loc[r.issues.variable == "income", "id"].tolist(), + "min-only range catches a negative") + df.loc[0, "income"] = 1e9 + check(validate(df, RULES).ok, "and has no upper bound") + + +def test_allowed_values_compare_as_trimmed_strings(): + df = clean(); df.loc[0, "gender"] = " F "; df.loc[1, "gender"] = "X" + r = validate(df, RULES) + bad = r.issues[r.issues.check == "not_allowed"] + check(bad["id"].tolist() == ["H2"], "whitespace is forgiven, X is not") + df2 = clean(); df2["gender"] = [1, 2, 1, 2] + r2 = validate(df2, Rules(id="id", allowed={"gender": ["1", "2"]})) + check(r2.ok, "an integer column matches string codes") + + +def test_type_checks_report_per_offending_value(): + df = clean(); df["age"] = [34, "forty", 29.5, 55] + r = validate(df, RULES) + bad = r.issues[r.issues.check == "type_integer"] + check(set(bad["id"]) == {"H2", "H3"}, "the word and the non-integer, not the column") + + +def test_binary_and_date_types(): + df = clean(); df["consent"] = [1, 0, 2, 1]; df["dob"] = ["2001-04-05", "not a date", "1999-12-31", "2005-01-01"] + r = validate(df, Rules(id="id", types={"consent": "binary", "dob": "date"})) + check(r.issues.loc[r.issues.check == "type_binary", "id"].tolist() == ["H3"], "2 is not binary") + check(r.issues.loc[r.issues.check == "type_date", "id"].tolist() == ["H2"], "one unreadable date") + + +def test_name_style(): + df = clean().rename(columns={"income": "Monthly Income"}) + r = validate(df, RULES) + check("Monthly Income" in r.issues.loc[r.issues.check == "name_style", "variable"].tolist(), + "snake_case violation reported") + check(validate(df, Rules(id="id", names="any")).ok, "and not when names are unconstrained") + + +def test_expected_columns_catch_a_renamed_question(): + df = clean().rename(columns={"income": "hh_income"}) + r = validate(df, Rules(id="id", expected_columns=["age", "gender", "income", "consent", "followup"])) + checks = r.issues.set_index("variable")["check"].to_dict() + check(checks.get("income") == "column_missing", "the old name is missing") + check(checks.get("hh_income") == "column_unexpected", "the new name is unexpected") + + +def test_rules_from_a_data_dictionary(): + rules = Rules.from_csv(HERE / "sample_data" / "data_dictionary.csv", id="id", names="snake_case") + check(rules.required == ["id", "age", "gender"], "required read from the yes column") + check(rules.ranges["age"] == (0.0, 110.0), "range read from min/max") + check(rules.ranges["income"] == (0.0, None), "open upper bound where max is blank") + check(rules.allowed["gender"] == ["M", "F", "O"], "allowed split on |") + check(rules.types["age"] == "integer", "type read") + sample = pd.read_csv(HERE / "sample_data" / "sample_data.csv") + r = validate(sample, rules) + check(isinstance(r.n_rows, int) and r.n_rows == len(sample), "runs on the bundled sample") + + +def test_bad_rules_are_refused_at_construction(): + for kwargs, msg in ((dict(id="id", names="camel"), "names"), + (dict(id="id", types={"a": "float"}), "unknown type"), + (dict(id="id", ranges={"a": (10, 1)}), "exceeds")): + try: + Rules(**kwargs) + except ValueError as exc: + check(msg in str(exc), f"refused: {msg}") + else: + raise AssertionError(f"Rules({kwargs}) should have been refused") + + +def test_cross_file_ids(): + hh = pd.DataFrame({"hh_id": ["H1", "H2", "H3"]}) + members = pd.DataFrame({"hh_id": ["H1", "H1", "H2", "H9"]}) + out = check_cross_file_ids(hh, members, parent_id="hh_id", child_key="hh_id") + check(out.loc[out.check == "orphan_child", "id"].tolist() == ["H9"], "the orphan") + check(out.loc[out.check == "childless_parent", "id"].tolist() == ["H3"], "the childless household") + check(out.loc[out.check == "orphan_child", "severity"].iloc[0] == "error", "orphan is an error") + check(out.loc[out.check == "childless_parent", "severity"].iloc[0] == "note", "childless is a note") + + +def test_report_summary_and_ordering(): + df = clean(); df.loc[3, "id"] = "H2"; df.loc[0, "age"] = 300 + r = validate(df, RULES) + check(r.issues["severity"].tolist()[0] == "error", "errors sort first") + s = r.summary() + check(set(s["check"]) == {"id_duplicate", "out_of_range"}, "summary counts by check") + check("2 issues" in repr(r), "repr states the count") + + +def main() -> int: + tests = [v for k, v in sorted(globals().items()) if k.startswith("test_")] + for fn in tests: + fn(); print(f" ok {fn.__name__}") + print(f"\n{len(tests)} tests, {PASSED} checks, all passed") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/data_validation/validate.py b/data_validation/validate.py new file mode 100644 index 0000000..2fc135e --- /dev/null +++ b/data_validation/validate.py @@ -0,0 +1,328 @@ +"""Rule-driven validation of a tabular file, with an issue row per problem. + +Design notes, so the choices survive a refactor. + +**Rules are data, not code.** A `Rules` object can be built from a data +dictionary CSV as easily as in Python, which is the point: the person who knows +that household size cannot exceed 30 in this survey is usually not the person +writing Python. `Rules.from_csv` reads the same dictionary that +`label_variables/` uses, so one file describes the data for both. + +**Missing is not wrong, unless the variable is required.** A blank in a skip- +pattern question is correct data. Only variables listed in `required` are +flagged for blanks; a range or allowed-values check silently ignores NA. This +is the opposite of what a naive check does, and the naive check produces +hundreds of false flags that teach people to ignore the report. + +**Sentinel codes are missing too.** -999, -99, 98, 99 arrive from ODK and Stata +as numbers, and a range check that does not know that reports every refusal as +an out-of-range age. `Rules.sentinels` lists them; the default covers the usual +conventions. + +**The id is reported, never the row number.** A row number changes when the +file is re-sorted; an id is what you phone the field team with. +""" + +from __future__ import annotations + +import re +from dataclasses import dataclass, field +from pathlib import Path + +import numpy as np +import pandas as pd + +__all__ = ["Rules", "Report", "validate", "check_cross_file_ids"] + +_NAME_STYLES = { + "snake_case": re.compile(r"^[a-z][a-z0-9_]*$"), + "stata": re.compile(r"^[A-Za-z_][A-Za-z0-9_]{0,31}$"), + "any": re.compile(r"^.+$"), +} + +_TYPE_CHECKS = ("numeric", "integer", "date", "text", "binary") + + +@dataclass +class Rules: + """What a valid file looks like. + + Parameters + ---------- + id + Identifier column. Checked for blanks and duplicates, and used to name + every issue. Required. + required + Columns that must never be blank. + ranges + ``{column: (min, max)}``, either end ``None`` for open. + allowed + ``{column: [values]}``. Compared as strings after trimming, so ``1`` + and ``"1"`` match; case is significant. + types + ``{column: "numeric" | "integer" | "date" | "text" | "binary"}``. + A column whose values cannot all be read as that type is reported per + offending value, not per column, so you can see whether it is one + stray entry or a wrong column. + names + ``"snake_case"``, ``"stata"`` (32 chars, letters/digits/underscore) + or ``"any"``. + sentinels + Numeric codes that mean missing. Treated as blank everywhere. + expected_columns + If given, columns present in the file but not listed are reported, + and listed columns absent from the file are reported. Catches a + renamed question between survey rounds. + """ + id: str + required: list[str] = field(default_factory=list) + ranges: dict[str, tuple] = field(default_factory=dict) + allowed: dict[str, list] = field(default_factory=dict) + types: dict[str, str] = field(default_factory=dict) + names: str = "any" + sentinels: list = field(default_factory=lambda: [-999, -998, -99, -88, -9]) + expected_columns: list[str] | None = None + + def __post_init__(self): + if self.names not in _NAME_STYLES: + raise ValueError(f"Rules.names must be one of {sorted(_NAME_STYLES)}") + bad = {c: t for c, t in self.types.items() if t not in _TYPE_CHECKS} + if bad: + raise ValueError(f"Rules.types: unknown type(s) {bad}; use {_TYPE_CHECKS}") + for c, r in self.ranges.items(): + if len(r) != 2: + raise ValueError(f"Rules.ranges[{c!r}] must be (min, max)") + lo, hi = r + if lo is not None and hi is not None and lo > hi: + raise ValueError(f"Rules.ranges[{c!r}]: min {lo} exceeds max {hi}") + + @classmethod + def from_csv(cls, path, *, id: str, names: str = "any", **overrides) -> "Rules": + """Build rules from a data dictionary. + + Expects columns ``variable`` and optionally ``type``, ``min``, ``max``, + ``allowed`` (pipe-separated), ``required`` (yes/no). Missing columns + are simply not used, so the `label_variables/` dictionary with only + ``variable`` and ``label`` works and produces name checks alone. + """ + d = pd.read_csv(path, dtype=str).fillna("") + if "variable" not in d.columns: + raise ValueError(f"{path}: a data dictionary needs a 'variable' column") + d["variable"] = d["variable"].str.strip() + req, ranges, allowed, types = [], {}, {}, {} + for _, row in d.iterrows(): + v = row["variable"] + if not v: + continue + if row.get("required", "").strip().lower() in ("yes", "y", "true", "1"): + req.append(v) + lo = row.get("min", "").strip(); hi = row.get("max", "").strip() + if lo or hi: + ranges[v] = (float(lo) if lo else None, float(hi) if hi else None) + if row.get("allowed", "").strip(): + allowed[v] = [x.strip() for x in row["allowed"].split("|") if x.strip()] + t = row.get("type", "").strip().lower() + if t in _TYPE_CHECKS: + types[v] = t + kwargs = dict(id=id, required=req, ranges=ranges, allowed=allowed, + types=types, names=names, + expected_columns=d["variable"][d["variable"] != ""].tolist()) + kwargs.update(overrides) + return cls(**kwargs) + + +_COLUMNS = ["check", "severity", "id", "variable", "value", "message"] + + +@dataclass +class Report: + issues: pd.DataFrame + n_rows: int + n_columns: int + + @property + def ok(self) -> bool: + return self.issues.empty + + def summary(self) -> pd.DataFrame: + if self.issues.empty: + return pd.DataFrame(columns=["check", "severity", "n"]) + out = (self.issues.groupby(["check", "severity"]).size() + .reset_index(name="n").sort_values(["severity", "n"], ascending=[True, False])) + return out.reset_index(drop=True) + + def by_variable(self) -> pd.DataFrame: + if self.issues.empty: + return pd.DataFrame(columns=["variable", "n"]) + return (self.issues.groupby("variable").size().reset_index(name="n") + .sort_values("n", ascending=False).reset_index(drop=True)) + + def __repr__(self) -> str: + if self.ok: + return f"Report: {self.n_rows} rows, {self.n_columns} columns, no issues" + return (f"Report: {self.n_rows} rows, {self.n_columns} columns, " + f"{len(self.issues)} issues\n{self.summary().to_string(index=False)}") + + +def _issue(check, severity, ids, variable, values, message) -> pd.DataFrame: + ids = list(ids) + values = list(values) if not isinstance(values, str) and values is not None else [values] * len(ids) + return pd.DataFrame({"check": check, "severity": severity, + "id": [str(i) for i in ids], "variable": variable, + "value": [None if v is None else str(v) for v in values], + "message": message}, columns=_COLUMNS) + + +def _blank_mask(s: pd.Series, sentinels) -> pd.Series: + """True where a value should be treated as missing.""" + m = s.isna() + if pd.api.types.is_numeric_dtype(s): + return m | s.isin(sentinels) + # Anything else is treated as text. Not `dtype == object`: pandas 3 gives + # string columns its own `str` dtype, and a check keyed on object silently + # skips every blank in every text column, which is what shipped first. + t = s.astype(str).str.strip() + m = m | t.isin(["", ".", "NA", "N/A", "n/a", "nan", "NaN", "None", ""]) + num = pd.to_numeric(t, errors="coerce") + return m | num.isin(sentinels) + + +def validate(df: pd.DataFrame, rules: Rules) -> Report: + """Apply every rule, return a Report with one issue row per problem.""" + if rules.id not in df.columns: + raise KeyError(f"validate: no id column {rules.id!r} in the frame") + + parts: list[pd.DataFrame] = [] + ids = df[rules.id] + + # id: blanks and duplicates. A blank id cannot be reported by id, so the + # row number is used there and only there, and the message says so. + blank_id = _blank_mask(ids, rules.sentinels) + if blank_id.any(): + parts.append(_issue("id_blank", "error", + [f"row {i}" for i in np.flatnonzero(blank_id)], + rules.id, None, "identifier is blank (reported by row number)")) + dup = ids[~blank_id].astype(str).duplicated(keep=False) + if dup.any(): + d = ids[~blank_id][dup].astype(str) + counts = d.value_counts() + parts.append(_issue("id_duplicate", "error", counts.index, rules.id, counts.values, + "identifier appears more than once")) + + # column set + if rules.expected_columns is not None: + expected = set(rules.expected_columns) + present = set(df.columns) + for c in sorted(expected - present): + parts.append(_issue("column_missing", "error", ["(file)"], c, None, + "listed in the dictionary but absent from the file")) + for c in sorted(present - expected - {rules.id}): + parts.append(_issue("column_unexpected", "warning", ["(file)"], c, None, + "present in the file but not in the dictionary")) + + # names + pat = _NAME_STYLES[rules.names] + for c in df.columns: + if not pat.match(str(c)): + parts.append(_issue("name_style", "warning", ["(file)"], c, c, + f"column name does not follow {rules.names}")) + + # required + for c in rules.required: + if c not in df.columns: + parts.append(_issue("column_missing", "error", ["(file)"], c, None, + "required column is absent")) + continue + m = _blank_mask(df[c], rules.sentinels) + if m.any(): + parts.append(_issue("required_blank", "error", ids[m], c, None, + f"{c} is required but blank")) + + # types + for c, t in rules.types.items(): + if c not in df.columns: + continue + s = df[c] + m = ~_blank_mask(s, rules.sentinels) + vals = s[m] + if t in ("numeric", "integer", "binary"): + num = pd.to_numeric(vals, errors="coerce") + bad = num.isna() + if t == "integer": + bad = bad | (~bad & (num != np.floor(num))) + if t == "binary": + bad = bad | (~bad & ~num.isin([0, 1])) + if bad.any(): + parts.append(_issue(f"type_{t}", "error", ids[m][bad], c, vals[bad], + f"{c} should be {t}")) + elif t == "date": + parsed = pd.to_datetime(vals, errors="coerce", format="mixed") + bad = parsed.isna() + if bad.any(): + parts.append(_issue("type_date", "error", ids[m][bad], c, vals[bad], + f"{c} is not a readable date")) + # text: anything goes + + # ranges, ignoring blanks and non-numeric (reported by the type check if asked) + for c, (lo, hi) in rules.ranges.items(): + if c not in df.columns: + continue + s = df[c] + m = ~_blank_mask(s, rules.sentinels) + num = pd.to_numeric(s[m], errors="coerce") + ok = num.notna() + out = pd.Series(False, index=num.index) + if lo is not None: + out = out | (ok & (num < lo)) + if hi is not None: + out = out | (ok & (num > hi)) + if out.any(): + parts.append(_issue("out_of_range", "warning", ids[m][out], c, num[out], + f"{c} outside [{lo if lo is not None else '-inf'}, " + f"{hi if hi is not None else 'inf'}]")) + + # allowed values + for c, allowed in rules.allowed.items(): + if c not in df.columns: + continue + s = df[c] + m = ~_blank_mask(s, rules.sentinels) + vals = s[m].astype(str).str.strip() + allowed_s = {str(a).strip() for a in allowed} + bad = ~vals.isin(allowed_s) + if bad.any(): + parts.append(_issue("not_allowed", "error", ids[m][bad], c, vals[bad], + f"{c} not one of {sorted(allowed_s)}")) + + issues = (pd.concat(parts, ignore_index=True) if parts + else pd.DataFrame(columns=_COLUMNS)) + sev = issues["severity"].map({"error": 0, "warning": 1}).fillna(2) + issues = issues.assign(_s=sev).sort_values(["_s", "check", "variable", "id"]).drop(columns="_s") + return Report(issues.reset_index(drop=True), len(df), df.shape[1]) + + +def check_cross_file_ids(parent: pd.DataFrame, child: pd.DataFrame, *, + parent_id: str, child_key: str) -> pd.DataFrame: + """Ids in a child file that have no parent, and parents with no children. + + A household roster with a member whose household id is not in the household + file is either a typo or a lost household; either way the member's data + cannot be joined and will drop silently at analysis. The second direction, + a household with no members, is usually fine and is reported as a note. + """ + for frame, col, name in ((parent, parent_id, "parent"), (child, child_key, "child")): + if col not in frame.columns: + raise KeyError(f"check_cross_file_ids: no column {col!r} in the {name} frame") + p = set(parent[parent_id].dropna().astype(str)) + c = child[child_key].dropna().astype(str) + orphans = c[~c.isin(p)] + parts = [] + if not orphans.empty: + counts = orphans.value_counts() + parts.append(_issue("orphan_child", "error", counts.index, child_key, counts.values, + f"{child_key} not found in the parent file")) + childless = sorted(p - set(c)) + if childless: + parts.append(_issue("childless_parent", "note", childless, parent_id, None, + "no rows in the child file")) + return pd.concat(parts, ignore_index=True) if parts else pd.DataFrame(columns=_COLUMNS) diff --git a/eval_docs/README.md b/eval_docs/README.md index 076c707..3906858 100644 --- a/eval_docs/README.md +++ b/eval_docs/README.md @@ -1,21 +1,34 @@ -# Evaluation Documents: School Health Clubs for Menstrual Health & Hygiene (MHH) - -This folder contains realistic MEL tools for a hypothetical education-health program focused on menstrual health and hygiene among adolescent girls. - -## 📄 Files Included - -- `mel_framework_mhh_program.pdf`: A 3-page MEL framework outlining the program theory, objectives, indicators, and learning plans. -- `logframe_mhh_program.xlsx`: A structured logical framework matrix covering goal, outcomes, outputs, indicators, verification sources, and assumptions. - -## 📌 Use Cases - -- Training MEL or program staff on real-world documentation -- Reference for designing new ToCs or MEL systems -- Adaptable templates for NGO proposals or donor reports - -## 🧩 Tips - -- Pair this with your ToC note (in `writing_guides/`) for better coherence -- Use the logframe to feed into your indicator tracking sheet or dashboard - -Licensed under MIT. Contributions welcome. +# eval_docs + +Templates and guidance for the three documents an evaluation is built on: the +logframe, the indicator reference sheet, and the MEL framework. + +| File | What it is | +|---|---| +| `logframe_template.md` | The four-row table, how to fill it in (outcome first), and the errors that recur | +| `logframe_template.csv` | The same table as a spreadsheet | +| `logframe_mhh_program.xlsx` | A completed example: school health clubs for menstrual health | +| `indicator_reference_sheet.md` | One sheet per indicator, with the three fields that matter most | +| `indicator_reference_sheet.csv` | The fields as columns, for a whole indicator set | +| `indicator_guidance.md` | Five tests for an indicator, the types and what each is for, and the errors | +| `mel_framework_outline.md` | Nine sections, with the question a reviewer asks of each | +| `mel_framework_mhh_program.pdf` | A completed example, same programme | + +The completed examples describe a hypothetical programme. The templates are +what to reuse. + +## The one thing to read if you read one + +The assumptions column of the logframe, and specifically the assumption +between outputs and outcomes. That cell is the programme's theory in one +sentence: "trained peer leaders will run sessions, girls will attend, and +attending changes what girls do". If it cannot be written, there is no +theory yet, and no indicator will supply one. + +## Related + +`../writing_guides/` for the theory of change note and the results chain that +the logframe summarises. `../annotated_research/` for five worked examples of +what each evaluation design in section 4 of the framework can and cannot +claim. FieldStack `survey_tools/` for the sample size calculation section 5 +needs, and `field_ops/` for the data quality checks. diff --git a/eval_docs/indicator_guidance.md b/eval_docs/indicator_guidance.md new file mode 100644 index 0000000..3303c98 --- /dev/null +++ b/eval_docs/indicator_guidance.md @@ -0,0 +1,68 @@ +# Choosing indicators, and the errors that make them useless + +An indicator is a number that will be quoted without its context. Choose it +knowing that. + +## Five tests, in the order to apply them + +**1. Does it measure the objective, or something near it?** "Number of health +sessions held" is near "girls know about menstrual hygiene" and is not it. +Sessions held is an output; knowledge is an outcome; practice is a further +outcome; the objective decides which is wanted. The most common indicator +error is measuring the thing that is easy to count and placing it in the row +of the thing that is hard to. + +**2. Can the denominator be stated in one sentence?** If not, the indicator +cannot be computed the same way twice. See `indicator_reference_sheet.md`. + +**3. Will it move in the time available, by enough to see?** A stunting rate +moves a point or two a year with everything going right. A programme +measured on it over eighteen months will show nothing whatever it did. +Choose an indicator on the causal path that responds faster: feeding +practice, dietary diversity, or, if the programme is about it, weight-for-age +in the specific cohort reached. + +**4. Can it be gamed by the people who report it?** Attendance registers +filled in by teachers whose school is being evaluated on attendance. Session +counts reported by the staff whose job depends on session counts. Where the +reporter has a stake, the indicator needs an independent check or a +different source, and the sheet should say which. + +**5. Is it worth what it costs to collect?** An anthropometric survey of two +thousand children costs what three field staff cost for a year. A +programme with eight outcome indicators each needing its own survey has +chosen not to collect any of them properly. + +## Types, and what each is for + +| Type | Example | Use it for | +|---|---|---| +| Count | Peer leaders trained | Outputs. Never as an outcome | +| Coverage | Share of target schools with an active club | Outputs where reach matters more than volume | +| Prevalence | Share of girls using a product | Outcomes. Needs a clean denominator | +| Mean | Mean days absent per girl per month | Outcomes where the distribution matters, and report the distribution too | +| Index | Composite hygiene practice score | Only when the components would be reported separately anyway and a summary is wanted on top. See EquityStack `social_sector/composite_index.py` for what an index has to decide | +| Rate | Deaths per thousand births | Goal-level, from sources you do not control | + +## Errors that recur + +**Percentages of small denominators.** "75 per cent of peer leaders" is three +of four. Report the count when the base is under fifty. + +**Self-reported behaviour, collected by the programme, after the programme +taught the right answer.** The indicator measures what respondents learned to +say. Prefer observed behaviour, records the respondent did not produce, or +an enumerator the respondent does not associate with the programme. + +**A target set from the baseline plus a round number.** Targets should come +from what comparable programmes achieved, and the sheet should cite one. +Where none exists, say the target is a guess. + +**Changing the definition between rounds.** Adding a product to the list of +"sanitary products" at endline raises the indicator without anything having +changed. Freeze the definition at baseline; if it must change, report both. + +**Reporting an outcome only for participants.** Share of *participating* +girls using a product is a different indicator from share of *all* girls in +the target schools, and the first is always higher. The logframe says which +was promised. diff --git a/eval_docs/indicator_reference_sheet.csv b/eval_docs/indicator_reference_sheet.csv new file mode 100644 index 0000000..cf3e0d3 --- /dev/null +++ b/eval_docs/indicator_reference_sheet.csv @@ -0,0 +1 @@ +indicator_name,logframe_level,logframe_row,definition,numerator,denominator,unit,disaggregation,data_source,collection_method,frequency,responsible,baseline_value,baseline_date,baseline_source,target_value,target_date,direction,known_limitations,calculation_notes diff --git a/eval_docs/indicator_reference_sheet.md b/eval_docs/indicator_reference_sheet.md new file mode 100644 index 0000000..a0653e0 --- /dev/null +++ b/eval_docs/indicator_reference_sheet.md @@ -0,0 +1,47 @@ +# Indicator reference sheet + +One of these per indicator. It is the document that stops two people +computing the same indicator two ways, and it is usually written after that +has already happened. Write it before. + +`indicator_reference_sheet.csv` holds the same fields for a whole indicator +set in one file. + +## The sheet + +| Field | What goes here | Example | +|---|---|---| +| **Indicator name** | Short, as it appears in the logframe | Share of girls reporting use of a sanitary product | +| **Logframe level and row** | Which objective it measures | Outcome 1 | +| **Precise definition** | The full sentence, with every term defined | Percentage of girls enrolled in classes 6 to 8 in programme schools who report, when asked in the endline survey, having used a commercially produced or programme-distributed sanitary pad during their most recent menstrual period | +| **Numerator** | Exactly who is counted | Girls in classes 6 to 8 who answered "pad" to Q14 | +| **Denominator** | Exactly who is the base | Girls in classes 6 to 8 who answered Q13 "yes" (have begun menstruating) and were surveyed at endline | +| **Unit** | Percentage, count, rupees, days | Percentage | +| **Disaggregation** | Every breakdown that will be reported | Class; school; whether the girl attended at least three sessions | +| **Data source** | The instrument and the question number | Endline survey, Q13 and Q14 | +| **Collection method** | Who asks, how, of whom | Trained female enumerator, face to face, in a private room, self-report | +| **Frequency** | When it is measured | Baseline (July), endline (March) | +| **Responsible** | A role, not a name | MEL officer | +| **Baseline** | Value, date, source | 38%, July 2024, baseline survey | +| **Target** | Value and date | 60%, March 2025 | +| **Direction** | Higher is better, or lower | Higher | +| **Known limitations** | What this number cannot tell you | Self-reported to programme staff; expect a demand effect; excludes girls absent on survey day, who are more likely to be the girls the programme is about | +| **Calculation notes** | Weighting, exclusions, edge cases | Unweighted; girls who have not begun menstruating are excluded from both numerator and denominator, not counted as zero | + +## The three fields that matter most + +**Precise definition.** Every word that could be read two ways is defined. +"Uses a sanitary product" becomes a sentence naming the products, the +reference period and the source of the report. If the definition changes +between rounds, the indicator is not comparable across rounds, and the sheet +is where the change is recorded. + +**Denominator.** More indicators are wrong in the denominator than in the +numerator. Girls who have not begun menstruating: excluded, or counted as +not using a product? Households that could not be found at endline: in the +denominator or out? The sheet answers it once. + +**Known limitations.** Written by the person who designed the measure, for +the person who will quote it in a report without having read anything else. +It is the only field that protects the number from being used to say more +than it can. diff --git a/eval_docs/logframe_template.csv b/eval_docs/logframe_template.csv new file mode 100644 index 0000000..8d47502 --- /dev/null +++ b/eval_docs/logframe_template.csv @@ -0,0 +1,10 @@ +level,objective,indicator,baseline,baseline_source,target,target_date,means_of_verification,assumptions +Goal,,,,,,,, +Outcome,,,,,,,, +Outcome,,,,,,,, +Output,,,,,,,, +Output,,,,,,,, +Output,,,,,,,, +Activity,,,,,,,, +Activity,,,,,,,, +Activity,,,,,,,, diff --git a/eval_docs/logframe_template.md b/eval_docs/logframe_template.md new file mode 100644 index 0000000..cb22456 --- /dev/null +++ b/eval_docs/logframe_template.md @@ -0,0 +1,66 @@ +# Logical framework template + +A logframe is a four-row table that forces one question at each level: what +we will do, what that produces, what changes as a result, and what larger +change that contributes to. Its value is in the discipline of filling in the +columns, and in the assumptions column most of all, which is the one that +gets left blank. + +`logframe_template.csv` is the same table as a file you can open in a +spreadsheet. `logframe_mhh_program.xlsx` is a completed example for a school +menstrual health programme. + +## The table + +| Level | Objective | Indicator | Baseline | Target | Means of verification | Assumptions | +|---|---|---|---|---|---|---| +| **Goal** | The long-term change the programme contributes to but cannot deliver alone | An outcome measured at population level, usually from a source you do not control | From a published survey | Directional, not numeric, unless you are the main contributor | NFHS, ASER, census, administrative data | What else has to hold for outcomes to add up to this goal | +| **Outcome** | The change in behaviour, practice or condition among the people the programme reaches, that the programme is responsible for | Measured among participants and a comparison, at baseline and endline | From your baseline survey | Numeric, with the date by which | Your own survey, with the design that supports the comparison | What has to hold for outputs to produce this change: the programme's theory, stated as a condition | +| **Output** | What the programme delivers: trained, built, distributed, reached | Counts and coverage | Zero, usually | Numeric, per quarter or year | Monitoring records, attendance sheets, distribution logs | What has to hold for activities to produce outputs: staff in post, materials arrive, schools open | +| **Activity** | What staff do | Completion, on schedule | | Dates | Work plan, reports | Budget released, contracts signed | + +## How to fill it in, in order + +1. **Outcome first.** Not the goal, not the activities. The outcome is the + change you are claiming responsibility for, and it decides everything + else: what the activities have to be, and what the goal can honestly be. +2. **Goal second**, as the larger change your outcome is one contributor to. + If you cannot name a contributor other than yourself, your goal is really + an outcome and belongs a row down. +3. **Outputs third**, as the things that must exist for the outcome to be + possible. Each output should be traceable to at least one outcome; an + output nothing depends on is an activity somebody likes. +4. **Activities last.** They fall out of the outputs. +5. **Assumptions, at every level, before indicators.** The assumption between + activities and outputs is operational (materials arrive). The assumption + between outputs and outcomes is the programme's theory (trained peer + leaders will run sessions, girls will attend, attending changes practice). + The assumption between outcomes and goal is about the rest of the world. + If the outputs-to-outcomes assumption cannot be stated in one sentence, + the programme does not yet have a theory, and no indicator will supply it. +6. **Indicators last**, one or two per row. See `indicator_guidance.md` for + what makes one usable and the errors that make one useless. + +## Errors that recur + +**Outputs dressed as outcomes.** "Number of girls trained" is an output. +"Share of girls using a sanitary product" is an outcome. A logframe whose +outcome row counts things the programme did is measuring effort and calling +it change. + +**A goal indicator the programme cannot move.** "District stunting rate" is a +fine goal indicator and a poor performance measure: fifty programmes and the +monsoon move it. Do not put a target on it that anyone will be held to. + +**Baseline blank.** A target with no baseline is a wish. If the baseline will +come from a survey you have not run yet, write "baseline survey, month X" in +the cell and a date, not nothing. + +**Assumptions written as risks.** "Risk: schools may not cooperate" is a risk. +The assumption is "schools will cooperate", and it belongs next to the +output it is a condition for, so that when a school does not cooperate the +logframe shows which output is now in question. + +**Means of verification that do not exist.** "Endline survey" is a means of +verification only if there is a budget line and a date for it. Check the +column against the budget. diff --git a/eval_docs/mel_framework_outline.md b/eval_docs/mel_framework_outline.md new file mode 100644 index 0000000..c1396dd --- /dev/null +++ b/eval_docs/mel_framework_outline.md @@ -0,0 +1,96 @@ +# MEL framework: an outline with the questions each section has to answer + +A monitoring, evaluation and learning framework is the document that says how +the programme will know whether it is working, and what it will do about it. +`mel_framework_mhh_program.pdf` is a completed example; this is the structure, +with what a reviewer checks in each section. + +## 1. Programme summary and theory of change + +Half a page. What the programme does, for whom, and the causal chain from +activities to outcome with the assumptions stated. If the theory of change is +a diagram elsewhere, the sentence version goes here anyway. + +*Reviewer's check:* can the outcome be reached from the activities by the +stated assumptions, or is there a step the diagram skips? + +## 2. Evaluation questions + +Three to five, ranked. "Did the programme increase product use among girls in +programme schools?" is a question. "Assess impact" is not. Each question +names its outcome and its comparison. + +*Reviewer's check:* does each question have a design in section 4 that can +answer it? A question with no design is a hope. + +## 3. Indicators + +The logframe's indicator column, each with a reference sheet +(`indicator_reference_sheet.md`). Say which are for monitoring (frequent, +from programme records, about delivery) and which for evaluation (baseline +and endline, from surveys, about change). + +*Reviewer's check:* is there at least one outcome indicator that is not +self-reported to programme staff? + +## 4. Evaluation design + +For each evaluation question: the comparison that will be made, why it +supports the claim the question makes, and its known weaknesses. Before- +after, with-without, difference-in-differences, randomised, regression +discontinuity: name it, and name what it cannot rule out. The annotated +briefs in `../annotated_research/` are five worked examples of what each +design can and cannot claim. + +*Reviewer's check:* if the design has no comparison group, does the +framework say so in this section, or only in a limitations paragraph at the +end of the final report? + +## 5. Data collection + +Instruments, samples, timing, who collects, how quality is checked in the +field. Sample size with the calculation shown, not asserted. FieldStack's +`survey_tools/sample_size_calculator.R` does the calculation and +`field_ops/` does the field checks. + +*Reviewer's check:* is the sample size justified for the outcome indicator's +expected change, with the clustering accounted for? A sample sized for a +simple random draw and collected by village is half the size it claims. + +## 6. Analysis plan + +Written before the data arrive. For each evaluation question: the estimator, +the standard errors (clustered at the level of assignment), the +disaggregations that will be reported, how attrition and missing data will be +handled. A pre-specified analysis plan is what separates a finding from a +search. + +*Reviewer's check:* does it say what will be done if attrition differs by +arm? If not, the answer at the time will be "nothing". + +## 7. Learning and use + +Who receives which result, when, in what form, and what decision it feeds. +"Quarterly review meeting, programme manager, one-page dashboard, decides +whether to reallocate sessions across schools" is a use. "Findings will be +disseminated" is not. `../learning_layers/` is about this section. + +*Reviewer's check:* is there a decision named that a result could change? If +every decision is already made, the MEL system is documentation. + +## 8. Ethics and data protection + +Consent, for adults and for minors with guardians. What happens to the +comparison group. Where identifiable data are stored, who can see them, when +they are deleted. Whether anyone can be identified from a reported +disaggregation (a table with one girl in a cell identifies her). + +*Reviewer's check:* is there an approval named, and a person responsible? + +## 9. Budget and schedule + +Every survey in section 5 with a cost and a month. Every analysis in section +6 with the person who will do it. + +*Reviewer's check:* does the endline survey have a budget line? Frameworks +routinely promise an endline nobody has funded. diff --git a/excalidraw_frameworks/README.md b/excalidraw_frameworks/README.md deleted file mode 100644 index 74ec82a..0000000 --- a/excalidraw_frameworks/README.md +++ /dev/null @@ -1,11 +0,0 @@ -# Excalidraw Frameworks - -This folder contains editable `.excalidraw` system logic maps and stakeholder networks, with screenshots. - -## 📂 Folder Contents -- `mel_framework.excalidraw` – Visual MEL flow: Inputs → Activities → Outputs -- `ecosystem_map.excalidraw` – PHC, CHW, Block Officer interaction map -- `mel_framework.png`, `ecosystem_map.png` – Preview PNGs of each map - -## 🧠 Use Case -[Excalidraw](https://excalidraw.com/) is ideal for field-based visual frameworks. You can sketch systems in planning workshops or turn theories of change into quick diagrams. These help MEL teams and frontline facilitators visualize complex interventions during co-creation or training sessions. \ No newline at end of file diff --git a/flourish_charts/README.md b/flourish_charts/README.md deleted file mode 100644 index a4c1eee..0000000 --- a/flourish_charts/README.md +++ /dev/null @@ -1,10 +0,0 @@ -# Flourish Chart Embeds - -This folder contains example embeds and visuals based on [Flourish](https://flourish.studio), a popular tool for visual storytelling and interactive dashboards. - -## 📂 Folder Contents -- `embed_example.md` – HTML iframe code to embed a Flourish chart -- `flourish_embed_screenshot.png` – Screenshot of a district chart with coverage indicators - -## 🧠 Use Case -Flourish helps NGOs, MEL teams, and policy advocates share data interactively — without needing coding skills. It's widely used in India for disaggregated visual storytelling, report supplements, and digital dashboards. \ No newline at end of file diff --git a/foundation_tools/README.md b/foundation_tools/README.md deleted file mode 100644 index c394e7b..0000000 --- a/foundation_tools/README.md +++ /dev/null @@ -1,57 +0,0 @@ -# Foundation Tools for Data Management & Reproducible Research - -This master bundle includes five fully built, reusable modules designed to support field research, MEL systems, and reproducible workflows across public interest data work in India. - ---- - -## 📁 Included Modules - -### 1. `data_validation/` -Reusable scripts for validating raw or cleaned datasets: -- Duplicate ID checks (Python, Stata, SPSS) -- Variable name validation -- Value range scans -- Code label checks - -### 2. `survey_to_codebook/` -Converts an XLSForm to a clean Markdown codebook: -- Input: XLSForm with `survey` and `choices` sheets -- Output: Human-readable `.md` codebook -- Language: Python - -### 3. `data_flow.md` -A Markdown template to document how your dataset moved from: -- Raw form → cleaned version → analysis-ready file -- Includes transformation logs, tool tracking, and derived variable mapping - -### 4. `label_variables/` -Templates to assign meaningful labels to variables in: -- Python (DataFrame metadata) -- SPSS syntax -- Stata `.do` format - -### 5. `replication/` -Basic structure to replicate your core analysis: -- Python regression example -- `requirements.txt` -- `README.md` for clear execution instructions - ---- - -## 🔧 How to Use - -Each folder is independent and reusable. Drop into your project directory, adapt paths, and run based on your toolchain (SPSS/Stata/Python). - ---- - -## 🔍 Why This Matters - -These tools promote better: -- Research reproducibility -- Field-level data cleaning workflows -- Open knowledge practices -- Institutional memory in long-term programs - -This bundle aligns with open standards like DIME (World Bank) and MEL frameworks used by IHCRF, Udaiti, and public health researchers. - ---- \ No newline at end of file diff --git a/foundation_tools/data_flow.md b/foundation_tools/data_flow.md deleted file mode 100644 index c0a7ffa..0000000 --- a/foundation_tools/data_flow.md +++ /dev/null @@ -1,54 +0,0 @@ -# Data Flow Documentation - -This file documents the transformation of a dataset from raw collection to final analysis-ready form. - ---- - -## 📌 Dataset Overview - -| Field | Description | -|------------------|-------------------------------------------------| -| Dataset Name | we_employment_survey.csv | -| Source | XLSForm via Kobo Toolbox | -| Format | CSV | -| Location | Jharkhand, India | -| Collection Date | Jan 2025 | -| Uploaded By | field_team_1 | - ---- - -## 🧹 Data Cleaning & Transformation Log - -| Step No. | Date | Action | Tool Used | Notes | -|----------|------------|-------------------------------------------|-------------|----------------------------------------| -| 1 | 2025-01-15 | Removed duplicates on `respondent_id` | Python | Used `check_duplicates.py` | -| 2 | 2025-01-16 | Recoded `education_level` to numeric | SPSS | Added ordinal labels | -| 3 | 2025-01-17 | Merged with wage_2021.csv | Stata | Merge on `respondent_id` | -| 4 | 2025-01-18 | Converted wide to long format | SPSS | Used `VARSTOCASES` | -| 5 | 2025-01-19 | Created `formal_sector` from employment | Python | New binary derived variable | - ---- - -## 🧮 Derived Variables - -| Variable Name | Description | Formula/Source | -|------------------|----------------------------------------------|-------------------------------------| -| formal_sector | Whether respondent is in formal employment | employment_type recode | -| age_group | Age bins: 18–25, 26–35, 36+ | Binned from `age` | -| edu_level_num | Ordinal numeric version of education_level | Recoded via SPSS | - ---- - -## 📁 Final Output - -| File Name | Description | -|-----------------------------|----------------------------------------| -| we_employment_survey_clean.csv | Cleaned dataset with derived vars | -| we_employment_survey_long.csv | Long-form reshaped dataset | - ---- - -## 🧾 Notes - -- This dataset is now ready for regression and stratified analysis. -- Use `data_validation/` and `spss_tools/` references for applied scripts. \ No newline at end of file diff --git a/foundation_tools/data_validation/README.md b/foundation_tools/data_validation/README.md deleted file mode 100644 index f24f2b5..0000000 --- a/foundation_tools/data_validation/README.md +++ /dev/null @@ -1,14 +0,0 @@ -# Data Validation Scripts - -This folder contains reusable scripts to validate datasets in `.csv`, `.sav`, and `.dta` formats. Checks include: -- Duplicate ID detection -- Variable naming conventions -- Value range checks -- Label inspection - -Languages used: -- Python (pandas) -- SPSS Syntax (.sps) -- Stata (.do) - -Each script is documented inside with usage examples. \ No newline at end of file diff --git a/foundation_tools/data_validation/check_code_labels.sps b/foundation_tools/data_validation/check_code_labels.sps deleted file mode 100644 index ea28382..0000000 --- a/foundation_tools/data_validation/check_code_labels.sps +++ /dev/null @@ -1,2 +0,0 @@ -* SPSS syntax to list and validate variable labels; -DISPLAY DICTIONARY. \ No newline at end of file diff --git a/foundation_tools/data_validation/check_duplicates.do b/foundation_tools/data_validation/check_duplicates.do deleted file mode 100644 index b679315..0000000 --- a/foundation_tools/data_validation/check_duplicates.do +++ /dev/null @@ -1,3 +0,0 @@ -* Stata .do file to check duplicates; -duplicates report respondent_id -duplicates list respondent_id \ No newline at end of file diff --git a/foundation_tools/data_validation/check_duplicates.py b/foundation_tools/data_validation/check_duplicates.py deleted file mode 100644 index 9db7e26..0000000 --- a/foundation_tools/data_validation/check_duplicates.py +++ /dev/null @@ -1,6 +0,0 @@ -import pandas as pd - -def check_duplicates(df, id_column): - dupes = df[df.duplicated(subset=[id_column], keep=False)] - print(f"Found {dupes.shape[0]} duplicate rows based on '{id_column}'") - return dupes \ No newline at end of file diff --git a/foundation_tools/data_validation/scan_value_ranges.py b/foundation_tools/data_validation/scan_value_ranges.py deleted file mode 100644 index 22d45bd..0000000 --- a/foundation_tools/data_validation/scan_value_ranges.py +++ /dev/null @@ -1,9 +0,0 @@ -def scan_ranges(df, rules): - errors = {} - for var, (min_val, max_val) in rules.items(): - if var in df.columns: - invalid = df[(df[var] < min_val) | (df[var] > max_val)] - if not invalid.empty: - errors[var] = invalid - print(f"{invalid.shape[0]} rows out of range in '{var}'") - return errors \ No newline at end of file diff --git a/foundation_tools/data_validation/validate_variable_names.py b/foundation_tools/data_validation/validate_variable_names.py deleted file mode 100644 index 7bf8d6c..0000000 --- a/foundation_tools/data_validation/validate_variable_names.py +++ /dev/null @@ -1,6 +0,0 @@ -import re - -def validate_names(df): - invalid = [col for col in df.columns if not re.match(r'^[a-z_][a-z0-9_]*$', col)] - print(f"Invalid variable names: {invalid}") - return invalid \ No newline at end of file diff --git a/foundation_tools/label_variables/README.md b/foundation_tools/label_variables/README.md deleted file mode 100644 index b283b60..0000000 --- a/foundation_tools/label_variables/README.md +++ /dev/null @@ -1,10 +0,0 @@ -# Variable Labeling Templates - -This folder contains reusable templates for assigning variable labels in: -- Python (with pandas) -- Stata (.do) -- SPSS (.sps) - -## How to Use - -Edit the files to match your dataset and run them in your preferred environment. \ No newline at end of file diff --git a/foundation_tools/label_variables/label_variables.do b/foundation_tools/label_variables/label_variables.do deleted file mode 100644 index ffa47c0..0000000 --- a/foundation_tools/label_variables/label_variables.do +++ /dev/null @@ -1,5 +0,0 @@ -* Label variables in Stata -label variable age "Age of respondent" -label variable wages "Monthly wages" -label variable edu_level_num "Education level (ordinal)" -label variable formal_sector "Formal employment (1=Yes, 0=No)" \ No newline at end of file diff --git a/foundation_tools/label_variables/label_variables.py b/foundation_tools/label_variables/label_variables.py deleted file mode 100644 index 8f06de9..0000000 --- a/foundation_tools/label_variables/label_variables.py +++ /dev/null @@ -1,10 +0,0 @@ -import pandas as pd - -def apply_labels(df, label_dict): - df.attrs['variable_labels'] = label_dict - print("Labels applied (stored in DataFrame metadata).") - return df - -# Example usage: -# labels = {'age': 'Age of respondent', 'wages': 'Monthly wages'} -# df = apply_labels(df, labels) \ No newline at end of file diff --git a/foundation_tools/replication/README.md b/foundation_tools/replication/README.md deleted file mode 100644 index d414cbb..0000000 --- a/foundation_tools/replication/README.md +++ /dev/null @@ -1,25 +0,0 @@ -# Replication Package - -This folder contains scripts and notes to replicate the main results from the analysis. - -## Structure -- `replication_analysis.py`: Sample regression script -- `we_employment_survey_clean.csv`: (Add your cleaned data here) -- `requirements.txt`: Python libraries needed -- `README.md`: This file - -## How to Replicate - -1. Install the packages: -```bash -pip install -r requirements.txt -``` - -2. Run the analysis: -```bash -python replication_analysis.py -``` - -## Notes -- This folder is useful for reviewers, co-authors, or future versions. -- Consider including citation metadata or LICENSE file as needed. \ No newline at end of file diff --git a/foundation_tools/replication/replication_analysis.py b/foundation_tools/replication/replication_analysis.py deleted file mode 100644 index 8b40573..0000000 --- a/foundation_tools/replication/replication_analysis.py +++ /dev/null @@ -1,11 +0,0 @@ -# Example replication analysis in Python (replace with your own model) -import pandas as pd -import statsmodels.api as sm - -df = pd.read_csv('we_employment_survey_clean.csv') -X = df[['age', 'edu_level_num', 'weekly_hours']] -X = sm.add_constant(X) -y = df['wages'] - -model = sm.OLS(y, X).fit() -print(model.summary()) \ No newline at end of file diff --git a/foundation_tools/replication/requirements.txt b/foundation_tools/replication/requirements.txt deleted file mode 100644 index a7cdbf6..0000000 --- a/foundation_tools/replication/requirements.txt +++ /dev/null @@ -1,2 +0,0 @@ -pandas -statsmodels \ No newline at end of file diff --git a/foundation_tools/survey_to_codebook/README.md b/foundation_tools/survey_to_codebook/README.md deleted file mode 100644 index b9fb61a..0000000 --- a/foundation_tools/survey_to_codebook/README.md +++ /dev/null @@ -1,14 +0,0 @@ -# XLSForm to Markdown Codebook Converter - -This tool converts an XLSForm with `survey` and `choices` sheets into a clean Markdown codebook. - -## How to Use - -1. Place your XLSForm in this folder. -2. Run the script: -```bash -python xlsform_to_codebook.py -``` -3. It will generate `output_codebook.md` with variable names, labels, types, and options. - -Works best with forms using `select_one` type choices. \ No newline at end of file diff --git a/foundation_tools/survey_to_codebook/output_codebook.md b/foundation_tools/survey_to_codebook/output_codebook.md deleted file mode 100644 index ed53d1d..0000000 --- a/foundation_tools/survey_to_codebook/output_codebook.md +++ /dev/null @@ -1 +0,0 @@ -Run the script to generate a Markdown codebook here. \ No newline at end of file diff --git a/foundation_tools/survey_to_codebook/sample_xlsform.xlsx b/foundation_tools/survey_to_codebook/sample_xlsform.xlsx deleted file mode 100644 index 67214e1bea8685b982731660f296fcf3ed3fce43..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 6159 zcmZ`-1yof3)*YImYv>jbq+2AUr5RcpsR4l@rKDj;r-KZ;QiQbE7RHG%pd#MCpj5>^x90e2woT zjA^su`j08UbKjeD?yeM}EPtNHPM3RFX5zhIM)vS)jbl|nlXBLPej=TG;hR$&j9OIl z)i2`?^*DaI&t0Fs4`-eO`WFZ2(Vrt26vvBA#EsaehtZB342ffTHc{j>-gAi8SZ$N- zKx6*eu6|ahFl!itPOR&?2&B+%Ge@6bmM`K+RJqMwqd-*6C`Dx@Ja`05zDO9>yg{|S zG#crZ&ZA4~k^5Y!@*t-vRhkCOAiB7Jq^M+^C@XRFbgT#_MJN|IBvlzChw7SgUYQW> z@W|_>tnzv6rv#9k4W|3qTT*(tNDyE=(X9ZjCw`C6W$|YEdSV@L(9!+9;rv(;clKRl zVi#4LeJAtG*XyDdwCV3z$z}(tLhs}7PgpxoC=IVseNGqF@xC<(Tq1-2O2Sc={LB`d zd__b6026*1J6YH|b8`H=md3TIbZ}vb?|aFtU8O?h&~x52(B{uG@rkuEKq+b>^F9Uc z?T5vPV?|n{MLF(nK>BwEMIyH&)z2ug*hoh-&vI!Yde?o-=1Npy{vKod>A^bdIS;_Y8;07G4cDTDnR3H9 z*|q6b_!>njEB$&cFDPdXfg7J+DN260{YaS#3%KW|3pBgt8M!Yw>;?Fnk?OpDjsrLo zvPb{`IsA0D<@`;Go4u2bslC0;&%*pgP*dFC^A0Y&pnY$QigUf%(L{Cn4RyqiwOdH& zHg+jfal=sG>oX8gnF_rCZS-|ANcgy7!1I{@WCiKoiv?64;Bm54gxOBd~#kHh+8mNoMt_~F*aiz}Jc(kLq9q4ltMOrc$8PvOCC?mOh_Jus32~c zjH%0X^7Z@r$twc8x&et+Jxkcks!wT-wdlY6_(+is!bQjZjiFi*mCP3XR8Zh6LOXftte$ERLe#}%PM z(UrC6acZj;+ZT`I(mjWCqc?3f;5Wh4qI()4<3|L0A0NQR@`Kw`1HPma)Of?@r%h%v zYzIN)frA{yyQ;`v=^I86zw9EOPo4V;|3LgZm*-2V1+U<_B>vyI5pPK?FD1E?iUtr|tj|ed7BIjTb)qMK)cC0lQk8IB5Y{ri-gdXEL)b^^LptQg` z9kb-4M`!?jPQOD%syci!ewS%AdI?!Uou-8vIwalabI?BPXxK}22Q!2Ov`nnypMRk3 z>zjJxt<$m`Tv*XAS}_{r>dfxe03d3ROu~ z)aIrMvF|P&`7YfN!G>sEIjo9$k|fWBRESzS>Q?+xZ;DW-nH1L1(@X2Ll9X|le1AnI zdMn}!XIcghPoXcj>Zg9{QIb5z=-$T3|H>c}N}dqo_5o=_Ra>p_bSecOYFhc*N*cWQ zh+REQ>0?lxpzrm_7TVt#oO?LA^%R~#E%+SA{nL&Ao>!4-{m?)zyvB1@wnd6o#4%}q zIq#y_;@S{Cp$2!L_Nz!+!nl_G)KGz@Q}fU6uC9pL6jj+LpBB>NuqiTm36yr9 zY&sqWqX+N9y=F|e{rhj>R4gCHr%rEu4&)%WBmuj+8z3V6& ztGG0%?FTW6oQiM}j0~eseQwu7QeL)~Yr#$Z0yRt|^VWfr_B@hvQ{?li3AN$J=&Ll9 z+-`@m4^JMhK1P8?9^WUmq&qa7qzB?mZWnXcNx3mXbeseP((5%1v`ED$^Eef&%vI#) zSx^1u8hrS@c1gkumc9!9orF*q5GFG`3CQqA$4{LDe{?uoJ#(@!*Kl#NhC-Zwwvcz6 zsv;6MUgy3~cua$GDLs9IzBIED@*ys@;#`I$Qm^^oK;qs9!J3t4gre(%X^V|Y>ZP~~ zVWQHN2sR!SPZbcRf`@X8zRvEfU&~^cMrODo!)+h5b>alP;n_a)PY>FTWx7;E!>Hdn1y}e*4UE`?{vn6L_A+t z_isGMI11Z%rIU_*lXC@|uy!;53YNI6zO|(EcO!z(<~t8%>#`uyD>$>FaIJ{_FJ@gl zY%P9K+N8M{k29AvU;Rx4aCFL?r;a|3QSd(<+=LVj&-@k!S(3N}dOvAhD)^zuOrGZP>C@r0E_ zzCd1ZfTd#gz&xf9Z2p6^yGUSD2fv<}`rkSPSU$hC5CY(;`N zlehQXBN3oQz7pbNVgVrp!C5mh{x2?QI}_g7#9y?OWVc+K(Ye`a=f%*Tu*T6tcMrtjc)2#tByEHc{1-LnZ-o<8XgG)loAhIlu#FMTm?p!**0D}eG+$QSvgo60jLpT$UHpu z@x7qh*sek2mU<#B*Y~>it3jD=D1#WT6+vyOaNy0GL|A^iydRmW+-xd=wc+gL(t`y} z{4gNmku-Fr7Q~VOovA%}M$z9d!U&wBaKtx5QN*9|Z|!asuyk$Gwwd8B2!jz&B2CWb z;hzG@&Dlyz$&RT5%cXm5J0L5=pBcA_gak;1r=!#RI0!e(sLe`Z?veA7Sde zO&tQHJx7;Z}US|GoNe88Rqw{4t2YZP7>^qCo zd#-7q#t*H6Ne4ezDU?5if0n?{>vacabUIU~2$_GH#=h5xaxlS$K+LS)Ult&yR^r6o zb^dV^i58bbn@{3-Y8?ITz^1NAFTdsN?@;FPx)8**kKGHIFdeEcg{(Z6Aoix?qEY%k8?-z=^M%x#QKG!Fc z#?4$v2{>MdQ-DQc_qQErd`txM^xd0&-6m-g0J897H7t51R-%W9 z`3O@QYAEhD;VHmLI<1X)&@e!$b?5aJ$Po7)x7rwBE^-CKA=-Ijx!r0A%A_a=i3_q^ zHqBkd;S%HN-OI$VvyYQx(je@IN3 zE>3}L64oX-B@8-Dy28WHf}T@$VY$z-?OJA9PMOi97t#XTA; zSP`zIK&=8ZcC6FH+lt|e@|-IVTnzh7Z(AcG;(&9tUG@G)v`c@-M8 zI`)=VnV#w@hf*rswEA;`8qlaT#unSLwYSe<@r`sw`MdpLE}r{xt`PdZGQ+JFZsnop zU9ThseHLFq4(#0dQWREPFpXeWGy)4$BJyXzwacsKtLFXY2yZtn_VQUf66}eIjg@rc zMkhIHwJx0T_}DZa`$2+uoXhtqoKCsll`>@=bi+%QB>j(V80s#Qi#NI2SwbD1fH`cs zAp~y>+T~mIdW9dWsB?)e9$P}h#m#EE15F!pvamO7kF=OyNJPGBt8XXQSx`yV08GZO zyIWc>q>R`Orh&21A!xpZx}4HdT_&;t1N$g}x=%SoqPt^O%dJNQ z$nEdjWzcS|%z;)KF+G4sozC@qx9`+6OrH&rO5 zD0^?E1SA|`;SsG$gdd{j@G97r9*1l58}T#<4E+4+(v~h;a-Qd~IKf!EZA*vG;)O|~ zY~8LrJNp@mNK58UOgZ1_E~vgGhb0n|uk)pwaxV`_*Mw{g&07;JJW>mxnc6xqeh5dr z?ASggDW#KcHc3EbkESHs%f<99g(zb0k|j01k6h|#e9{4ug&MK6=Ee!G&A}-elUe1w zvKCn>C9I<>ZgNrR6z1cz1G^ueX?($+OwMS+};fAWbfe2X=d+a z@l#!8{8`EK8rY|y3f#3wcD+UZk{WtWgd<(}H9j$8k36pWqP{W2Xh@J)Tef5vYI5LK zM*6fp%F%+21ZATWjT*cykfce|x(=n^C|ao=$9Uj^9Flly*sIi54HGQEe#+S@gOG2~ zrUW}%NJBL1KBFyC43I)`EG*3zGq4N9wpgr2NXI|&V|4E;Dy&26vSiA|@11_d_qdh{ zkp9d{BH#$8P(J*#5w3Q98cLwka(p`t_FT2f%C6W$ZiI^oQ$qcFbrf)Q!a^=6d&8Tu>9ZU$_byme})iO3sEUB%c*1teHFXAb6+@g=UoWG@z&wt z@>WWIy|8Q9CU-1$Gr@-lb)rq>x220ZD<0?H)=QW}+a{L+x)su6K4skG3$REl*~W(N zE)@~5I~&N$7j)uS#m61CbXdHze#-e;pG&B*Lo{Q4WE9pCb#l^Du4v|ln736u>>uh3HDpk^<81qQ=PPs1k{)v*Et&C?IbBz|$7$FMH##fafyoy6Qrm}op9SeggBzox zr#V?}Jl&}3(Zz6eR5lw@T;oeD~S7#04uNAQB<{Z_^Z>uwO4f|DVS{Sg3aa?izl-!vO$4 ze;B+7{>%Qm8+g|j`xBT1@5cY_lHJ9*>l^*SX@OTD9OoZhq`N409g9CG5pV(c4=BI< zi@PXyx8^@6%4q*V`S%WeH~8+h_a}G+Zg9i@!vEX-?jqdXYW^UwVf;q;&yI69^sdzX z6RM8&EA+1Ly$f)6l>PzWCHRNIdN=&;)c6w~0{8O&GnfB0L%@nCs6X$(gulDs)xrz< H`R)Gz22+<7 diff --git a/foundation_tools/survey_to_codebook/xlsform_to_codebook.py b/foundation_tools/survey_to_codebook/xlsform_to_codebook.py deleted file mode 100644 index 6012d95..0000000 --- a/foundation_tools/survey_to_codebook/xlsform_to_codebook.py +++ /dev/null @@ -1,26 +0,0 @@ -import pandas as pd - -def xlsform_to_codebook(xls_path, output_path="output_codebook.md"): - survey = pd.read_excel(xls_path, sheet_name="survey") - choices = pd.read_excel(xls_path, sheet_name="choices") - - grouped_choices = choices.groupby('list_name') - - with open(output_path, "w") as f: - f.write("# Codebook\n\n") - for _, row in survey.iterrows(): - f.write(f"## {row['name']}\n") - f.write(f"**Label**: {row['label']}\n\n") - f.write(f"**Type**: {row['type']}\n\n") - - if "select_one" in row['type']: - list_name = row['type'].split()[1] - if list_name in grouped_choices.groups: - f.write("**Choices:**\n") - for _, c_row in grouped_choices.get_group(list_name).iterrows(): - f.write(f"- {c_row['name']}: {c_row['label']}\n") - f.write("\n") - f.write("---\n\n") - -# Example usage: -# xlsform_to_codebook("sample_xlsform.xlsx") \ No newline at end of file diff --git a/index.html b/index.html index 15bc9e6..97e92c0 100644 --- a/index.html +++ b/index.html @@ -137,57 +137,33 @@

Code and tooling

Who influences whom in a self-help group, how far a practice spreads, and why the obvious peer-effects regression cannot answer the question it looks like it answers. - Replication templatePython, R - Hand your analysis to a reviewer and have it run on their machine, in Python or in R. + ReplicationPython, R + A package that checks itself: one command from raw data to results, a recorded answer, R against Python, and a test that the check can fail. - Data validationPy, R, Stata - Catch the problems in field data before they reach a finding. Ships with deliberately messy data to practise on. + Data validationPy, R, Stata, SPSS + Catch what is wrong with a file, by household id, before it reaches a finding. Rules come from the data dictionary; blanks in skip-pattern questions are not flagged; sentinel codes count as missing. - Label variablesStata, R - Turn cryptic variable names into a codebook a colleague can actually read. + Label variablesPy, R, Stata, SPSS + Put the labels from the dictionary into a Stata or SPSS file, where they survive. A CSV loses them on every round trip. Survey to codebookPython - Go from questionnaire to codebook without retyping it. + Turn an XLSForm into a codebook a reviewer can read top to bottom, and into the label dictionary for its own export. Reports the choice list the form names and does not have. System dynamicsVensim Where an intervention pushes back on itself: eight system models across health, agriculture, climate, migration and education. - - Observable notebooksObservable - Compare districts on vulnerability, interactively. - - - Kumu mapsKumu - Map who is connected to whom, and where the influence sits. - - - Excalidraw frameworksExcalidraw - Framework and ecosystem diagrams you can edit and drop into a deck. - - - Flourish chartsFlourish - The embed snippet for putting a Flourish chart into a page, with a screenshot of one. + + Tool notes7 tools + Excalidraw, Kumu, Observable, RawGraphs, Flourish, Power BI and Miro: when each is worth the setup, when it is not, and what goes wrong, with one worked file apiece. Excel visualsExcel Charts for the colleague who will only ever open Excel. - - Power BI reportsPower BI - Notes on adapting Microsoft's sample dashboard for MEL reporting. No .pbix file here; the README links to theirs. - - - Miro boardsMiro - An outline for running a human-centred design session, a template PDF and board links. - - - Open Data EditorWorkflows - A district health-gaps CSV shaped for RawGraphs and Datawrapper, with an example chart. - Taguette codingQualitative Code interview transcripts and get the tags back out for analysis. @@ -197,20 +173,28 @@

Code and tooling

A Descript storyboard and a climate fieldwork transcript, showing how a recording becomes notes. - Annotated researchReading - One annotated brief on menstrual health and school attendance in Jharkhand, marked up to show how design and limitations get reported. + Annotated research5 briefs + Five research briefs marked up with the same eight questions, weakest design to strongest. Hand one out unmarked, have readers tag it, compare. Learning libraryReference A reading shelf: programming, data science, research methods, AI tools. - Learning layersFrameworks - A framework for how a health system builds adaptive capacity by learning at each stage of implementation. + Learning layersPractice + A MEL system that changes nothing has no learning in the calendar. Three layers at three tempos, each a decision with a date, a person and the evidence in front of them. - Writing guidesDocumentation - Notes and a Quarto template for writing a theory of change, with a worked example. + Writing guides5 guides + Theory of change, results chain, the evaluation report, the policy brief, and how to write uncertainty into the same sentence as the number. The verb is decided by the design. + + + Evaluation documentsTemplates + Logframe, indicator reference sheet and MEL framework: the template, how to fill it in outcome-first, and the errors that recur. The assumptions column is the programme's theory. + + + Knowledge managementConventions + One folder structure and a README template, so a file's location says what it is. Raw data is never edited; every table came from a script still in the folder. LaTeX templatesLaTeX diff --git a/kumu_maps/README.md b/kumu_maps/README.md deleted file mode 100644 index a452036..0000000 --- a/kumu_maps/README.md +++ /dev/null @@ -1,10 +0,0 @@ -# Kumu Stakeholder Map - -This folder contains a JSON-format stakeholder map and a preview visual representing key actors in a district public health delivery network. - -## 📂 Folder Contents -- `district_health_network.json` – Simplified map structure in Kumu-compatible JSON format -- `district_health_network.png` – Visual rendering of map for documentation or sharing - -## 🧠 Use Case -Kumu helps visualize complex systems such as community health, education delivery, or program stakeholder networks. This example shows how a district health MEL team can map supervisory and service-delivery relationships between PHCs, CHWs, and pregnant women. \ No newline at end of file diff --git a/label_variables/README.md b/label_variables/README.md index f49e36b..170aa8c 100644 --- a/label_variables/README.md +++ b/label_variables/README.md @@ -1,18 +1,57 @@ +# label_variables -# Label Variables Folder +Variable and value labels from a data dictionary, applied to a frame and +written into a file that can carry them. -This folder provides scripts to apply variable labels using a shared dictionary. +```python +from label_variables import read_dictionary, apply_labels, write_labelled -## Files Included: -- `input/data_dictionary.csv` – variable names and labels -- Scripts: - - `label_data_stata.do` - - `label_data_r.R` - - `label_data_python.py` +d = read_dictionary("input/data_dictionary.csv") +df, cov = apply_labels(df, d) # cov says what is unlabelled, and what has no column +write_labelled(df, "out/survey.dta") # labels are now in the file +write_labelled(df, "out/survey.sav") +``` -## Instructions: -1. Place your data file (CSV, DTA) in the same folder. -2. Run the appropriate script to apply labels. -3. Output: a labelled dataset file. +``` +python label_variables/test_label_variables.py # 8 tests, 20 checks +``` -These scripts are reusable for survey cleaning and documentation. +## The dictionary + +``` +variable,label,values +gender,Gender of respondent,1=Male|2=Female|3=Other +consent,Consent to interview,1=Yes|0=No|-99=Refused +``` + +`values` is optional and holds value labels as `code=label` pairs separated by +`|`. The same file serves `data_validation/`, which reads its `min`, `max`, +`allowed` and `required` columns; and `survey_to_codebook/` writes this exact +shape from an XLSForm, so a form's own choice lists become the value labels on +its export without retyping. + +## Why the export step is the point + +A CSV has no labels. A dataset that lives as CSV loses its labels on every +round-trip, and the previous version of this folder attached labels to +`df.attrs` and then wrote a CSV, which discards them on the same line. Stata +and SPSS files carry both variable and value labels in the file, so +`write_labelled` writes `.dta` or `.sav` and refuses `.csv` with an +explanation. + +`dictionary_from_file` goes the other way: given a labelled `.dta` or `.sav`, +it emits the dictionary, so a file you were handed becomes a dictionary you +can keep. The tests check the round trip in both formats. + +## Two Stata limits, enforced rather than truncated + +Variable names are limited to 32 characters and variable labels to 80. The +writer would truncate silently; `write_labelled` raises instead, because a +truncated label reads as a mistake to whoever opens the file next. + +## Companions + +`companions/label.do` and `companions/label.R` apply the same dictionary in +Stata and R. `label.do` reads the `values` column and builds `label define` +statements from it, so the value labels come along, which the previous +Stata script did not do. diff --git a/label_variables/__init__.py b/label_variables/__init__.py new file mode 100644 index 0000000..e1a7797 --- /dev/null +++ b/label_variables/__init__.py @@ -0,0 +1,20 @@ +"""Variable and value labels from a data dictionary, applied and written out. + + from label_variables import read_dictionary, apply_labels, write_labelled + + dictionary = read_dictionary("input/data_dictionary.csv") + df, coverage = apply_labels(df, dictionary) + write_labelled(df, "out/survey.dta") # labels travel into Stata + write_labelled(df, "out/survey.sav") # or SPSS + +A CSV has no labels, so a dataset that lives as CSV loses them on every +round-trip. Stata and SPSS files carry them, which is why the export functions +here matter more than the apply step. The version this replaced stored labels +in `df.attrs` and wrote a CSV, which discards them on the same line. +""" + +from .labels import (apply_labels, coverage, dictionary_from_file, read_dictionary, + write_labelled) + +__all__ = ["read_dictionary", "apply_labels", "coverage", "write_labelled", + "dictionary_from_file"] diff --git a/label_variables/companions/label.R b/label_variables/companions/label.R new file mode 100644 index 0000000..3843ae3 --- /dev/null +++ b/label_variables/companions/label.R @@ -0,0 +1,37 @@ +# Apply variable and value labels from a data dictionary CSV, and write a +# labelled .dta or .sav. R companion to label_variables/labels.py. Needs haven. +# +# source("label_variables/companions/label.R") +# df <- label_from_dictionary(df, "label_variables/input/data_dictionary.csv") +# haven::write_dta(df, "out.dta") + +label_from_dictionary <- function(df, dictionary_path, strict = FALSE) { + dict <- read.csv(dictionary_path, stringsAsFactors = FALSE, colClasses = "character") + dict[is.na(dict)] <- "" + dict$variable <- trimws(dict$variable) + dict <- dict[nzchar(dict$variable), ] + + missing_col <- setdiff(dict$variable, names(df)) + unlabelled <- setdiff(names(df), dict$variable) + if (strict && (length(missing_col) || length(unlabelled))) { + stop("dictionary entries with no column: ", paste(missing_col, collapse = ", "), + "; columns with no entry: ", paste(unlabelled, collapse = ", "), call. = FALSE) + } + + for (i in seq_len(nrow(dict))) { + v <- dict$variable[i] + if (!v %in% names(df)) next + attr(df[[v]], "label") <- dict$label[i] + spec <- if ("values" %in% names(dict)) trimws(dict$values[i]) else "" + if (nzchar(spec) && is.numeric(df[[v]])) { + pairs <- strsplit(strsplit(spec, "|", fixed = TRUE)[[1]], "=", fixed = TRUE) + codes <- as.numeric(vapply(pairs, `[`, "", 1)) + labs <- vapply(pairs, function(p) paste(p[-1], collapse = "="), "") + df[[v]] <- haven::labelled(df[[v]], stats::setNames(codes, labs), + label = dict$label[i]) + } + } + if (length(unlabelled)) message("unlabelled: ", paste(unlabelled, collapse = ", ")) + if (length(missing_col)) message("no such column: ", paste(missing_col, collapse = ", ")) + df +} diff --git a/label_variables/companions/label.do b/label_variables/companions/label.do new file mode 100644 index 0000000..57e0163 --- /dev/null +++ b/label_variables/companions/label.do @@ -0,0 +1,57 @@ +* Apply variable and value labels from a data dictionary CSV. +* Stata companion to label_variables/labels.py. +* +* The dictionary has columns variable, label, values, with values written as +* code=label pairs separated by |, e.g. 1=Yes|0=No|-99=Refused. This reads it, +* labels every variable it can find, builds a value label from the values +* column, and tells you which variables in the data have no entry. + +local dictionary "label_variables/input/data_dictionary.csv" +local datafile "your_data.dta" // set this + +preserve +import delimited using "`dictionary'", clear varnames(1) stringcols(_all) +local n = _N +forvalues i = 1/`n' { + local v`i' = trim(variable[`i']) + local lab`i' = trim(label[`i']) + capture local val`i' = trim(values[`i']) + if _rc local val`i' "" +} +restore + +use "`datafile'", clear + +forvalues i = 1/`n' { + capture confirm variable `v`i'' + if _rc { + display as error "no variable `v`i'' in the data (in the dictionary)" + continue + } + label variable `v`i'' "`lab`i''" + + if "`val`i''" != "" { + capture confirm numeric variable `v`i'' + if _rc { + display as text "`v`i'': value labels only attach to numeric variables; skipped" + continue + } + local pairs : subinstr local val`i' "|" " ", all + local define "" + foreach pair of local pairs { + gettoken code lbl : pair, parse("=") + local lbl = subinstr("`lbl'", "=", "", 1) + local define `"`define' `code' "`lbl'""' + } + capture label drop `v`i''_lbl + label define `v`i''_lbl `define' + label values `v`i'' `v`i''_lbl + } +} + +* Coverage: variables in the data with no dictionary entry. +display as text _n "Unlabelled variables:" +foreach v of varlist _all { + local l : variable label `v' + if "`l'" == "" display as result " `v'" +} diff --git a/foundation_tools/label_variables/label_variables.sps b/label_variables/companions/label.sps similarity index 100% rename from foundation_tools/label_variables/label_variables.sps rename to label_variables/companions/label.sps diff --git a/label_variables/input/data_dictionary.csv b/label_variables/input/data_dictionary.csv index 65b5a29..29c9593 100644 --- a/label_variables/input/data_dictionary.csv +++ b/label_variables/input/data_dictionary.csv @@ -1,5 +1,6 @@ -variable,label -id,Unique ID of respondent -age,Age in completed years -gender,Gender of respondent -income,Monthly income in INR +variable,label,values +id,Unique ID of respondent, +age,Age in completed years, +gender,Gender of respondent,1=Male|2=Female|3=Other +income,Monthly income in INR, +consent,Consent to interview,1=Yes|0=No|-99=Refused diff --git a/label_variables/labels.py b/label_variables/labels.py new file mode 100644 index 0000000..1683788 --- /dev/null +++ b/label_variables/labels.py @@ -0,0 +1,175 @@ +"""Labels: read them from a dictionary, apply, report what is missing, write out. + +The dictionary is a CSV with a ``variable`` column, a ``label`` column, and an +optional ``values`` column holding value labels as ``code=label`` pairs +separated by ``|``, for example ``1=Yes|0=No|-99=Refused``. The same file +serves `data_validation/`, which reads its ``min``, ``max``, ``allowed`` and +``required`` columns and ignores the rest, so one dictionary describes the +data for both. + +Labels live in ``df.attrs["variable_labels"]`` and ``df.attrs["value_labels"]`` +while in pandas, and are written into the file's own metadata by +`write_labelled`. pandas does not preserve ``attrs`` through most operations, +and that is fine: the dictionary is the source of truth, and `apply_labels` +is cheap to call again just before writing. +""" + +from __future__ import annotations + +from pathlib import Path + +import pandas as pd + +__all__ = ["read_dictionary", "apply_labels", "coverage", "write_labelled", + "dictionary_from_file"] + + +def _parse_values(spec: str) -> dict: + """``"1=Yes|0=No"`` to ``{1: "Yes", 0: "No"}``. Numeric codes become numbers.""" + out = {} + for pair in str(spec).split("|"): + pair = pair.strip() + if not pair: + continue + if "=" not in pair: + raise ValueError(f"value label {pair!r} is not code=label") + code, label = pair.split("=", 1) + code = code.strip() + try: + code_v = int(code) + except ValueError: + try: + code_v = float(code) + except ValueError: + code_v = code + out[code_v] = label.strip() + return out + + +def read_dictionary(path) -> pd.DataFrame: + """Read and check a dictionary CSV. Returns variable, label, values (dict).""" + d = pd.read_csv(path, dtype=str).fillna("") + for col in ("variable", "label"): + if col not in d.columns: + raise ValueError(f"{path}: a data dictionary needs a {col!r} column") + d["variable"] = d["variable"].str.strip() + d = d[d["variable"] != ""].copy() + dup = d["variable"][d["variable"].duplicated()] + if not dup.empty: + raise ValueError(f"{path}: variable(s) listed twice: {sorted(set(dup))}") + d["label"] = d["label"].str.strip() + d["values"] = (d["values"].map(_parse_values) if "values" in d.columns + else [dict() for _ in range(len(d))]) + return d[["variable", "label", "values"]].reset_index(drop=True) + + +def coverage(df: pd.DataFrame, dictionary: pd.DataFrame) -> pd.DataFrame: + """One row per variable in either place, saying where it is and is not. + + A column with no dictionary entry ships unlabelled. A dictionary entry with + no column is a renamed question, or a question dropped between rounds, and + either way somebody should know. Both are listed; neither raises, because + a partial dictionary is normal mid-cleaning. + """ + cols = list(df.columns) + dict_vars = dictionary["variable"].tolist() + labels = dict(zip(dictionary["variable"], dictionary["label"])) + rows = [] + for c in cols: + rows.append({"variable": c, "in_data": True, "in_dictionary": c in labels, + "label": labels.get(c, ""), + "status": "labelled" if labels.get(c) else + ("blank label" if c in labels else "no dictionary entry")}) + for v in dict_vars: + if v not in cols: + rows.append({"variable": v, "in_data": False, "in_dictionary": True, + "label": labels[v], "status": "no such column"}) + return pd.DataFrame(rows) + + +def apply_labels(df: pd.DataFrame, dictionary: pd.DataFrame, *, + strict: bool = False) -> tuple[pd.DataFrame, pd.DataFrame]: + """Attach labels to ``df.attrs`` and return the frame with a coverage report. + + ``strict=True`` raises if any column is unlabelled or any dictionary entry + has no column, for the final export where a gap is a defect. + """ + cov = coverage(df, dictionary) + if strict: + bad = cov[cov["status"] != "labelled"] + if not bad.empty: + raise ValueError("apply_labels: unlabelled or unmatched variables:\n" + + bad[["variable", "status"]].to_string(index=False)) + labels = dict(zip(dictionary["variable"], dictionary["label"])) + values = dict(zip(dictionary["variable"], dictionary["values"])) + out = df.copy() + out.attrs["variable_labels"] = {c: labels[c] for c in df.columns if labels.get(c)} + out.attrs["value_labels"] = {c: values[c] for c in df.columns if values.get(c)} + return out, cov + + +def write_labelled(df: pd.DataFrame, path, *, dictionary: pd.DataFrame | None = None) -> Path: + """Write ``.dta`` or ``.sav`` with variable and value labels in the file. + + Uses the labels in ``df.attrs`` unless a dictionary is passed, in which case + it is applied first. Stata variable names are limited to 32 characters and + labels to 80; both are enforced here with an error rather than silently + truncated by the writer. + """ + import pyreadstat + + path = Path(path) + if dictionary is not None: + df, _ = apply_labels(df, dictionary) + var_labels = df.attrs.get("variable_labels", {}) + val_labels = df.attrs.get("value_labels", {}) + + if path.suffix.lower() == ".dta": + too_long = [c for c in df.columns if len(str(c)) > 32] + if too_long: + raise ValueError(f"Stata variable names are limited to 32 characters: {too_long}") + long_labels = {c: l for c, l in var_labels.items() if len(l) > 80} + if long_labels: + raise ValueError(f"Stata variable labels are limited to 80 characters: " + f"{list(long_labels)}") + # Stata value labels attach to numeric variables only. + numeric_val = {c: v for c, v in val_labels.items() + if pd.api.types.is_numeric_dtype(df[c])} + pyreadstat.write_dta(df, str(path), column_labels=var_labels or None, + variable_value_labels=numeric_val or None) + elif path.suffix.lower() == ".sav": + pyreadstat.write_sav(df, str(path), column_labels=var_labels or None, + variable_value_labels=val_labels or None) + else: + raise ValueError(f"write_labelled: {path.suffix!r} cannot carry labels; " + "use .dta or .sav, or keep the dictionary CSV beside a CSV export") + return path + + +def dictionary_from_file(path) -> pd.DataFrame: + """Read a labelled ``.dta`` or ``.sav`` and emit a dictionary frame. + + The reverse direction: when the labelled file is what you were given and + the dictionary is what you need to write down. Round-trips with + `write_labelled`, which the tests check. + """ + import pyreadstat + + path = Path(path) + reader = {".dta": pyreadstat.read_dta, ".sav": pyreadstat.read_sav}.get(path.suffix.lower()) + if reader is None: + raise ValueError(f"dictionary_from_file: cannot read labels from {path.suffix!r}") + _, meta = reader(str(path), metadataonly=True) + rows = [] + def tidy(code): + # SPSS stores numeric codes as doubles, so 1 comes back as 1.0. Write + # it as 1, so the dictionary round-trips through both formats identically. + if isinstance(code, float) and code.is_integer(): + return int(code) + return code + + for name, label in zip(meta.column_names, meta.column_labels): + vals = meta.variable_value_labels.get(name, {}) + rows.append({"variable": name, "label": label or "", + "values": "|".join(f"{tidy(k)}={v}" for k, v in vals.items())}) + return pd.DataFrame(rows) diff --git a/label_variables/scripts/label_data_python.py b/label_variables/scripts/label_data_python.py deleted file mode 100644 index 70ac926..0000000 --- a/label_variables/scripts/label_data_python.py +++ /dev/null @@ -1,14 +0,0 @@ -import pandas as pd - -# Load your dataset and dictionary -df = pd.read_csv('your_data_file.csv') -dict_df = pd.read_csv('../input/data_dictionary.csv') - -# Create label mapping -labels = dict(zip(dict_df['variable'], dict_df['label'])) - -# Apply as metadata (example: store in df.attrs or export to separate JSON) -df.attrs['variable_labels'] = labels - -# Save as CSV again or export labels separately -df.to_csv('your_data_file_labelled.csv', index=False) \ No newline at end of file diff --git a/label_variables/scripts/label_data_r.R b/label_variables/scripts/label_data_r.R deleted file mode 100644 index 6ee021a..0000000 --- a/label_variables/scripts/label_data_r.R +++ /dev/null @@ -1,15 +0,0 @@ -library(haven) -library(dplyr) - -# Load your data and dictionary -data <- read_dta("your_data_file.dta") -dict <- read.csv("../input/data_dictionary.csv") - -# Apply labels -for (i in 1:nrow(dict)) { - varname <- dict$variable[i] - label <- dict$label[i] - attr(data[[varname]], "label") <- label -} - -write_dta(data, "your_data_file_labelled.dta") \ No newline at end of file diff --git a/label_variables/scripts/label_data_stata.do b/label_variables/scripts/label_data_stata.do deleted file mode 100644 index b387262..0000000 --- a/label_variables/scripts/label_data_stata.do +++ /dev/null @@ -1,16 +0,0 @@ -* Apply labels using data_dictionary.csv -import delimited using "../input/data_dictionary.csv", clear -tempfile labels -save `labels' - -use your_data_file.dta, clear - -foreach var in id age gender income { - preserve - use `labels', clear - keep if variable == "`var'" - local lbl = label[1] - restore - label variable `var' "`lbl'" -} -save your_data_file_labelled.dta, replace \ No newline at end of file diff --git a/label_variables/test_label_variables.py b/label_variables/test_label_variables.py new file mode 100644 index 0000000..61a4d50 --- /dev/null +++ b/label_variables/test_label_variables.py @@ -0,0 +1,141 @@ +"""Tests for label_variables. Run: python label_variables/test_label_variables.py + +The check that matters is the round trip: labels written into a .dta come back +out of it. A label that lives only in `df.attrs` is one that vanishes the first +time somebody saves to CSV, which is what the previous version of this folder +did on the line after applying them. +""" + +import sys +import tempfile +from pathlib import Path + +import pandas as pd + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) +from label_variables import (apply_labels, coverage, dictionary_from_file, # noqa: E402 + read_dictionary, write_labelled) + +HERE = Path(__file__).resolve().parent +PASSED = 0 + + +def check(cond, label): + global PASSED + if not cond: + raise AssertionError(label) + PASSED += 1 + + +def frame(): + return pd.DataFrame({"id": ["H1", "H2", "H3"], "age": [34, 41, 29], + "gender": [1, 2, 3], "income": [1000.0, 2500.5, 800.0], + "consent": [1, 1, -99]}) + + +def test_dictionary_reads_and_parses_value_labels(): + d = read_dictionary(HERE / "input" / "data_dictionary.csv") + check(d["variable"].tolist() == ["id", "age", "gender", "income", "consent"], "variables") + g = d.set_index("variable")["values"]["gender"] + check(g == {1: "Male", 2: "Female", 3: "Other"}, "codes parsed as integers") + c = d.set_index("variable")["values"]["consent"] + check(c[-99] == "Refused", "negative code parsed") + check(d.set_index("variable")["values"]["age"] == {}, "no values is an empty dict") + + +def test_duplicate_dictionary_entries_are_refused(): + with tempfile.TemporaryDirectory() as tmp: + p = Path(tmp) / "d.csv" + p.write_text("variable,label\nage,A\nage,B\n") + try: + read_dictionary(p) + except ValueError as exc: + check("listed twice" in str(exc), "duplicate refused") + else: + raise AssertionError("duplicate should be refused") + + +def test_coverage_reports_both_directions(): + d = read_dictionary(HERE / "input" / "data_dictionary.csv") + df = frame().drop(columns=["consent"]); df["extra"] = 1 + cov = coverage(df, d).set_index("variable")["status"] + check(cov["extra"] == "no dictionary entry", "column without an entry") + check(cov["consent"] == "no such column", "entry without a column") + check(cov["age"] == "labelled", "the ordinary case") + + +def test_apply_puts_labels_in_attrs_and_strict_refuses_gaps(): + d = read_dictionary(HERE / "input" / "data_dictionary.csv") + out, cov = apply_labels(frame(), d) + check(out.attrs["variable_labels"]["age"] == "Age in completed years", "variable label") + check(out.attrs["value_labels"]["gender"][2] == "Female", "value label") + check((cov["status"] == "labelled").all(), "everything labelled") + df = frame(); df["extra"] = 1 + try: + apply_labels(df, d, strict=True) + except ValueError as exc: + check("extra" in str(exc), "strict mode names the gap") + else: + raise AssertionError("strict should refuse an unlabelled column") + + +def test_labels_round_trip_through_stata(): + d = read_dictionary(HERE / "input" / "data_dictionary.csv") + with tempfile.TemporaryDirectory() as tmp: + p = write_labelled(frame(), Path(tmp) / "out.dta", dictionary=d) + back = dictionary_from_file(p).set_index("variable") + check(back.loc["age", "label"] == "Age in completed years", "variable label survives") + check("2=Female" in back.loc["gender", "values"], "value label survives") + check("-99=Refused" in back.loc["consent", "values"], "negative code survives") + import pyreadstat + df2, meta = pyreadstat.read_dta(str(p), apply_value_formats=True) + check(df2["gender"].tolist() == ["Male", "Female", "Other"], "Stata applies the labels") + + +def test_labels_round_trip_through_spss(): + d = read_dictionary(HERE / "input" / "data_dictionary.csv") + with tempfile.TemporaryDirectory() as tmp: + p = write_labelled(frame(), Path(tmp) / "out.sav", dictionary=d) + back = dictionary_from_file(p).set_index("variable") + check(back.loc["income", "label"] == "Monthly income in INR", "sav variable label") + check("1=Yes" in back.loc["consent", "values"], "sav value label") + + +def test_stata_limits_are_errors_not_truncation(): + d = read_dictionary(HERE / "input" / "data_dictionary.csv") + with tempfile.TemporaryDirectory() as tmp: + df = frame().rename(columns={"income": "a_variable_name_that_is_far_too_long_for_stata"}) + try: + write_labelled(df, Path(tmp) / "x.dta", dictionary=d) + except ValueError as exc: + check("32 characters" in str(exc), "long name refused") + else: + raise AssertionError("should refuse a 45-character name") + d2 = d.copy(); d2.loc[d2.variable == "age", "label"] = "x" * 81 + try: + write_labelled(frame(), Path(tmp) / "y.dta", dictionary=d2) + except ValueError as exc: + check("80 characters" in str(exc), "long label refused") + else: + raise AssertionError("should refuse an 81-character label") + + +def test_csv_export_is_refused_with_an_explanation(): + try: + write_labelled(frame(), "out.csv") + except ValueError as exc: + check("cannot carry labels" in str(exc), "CSV refused, reason given") + else: + raise AssertionError("CSV should be refused") + + +def main() -> int: + tests = [v for k, v in sorted(globals().items()) if k.startswith("test_")] + for fn in tests: + fn(); print(f" ok {fn.__name__}") + print(f"\n{len(tests)} tests, {PASSED} checks, all passed") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/learning_layers/README.md b/learning_layers/README.md index a06fd02..6336192 100644 --- a/learning_layers/README.md +++ b/learning_layers/README.md @@ -1,22 +1,21 @@ -# 🧠 Learning Layers - -This folder curates conceptual tools and frameworks that explain *how* programs learn and adapt over time — especially in complex systems like public health, climate resilience, or education. - ---- - -## 📄 Included Files - -- `learning_layer_framework.pdf` — A framework authored to explain how health systems can build adaptive capacity by embedding learning at every stage of implementation -- `README.md` — This file, explaining use cases and applications -- *(Optional: slide or visual coming soon)* - ---- - -## 🧠 Why “Learning Layers”? - -Programs often focus on outputs and outcomes, but the *learning systems* that link design to adaptation are underdeveloped. This folder captures tools that help make learning tangible — across program design, MEL, research, and KM. - -Use these when you want to: -- Build in feedback loops during implementation -- Make your MEL system more adaptive -- Help teams reflect systematically +# learning_layers + +How a programme learns from its own data, treated as a set of scheduled +decisions rather than a section heading. + +| File | What it is | +|---|---| +| `embedding_learning_in_the_mel_cycle.md` | The practical note: three layers at three tempos, each a decision with a date, a person and the evidence in front of them | +| `learning_layer_framework.pdf` | The framework the note is drawn from, written for health systems building adaptive capacity | +| `why_cite_insightstack.md` | Why a repository of methods and templates carries a DOI | + +The argument in one paragraph: a MEL system collects a great deal and changes +nothing when "learning" has no place in the calendar. Start from the +decisions the programme will actually take in the next year, write down who +takes each, when, and what evidence could change it, and design the data +collection backwards from that. An evaluation cannot inform a monthly +decision, and monitoring data cannot answer an annual one; most systems try +to make each do the other's job. + +Related: `../eval_docs/mel_framework_outline.md` section 7, which is where +this goes in a framework document. diff --git a/learning_layers/embedding_learning_in_the_mel_cycle.md b/learning_layers/embedding_learning_in_the_mel_cycle.md new file mode 100644 index 0000000..05db692 --- /dev/null +++ b/learning_layers/embedding_learning_in_the_mel_cycle.md @@ -0,0 +1,93 @@ +# Embedding learning in the MEL cycle + +Most MEL systems collect a great deal and change nothing. The data arrive, +the report is written, the report is filed. "Learning" appears in the acronym +and in a section heading and nowhere in the calendar. + +This note is about the calendar. A learning system is a set of decisions, +each with a date, a person, and the evidence that will be in front of them. +If any of the three is missing, the decision will be made the way it would +have been made without the data. + +## The unit is a decision, not a finding + +Start from the decisions the programme will actually take in the next year. +Not the ones it could take: the ones that are already scheduled or that +recur. Typically: + +- Which schools or villages get the next round of the intervention +- Whether to keep, change or drop a component +- How field staff time is allocated across sites +- What goes in the next proposal +- Whether to continue at all + +For each, write down who takes it, when, and what evidence would change it. +The last part is the test: if no result could change the decision, it is +not a decision the MEL system should be serving, and collecting data for it +is documentation. + +## Match the evidence to the decision's tempo + +| Decision | Tempo | Evidence that can reach it in time | Evidence that cannot | +|---|---|---|---| +| Reallocate field staff across sites | Monthly | Monitoring data: sessions held, attendance, the field checks in FieldStack `field_ops/` | An endline survey | +| Change a component | Quarterly | Monitoring trends; a rapid qualitative check on why a component is or is not working | An impact estimate | +| Continue or scale | Annually or at the end | The evaluation | Anything not designed with a comparison group | +| What goes in the next proposal | Once | The evaluation plus everything above | | + +The commonest failure is an evaluation designed to answer the annual +question and then expected to inform the monthly ones. It cannot; the data +arrive a year late for that. The second commonest is monitoring data used to +answer the annual question, which it cannot either; it has no comparison. + +## The three layers + +**Layer 1, operational.** Monitoring data, reviewed monthly by the people who +can act on it, in a form they can act on. A one-page sheet: what was +delivered against plan, by site, with the outliers marked. The review is a +meeting with a fixed agenda, and its output is a list of changes to the next +month's plan. If the meeting produces no changes three months running, either +the programme is running perfectly or the sheet is not showing anything +anyone can act on. Assume the second. + +**Layer 2, adaptive.** Quarterly. Whether the components are working as the +theory of change said they would, using the results chain +(`../writing_guides/results_chain.md`): which links are holding and which +are not. Evidence is monitoring trends plus deliberate short qualitative +work on the weak link. The output is a decision to keep, change or drop a +component, and a record of why, so that the endline can be read against +what was actually delivered rather than what was planned. + +**Layer 3, evaluative.** Once, or at a milestone. Did it work. Evidence is +the evaluation, with the design that supports the claim. The output feeds +the continue-or-scale decision and the next design. This layer is the one +everyone plans for and the only one most programmes have, and on its own it +arrives too late to change anything about the programme it evaluates. + +## What each layer needs from the other two + +Layer 3 needs Layer 2's record of what changed and when, or it evaluates a +programme that no longer exists. Layer 2 needs Layer 1's data to see trends. +Layer 1 needs Layer 3's baseline to know what a normal month looks like. A +MEL framework that plans Layer 3 alone has planned a document. + +## The learning agenda, as a page + +One page, revised quarterly: + +1. The five decisions and their dates +2. For each, the evidence that will be in front of the person taking it, + and who is responsible for it being there +3. The weakest link in the results chain, and what is being done this + quarter to find out whether it holds +4. The three things learned last quarter and what changed because of them + +If item 4 is empty two quarters running, the system is not learning, and the +honest response is to stop collecting whatever feeds nothing. + +## What this is not + +It is not a knowledge management system, which is about storing and finding +documents (`../KM_tools/`). It is not a theory of learning organisations. It +is a calendar with names on it, and it works to the extent that the meetings +happen and produce changes. diff --git a/learning_layers/why_cite_insightstack.md b/learning_layers/why_cite_insightstack.md index df5627b..598c115 100644 --- a/learning_layers/why_cite_insightstack.md +++ b/learning_layers/why_cite_insightstack.md @@ -1,31 +1,31 @@ -# 📘 Why Cite InsightStack? +# Why Cite InsightStack? -You might wonder — it’s just a GitHub repo, why does it have a citation? +You might wonder , it’s just a GitHub repo, why does it have a citation? The answer is simple: **good documentation deserves credit.** --- -## 🔍 This Isn't Just Code +## This Isn't Just Code InsightStack contains: - Templates and tools built over years of work in public health, WEE, education, and climate - Reusable logic, planning calculators, and knowledge SOPs - Institutional workflows for MEL and research projects -Every method, assumption, and format in here has been used in real programs. Citing it acknowledges that this thinking — not just code — has value. +Every method, assumption, and format in here has been used in real programs. Citing it acknowledges that this thinking , not just code , has value. --- -## 🧪 Research Repos Are Still Research +## Research Repos Are Still Research -Open code is increasingly cited in academic and policy work. GitHub + Zenodo integration allows repositories like InsightStack to have a **DOI** — a Digital Object Identifier — just like journal articles or datasets. +Open code is increasingly cited in academic and policy work. GitHub + Zenodo integration allows repositories like InsightStack to have a **DOI** , a Digital Object Identifier , just like journal articles or datasets. This lets others reference it properly in reports, evaluations, and learning notes. --- -## 🧭 When You Should Cite This +## When You Should Cite This - If you're using a calculator from this repo in your own program design - If you're adapting SOPs or logic templates for MEL or learning systems @@ -35,7 +35,7 @@ Use the citation block from the README, or just link to the [Zenodo record](http --- -## 🎓 Citation (copy-paste version) +## Citation (copy-paste version) > Varna Sri Raman. (2025). *InsightStack: Knowledge Documentation for Development Research* [Data & Tools]. Zenodo. [https://doi.org/10.5281/zenodo.15245182](https://doi.org/10.5281/zenodo.15245182) diff --git a/miro/README.md b/miro/README.md deleted file mode 100644 index 96d1ecb..0000000 --- a/miro/README.md +++ /dev/null @@ -1,15 +0,0 @@ -# miro - -Notes and materials for running a human-centred design session on a Miro board. - -| File | What it is | -|---|---| -| `miro_use_case_india.md` | When a board is worth the setup, and when a room and paper is faster | -| `HCD_Miro.pdf` | A session structure | -| `Miro HCD Template Example.png` | What the finished board looks like | -| `Miro Purpose.png` | The framing slide | -| `sample_miro_board_links.txt` | Links to public boards worth copying from | - -No board file ships here. Miro boards live in Miro and are copied from a -template rather than imported from a repository, so what is useful to keep in -version control is the session design, not the artefact. diff --git a/observable_notebooks/README.md b/observable_notebooks/README.md deleted file mode 100644 index 4466533..0000000 --- a/observable_notebooks/README.md +++ /dev/null @@ -1,11 +0,0 @@ -# Observable Notebooks - -Interactive, real-time data storytelling with embedded charts and logic flows. Built using [Observable](https://observablehq.com/), these notebooks help visualize district health vulnerabilities and MEL results. - -## 📂 Folder Contents -- `district_vulnerability_dashboard.ojs` – A notebook that visualizes vaccine coverage and availability across districts -- `public_health_data.csv` – Dataset with coverage and population data -- `dashboard_screenshot.png` – Screenshot of the dashboard view - -## 🧠 Use Case -Observable is powerful for creating real-time dashboards and exploring spatial or temporal field data. Useful in MEL teams, participatory planning, or district dashboards in India. \ No newline at end of file diff --git a/open_data_editor/README.md b/open_data_editor/README.md deleted file mode 100644 index 2944c33..0000000 --- a/open_data_editor/README.md +++ /dev/null @@ -1,10 +0,0 @@ -# Open Data Editor Tools - -This folder includes real visual outputs and dataset examples from RawGraphs, Datawrapper, and the Open Data Editor. - -## 📂 Folder Contents -- `district_health_gaps.csv` – Multivariate development indicators for Indian districts -- `stacked_bar_chart.png` – Exported bar chart visual created using pandas/matplotlib, compatible with RawGraphs formats - -## 🧠 Use Case -Tools like RawGraphs and Datawrapper enable field teams and MEL specialists to create fast, explainable visualizations from survey and administrative data. This supports storytelling, advocacy, and participatory planning in development programs across India. \ No newline at end of file diff --git a/powerbi_reports/README.md b/powerbi_reports/README.md deleted file mode 100644 index bbe7c3c..0000000 --- a/powerbi_reports/README.md +++ /dev/null @@ -1,62 +0,0 @@ -# Power BI Reports (Curated for MEL in India) - -This folder does not contain a live `.pbix` file, but instead links to curated, real downloadable Power BI reports and a detailed MEL-specific adaptation note. - -## 🔗 Download a Working Power BI File -We recommend starting with this official sample: -➡️ [Human Resources Sample PBIX from Microsoft](https://learn.microsoft.com/en-us/power-bi/create-reports/sample-datasets#human-resources-sample-pbix-file) - -It opens in Power BI Desktop and contains layout logic that is easily repurposable. - ---- - -## 🧠 How to Adapt for MEL (Monitoring, Evaluation & Learning) - -Power BI is an excellent tool for MEL reporting in India. Here's how to make the most of it: - -### 1. **What to Use It For** -- Dashboarding program indicators (coverage, dropout, access, supply chains) -- Summarizing evaluation rounds (baseline–endline comparisons) -- Real-time performance monitoring for government/NGO programs -- Gender or equity disaggregated tracking - ---- - -### 2. **Key Power BI Features to Leverage** -| Feature | MEL Adaptation | -|--------|----------------| -| **Slicers** | Filter indicators by gender, district, age group | -| **Tooltips** | Add insights on survey rounds, confidence intervals | -| **Drillthrough** | Move from summary indicator → village/block view | -| **Bookmarks** | Save different MEL views (e.g., donor vs internal dashboard) | -| **Conditional Formatting** | Color-code whether program targets were met or missed | -| **Dynamic Titles** | Reflect user filters in chart titles automatically | - ---- - -### 3. **Steps to Adapt a PBIX File** -1. Open the sample `.pbix` in Power BI Desktop -2. Replace the dummy dataset with your own (from Excel, CSV, SQL, etc.) -3. Customize visuals: - - Change indicator names - - Apply disaggregation by caste, gender, block - - Use conditional formatting for thresholds -4. Add **navigation tabs** (program overview, district performance, field reports) -5. Export the dashboard as PDF, PNG, or share via Power BI Service - ---- - -### 4. **Tips for Field Teams in India** -- Work in local languages: Power BI allows Unicode fonts -- Use offline Excel exports for teams without internet access -- Align visual colors with government schemes or SDG colors -- Share via WhatsApp as image exports when needed - ---- - -## 📂 Folder Contents -- `mel_dashboard_screenshot.png` – Screenshot of a MEL-style Power BI layout -- `README.md` – This guide - -## 🧾 License -This curation and documentation is public domain. The linked `.pbix` belongs to Microsoft under their sample datasets license. \ No newline at end of file diff --git a/replication/.gitignore b/replication/.gitignore new file mode 100644 index 0000000..fd6aad7 --- /dev/null +++ b/replication/.gitignore @@ -0,0 +1,2 @@ +output/ +__pycache__/ diff --git a/replication/README.md b/replication/README.md index 34393ff..2283fa8 100644 --- a/replication/README.md +++ b/replication/README.md @@ -1,16 +1,66 @@ -# Replication Folder (InsightStack) +# replication -This folder allows replication of a simple linear regression analysis using synthetic data. +A replication package that can be verified, not just re-run. -## Folder Structure: -- `data/simulated_study_data.csv` – synthetic dataset (100 rows) -- `scripts/run_regression.py` – Python script using `statsmodels` -- `scripts/run_regression.R` – R script using `lm()` -- `output/` – regression result summaries will be saved here +``` +python replication/run_all.py # run from raw data, write output/, verify against the record +Rscript replication/run_regression.R # the same model in R; run_all then checks the two agree +python replication/test_replication.py # 4 tests, 12 checks +``` -## How to Use -1. Run either script inside `scripts/`. -2. Check `output/` for results. -3. Modify or reuse this structure for your own replications. +## What "verified" means here -This structure supports transparent, reproducible workflows. \ No newline at end of file +`run_all.py` fits the model, writes every output, and compares the numbers a +reader would quote against `expected_results.json`. Any difference exits 1 and +names the result that moved. `--record` overwrites the record after a +deliberate change, and that change then shows in the diff of one small file. + +`run_regression.R` fits the same model and writes the same key results, and +`verify.py` checks that Python and R agree on the point estimates, R-squared +and group means. They do, to six places. Standard errors are not compared, +because the Python uses HC1 and the R uses conventional errors; installing +`sandwich` was not worth it for a check on point estimates. + +`test_replication.py` corrupts one outcome value in a copy and confirms the +verification fails. A verification step that cannot fail is not verifying +anything, and that test is the reason to trust the green run. + +## Layout + +| File | What it is | +|---|---| +| `data/simulated_study_data.csv` | 100 rows, synthetic. `id`, `treatment`, `age`, `income`, `outcome` | +| `analysis.py` | The model as functions: load, descriptives, fit, coefficient table, key results | +| `run_all.py` | Runs it end to end and verifies | +| `verify.py` | The comparison, and the Python-against-R check | +| `expected_results.json` | The record: eight numbers and the model spec | +| `run_regression.R` | The R cross-check | +| `output/` | Written on each run; not committed | + +## Checklist for your own package + +The structure is the point; the model is a placeholder. When you replace it: + +1. **Data.** Raw data in `data/`, never edited by hand. If it cannot be + shared, a `make_fixture.py` that builds a synthetic file with the same + structure, as `data_starters/` does in this repository. +2. **One entry point.** A reader runs one command. Every intermediate file is + produced by it, so nothing depends on a step that happened once on your + laptop. +3. **Environment.** `output/environment.txt` records exact versions on every + run. Pin them in a `requirements.txt` when you publish. +4. **The record.** Round the key results to what the data supports, store + them, and compare exactly. A tolerance hides drift; rounding in one place + does not. +5. **A second implementation** of the headline number, in another language or + at least another library. Agreement between two independent routes is the + cheapest strong evidence that neither is wrong. +6. **A test that the verification can fail.** Corrupt a value in a copy and + confirm the run goes red. + +## What this replaced + +Two scripts, one Python and one R, each fitting the regression and writing a +text summary to a path that only resolved from inside `scripts/`. Nothing +compared them to each other or to anything, so a changed dataset produced a +different summary and no signal. diff --git a/replication/analysis.py b/replication/analysis.py new file mode 100644 index 0000000..2bf5af6 --- /dev/null +++ b/replication/analysis.py @@ -0,0 +1,79 @@ +"""The analysis, as functions, so `run_all.py` can run it and `verify.py` can check it. + +One model: outcome on treatment, age and income, OLS with heteroskedasticity- +robust (HC1) standard errors. The point of this folder is not the model, which +is deliberately plain; it is that the model, its data, its environment and its +expected output sit together and a stranger can confirm they still agree. + +Nothing here prints. Functions return frames and dicts; the runner writes them. +""" + +from __future__ import annotations + +from pathlib import Path + +import numpy as np +import pandas as pd + +HERE = Path(__file__).resolve().parent +DATA = HERE / "data" / "simulated_study_data.csv" + +MODEL = {"outcome": "outcome", "predictors": ["treatment", "age", "income"], "cov": "HC1"} + + +def load_data(path=DATA) -> pd.DataFrame: + df = pd.read_csv(path) + needed = [MODEL["outcome"], *MODEL["predictors"], "id"] + missing = [c for c in needed if c not in df.columns] + if missing: + raise ValueError(f"{path}: missing column(s) {missing}") + if df["id"].duplicated().any(): + raise ValueError(f"{path}: duplicate ids; the analysis assumes one row per unit") + return df + + +def descriptives(df: pd.DataFrame) -> pd.DataFrame: + cols = [MODEL["outcome"], *MODEL["predictors"]] + out = df[cols].agg(["count", "mean", "std", "min", "max"]).T + out.index.name = "variable" + return out.reset_index() + + +def fit(df: pd.DataFrame): + import statsmodels.api as sm + + X = sm.add_constant(df[MODEL["predictors"]].astype(float)) + y = df[MODEL["outcome"]].astype(float) + return sm.OLS(y, X).fit(cov_type=MODEL["cov"]) + + +def coefficient_table(result) -> pd.DataFrame: + ci = result.conf_int() + return pd.DataFrame({ + "term": result.params.index, + "estimate": result.params.values, + "std_error": result.bse.values, + "t": result.tvalues.values, + "p_value": result.pvalues.values, + "ci_low": ci[0].values, + "ci_high": ci[1].values, + }) + + +def key_results(result, df: pd.DataFrame) -> dict: + """The numbers a reader would quote, rounded to what the data can support. + + These are what `expected_results.json` stores and `verify.py` compares. Six + decimal places, which is far inside floating-point agreement across + platforms and far outside anything a substantive change would leave alone. + """ + return { + "n": int(result.nobs), + "r_squared": round(float(result.rsquared), 6), + "treatment_estimate": round(float(result.params["treatment"]), 6), + "treatment_std_error": round(float(result.bse["treatment"]), 6), + "treatment_p_value": round(float(result.pvalues["treatment"]), 6), + "outcome_mean_control": round(float(df.loc[df.treatment == 0, "outcome"].mean()), 6), + "outcome_mean_treated": round(float(df.loc[df.treatment == 1, "outcome"].mean()), 6), + "model": MODEL, + } diff --git a/replication/expected_results.json b/replication/expected_results.json new file mode 100644 index 0000000..c30000a --- /dev/null +++ b/replication/expected_results.json @@ -0,0 +1,18 @@ +{ + "n": 100, + "r_squared": 0.665421, + "treatment_estimate": 11.319997, + "treatment_std_error": 1.844772, + "treatment_p_value": 0.0, + "outcome_mean_control": 166.14667, + "outcome_mean_treated": 176.213045, + "model": { + "outcome": "outcome", + "predictors": [ + "treatment", + "age", + "income" + ], + "cov": "HC1" + } +} diff --git a/replication/run_all.py b/replication/run_all.py new file mode 100644 index 0000000..d3c9f7b --- /dev/null +++ b/replication/run_all.py @@ -0,0 +1,64 @@ +"""Run the whole analysis from raw data to outputs, then verify against the record. + + python replication/run_all.py # run, write output/, verify + python replication/run_all.py --record # run, and overwrite expected_results.json + +Writes to `output/`: descriptives.csv, coefficients.csv, model_summary.txt, +key_results.json, and environment.txt (the exact package versions that +produced them). Then compares key_results.json with expected_results.json and +exits 1 on any difference, so a replication that has drifted fails loudly. +""" + +from __future__ import annotations + +import argparse +import json +import platform +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent)) +import analysis # noqa: E402 +import verify # noqa: E402 + +HERE = Path(__file__).resolve().parent +OUT = HERE / "output" +EXPECTED = HERE / "expected_results.json" + + +def environment() -> str: + import numpy, pandas, statsmodels, scipy + return "\n".join([ + f"python {platform.python_version()} ({platform.system()} {platform.machine()})", + f"numpy {numpy.__version__}", f"pandas {pandas.__version__}", + f"scipy {scipy.__version__}", f"statsmodels {statsmodels.__version__}", + ]) + "\n" + + +def main(argv=None) -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--record", action="store_true", + help="overwrite expected_results.json with this run's results") + args = ap.parse_args(argv) + + OUT.mkdir(exist_ok=True) + df = analysis.load_data() + result = analysis.fit(df) + + analysis.descriptives(df).to_csv(OUT / "descriptives.csv", index=False) + analysis.coefficient_table(result).to_csv(OUT / "coefficients.csv", index=False) + (OUT / "model_summary.txt").write_text(result.summary().as_text() + "\n") + key = analysis.key_results(result, df) + (OUT / "key_results.json").write_text(json.dumps(key, indent=2) + "\n") + (OUT / "environment.txt").write_text(environment()) + print(f"ran on {len(df)} rows; outputs in {OUT.relative_to(HERE.parent)}/") + + if args.record: + EXPECTED.write_text(json.dumps(key, indent=2) + "\n") + print(f"recorded {EXPECTED.name}") + return 0 + return verify.compare(key, json.loads(EXPECTED.read_text())) or verify.compare_with_r(key) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/replication/run_regression.R b/replication/run_regression.R new file mode 100644 index 0000000..197ffd3 --- /dev/null +++ b/replication/run_regression.R @@ -0,0 +1,31 @@ +# The same model in R, writing the same key results, so the two can be compared. +# +# Rscript replication/run_regression.R +# +# Coefficients and R-squared must match the Python run to six places; standard +# errors are not compared, because this uses lm()'s conventional errors and the +# Python uses HC1, and installing `sandwich` is not worth it for a check that +# the point estimates agree. + +args <- commandArgs(trailingOnly = TRUE) +root <- if (file.exists("replication/data/simulated_study_data.csv")) "replication" else "." +data <- read.csv(file.path(root, "data", "simulated_study_data.csv")) +stopifnot(!any(duplicated(data$id))) + +model <- lm(outcome ~ treatment + age + income, data = data) +s <- summary(model) + +dir.create(file.path(root, "output"), showWarnings = FALSE) +sink(file.path(root, "output", "model_summary_r.txt")); print(s); sink() + +key <- list( + n = nobs(model), + r_squared = round(unname(s$r.squared), 6), + treatment_estimate = round(unname(coef(model)["treatment"]), 6), + outcome_mean_control = round(mean(data$outcome[data$treatment == 0]), 6), + outcome_mean_treated = round(mean(data$outcome[data$treatment == 1]), 6) +) +json <- paste0("{\n", paste(sprintf(' "%s": %s', names(key), + vapply(key, function(v) format(v, digits = 15), "")), collapse = ",\n"), "\n}\n") +writeLines(json, file.path(root, "output", "key_results_r.json")) +cat("R: treatment estimate", key$treatment_estimate, " R2", key$r_squared, "\n") diff --git a/replication/scripts/run_regression.R b/replication/scripts/run_regression.R deleted file mode 100644 index 5e749a6..0000000 --- a/replication/scripts/run_regression.R +++ /dev/null @@ -1,8 +0,0 @@ -data <- read.csv('../data/simulated_study_data.csv') - -model <- lm(outcome ~ treatment + age + income, data=data) -summary_out <- summary(model) - -sink("../output/results_r.txt") -print(summary_out) -sink() \ No newline at end of file diff --git a/replication/scripts/run_regression.py b/replication/scripts/run_regression.py deleted file mode 100644 index 7f0c6a4..0000000 --- a/replication/scripts/run_regression.py +++ /dev/null @@ -1,11 +0,0 @@ -import pandas as pd -import statsmodels.api as sm - -df = pd.read_csv('../data/simulated_study_data.csv') -X = df[['treatment', 'age', 'income']] -X = sm.add_constant(X) -y = df['outcome'] - -model = sm.OLS(y, X).fit() -with open('../output/results_python.txt', 'w') as f: - f.write(model.summary().as_text()) \ No newline at end of file diff --git a/replication/test_replication.py b/replication/test_replication.py new file mode 100644 index 0000000..4ad6f37 --- /dev/null +++ b/replication/test_replication.py @@ -0,0 +1,88 @@ +"""Tests for the replication package. Run: python replication/test_replication.py + +Checks that a clean run reproduces the record, and that a changed dataset is +caught. A verification step that cannot fail is not verifying anything. +""" + +import json +import shutil +import subprocess +import sys +import tempfile +from pathlib import Path + +HERE = Path(__file__).resolve().parent +PASSED = 0 + + +def check(cond, label): + global PASSED + if not cond: + raise AssertionError(label) + PASSED += 1 + + +def run(cwd, *args): + return subprocess.run([sys.executable, str(Path(cwd) / "run_all.py"), *args], + capture_output=True, text=True) + + +def test_clean_run_reproduces_the_record(): + with tempfile.TemporaryDirectory() as tmp: + shutil.copytree(HERE, Path(tmp) / "rep", ignore=shutil.ignore_patterns("output", "__pycache__")) + r = run(Path(tmp) / "rep") + check(r.returncode == 0, f"clean run verifies: {r.stdout}{r.stderr}") + check("REPLICATION OK" in r.stdout, "says so") + out = Path(tmp) / "rep" / "output" + for f in ("descriptives.csv", "coefficients.csv", "model_summary.txt", + "key_results.json", "environment.txt"): + check((out / f).exists(), f"writes {f}") + + +def test_a_changed_dataset_fails_verification(): + with tempfile.TemporaryDirectory() as tmp: + dst = Path(tmp) / "rep" + shutil.copytree(HERE, dst, ignore=shutil.ignore_patterns("output", "__pycache__")) + data = dst / "data" / "simulated_study_data.csv" + lines = data.read_text().splitlines() + parts = lines[1].split(","); parts[-1] = str(float(parts[-1]) + 50); lines[1] = ",".join(parts) + data.write_text("\n".join(lines) + "\n") + r = run(dst) + check(r.returncode == 1, "a changed outcome value fails verification") + check("REPLICATION FAILED" in r.stdout and "treatment_estimate" in r.stdout, + "and names the result that moved") + + +def test_record_mode_rewrites_the_expected_file(): + with tempfile.TemporaryDirectory() as tmp: + dst = Path(tmp) / "rep" + shutil.copytree(HERE, dst, ignore=shutil.ignore_patterns("output", "__pycache__")) + (dst / "expected_results.json").write_text("{}") + check(run(dst).returncode == 1, "an empty record fails") + check(run(dst, "--record").returncode == 0, "--record rewrites it") + check(run(dst).returncode == 0, "and the next run passes") + rec = json.loads((dst / "expected_results.json").read_text()) + check(rec["n"] == 100, "with the real results") + + +def test_duplicate_ids_are_refused(): + with tempfile.TemporaryDirectory() as tmp: + dst = Path(tmp) / "rep" + shutil.copytree(HERE, dst, ignore=shutil.ignore_patterns("output", "__pycache__")) + data = dst / "data" / "simulated_study_data.csv" + lines = data.read_text().splitlines(); lines.append(lines[1]) + data.write_text("\n".join(lines) + "\n") + r = run(dst) + check(r.returncode != 0 and "duplicate ids" in r.stderr, "duplicate id refused") + + +def main() -> int: + tests = [v for k, v in sorted(globals().items()) if k.startswith("test_")] + for fn in tests: + fn(); print(f" ok {fn.__name__}") + print(f"\n{len(tests)} tests, {PASSED} checks, all passed") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/replication/verify.py b/replication/verify.py new file mode 100644 index 0000000..394d09a --- /dev/null +++ b/replication/verify.py @@ -0,0 +1,68 @@ +"""Compare a run's key results with the recorded ones. Exit 1 on any difference. + + python replication/verify.py # compares output/key_results.json to expected_results.json + +Exact comparison on integers and the model spec, and on the rounded floats, +because both sides were rounded to six places by `analysis.key_results` before +being written. If a platform difference ever shows up in the sixth decimal, +loosen the rounding there, in one place, rather than adding a tolerance here. +""" + +from __future__ import annotations + +import json +import sys +from pathlib import Path + +HERE = Path(__file__).resolve().parent + + +def compare(got: dict, expected: dict) -> int: + problems = [] + for k, v in expected.items(): + if k not in got: + problems.append(f"{k}: missing from this run") + elif got[k] != v: + problems.append(f"{k}: expected {v!r}, got {got[k]!r}") + for k in got: + if k not in expected: + problems.append(f"{k}: produced by this run but not in the record") + if problems: + print("REPLICATION FAILED") + for p in problems: + print(" " + p) + return 1 + print(f"REPLICATION OK: {len(expected)} recorded results reproduced") + return 0 + + +def compare_with_r(got: dict) -> int: + """If run_regression.R has written its results, check the two agree. + + Only the fields both produce: n, r_squared, the treatment estimate, and the + two group means. Standard errors differ by design (HC1 against + conventional) and are not compared. + """ + path = HERE / "output" / "key_results_r.json" + if not path.exists(): + print("R results not present; skipped (run Rscript replication/run_regression.R)") + return 0 + r = json.loads(path.read_text()) + problems = [f"{k}: python {got[k]!r}, R {r[k]!r}" for k in r if k in got and got[k] != r[k]] + if problems: + print("PYTHON AND R DISAGREE") + for p in problems: + print(" " + p) + return 1 + print(f"Python and R agree on {len(r)} results") + return 0 + + +def main() -> int: + got = json.loads((HERE / "output" / "key_results.json").read_text()) + expected = json.loads((HERE / "expected_results.json").read_text()) + return compare(got, expected) or compare_with_r(got) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/survey_to_codebook/README.md b/survey_to_codebook/README.md index dee38e9..2372e4d 100644 --- a/survey_to_codebook/README.md +++ b/survey_to_codebook/README.md @@ -1,14 +1,48 @@ +# survey_to_codebook -# Survey to Codebook Generator +An XLSForm already contains everything a codebook needs, question by question. +This writes it out as one document a reviewer or analyst can read top to +bottom, and reports the defects it finds in the form on the way. -This folder contains: -- A real XLSForm (`input/survey.xlsx`) -- A Python notebook to convert the XLSForm into a readable Markdown codebook -- A generated example: `output/generated_codebook.md` +``` +python -m survey_to_codebook input/survey.xlsx -o output/codebook.md --csv output/dictionary.csv +python survey_to_codebook/test_survey_to_codebook.py # 9 tests, 27 checks +``` -## How to Use -1. Open `convert_xlsform_to_codebook.ipynb`. -2. Run all cells. -3. A Markdown file with variables, types, hints, constraints, and choices will be saved in `/output`. +Exit status is 1 when the form has a problem, so it can gate a deployment. -This helps automate documentation for field surveys and make forms easier to understand. +## What the codebook records + +For every variable: the export path (`group/question`, matching the column name +the platform will produce), type, label, hint, whether it is required, the +relevance expression it is asked under, its constraint, and for a select the +full choice list with codes. Metadata rows (`start`, `end`, `deviceid`, +`calculate`, `note`) are kept and marked, so a reader knows which columns in +the export are not answers. + +The `--csv` output is a flat dictionary in the shape `label_variables/` reads, +with the choice lists as `code=label|...` value labels. So a form's own choices +become the value labels on its exported data, with nothing retyped. + +## What it catches + +| Problem | Why it matters | +|---|---| +| A select naming a choice list that is not on the choices sheet | The form will not deploy, and a codebook that silently prints an empty list hides why | +| A choice list no question uses | Usually a renamed question; the old list is a fossil | +| A group opened and never closed | Every later path is wrong | +| The same variable name twice | The export overwrites one column with the other | +| A group name written in the `type` cell (`begin_group personal`) with `name` blank | Some exports do this; both spellings are read | + +The bundled `input/survey.xlsx` shipped with the first of these: the form said +`select_multiple hobbies` and the choices sheet named the list `hobby`. The +previous notebook printed a codebook with an empty choice list and no comment. +This tool reported it on the first run, and the sample is now fixed. + +## What this replaced + +A notebook that iterated the survey sheet and printed a heading per row. It did +not know about groups, so every path was wrong for a grouped question; it +handled `select_one` and not `select_multiple`; and it looked up choice lists +by splitting on whitespace and taking the second token, which works until a +list name has a space in it. diff --git a/survey_to_codebook/__init__.py b/survey_to_codebook/__init__.py new file mode 100644 index 0000000..f78c1a5 --- /dev/null +++ b/survey_to_codebook/__init__.py @@ -0,0 +1,14 @@ +"""XLSForm to codebook: the documentation the form already contains, written down. + + python -m survey_to_codebook input/survey.xlsx -o output/codebook.md + +An XLSForm carries everything a codebook needs, question by question: the +variable name, the label, the type, the choice list, the skip logic, the +constraint, the hint. Almost nobody reads it as a codebook because it is a +spreadsheet with the choices on a different sheet. This turns it into one +document a reviewer or an analyst can read top to bottom. +""" + +from .xlsform_codebook import build_codebook, read_xlsform, to_csv, to_markdown + +__all__ = ["read_xlsform", "build_codebook", "to_markdown", "to_csv"] diff --git a/survey_to_codebook/__main__.py b/survey_to_codebook/__main__.py new file mode 100644 index 0000000..d4d1c22 --- /dev/null +++ b/survey_to_codebook/__main__.py @@ -0,0 +1,3 @@ +from .xlsform_codebook import main +import sys +sys.exit(main()) diff --git a/survey_to_codebook/convert_xlsform_to_codebook.ipynb b/survey_to_codebook/convert_xlsform_to_codebook.ipynb deleted file mode 100644 index 64973e4..0000000 --- a/survey_to_codebook/convert_xlsform_to_codebook.ipynb +++ /dev/null @@ -1 +0,0 @@ -{"nbformat": 4, "nbformat_minor": 5, "metadata": {}, "cells": [{"id": "123edb9b", "cell_type": "markdown", "source": "# XLSForm to Codebook Generator\nThis notebook parses an XLSForm survey and generates a readable Markdown codebook.", "metadata": {}}, {"id": "2e88b240", "cell_type": "code", "metadata": {}, "execution_count": null, "source": "import pandas as pd\n\n# Load survey and choices\nsurvey_path = '../input/survey.xlsx'\nsurvey_df = pd.read_excel(survey_path, sheet_name='survey')\nchoices_df = pd.read_excel(survey_path, sheet_name='choices')\n\n# Preview\nsurvey_df.head()", "outputs": []}, {"id": "fd6e6fd7", "cell_type": "code", "metadata": {}, "execution_count": null, "source": "# Create choices lookup\nchoices_dict = choices_df.groupby('list_name').apply(\n lambda df: dict(zip(df['name'], df['label']))\n).to_dict()\n\nchoices_dict", "outputs": []}, {"id": "fface476", "cell_type": "code", "metadata": {}, "execution_count": null, "source": "# Process survey to extract codebook\ncodebook_lines = [\"# Generated Codebook\\n\"]\n\nfor _, row in survey_df.iterrows():\n if pd.isna(row['name']) or row['type'].startswith(\"begin_group\") or row['type'].startswith(\"end_group\"):\n continue\n line = f\"## `{row['name']}`\\n\"\n line += f\"**Label:** {row['label']}\\n\\n\"\n line += f\"**Type:** {row['type']}\\n\\n\"\n if pd.notna(row.get('hint')):\n line += f\"**Hint:** {row['hint']}\\n\\n\"\n if pd.notna(row.get('constraint')):\n line += f\"**Constraint:** `{row['constraint']}`\\n\\n\"\n if pd.notna(row.get('relevant')):\n line += f\"**Relevant if:** `{row['relevant']}`\\n\\n\"\n if \"select_one\" in row['type'] or \"select_multiple\" in row['type']:\n list_name = row['type'].split()[-1]\n options = choices_dict.get(list_name, {})\n line += \"**Choices:**\\n\" + \"\\n\".join([f\"- `{k}`: {v}\" for k, v in options.items()]) + \"\\n\"\n codebook_lines.append(line)\n\n# Save Markdown codebook\nwith open('../output/generated_codebook.md', 'w') as f:\n f.write(\"\\n\\n\".join(codebook_lines))", "outputs": []}, {"id": "99e9f6c3", "cell_type": "markdown", "source": "Run all cells to generate `generated_codebook.md` in the `/output` folder.", "metadata": {}}]} \ No newline at end of file diff --git a/survey_to_codebook/input/survey.xlsx b/survey_to_codebook/input/survey.xlsx index c61d98e7f3fbae0fc009c50d71fc6315d637d044..8e9dff12f093a848e838d218f21ebd41c1c0d110 100644 GIT binary patch delta 4130 zcmZ`+1z1#D*Pa1}k{CKKCDJ_u2qN7fDGh@Hg2d1%IDpd90uC{>QWBC<7o~=f5T#+1 zMnFQ4hHtp{_rDjP?>px?XFun8*4lfoz25bn^&W~=3L^BhaPg=C001Gt$$`g8wMR!i z75nu8`+%{Jotv$mr<=Q%@I7~TA%9mFogtlT??p)Pzx-+7+klK=DI`Zq~9HW@7M4;0@VWDlMsW ze$Yj%VLy!_YPrwt)DH9~K@;8880-oNX=ku|e+&cwC;%x*^i+uOVFl+7HfnPLpY8CQA<7Sm%7G^riFpZPV6~&aeO_A;0!;7<{DhUOo?7YxV;B_hO*fKdk z4P=!)cYZD)St@V*A(F|8j68TRW0yVwTzfREhR3T6V$rbBVQiCUd5|=#;Y?VdTf-!@ zBf~e={-XU;3l~DNA>y7XSy^>DjQ7jtQh=OLr&YSTAHJC(=Ak`@lVUSEB;TT4Ff)1l zbBFCoCnK+2uZrwl3&I;?nrU0&r%e~6$8$|jPISu~C)}&y*Gl^0lIJ6D0(D3cZB0jb zSEHvtRyC>u0RRR}000&C6#QL;y=$yUdqI?!`L%<(xn|)20NsQD06ic@kB%l~8yAEK zPMd~jiGU+uT`G5lJC<8C%}OKeQZ0>Ql55{SBN+&*>wfvuMWXmyvmZ$_r@m`NP&;M| z$qM0`bgHQr2fP_q|>io$O@2aoVu^9?i7)eQ$AXQ_`;H!TPwEn4geW74-Ljc~W zr=PLs<;SJ7hhL=U?(kw3N-8;fL%kJYr5ri(c;4Db{3n%1Mr}(NL3@rM5c*6l+W?wd zJ{-efDww|&N!K5syT|{SXkyAQI6?*iAOa~)YBeuS*{TaaC{I3T7K%NWS#D5qWR%pRRI@4nDDp`Skbd_FFr?XAE`DYvvj+&baBG=D04Y9 zj-!IjB1JBC1!_+t-jBSon+dr#m%e*9>g+U9;-tUL=+NiggEj~w2RT=+Jz>+`@?jnR z&~M=N5T;vSx7Jp-9?HXuO^3{ZW%8+ud&7qVWDoJ81(Zp`y>D{M9xG8gAZ#feMm3;D zj3B8Fr|cCR7mRP4wxS|C+cjXbvMw6r3OsFT+dG;M~}m7WsqTbgk3x6<&TP z4R0CWk!9;6MdZv2Ai3vm`P2;y_3-9+#Xo^9_v+% zW7eGUda>2ri^ch0C}u>YKA(kWicp8ZKu5rDP}eslIEAStQ$iefqwB4V@yTpH%MHjf zOivH_Qb~@uzqIs--`qO96_Twx)iMPH)L;yFEE0hpLAjqKaMd5U?Gg5U@SakS5E8## zR{$KtS0`?=6c{rSt58QqRy!Cj2PTB^`%55awR#6~)WCs!XhDP&nM;724SlY?>>C!} z5R588j{S2qvw$^>#Pl~&F_R!KoNxja0669nJi1CiZ+Jx3!4v9pKo zJ!inUASGo9n-P)Avz`^FWm9<`H8Y!$MPF!P)G3LE-eQjvB?$I*#4>VAFI>=0;1znG zLH!_xuJmZ)+r_j4S#B|jUKu4_Ok7{WFjjlb%~QfE-d>MOx^%ynTS8kBA`YS{pHw)L z`Q8x?KeR>PmE+D(toK4A*Z49jDkOE-eLvx^MDeL?JU~Q2hJ5xcVOvPW#kt)FcOW2_ zbI_c?TJO%m_;dsk#h*#EYMIydfgl;#u+Gt0+yj0)uJ5cYm@xT8mYzQ6EuwOi(RO~T z-2YKJ0(dpq*ZI}?>adh0Kmq`e{!M6J-T^N5UYGx)S(EAbOcC%If~J$;IcG7C5wIkV zy2QZZb?wQp2{--yUOK0Nn2mA2Zmxz|k~QOCKA!tB4udl?`1YUQK3Pn7G-w6oDQV`I zyEXn;Ga25n663?F-7n5oR~;i|I>ohS-f#AqU8C1dG}Au%zGBak=7CdzCEccWr4_}{ zJP)r)-H5|y4l?2@dqj-dkT&g3hDLQA2R-D*BKs1X@9&C2Ga`b1UAHqsC4n zHJGNOZL0*)y*WCUThixrNoNI5coh5Aj}`PQ%$J!P0&!#Icc&UAc;x;tN9R{>h`@rL z3Dchg@6~r3&sc+qOFeG{Y$o1QsW}LPysjL(=|HWSY7;jiV2AK5v>G|frLLYni6eV` zuNBfW$m!@q;xRsv87}jEDr|)Jb948*cpgXUT(et3OV^K&&2pqA2ojY32xmAqKicUM zl!+N$)+d2?gYX-@>9SylR;Sq*V;QD{qDaqL*u}TF1@g4<``hbzvfT1-bEv*63{4rc z$ZCx^b}8*6n5z+%)z^imnE0*P7*QdA@Z4k$b%1A2Ggt8eIr+q%3q)5CU zHSd($PQ5NnJo{5oI?rwYk1Xl>aLLJ3ld4kx4}oi_J2n(KOIt0Jt$n)2wV+$W8EQD3rvmhBtxvtk~_tXm4J)~43l&F$G(itZz9biT+B73b^7oDkjx7d=E#fy0y;A8IQ4+i}J>sPGCa+q1KVs^A;9*5a#&u04^GS~oh}Ol2 znIYm&8QGQP8>g?tw_gtY*k?=3S5t8p*h?zZj`e+BDT|tID@6P#vb|sHk6FF=FVXk)r62<0?d4ZdFI1tDz42`ybk3**gszD*4|9?dRs{eBaH@`Ld+r^sRI|L`Y8g z?>+`>mg+woZK4CSQsUqdqH4OW1d|&zXHs|8JuGOr*x6C@xBAn61dJN`BQO-L+^mDg zjbuemFQ$Dmx=Vqt78vTj6qy*hXMiGtHGSe}EDb(OZD%c;85aV}A8@iVxfEo4IxnD^ z226G0hh%3LW~o-zXHHJqs(;*kP;5&j%YVM%g-TD5)PzOTV5b-&1*9{GT4iUm}iw0vaU-)X9)G} z%G|!=yJOr-j8)i`%Kdbea2r<)o81 zKc?_>&n!#sOevJyrz3M`9s_?L)8?-x%*LZ000Ad#3MCzrSe%Y%aEBeZ0%8CV<4c^~vu!WKv+}j@R4KoUG zxA(HT1X`C-|D6sI@M$V~oX>YH6xS&*&oYnTaXTj>vpWX13)eb6aG^9OGBPh^^N6O| z+8FTf+1=d-18WA&62cv$=ARgtN_>I#f!}!*k@Y^D$JE}{Pk%iZW9RyJT0QIKn+sx}>Ui^rmj*pzJ736~OaKyVJUO$i zlDJsJo?)7M@$oZaV@PHT!AXW%?#*+eX>GR>wDF<`Qk77g$Vaia=t6k#1okEGRPF#q~<$v!kewl>;L$Lh@Ry_UFYy3X{@09(Is|}Xc{zc%wJN_R5 z8vH*4{^O~E72ViH|ILTLd;e3b0084JPwW^1oTNy3Ps^iq8P@Oc|78~f?4kq!{!20F UX@T%Bjfk=LQ!FvlTxK2cAFa9lrvLx| delta 4598 zcmZu#1yoes8l9m#hi*i=OOQ?(X`}@~x`&X48A^J9p+rJD1w>Fl8U>{VB^+8nT96tV z-spP&^Vjw6TKArH*4^i%2%Lz}>S|i{hX7{91spdPAPpbat`69l z_4!aQ%lN`2M}Vzpuhu$r-j+`4+iS0eh<5G#{rj10;PTQV5`sn?yXCP7&lZv}GZ#P8 zx_Itc{P3y>bM|APMOC`|Wa=kKZXD}`#RyE1D5#yGu?`3-`*Oan(xk7 zrr=SV`7Pix7%=P1v7Q73sx8~bn|L(_$Yte#?Ly*7&W}YaXWkPsb962EvHZ8 zW4#VWK*zTQJd%L^Rn(flge_g2X>tUnpw&j(L$pt;w?000Qlt(CXE zn~#v-&u0xRwOi+%Fp=VJh}z0|wuc5@VQDLC$sDJ+d>4lY!)thPPvp*ST#6zQ+!;5? zYkSRcaBEl=z6m!#F%t1Vrw3sVj>L>~4PH5Wk@6Da#N5CY^(-OES#`M7x^cI%zdhNE zN}~@a2ys&2+bE}<#g_yc&>eJE=J`0j5$o5LbXWze-Fm?oYp@j(Wf@P*nKZOn7CupVP9aWDZ^q z1szBnE@9sAo#)}MXT)lm&KY@4Fprv8aVh!o%zfdb)tXk31T18<;{RaE(LcW#1`2E{w44ph>G;;NHCr=p#a{+-ID$G6SD57EUeh zzbLFhitY3HKlnCccv}eN6kbW$i)GuyC5iJv)%-*`VCEh<(I~fmvqAK!@5qpAe_3GZ zvz49_yXBeMdkxgh^OY~=g*g`_gL3D}{P$@7RDvAQB11k}2#MHO|5XANOjIzy-3^}J z?U5@?7LCr3!Q)z>kXs8zk+}XiGp~`+bSoyNDUh^2Zw~d{wPAQbv)Ty4e7$^~LuWfM zv+!AiqyVmJH1;S>52LEvxT>Q_A)i@cUeBgluZC7%N$onUu<0mfKlZGb2#_g-Y0rDx zRA!e>Y#5Lx%#f=jU8(|;!{1V#6BBbK`|j;TZWg8o z%So^P(DKwFrJ>LldAIx&8XH|(4li)%F#&nchLJ{W2d ze^Lu3%nM-pPAUrpB1tES1r%Bp@c1?jmJ!V!2DjdroghnscQ#=z7pr*-mg4@OOoA&K zEc;24kQbV#7xFjThnkbKa>cUV<-7J3N!S%0`KgiiZaZL;sEsUZG28FE4(Sca*JKs; zHF>Q{4$MXmh4d1sJ*&>5v0-*x)pvlx>Ec5|D`{-V9jWylywE+_`yxU;bMgHVkRWQg z5XzP(11aRU{pGol+uUR2Ty@m~BZC7+dx?`vht$M7$Ve2Tb%1271N*8p55{q1^=BYw zrKUDa(SbBdvnHd34SGZhleHc5@h>oRVT`MWYlK!Jsv2X&rCS5=jT7N+l&PJ&*|C!C zM|K|r{QMl-E%MCOlC1N@TMGL5%dj5VM~$CHKs}y8cDcQtxprra>5Ym>weUhRBVWb{ z*^*#wz2SvnjHF&>YKvY!|)qP2gt)U}R`IwGiz+iSeP#LV= zFFfyWaDo#DlYMsWDa3jV7h0G7xNOaA`6l^1N2ln6XKm9RsbvLh5BLEWtpnSh%@jL6 z$<$_*NVAInO-B=NNy*$6LklBXdB$QP%?3Li@EqS!*la6QB4nE;zHIT^tv@**|9YtU z0?m0gI(L3Lo&Yx?AE$@j_I9v)kG!2d9DRN!lCPmQWL%i6{eTaL(95Kicdy- z>Fd?%@ar&*PbdSK1bCK6yybaO5+Y2~-K|GsMZwr;TU}h#BlphW5Nr(lMqKj4iXu|( zb1C-Jv5D62s47tdGhkj0zr@nJjIjQyd`jqp@Z+hXnqn7M7iW~I$@t`i_X80=?P%vf z#k>fh#Iur8&~7dqWUBl=5qmo*rF%p}v>*{*UUS)ZZ;uOa6x zGTSnHZ{iZ8JU9jTq&l%v26i%CP`xI-j6|fjvy_Hf41_l!o($Io_v8?Zv-j}idP1L0 zww(<-T!c?yhPqhi)aPjS&~&WU@yh9#)0lkCxhw|;XX!^UcG9V9%3$lrz&b3jr z=jb5IVFLif{|)k^KsWne!EHBONI>tB&|NF?edrqRp;JQ8ApzlISR=@TFceNNk;0Oi zm@sd(0A8vh^yi;du^Jj$$$>a@^sFSOrLGLsP^#02w=3wq^>wE&Y>l{*X4;k4UkclZ z1{Vh^Iqt06p0fE1YhKOzkYLfh;&UPu4gph`99xL^LvU)n$|a0N6sP>+82BCG2jIJ$ z$ge+AxH-iI4kn#oo+XmS(Gd=s1w(U$(qeJ$rQ$pX(;Bw;ta2!F_G}(z@`Wr=29^95 z(VHub@1BxIB~j;6Oq;T3**rd}NxjzR<0narN}qaXQ!s>!(^Vs1q(XfzPHOs~_y>Y> z(W5|Sa8>-gSrdO=F)9b0mSwK% zGf{;=2;w_q75Wq6mINJ2;*!=D8q{`}+);w&qObX`?;|%DkQnPqt#;vvDNd2c=`z0 zLcHyNZV!3CooI2Z0FsRjZzjBPWn8h4N1z?wIW5*)LBoZqGn7iki{eLRL4tV0!iXTR z$(;#~&c!zI7Ue*7J&bGywxbT^-5K8&)i_cu!bw_Us?DxqBiN;$x8@tKADfBTG#n=! z5wFT;jn~N{N@;_kfZh;lrrYr{%OJwNP1G0Bt6@|ap)$3pl63@jv~JgUlIkOM0(tiD z$*w_Ri?sB51(6>DGnb#=3cClbFEH`Qy&FC9w&aLCX{Y!}@k{ra>b#@LEQJsGIZv9P zY#{>|z` zPtRYZ8gzTiz=Vm~cln6s7~XUX6NOh7X~-PbJp`sUI_i|!7S?mbCL*@CxMX6ts$+l$ z8+&_SH=dTX$oSQ+iy(+QD4G%sK} zhKS!@tfUa|u>gb1dP$rRGxGk~DYq9b%G!yvZHeWvU_R@k4HlW-XyWduTiJ$Ea;bRv zSsYz-miY>+$Oj?wY*3#!<<%XDJ+Hu*@3KMM;uf;Z#J6v*X=lJ2ok;rk!B6gvXug(M z^vr|%VkYVJ`zy<_5sA%&X_pv%e9lZ9W0-Q8SKbKf6=G#vGAxR*>u!)59O;d<l=69qB(atE1lJySu;Nkg&bmzE0ZOW$SnI3mCZfzkGI%9 zN4F&>BZYXbg8Z3U@Z?ENbkMURv-ktN3WKV`e*Myw#(%Q diff --git a/survey_to_codebook/output/dictionary.csv b/survey_to_codebook/output/dictionary.csv new file mode 100644 index 0000000..6e934de --- /dev/null +++ b/survey_to_codebook/output/dictionary.csv @@ -0,0 +1,6 @@ +variable,path,type,label,values,required,relevant,constraint,metadata +name,personal/name,text,What is your name?,,no,,,no +age,personal/age,integer,How old are you?,,no,${age} >= 0,. >= 0,no +gender,personal/gender,select_one,Select your gender,m=Male|f=Female,no,,,no +hobby,personal/hobby,select_multiple,Select hobbies,reading=Reading|sports=Sports|music=Music,no,,,no +endnote,endnote,note,Thank you for completing the survey!,,no,,,yes diff --git a/survey_to_codebook/output/generated_codebook.md b/survey_to_codebook/output/generated_codebook.md index 4645865..b6d0668 100644 --- a/survey_to_codebook/output/generated_codebook.md +++ b/survey_to_codebook/output/generated_codebook.md @@ -1,73 +1,54 @@ -# Generated Codebook +# Codebook +5 variables, of which 4 are questions and 1 are form metadata or calculations. -## `name` -**Label:** What is your name? +| Variable | Type | Label | +|---|---|---| +| `personal/name` | text | What is your name? | +| `personal/age` | integer | How old are you? | +| `personal/gender` | select_one | Select your gender | +| `personal/hobby` | select_multiple | Select hobbies | +| `endnote` | note | Thank you for completing the survey! *(metadata)* | -**Type:** text name +## `personal/name` -**Hint:** Enter your full name +**What is your name?** +Type: `text` +Hint: Enter your full name -**Constraint:** `` +## `personal/age` -**Relevant if:** `` +**How old are you?** +Type: `integer` +Hint: In completed years +Asked when: `${age} >= 0` +Constraint: `. >= 0` +## `personal/gender` +**Select your gender** +Type: `select_one` from list `gender` -## `age` -**Label:** How old are you? +| Code | Label | +|---|---| +| `m` | Male | +| `f` | Female | -**Type:** integer age +## `personal/hobby` -**Hint:** In completed years +**Select hobbies** +Type: `select_multiple` from list `hobbies` +Hint: Tick all that apply -**Constraint:** `. >= 0` - -**Relevant if:** `${age} >= 0` - - - -## `gender` -**Label:** Select your gender - -**Type:** select_one gender gender - -**Hint:** - -**Constraint:** `` - -**Relevant if:** `` - -**Choices:** -- `m`: Male -- `f`: Female - - -## `hobby` -**Label:** Select hobbies - -**Type:** select_multiple hobbies hobby - -**Hint:** Tick all that apply - -**Constraint:** `` - -**Relevant if:** `` - -**Choices:** -- `reading`: Reading -- `sports`: Sports -- `music`: Music +| Code | Label | +|---|---| +| `reading` | Reading | +| `sports` | Sports | +| `music` | Music | +Exported as one space-separated string of selected codes; expand to one column per option before tabulating. ## `endnote` -**Label:** Thank you for completing the survey! - -**Type:** note endnote - -**Hint:** - -**Constraint:** `` - -**Relevant if:** `` +**Thank you for completing the survey!** +Type: `note` diff --git a/survey_to_codebook/test_survey_to_codebook.py b/survey_to_codebook/test_survey_to_codebook.py new file mode 100644 index 0000000..537a07d --- /dev/null +++ b/survey_to_codebook/test_survey_to_codebook.py @@ -0,0 +1,174 @@ +"""Tests for survey_to_codebook. Run: python survey_to_codebook/test_survey_to_codebook.py + +The fixture XLSForm is built in code with openpyxl, so the test does not depend +on the bundled example and can exercise the awkward cases on purpose: a group +name written in the type cell, a select naming a list that does not exist, a +choice list nobody uses, a repeat, and a duplicate variable name. +""" + +import sys +import tempfile +from pathlib import Path + +import openpyxl + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) +from survey_to_codebook import build_codebook, read_xlsform, to_csv, to_markdown # noqa: E402 + +HERE = Path(__file__).resolve().parent +PASSED = 0 + + +def check(cond, label): + global PASSED + if not cond: + raise AssertionError(label) + PASSED += 1 + + +def write_form(path, survey_rows, choices_rows, + survey_cols=("type", "name", "label", "hint", "relevant", "constraint", "required")): + wb = openpyxl.Workbook() + ws = wb.active; ws.title = "survey" + ws.append(list(survey_cols)) + for r in survey_rows: + ws.append(list(r) + [""] * (len(survey_cols) - len(r))) + wc = wb.create_sheet("choices") + wc.append(["list_name", "name", "label"]) + for r in choices_rows: + wc.append(list(r)) + wb.save(path) + return path + + +def standard_form(path): + return write_form(path, [ + ("start", "start", ""), + ("begin_group", "hh", "Household"), + ("text", "head_name", "Name of household head", "Full name", "", "", "yes"), + ("integer", "hh_size", "Household size", "", "", ". >= 1 and . <= 30", "yes"), + ("select_one gender", "head_sex", "Sex of head"), + ("begin_repeat", "member", "Household member"), + ("integer", "age", "Age", "In completed years", "", ". >= 0"), + ("select_multiple asset", "owns", "Assets owned", "", "${age} >= 18"), + ("end_repeat", "", ""), + ("end_group", "", ""), + ("note", "thanks", "Thank you"), + ], [ + ("gender", "m", "Male"), ("gender", "f", "Female"), + ("asset", "1", "Bicycle"), ("asset", "2", "Phone"), ("asset", "3", "Land"), + ]) + + +def test_paths_follow_groups_and_repeats(): + with tempfile.TemporaryDirectory() as tmp: + survey, choices = read_xlsform(standard_form(Path(tmp) / "f.xlsx")) + book = build_codebook(survey, choices) + paths = {e["name"]: e["path"] for e in book["entries"]} + check(paths["head_name"] == "hh/head_name", "group prefix") + check(paths["age"] == "hh/member/age", "nested repeat prefix") + check(paths["start"] == "start", "top-level metadata has no prefix") + rep = {e["name"]: e["in_repeat"] for e in book["entries"]} + check(rep["age"] and not rep["hh_size"], "in_repeat is set only inside the repeat") + check(book["problems"] == [], "a well-formed form has no problems") + + +def test_choices_are_resolved_and_required_and_relevance_kept(): + with tempfile.TemporaryDirectory() as tmp: + survey, choices = read_xlsform(standard_form(Path(tmp) / "f.xlsx")) + e = {x["name"]: x for x in build_codebook(survey, choices)["entries"]} + check(e["head_sex"]["choices"] == [("m", "Male"), ("f", "Female")], "select_one list") + check(len(e["owns"]["choices"]) == 3, "select_multiple list") + check(e["hh_size"]["required"] and not e["age"]["required"], "required parsed") + check(e["owns"]["relevant"] == "${age} >= 18", "relevance kept") + check(e["hh_size"]["constraint"] == ". >= 1 and . <= 30", "constraint kept") + check(e["start"]["is_metadata"] and e["thanks"]["is_metadata"], "metadata flagged") + + +def test_group_name_in_the_type_cell_is_read(): + """Some exports write `begin_group personal` in type and leave name blank.""" + with tempfile.TemporaryDirectory() as tmp: + p = write_form(Path(tmp) / "f.xlsx", + [("begin_group personal", "", ""), ("text", "name", "Name"), ("end_group", "", "")], []) + survey, choices = read_xlsform(p) + book = build_codebook(survey, choices) + check(book["entries"][0]["path"] == "personal/name", "group name recovered from the type cell") + check(book["problems"] == [], "and not reported as a problem") + + +def test_missing_and_unused_choice_lists_are_reported(): + with tempfile.TemporaryDirectory() as tmp: + p = write_form(Path(tmp) / "f.xlsx", + [("select_one colour", "fav", "Favourite colour")], + [("size", "s", "Small"), ("size", "l", "Large")]) + survey, choices = read_xlsform(p) + book = build_codebook(survey, choices) + msgs = " ".join(book["problems"]) + check("'colour'" in msgs and "not on the choices sheet" in msgs, "missing list reported") + check("'size'" in msgs and "no question uses it" in msgs, "unused list reported") + check(book["entries"][0]["choices"] == [], "and the entry has no choices rather than wrong ones") + + +def test_unclosed_group_is_reported(): + with tempfile.TemporaryDirectory() as tmp: + p = write_form(Path(tmp) / "f.xlsx", [("begin_group", "g", "G"), ("text", "q", "Q")], []) + survey, choices = read_xlsform(p) + book = build_codebook(survey, choices) + check(any("never closed" in m for m in book["problems"]), "unclosed group reported") + + +def test_duplicate_variable_name_is_refused(): + with tempfile.TemporaryDirectory() as tmp: + p = write_form(Path(tmp) / "f.xlsx", [("text", "q", "Q1"), ("integer", "q", "Q2")], []) + survey, choices = read_xlsform(p) + try: + build_codebook(survey, choices) + except ValueError as exc: + check("appears twice" in str(exc), "duplicate name refused") + else: + raise AssertionError("duplicate should be refused") + + +def test_markdown_and_csv_outputs(): + with tempfile.TemporaryDirectory() as tmp: + survey, choices = read_xlsform(standard_form(Path(tmp) / "f.xlsx")) + book = build_codebook(survey, choices) + md = to_markdown(book, title="Test form") + check(md.startswith("# Test form"), "title") + check("| `hh/member/owns` | select_multiple |" in md, "summary table row") + check("space-separated string" in md, "select_multiple export note") + check("| `2` | Phone |" in md, "choice rows") + csv = to_csv(book).set_index("variable") + check(csv.loc["owns", "values"] == "1=Bicycle|2=Phone|3=Land", "csv values in label_variables format") + check(csv.loc["hh_size", "required"] == "yes", "csv required") + check(csv.loc["start", "metadata"] == "yes", "csv metadata flag") + + +def test_bundled_example_runs(): + survey, choices = read_xlsform(HERE / "input" / "survey.xlsx") + book = build_codebook(survey, choices) + check(any(e["path"] == "personal/gender" for e in book["entries"]), "bundled form parses") + + +def test_missing_sheet_is_a_clear_error(): + with tempfile.TemporaryDirectory() as tmp: + p = Path(tmp) / "bad.xlsx" + wb = openpyxl.Workbook(); wb.active.title = "survey"; wb.active.append(["type", "name"]); wb.save(p) + try: + read_xlsform(p) + except ValueError as exc: + check("choices" in str(exc), "names the missing sheet") + else: + raise AssertionError("should refuse a form with no choices sheet") + + +def main() -> int: + tests = [v for k, v in sorted(globals().items()) if k.startswith("test_")] + for fn in tests: + fn(); print(f" ok {fn.__name__}") + print(f"\n{len(tests)} tests, {PASSED} checks, all passed") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/survey_to_codebook/xlsform_codebook.py b/survey_to_codebook/xlsform_codebook.py new file mode 100644 index 0000000..4fb6622 --- /dev/null +++ b/survey_to_codebook/xlsform_codebook.py @@ -0,0 +1,234 @@ +"""Read an XLSForm, resolve its structure, write a codebook. + +Handles the parts of the format that a first attempt gets wrong: + +- **Groups and repeats.** `begin_group` / `end_group` and `begin_repeat` / + `end_repeat` nest, and the exported column name for a question inside them is + `group/question`. The codebook records the path so it matches the export. +- **A group name written in the type cell.** Some exports carry + `begin_group personal` in `type` with `name` blank. Both spellings are read. +- **select_one and select_multiple.** Each names a choice list; the list is + looked up and every option listed. A select naming a list that does not + exist is reported, since the form would fail to deploy and the codebook + should say so rather than print an empty list. +- **Unused choice lists** are reported at the end, because a list nobody + references is usually a renamed question. +- **Duplicate names** are refused, since the export would overwrite one column + with another. +- **Metadata rows** (`start`, `end`, `deviceid`, `today`, `username`, + `calculate`, `note`) are kept but marked, so a reader knows which columns in + the export are not answers. +""" + +from __future__ import annotations + +import argparse +import re +import sys +from pathlib import Path + +import pandas as pd + +__all__ = ["read_xlsform", "build_codebook", "to_markdown", "to_csv", "main"] + +_STRUCTURE = {"begin_group", "end_group", "begin_repeat", "end_repeat", "begin group", + "end group", "begin repeat", "end repeat"} +_META = {"start", "end", "today", "deviceid", "subscriberid", "simserial", + "phonenumber", "username", "email", "audit", "calculate", "note", "hidden"} + + +def read_xlsform(path) -> tuple[pd.DataFrame, pd.DataFrame]: + """Return the survey and choices sheets as string frames, blanks as ''.""" + x = pd.ExcelFile(path) + missing = [s for s in ("survey", "choices") if s not in x.sheet_names] + if missing: + raise ValueError(f"{path}: an XLSForm needs sheet(s) {missing}; found {x.sheet_names}") + survey = x.parse("survey", dtype=str).fillna("") + choices = x.parse("choices", dtype=str).fillna("") + survey.columns = [str(c).strip() for c in survey.columns] + choices.columns = [str(c).strip() for c in choices.columns] + for col in ("type", "name"): + if col not in survey.columns: + raise ValueError(f"{path}: the survey sheet needs a {col!r} column") + for col in ("list_name", "name"): + if col not in choices.columns: + raise ValueError(f"{path}: the choices sheet needs a {col!r} column") + if "label" not in survey.columns: + lab = [c for c in survey.columns if c.startswith("label")] + if not lab: + raise ValueError(f"{path}: the survey sheet has no label column") + survey["label"] = survey[lab[0]] + if "label" not in choices.columns: + lab = [c for c in choices.columns if c.startswith("label")] + choices["label"] = choices[lab[0]] if lab else "" + return survey, choices + + +def _split_type(raw: str) -> tuple[str, str]: + """``"select_one gender"`` to ``("select_one", "gender")``; plain types to ``(t, "")``.""" + parts = str(raw).strip().split() + if not parts: + return "", "" + head = parts[0].lower() + if head in ("begin", "end") and len(parts) > 1 and parts[1].lower() in ("group", "repeat"): + head = f"{head}_{parts[1].lower()}" + rest = " ".join(parts[2:]) + else: + rest = " ".join(parts[1:]) + return head, rest.strip() + + +def build_codebook(survey: pd.DataFrame, choices: pd.DataFrame) -> dict: + """Resolve the form into entries. Returns ``{"entries": [...], "problems": [...]}``.""" + lists: dict[str, list] = {} + for _, row in choices.iterrows(): + ln = str(row["list_name"]).strip() + if ln: + lists.setdefault(ln, []).append((str(row["name"]).strip(), str(row.get("label", "")).strip())) + + entries, problems, stack, seen = [], [], [], {} + used_lists = set() + + def col(row, name): + return str(row[name]).strip() if name in row.index else "" + + for i, row in survey.iterrows(): + kind, rest = _split_type(row["type"]) + name = col(row, "name") + if not kind: + continue + if kind in ("begin_group", "begin_repeat"): + gname = name or rest + if not gname: + problems.append(f"row {i + 2}: {kind} with no name") + gname = f"unnamed_{i + 2}" + stack.append((gname, kind == "begin_repeat")) + continue + if kind in ("end_group", "end_repeat"): + if stack: + stack.pop() + else: + problems.append(f"row {i + 2}: {kind} with nothing open") + continue + if not name: + problems.append(f"row {i + 2}: {kind} question with no name") + continue + + path = "/".join([g for g, _ in stack] + [name]) + if name in seen: + raise ValueError(f"survey sheet: variable name {name!r} appears twice " + f"(rows {seen[name]} and {i + 2}); the export would overwrite one") + seen[name] = i + 2 + + entry = { + "name": name, "path": path, "type": kind, + "label": col(row, "label"), + "hint": col(row, "hint"), + "required": col(row, "required").lower() in ("yes", "true", "true()", "1"), + "relevant": col(row, "relevant"), + "constraint": col(row, "constraint"), + "constraint_message": col(row, "constraint_message"), + "in_repeat": any(r for _, r in stack), + "is_metadata": kind in _META, + "choices": [], + "list_name": "", + } + if kind in ("select_one", "select_multiple", "select_one_from_file", "rank"): + ln = rest.split()[0] if rest else "" + entry["list_name"] = ln + if ln in lists: + entry["choices"] = lists[ln] + used_lists.add(ln) + elif kind != "select_one_from_file": + problems.append(f"{path}: {kind} names choice list {ln!r}, which is not on the choices sheet") + entries.append(entry) + + if stack: + problems.append(f"{len(stack)} group(s) never closed: {[g for g, _ in stack]}") + for ln in lists: + if ln not in used_lists: + problems.append(f"choice list {ln!r} is defined but no question uses it") + return {"entries": entries, "problems": problems} + + +def to_markdown(codebook: dict, *, title: str = "Codebook") -> str: + e = codebook["entries"] + n_q = sum(1 for x in e if not x["is_metadata"]) + out = [f"# {title}", "", + f"{len(e)} variables, of which {n_q} are questions and {len(e) - n_q} are " + "form metadata or calculations.", ""] + out += ["| Variable | Type | Label |", "|---|---|---|"] + for x in e: + tag = " *(metadata)*" if x["is_metadata"] else (" *(in repeat)*" if x["in_repeat"] else "") + out.append(f"| `{x['path']}` | {x['type']} | {x['label']}{tag} |") + out.append("") + for x in e: + out.append(f"## `{x['path']}`") + out.append("") + out.append(f"**{x['label'] or '(no label)'}** ") + out.append(f"Type: `{x['type']}`" + (f" from list `{x['list_name']}`" if x["list_name"] else "") + + ("; required" if x["required"] else "") + ("; inside a repeat" if x["in_repeat"] else "") + + " ") + if x["hint"]: + out.append(f"Hint: {x['hint']} ") + if x["relevant"]: + out.append(f"Asked when: `{x['relevant']}` ") + if x["constraint"]: + msg = f" ({x['constraint_message']})" if x["constraint_message"] else "" + out.append(f"Constraint: `{x['constraint']}`{msg} ") + if x["choices"]: + out.append("") + out.append("| Code | Label |"); out.append("|---|---|") + out += [f"| `{c}` | {l} |" for c, l in x["choices"]] + if x["type"] == "select_multiple": + out.append("") + out.append("Exported as one space-separated string of selected codes; " + "expand to one column per option before tabulating.") + out.append("") + if codebook["problems"]: + out += ["## Problems found in the form", ""] + out += [f"- {p}" for p in codebook["problems"]] + out.append("") + return "\n".join(out) + + +def to_csv(codebook: dict) -> pd.DataFrame: + """A flat frame, one row per variable, choices joined as ``code=label|...``. + + This is the shape `label_variables/` reads, so a codebook made here can be + applied straight back onto the exported data as labels. + """ + rows = [] + for x in codebook["entries"]: + rows.append({"variable": x["name"], "path": x["path"], "type": x["type"], + "label": x["label"], + "values": "|".join(f"{c}={l}" for c, l in x["choices"]), + "required": "yes" if x["required"] else "no", + "relevant": x["relevant"], "constraint": x["constraint"], + "metadata": "yes" if x["is_metadata"] else "no"}) + return pd.DataFrame(rows) + + +def main(argv=None) -> int: + ap = argparse.ArgumentParser(description="XLSForm to Markdown codebook") + ap.add_argument("xlsform") + ap.add_argument("-o", "--output", default=None, help="markdown path (default: stdout)") + ap.add_argument("--csv", default=None, help="also write a flat CSV dictionary here") + ap.add_argument("--title", default="Codebook") + args = ap.parse_args(argv) + survey, choices = read_xlsform(args.xlsform) + book = build_codebook(survey, choices) + md = to_markdown(book, title=args.title) + if args.output: + Path(args.output).write_text(md, encoding="utf-8") + print(f"wrote {args.output}: {len(book['entries'])} variables, {len(book['problems'])} problem(s)") + else: + sys.stdout.write(md) + if args.csv: + to_csv(book).to_csv(args.csv, index=False) + print(f"wrote {args.csv}") + return 1 if book["problems"] else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tool_notes/README.md b/tool_notes/README.md new file mode 100644 index 0000000..5dd3f7f --- /dev/null +++ b/tool_notes/README.md @@ -0,0 +1,39 @@ +# tool_notes + +Short, honest notes on seven visual and collaboration tools, with the one +worked file each note needs to make its point. When the tool is worth the +setup, when it is not, and what will go wrong. + +These were seven top-level folders, each presented as a module. None of them +is: the tools live on their own sites, and what a repository can usefully hold +about them is a page of judgement plus a small example, which is what each +folder now contains. The seven are here together so nobody mistakes a +screenshot for a template again. + +| Note | Tool | What is actually in the folder | +|---|---|---| +| [`excalidraw_frameworks/`](excalidraw_frameworks/) | Excalidraw | Two editable `.excalidraw` diagrams, a MEL results chain and a district ecosystem map, with PNG previews | +| [`kumu_maps/`](kumu_maps/) | Kumu | One stakeholder map of a district health network as Kumu JSON, with a preview | +| [`observable_notebooks/`](observable_notebooks/) | Observable | One `.ojs` notebook plotting district coverage, its CSV, and a screenshot | +| [`open_data_editor/`](open_data_editor/) | RawGraphs, Datawrapper, Open Data Editor | A district indicators CSV shaped for those tools, and one chart made from it | +| [`flourish_charts/`](flourish_charts/) | Flourish | The iframe embed snippet, and a screenshot of a chart | +| [`powerbi_reports/`](powerbi_reports/) | Power BI | An adaptation note. No `.pbix` ships; the note links to Microsoft's sample | +| [`miro/`](miro/) | Miro | A human-centred design session outline and links. No board ships | + +## The one judgement that applies to all seven + +Each of these tools is quicker than code for the first version and slower than +code for the tenth. A Flourish chart takes twenty minutes to make and twenty +minutes to remake when the data changes; a script takes two hours the first +time and two seconds thereafter. So the deciding question is how many times the +output will change. A one-off workshop artefact belongs in Miro or Excalidraw. +A quarterly indicator chart belongs in `ggplot2` or `matplotlib`, and this +repository's `stata_snippets/`, `econometrics/` and FieldStack's +`visualisation/` are where to start. + +The second judgement is where the file lives. Everything in Kumu, Flourish, +Observable and Miro lives on their servers under their terms. A district health +network map with named officials is data about people, and it should not sit on +a third-party server without someone having decided that it can. Excalidraw and +RawGraphs run in the browser without an account and keep nothing, which is why +they are the two of the seven that this repository recommends without a caveat. diff --git a/tool_notes/excalidraw_frameworks/README.md b/tool_notes/excalidraw_frameworks/README.md new file mode 100644 index 0000000..842bce5 --- /dev/null +++ b/tool_notes/excalidraw_frameworks/README.md @@ -0,0 +1,33 @@ +# Excalidraw + +Two editable diagrams and their previews. Open a `.excalidraw` file at +[excalidraw.com](https://excalidraw.com) (File, Open) or in the VS Code +extension; nothing is uploaded unless you choose to share. + +| File | What it shows | +|---|---| +| `mel_framework.excalidraw` | A results chain: inputs, activities, outputs, outcomes, with the assumptions between each pair drawn as labelled links rather than left implicit | +| `ecosystem_map.excalidraw` | A district health delivery map: PHC, CHWs, block officer, ASHA supervisors, and who reports to whom | +| `*.png` | Previews, for a README or a slide when nobody needs to edit | + +## When to use it + +A workshop, where the diagram is being built in the room and the hand-drawn +look tells participants it is still a draft and they are allowed to move the +boxes. A theory of change at the stage where the causal claims are still being +argued about. Anything that will be redrawn three times in an afternoon. + +## When not to + +Anything that will be regenerated from data. A results chain whose boxes carry +indicator values is a chart, and it should be a chart, so the numbers update +when the data does. And anything going to a donor as a final document: export +to SVG and place it in the document rather than sending a screenshot, which +prints badly. + +## What goes wrong + +The hand-drawn font (Virgil) is not installed on the reader's machine, so a +PNG export looks right and an SVG opened elsewhere may not. Export PNG for +sharing, keep the `.excalidraw` for editing. And a file with a hundred elements +becomes slow to open in the browser; split a large ecosystem map by block. diff --git a/excalidraw_frameworks/ecosystem_map.excalidraw b/tool_notes/excalidraw_frameworks/ecosystem_map.excalidraw similarity index 100% rename from excalidraw_frameworks/ecosystem_map.excalidraw rename to tool_notes/excalidraw_frameworks/ecosystem_map.excalidraw diff --git a/excalidraw_frameworks/ecosystem_map.png b/tool_notes/excalidraw_frameworks/ecosystem_map.png similarity index 100% rename from excalidraw_frameworks/ecosystem_map.png rename to tool_notes/excalidraw_frameworks/ecosystem_map.png diff --git a/excalidraw_frameworks/mel_framework.excalidraw b/tool_notes/excalidraw_frameworks/mel_framework.excalidraw similarity index 100% rename from excalidraw_frameworks/mel_framework.excalidraw rename to tool_notes/excalidraw_frameworks/mel_framework.excalidraw diff --git a/excalidraw_frameworks/mel_framework.png b/tool_notes/excalidraw_frameworks/mel_framework.png similarity index 100% rename from excalidraw_frameworks/mel_framework.png rename to tool_notes/excalidraw_frameworks/mel_framework.png diff --git a/tool_notes/flourish_charts/README.md b/tool_notes/flourish_charts/README.md new file mode 100644 index 0000000..05e492f --- /dev/null +++ b/tool_notes/flourish_charts/README.md @@ -0,0 +1,32 @@ +# Flourish + +The embed snippet, and a screenshot of one chart. There is no template file to +download, because Flourish templates live in Flourish; what you can keep is the +embed code and the knowledge of what the tool is for. + +| File | What it is | +|---|---| +| `embed_example.md` | The iframe you paste into a page to show a published Flourish chart | +| `flourish_embed_screenshot.png` | A district coverage chart, as it renders | + +## When to use it + +Something going out to readers. Flourish's animated bar races, scrollytelling +stories and maps are built for an audience, not for analysis, and for a +newsletter or a public-facing page they are hard to beat for the time they +take. + +## When not to + +The chart in the report itself, where an animation is a distraction and a +static SVG prints properly. And anything with more than a few thousand rows: +Flourish is a presentation tool, and a large dataset makes the editor slow and +the embed slower. + +## What goes wrong + +Published charts are public, at a URL anyone can guess from the embed, whatever +the page around them says. Do not put anything in a Flourish chart that you +would not put on the front page. And the free tier watermarks the embed; if the +chart is going into a donor deliverable, that is a conversation to have before +building it. diff --git a/flourish_charts/embed_example.md b/tool_notes/flourish_charts/embed_example.md similarity index 100% rename from flourish_charts/embed_example.md rename to tool_notes/flourish_charts/embed_example.md diff --git a/flourish_charts/flourish_embed_screenshot.png b/tool_notes/flourish_charts/flourish_embed_screenshot.png similarity index 100% rename from flourish_charts/flourish_embed_screenshot.png rename to tool_notes/flourish_charts/flourish_embed_screenshot.png diff --git a/tool_notes/kumu_maps/README.md b/tool_notes/kumu_maps/README.md new file mode 100644 index 0000000..4c26871 --- /dev/null +++ b/tool_notes/kumu_maps/README.md @@ -0,0 +1,41 @@ +# Kumu + +One stakeholder map of a district health delivery network, as Kumu-importable +JSON, with a rendered preview. + +| File | What it is | +|---|---| +| `district_health_network.json` | Elements (PHC, CHWs, ASHA supervisor, block officer, pregnant women as a group) and the connections between them, in the `elements` / `connections` shape Kumu imports | +| `district_health_network.png` | The rendered map | + +Import at kumu.io: create a project, then Import, then choose the JSON. Kumu +lays it out; you adjust. + +## When to use it + +Stakeholder and influence mapping where the point is the *structure* of a +system rather than any number in it, and where the map will be presented and +talked through rather than analysed. Kumu's strength is that a non-technical +audience reads it immediately. Power and interest analysis, supervision chains, +who-talks-to-whom in a district health system. + +## When not to + +Anything where you need a network *statistic*. Kumu will show centrality as +node size but will not give you the number in a form you can put in a table or +compare across districts. For that, the data goes into `networkx` (see +`../../network_effects_sni/`) or R's `igraph`, and Kumu is at most the +presentation layer at the end. + +## What goes wrong + +**Named people on a third-party server.** A map of a district health system +with the block medical officer's name on it is personal data and it sits on +Kumu's servers under Kumu's terms. Free projects are public by default. Use +roles rather than names, or use a private project and decide who is allowed to +see it, before importing anything real. + +**The JSON is easy to hand-write and easy to get subtly wrong.** A connection +whose `from` or `to` names an element that does not exist is silently dropped +on import, and the map looks fine with an edge missing. Generate the JSON from +a spreadsheet rather than typing it, and count the connections after import. diff --git a/kumu_maps/district_health_network.json b/tool_notes/kumu_maps/district_health_network.json similarity index 100% rename from kumu_maps/district_health_network.json rename to tool_notes/kumu_maps/district_health_network.json diff --git a/kumu_maps/district_health_network.png b/tool_notes/kumu_maps/district_health_network.png similarity index 100% rename from kumu_maps/district_health_network.png rename to tool_notes/kumu_maps/district_health_network.png diff --git a/miro/HCD_Miro.pdf b/tool_notes/miro/HCD_Miro.pdf similarity index 100% rename from miro/HCD_Miro.pdf rename to tool_notes/miro/HCD_Miro.pdf diff --git a/miro/Miro HCD Template Example.png b/tool_notes/miro/Miro HCD Template Example.png similarity index 100% rename from miro/Miro HCD Template Example.png rename to tool_notes/miro/Miro HCD Template Example.png diff --git a/miro/Miro Purpose.png b/tool_notes/miro/Miro Purpose.png similarity index 100% rename from miro/Miro Purpose.png rename to tool_notes/miro/Miro Purpose.png diff --git a/tool_notes/miro/README.md b/tool_notes/miro/README.md new file mode 100644 index 0000000..0f038b0 --- /dev/null +++ b/tool_notes/miro/README.md @@ -0,0 +1,36 @@ +# Miro + +A human-centred design session outline and links to boards worth copying from. +No board file ships: a Miro board lives in Miro and is copied from a template +there, so what a repository can usefully hold is the session design. + +| File | What it is | +|---|---| +| `miro_use_case_india.md` | When a board is worth the setup, and when a room and paper is faster | +| `HCD_Miro.pdf` | A session structure: framing, empathy mapping, ideation, prioritisation | +| `Miro HCD Template Example.png` | What the finished board looks like | +| `Miro Purpose.png` | The framing slide | +| `sample_miro_board_links.txt` | Public boards to copy from | + +## When to use it + +A remote or hybrid workshop where participants are in three cities and the +alternative is a shared slide deck. Miro's sticky-note-and-vote workflow is a +good substitute for a wall and a stack of Post-its when there is no wall. + +## When not to + +A workshop where everyone is in the room. Paper is faster, needs no login, and +the people least comfortable with a laptop are often the people whose input +matters most. Miro in a physical room is a way of making the facilitator the +only person who can write on the wall. + +And anything that needs to survive: a Miro board is a snapshot on a +third-party server, and the free plan limits how many you can keep. Export the +board to PDF at the end of the session and put the PDF somewhere you control. + +## What goes wrong + +Participants on a phone. Miro on a small screen is unusable for anything beyond +reading, so a session where some people are joining from the field needs a +second channel for them, or their contribution disappears. diff --git a/miro/miro_use_case_india.md b/tool_notes/miro/miro_use_case_india.md similarity index 100% rename from miro/miro_use_case_india.md rename to tool_notes/miro/miro_use_case_india.md diff --git a/miro/sample_miro_board_links.txt b/tool_notes/miro/sample_miro_board_links.txt similarity index 100% rename from miro/sample_miro_board_links.txt rename to tool_notes/miro/sample_miro_board_links.txt diff --git a/tool_notes/observable_notebooks/README.md b/tool_notes/observable_notebooks/README.md new file mode 100644 index 0000000..562498b --- /dev/null +++ b/tool_notes/observable_notebooks/README.md @@ -0,0 +1,37 @@ +# Observable + +One notebook, its data, and a screenshot. + +| File | What it is | +|---|---| +| `district_vulnerability_dashboard.ojs` | Plots vaccine coverage and vaccine availability across districts, with a district selector | +| `public_health_data.csv` | The data it reads | +| `dashboard_screenshot.png` | What it looks like rendered | + +Run it by pasting the cells into a notebook at observablehq.com, or with the +Observable CLI (`npm install -g @observablehq/cli`, then `observable preview`) +if you want it local. + +## When to use it + +An exploratory dashboard that a small team will poke at for a fortnight while +deciding what the real chart should be. Observable is the fastest way to get a +slider attached to a chart, and the reactive model means a change to one cell +updates everything downstream without a rerun. + +## When not to + +A dashboard anyone outside the team will rely on. Observable notebooks depend +on the platform, an account, and JavaScript loaded from Observable's CDN; a +government partner opening it on a locked-down machine sees nothing. For a +durable dashboard, a static HTML page with the chart library inlined, like +InsightStack's own `calculators/`, works anywhere and needs no account. + +## What goes wrong + +The CSV is read at runtime by URL, so a notebook that works on your account +breaks when someone forks it and the file attachment does not come along. And +`.ojs` is not JavaScript, whatever the extension suggests: cells are +declarative and execution order is by dependency, so a cell that uses a +variable defined "later" in the file is fine and a loop that mutates one is +not. If you are debugging something that "should work", that is usually why. diff --git a/observable_notebooks/dashboard_screenshot.png b/tool_notes/observable_notebooks/dashboard_screenshot.png similarity index 100% rename from observable_notebooks/dashboard_screenshot.png rename to tool_notes/observable_notebooks/dashboard_screenshot.png diff --git a/observable_notebooks/district_vulnerability_dashboard.ojs b/tool_notes/observable_notebooks/district_vulnerability_dashboard.ojs similarity index 100% rename from observable_notebooks/district_vulnerability_dashboard.ojs rename to tool_notes/observable_notebooks/district_vulnerability_dashboard.ojs diff --git a/observable_notebooks/public_health_data.csv b/tool_notes/observable_notebooks/public_health_data.csv similarity index 100% rename from observable_notebooks/public_health_data.csv rename to tool_notes/observable_notebooks/public_health_data.csv diff --git a/tool_notes/open_data_editor/README.md b/tool_notes/open_data_editor/README.md new file mode 100644 index 0000000..6353988 --- /dev/null +++ b/tool_notes/open_data_editor/README.md @@ -0,0 +1,36 @@ +# RawGraphs, Datawrapper and the Open Data Editor + +A district indicators CSV shaped for the no-code chart tools, and one chart +made from it so you can see what the shape produces. + +| File | What it is | +|---|---| +| `district_health_gaps.csv` | One row per district, one column per indicator: the tidy-wide shape these tools want | +| `stacked_bar_chart.png` | A stacked bar made from it in matplotlib, which is what RawGraphs produces from the same input | + +## When to use it + +A chart for a report or a slide, made by someone who does not write code, from +data that is already clean. RawGraphs (rawgraphs.io) runs in the browser and +keeps nothing, so it is safe with real data. Datawrapper hosts the chart and +gives you an embed, so it is the right choice when the chart is going on a web +page. The Open Data Editor (from Frictionless Data) is for checking a CSV's +structure before charting it, which is more useful than it sounds. + +## When not to + +A chart that will be remade when the data updates. All three tools start from +a pasted table and none remembers a pipeline, so the fourth quarterly version +costs the same twenty minutes as the first. And anything that needs a survey +weight: they plot the numbers they are given, and a weighted mean has to be +computed elsewhere first. + +## What goes wrong + +**Datawrapper hosts your data.** The embed loads from their servers, the data +is on their servers, and a free account's charts are public. District-level +aggregates are usually fine; anything at a finer grain than that, decide first. + +**The shape matters more than the tool.** These tools want one row per unit and +one column per series. A "long" file with an `indicator` column and a `value` +column will chart as one stripe. Pivot before pasting. diff --git a/open_data_editor/district_health_gaps.csv b/tool_notes/open_data_editor/district_health_gaps.csv similarity index 100% rename from open_data_editor/district_health_gaps.csv rename to tool_notes/open_data_editor/district_health_gaps.csv diff --git a/open_data_editor/stacked_bar_chart.png b/tool_notes/open_data_editor/stacked_bar_chart.png similarity index 100% rename from open_data_editor/stacked_bar_chart.png rename to tool_notes/open_data_editor/stacked_bar_chart.png diff --git a/tool_notes/powerbi_reports/README.md b/tool_notes/powerbi_reports/README.md new file mode 100644 index 0000000..494087b --- /dev/null +++ b/tool_notes/powerbi_reports/README.md @@ -0,0 +1,41 @@ +# Power BI + +An adaptation note. No `.pbix` file ships here, and none is going to: a Power BI +report is bound to its data source, and a template with no data behind it is a +blank canvas with a menu. + +| File | What it is | +|---|---| +| `mel_dashboard_screenshot.png` | What a MEL dashboard in Power BI looks like, for reference | + +Start from Microsoft's own Human Resources sample (`.pbix`, linked from +learn.microsoft.com under "sample datasets"), which opens in Power BI Desktop +and has a layout worth repurposing. + +## When to use it + +A dashboard for an organisation that already runs on Microsoft 365. If the +team's data is in SharePoint and Excel and the licences are already paid for, +Power BI is the path of least resistance and the IT department will support it. +It handles disaggregation well: slicers on gender, district and age group, and +drillthrough from a summary to the block. + +## When not to + +A dashboard for anyone *outside* that organisation. Sharing a Power BI report +externally needs a Pro licence on the viewing side too, and a government +partner or a community organisation generally does not have one. For an +external audience, a static HTML page (see InsightStack's `calculators/`) or a +PDF export works without asking anyone to buy anything. + +## What goes wrong + +**The refresh schedule is the product.** A dashboard is only as current as its +last refresh, and a `.pbix` on someone's laptop refreshes when they open it. +Publishing to the service with a scheduled refresh against a source the +organisation controls is the whole setup; without it you have a screenshot +with slicers. + +**Unicode.** Power BI renders Indic scripts, but the default fonts do not +include them all, so a Hindi label renders as boxes until the font is changed +in the visual. Check on the machine the audience will use, not on yours. diff --git a/powerbi_reports/mel_dashboard_screenshot.png b/tool_notes/powerbi_reports/mel_dashboard_screenshot.png similarity index 100% rename from powerbi_reports/mel_dashboard_screenshot.png rename to tool_notes/powerbi_reports/mel_dashboard_screenshot.png diff --git a/writing_guides/README.md b/writing_guides/README.md index f3c8b73..7ca2c58 100644 --- a/writing_guides/README.md +++ b/writing_guides/README.md @@ -1,15 +1,25 @@ # writing_guides -Notes and a template for writing a theory of change. +Guides for the documents an evaluation produces, in the order they are +written. -| File | What it is | +| Guide | What it is for | |---|---| -| `theory_of_change_notes.md` | The notes: what a theory of change is for, where they go wrong, what a reader should be able to check | -| `theory_of_change_notes.qmd` | The same as a Quarto source, if you want to render it or adapt it | -| `sample_theory_of_change_note.pdf` | A worked example, rendered | - -The writing-up is usually the part that decides whether an evaluation is used, -and it is the part least often planned for. A theory of change written after the -results are in tends to be a description of what happened rather than a claim -anyone could have disagreed with beforehand, which is the whole point of having -one. +| `theory_of_change_notes.md` | The programme's theory as a short note: problem, pathway, assumptions, outcomes. `sample_theory_of_change_note.pdf` is a worked example; the `.qmd` is the same note as a Quarto source | +| `results_chain.md` | The theory with the arrows made explicit: "if A then B because C", one link at a time, to find the link everyone assumed | +| `evaluation_report_structure.md` | The report: what goes in which section, and the table of which verb each design permits in the summary | +| `policy_brief.md` | Two pages, one message, one ask, for a reader who decides things | +| `writing_about_uncertainty.md` | Sampling, design and measurement uncertainty, and how to write each in the same sentence as the number | + +## The idea that runs through all five + +The verb is decided by the design. "Rose among participants", "was higher +than in comparison areas", "increased relative to comparison areas", +"caused": these are not stylistic choices, they are claims of different +strength, and the design in the methods section fixes which one the summary +is allowed. Most misleading evaluation writing is a strong verb on a weak +design, and it usually happens in the summary or the brief, written last, by +someone who did not write the methods. + +`../annotated_research/` has five reports read this way, with the verb +errors marked. diff --git a/writing_guides/evaluation_report_structure.md b/writing_guides/evaluation_report_structure.md new file mode 100644 index 0000000..021a337 --- /dev/null +++ b/writing_guides/evaluation_report_structure.md @@ -0,0 +1,79 @@ +# Structure of an evaluation report + +The reader is a programme manager or a donor officer with twenty minutes. +They will read the summary, look at one table, and skim the recommendations. +Structure the report so that those three things are true on their own and +consistent with everything else. + +## The order + +**1. Summary (one page).** The evaluation question, the design in one +sentence, the headline result with its uncertainty, the main limitation, and +the recommendation. Every number in the summary appears in a table later with +the same value. The summary's verbs match the design: see "The verb" below. + +**2. The programme and the question.** What was done, for whom, where, when. +The evaluation question stated as a question. What would count as the +programme having worked, decided before the data. + +**3. Design.** The comparison being made and why it supports the claim. What +it cannot rule out, stated here and not saved for the limitations. The +sample: who, how selected, how many, and how many were lost between rounds, +by arm. + +**4. Data.** Instruments, when collected, by whom, how quality was checked. +Attrition and missingness handled explicitly. A table of baseline balance +between arms, including anything that would have predicted the outcome. + +**5. Results.** One table per evaluation question, with the estimate, its +standard error or interval, the sample size behind it, and the level at +which errors are clustered. Disaggregations promised in the MEL framework, +each reported whether or not it is interesting. A null result reported in the +same format as a positive one. + +**6. Limitations.** The real ones. Ranked by how much they threaten the +headline, with the direction of bias where it is known. A limitations +section that lists "sample size" and "short duration" and omits "no +comparison group" is doing public relations. + +**7. Recommendations.** Each one traceable to a specific result. "Scale up" +requires a result that supports scale-up; a before-after study does not +provide one, and the recommendation from a before-after study is "run a +comparison-group study before scaling". Cost per unit of effect where it can +be computed. + +**8. Annexes.** Instruments, the analysis plan as pre-specified, the +deviations from it, the full tables, the code or a pointer to it. + +## The verb + +The single most consequential choice in the report is the verb in the +summary's headline sentence, and it is decided by section 3, not by the size +of the number. + +| Design | The verb the summary may use | +|---|---| +| Cross-section, one arm | "was", "reported", "among participants" | +| Before-after, one arm | "rose from X to Y among participants". Never "increased" without "among participants"; never "the programme increased" | +| With-without, not randomised | "was higher in programme areas than comparison areas". "Associated with" | +| Difference-in-differences | "increased relative to comparison areas", with the parallel-trends caveat in the same paragraph | +| Randomised | "increased", "caused", "reduced", subject to attrition and clustering being handled | +| Regression discontinuity | "increased, for households near the threshold" | + +The annotated briefs in `../annotated_research/` are five reports read +against this table. + +## Things to cut + +Background on the sector that the reader knows. A methods section that +explains what a survey is. A findings section organised by instrument rather +than by question. Recommendations that are not connected to a finding. +Anything in the summary that is not in the body. + +## Two checks before it goes out + +Read only the summary and the tables. Does the summary claim anything the +tables do not show? + +Read only the limitations and the recommendations. Does any recommendation +survive its own limitations? diff --git a/writing_guides/policy_brief.md b/writing_guides/policy_brief.md new file mode 100644 index 0000000..27b5f9e --- /dev/null +++ b/writing_guides/policy_brief.md @@ -0,0 +1,60 @@ +# Writing a policy brief + +A policy brief is two to four pages for a reader who decides things and does +not read reports. It has one message, evidence for it, and one ask. + +## The structure + +**Title.** The message, as a sentence. "Community early warning reached +nine in ten households; district loss records should confirm it before +scale-up" is a title. "Evaluation of the Community Early Warning Programme" +is a label. + +**The message, in the first paragraph.** What was found, how sure you are, +and what should happen. A reader who stops after the first paragraph has the +brief. + +**The evidence.** Two or three findings, each a sentence with a number and +its source. One chart or table if it carries a finding better than the +sentence does, and none if not. The design in one clause: "in a survey of +842 households in programme villages" tells a careful reader what kind of +evidence this is. + +**What it does not show.** One paragraph. The main limitation and what it +means for the ask. Readers who decide things have seen many briefs that +omitted this paragraph and were wrong, and a brief that includes it is read +as more credible, not less. + +**The ask.** One thing, addressed to someone who can do it, with a cost if it +has one. "The Disaster Management Authority should compare 2024 loss records +between programme and non-programme villages before the scale-up decision in +March" is an ask. "Stakeholders should consider replication" is not. + +**Where to read more.** The full report, with a link. + +## Rules + +**One message.** A brief with three messages has none. If the evaluation +found three things, write three briefs or pick the one that matters. + +**Numbers with their uncertainty and their base.** "87 per cent (of 842 +households in programme villages)" not "87 per cent". "A 31 per cent +increase" should say what the increase is of, over what period, against +what comparison. + +**The verb matches the design.** See `evaluation_report_structure.md`. A brief +is where causal verbs most often get attached to non-causal designs, because +the brief is written for people who want a causal answer. + +**No jargon the reader would have to look up.** "Difference-in-differences" +becomes "compared with similar schools that did not have the programme". +"Statistically significant" becomes "unlikely to be chance" or is dropped in +favour of the interval. + +**Sentence case, plain words, short paragraphs.** A brief is read on a phone +between meetings. + +## Length + +Two pages is right. Four is the maximum. Six is a report with a different +cover. diff --git a/writing_guides/results_chain.md b/writing_guides/results_chain.md new file mode 100644 index 0000000..bdb55b1 --- /dev/null +++ b/writing_guides/results_chain.md @@ -0,0 +1,64 @@ +# Writing a results chain + +A results chain is the theory of change with the arrows made explicit: each +link is a sentence of the form "if A, then B, because C". Written out, it +exposes the step that everyone assumed and nobody stated, which is where the +programme usually fails. + +## The form + +Inputs, then activities, then outputs, then outcomes, then impact. Between +each pair, a link written as a conditional with its mechanism: + +> If peer leaders are trained and given materials (output), then monthly +> sessions will be held in each school (output), because trained peer leaders +> have the content and the school has agreed to the timetable. + +> If monthly sessions are held (output), then girls in classes 6 to 8 will +> know the recommended hygiene practices (outcome), because the sessions +> cover them and girls attend. + +> If girls know the practices (outcome), then more girls will use a sanitary +> product during menstruation (outcome), because knowing what to do is what +> stops them. + +Read the third link again. It claims knowledge is the binding constraint. If +the real constraint is that pads cost money or that there is nowhere to +change one at school, the chain breaks at that link and no amount of +training fixes it. That is the sentence the exercise exists to produce. + +## Rules + +**One "because" per link, and it names a mechanism.** "Because the programme +is well designed" is not a mechanism. "Because knowing what to do is what +stops them" is a mechanism, and it can be wrong, which is the point. + +**Every link is a hypothesis somebody could test.** If a link cannot be false, +it is not carrying any weight. "If sessions are held, then sessions will have +been held" is a tautology; "if sessions are held, then girls will attend" can +fail and the programme has to say why it will not. + +**The weakest link decides the programme.** Find it, say it is the weakest, +and put the monitoring there. A results chain with no weak link has been +written to reassure rather than to plan. + +**Outcomes are changes among people, not things the programme did.** "Girls +trained" is an output whatever column it is written in. + +**Stop at the level the programme can be held to.** Impact, the population- +level change many programmes contribute to, belongs at the top as context. +The programme's claim ends at the outcome. + +## From chain to logframe + +The outputs and outcomes become the rows of the logframe +(`../eval_docs/logframe_template.md`). The "because" clauses become the +assumptions column. Nothing in the logframe should be absent from the chain, +and the chain should have nothing the logframe leaves out except the +mechanisms, which the logframe compresses into assumptions. + +## A test + +Give the chain to someone who has not seen the programme and ask them to +find the link they believe least. If they pick a different one from you, +one of you is wrong about the programme, and it is cheaper to find out now. diff --git a/writing_guides/theory_of_change_notes.md b/writing_guides/theory_of_change_notes.md index 9bda909..cbb71c2 100644 --- a/writing_guides/theory_of_change_notes.md +++ b/writing_guides/theory_of_change_notes.md @@ -5,7 +5,7 @@ A Theory of Change (ToC) note is a short, structured explanation of *how* and *w --- -## ✍🏽 Core Components +## Core Components **1. Problem Statement** Briefly outline the core issue you're addressing. Ground it in evidence. @@ -20,14 +20,14 @@ What are the key steps/activities that will trigger change? Use causal logic, no List what needs to be true for the ToC to work (e.g. policy stability, stakeholder buy-in). **5. Outcomes** -Short-term, medium-term, and long-term — and how they connect. +Short-term, medium-term, and long-term , and how they connect. **6. Indicators (optional)** You may include early thoughts on how you’ll measure each outcome. --- -## ✏️ Sample Structure +## Sample Structure ``` Problem: High adolescent anemia in tribal blocks of X district, with poor uptake of IFA tablets. @@ -50,18 +50,18 @@ Long-term: Reduced prevalence of anemia --- -## ✅ Do's and Don'ts +## Do's and Don'ts | Do's | Don'ts | |------|--------| | Use plain language | Overload with jargon | | Ground assumptions in reality | Leave out local risks/barriers | -| Align with MEL plans | Treat ToC as fixed — it evolves | +| Align with MEL plans | Treat ToC as fixed , it evolves | | Make outcomes measurable | Use vague claims like “empowerment” with no link | --- -## 🔍 Quick Checklist +## Quick Checklist - [x] Clear problem + target group - [x] Plausible, evidence-based causal pathway diff --git a/writing_guides/theory_of_change_notes.qmd b/writing_guides/theory_of_change_notes.qmd index 1a03abb..0e2be02 100644 --- a/writing_guides/theory_of_change_notes.qmd +++ b/writing_guides/theory_of_change_notes.qmd @@ -41,7 +41,7 @@ Activities: |------|--------| | Use plain language | Overload with jargon | | Ground assumptions in reality | Leave out local risks/barriers | -| Align with MEL plans | Treat ToC as fixed — it evolves | +| Align with MEL plans | Treat ToC as fixed , it evolves | | Make outcomes measurable | Use vague claims like “empowerment” with no link | --- diff --git a/writing_guides/writing_about_uncertainty.md b/writing_guides/writing_about_uncertainty.md new file mode 100644 index 0000000..8395d51 --- /dev/null +++ b/writing_guides/writing_about_uncertainty.md @@ -0,0 +1,68 @@ +# Writing about uncertainty + +Every number in a report is uncertain, and most reports are written as if +none were. The result is that readers cannot tell a finding from a guess, +and learn to treat all numbers as guesses, which is worse for the findings +than for the guesses. + +## The three kinds, and how each is written + +**Sampling uncertainty.** The number would have been different with a +different sample. Written as an interval: "64 per cent (95 per cent interval +58 to 70)". The interval, not the p-value, is what the reader needs; a +p-value says whether the interval excludes zero, which is one fact about it. + +**Design uncertainty.** The number might not mean what the sentence says, +because the comparison cannot rule out other explanations. Written as a +verb choice and a caveat in the same sentence: "product use rose from 38 to +64 per cent among participating girls; without a comparison group the study +cannot say how much of this is the programme". This is the uncertainty +intervals do not capture and it is usually the larger one. + +**Measurement uncertainty.** The number might not be the thing it is named +for. Written as a description of what was actually recorded: "girls' own +reports, to programme staff, of product use during their last period". +The reader can then judge the demand effect. + +## Sentences that work + +"Between 58 and 70 per cent of girls, most likely around 64, reported using +a product." + +"The estimate is 18 percentage points. Given the way schools were selected, +the true effect is probably smaller; it is unlikely to be zero." + +"We do not know whether the programme caused the change. We know the change +happened in programme schools and we did not measure schools without it." + +"This is a local estimate: it applies to households near the eligibility +threshold and may not apply to poorer households further below it." + +## Sentences that do not + +"The programme significantly improved outcomes." Significant in which sense, +by how much, compared with what. + +"Results indicate a positive trend." Every number is a positive trend or a +negative one; the sentence says nothing. + +"Further research is needed." Say what research, on what question, and why +the present study could not answer it. + +"Robust." The word has a technical meaning (the result survives a specific +alternative specification) and a colloquial one (I am confident), and +readers cannot tell which is meant. Say which specifications were tried. + +## Rounding + +Report to the precision the data support, which is usually less than the +software prints. A percentage from a sample of 800 is known to about two +points; report "64 per cent", not "64.3". A coefficient with a standard +error of 4.1 is "about 18", not "18.24". Excess digits are a claim to +precision that the interval contradicts. + +## The one rule + +State the uncertainty in the same sentence as the number, not in a +limitations section three pages later. A reader who takes the number and +leaves the caveat has been given permission to by the layout.