Skip to content

SHA-256 digest capture and verification for all downloads (Provenance Phase 1) #1296

Open
#1310
Open
SHA-256 digest capture and verification for all downloads (Provenance Phase 1)#1296
#1310
Assignees
LalatenduMohanty
Labels
enhancementNew feature or request

Description

@LalatenduMohanty
LalatenduMohanty
opened on Aug 6, 2026

Summary

fromager downloads sdists and wheels without computing or verifying any cryptographic hash. A compromised upstream can replace artifact bytes between runs and fromager will accept them silently. This is Phase 1 of #1238.

Current state

  • download_url() streams bytes to disk with no hashlib computation
  • #sha256= URL fragments from PyPI indexes are carried in URL strings but never verified
  • pypi_simple.DistributionPackage.digests is available but never read
  • Candidate, DependencyNode, and build-order entries have no digest field

Proposed changes

  1. Compute SHA-256 during download — add streaming hash in download_url(), the single chokepoint for all HTTP downloads; return the digest alongside the file path
  2. Verify URL fragment hashes — extract #sha256= from download URLs and hard-fail on mismatch with computed digest
  3. Read index digests — use dp.digests from pypi_simple as a cross-check when creating Candidate objects
  4. Store digests in data structures — add optional digest field to Candidate, DependencyNode/DependencyNodeDict, and build-order entries (backward-compatible via NotRequired/defaults)
  5. Write provenance-index.json — aggregate per-package digests during finalize(), alongside build-order.json
  6. Re-download verification — when a digest is already recorded, verify new downloads produce identical bytes (trust-on-first-use)

Activity

  1. LalatenduMohanty
    added
    enhancementNew feature or request
    on Aug 6, 2026
  2. LalatenduMohanty
    self-assigned this
    on Aug 6, 2026
  3. LalatenduMohanty
    added a commit that references this issue on Aug 6, 2026

    docs(proposals): add SHA-256 digest capture and verification proposal

    27fce3b
  4. LalatenduMohanty
    added a commit that references this issue on Aug 31, 2026

    docs(proposals): add SHA-256 digest capture and verification proposal

    c225f2e

  5. LalatenduMohanty commented on Sep 1, 2026

    @LalatenduMohanty
    LalatenduMohanty
    on Sep 1, 2026
    MemberAuthor

    I propose breaking the provenance work into the following phases:

    Phase 1 — Capture build lineage

    Focus on answering: “Which upstream bytes, source changes, downstream sdist, and wheel produced this build?”

    • Record upstream archive URL and digest.
    • For Git sources, record repository, requested ref/tag, resolved commit, and source-tree identity.
    • Record unpacking, repacking, patches, overrides, vendoring, plugin changes, and PKG-INFO status.
    • Record normalized build context such as variant, platform, interpreter, and relevant configuration.
    • Record downstream sdist and wheel digests.
    • Link the artifacts as upstream input → downstream sdist → wheel.
    • Write the complete lineage to provenance-index.json.
    • Add lightweight artifact and lineage references to graph.json and build-order.json.

    Phase 1 should preserve existing build behavior and plugin APIs. It should record current repacking and stub PKG-INFO behavior rather than changing them.

    Phase 2 — Verify and safely reuse artifacts

    • Track expected and observed digests separately.
    • Verify downloads, existing files, and cache entries.
    • Add continuity checks for previous build metadata.
    • Reject conflicting digest claims.
    • Make downstream sdist reuse depend on matching digest, source changes, and build context.
    • Fail closed on integrity mismatches.

    Phase 3 — Integrate downstream consumers

    • Update builder onboarding to preserve upstream identity.
    • Model source-mirror uploads as publications rather than replacements for the upstream artifact.
    • Add optional upstream evidence when already available, such as publisher, repository, workflow, ref, commit, release date, wheel tags, or attestation presence.
    • Do not fetch or verify attestations in this phase.

    Future work — Improve sdist quality

    The following should be handled separately because they change build behavior:

    • avoid unnecessary sdist repacking where possible;
    • generate correct PKG-INFO for Git and raw-tarball sources; and
    • make sdist generation deterministic across variants and environments where practical.

    This phased approach keeps the primary objective focused on recording complete upstream-to-downstream provenance while allowing verification, integration, and sdist-generation improvements to evolve independently.

  6. LalatenduMohanty
    added 4 commits that reference this issue on Sep 1, 2026

    docs(proposals): add SHA-256 digest capture and verification proposal

    c4fbe16

    docs(proposals): propose artifact digest verification

    4e4944a

    docs(proposals): propose artifact digest verification

    6f48046

    docs(proposals): propose artifact digest verification

    f842140
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions