Strategy-to-backtest validation and traceability

[Shubhyam-f]

What this PR does

Adds reproducible validation and traceability documentation for strategy candidates, backtest runs, and simulated fills, addressing #165.

The contribution includes SQL validation checks, metric definitions, data lineage documentation, Power BI measures, and Power Query transformations.

Scope

• Validates candidate → backtest run → fill relationships
• Documents 1:N candidate-to-run lineage and latest-run semantics
• Reconciles fill counts, fees, initial cash, and return calculations
• Documents annualized return and risk-adjusted metric conventions
• Documents metric reproducibility limitations caused by the absence of a per-bar equity curve
• Provides Power BI lineage/inspection logic for tracing results from runs to fills

Validation

Validation was performed against the provided sanitized snapshot using both Excel and SQLite.

The Power BI prototype provides an investigation flow from aggregated realized P&L to candidate/run/fill-level details.

Data boundary

No private production data, account information, credentials, internal IDs, or raw private rows are included in this PR.

The contribution contains analytical logic and documentation intended to be reproducible against an equivalent sanitized dataset.

Related issue

Closes #165

[mirror29]

Thanks, Shubhyam, for packaging this work and incorporating the confirmed metric conventions. The candidate → run → fill scope is useful: it can become a repeatable reconciliation baseline for backtest outputs, with Power BI as an inspection interface.

I reviewed the eight changed files, compared the metric definitions with the engine, and exercised the SQLite queries against small synthetic cases. I have not rerun your private dataset or the PBIX, so the reported snapshot PASS results remain unverified in this review. Please keep this as a Draft and address the following before the next review.

Required corrections

1. Restore the project README. README.md replaces the repository's project overview with this contribution's package notes. Restore the project README and place the analytical package in a dedicated directory, for example examples/backtest-validation/, with its own README and relative links.

2. Correct the realized-P&L measure. In powerbi/DAX_MEASURES.md:6–7, metric_initial_cash - final_equity reverses the sign of equity gains. Reversing the subtraction alone is insufficient: equity change is not gross realized P&L from fills, because fees and unrealized P&L affect equity. For the stated metric, use a proper aggregate measure such as:

   Total Gross Realized P&L = SUM(backtest_trades[realized_pnl])
   

   The current unaggregated column references also do not support the described multi-run measure context. Export the actual measures used by the prototype, and verify candidate/run filtering against known synthetic results.

3. Make the SQL file executable end to end. sql/validation_checks.sql:44 contains the uncommented text docs(data): add SQL validation checks. Executing the submitted script against a synthetic schema fails with near "docs": syntax error. Remove it and rerun the complete checked-in script.

4. Explicitly detect missing values and invalid denominators. The comparisons at sql/validation_checks.sql:12–29 silently exclude NULL values. In synthetic cases, NULL reported counts/fees and zero metric_initial_cash produced no rows in the relevant exception queries. Add checks for required fields and non-positive initial cash; distinguish invalid or uncheckable values from successful reconciliation. Preserve the documented, expected NULL behavior for nullable candidate fields and legitimately undefined metrics. FINDINGS.md should only claim NULL checks that are actually included.

5. Include executable annualization validation. METRIC_CATALOGUE.md:6 says annualized return was independently validated, but the SQL contains no corresponding check. Include the timeframe mapping, calculation, tolerance, and discrepancy output. Handle zero/missing bar counts and unsupported market/timeframe factors explicitly rather than treating them as passes. The agreed formula is linear annualization, not CAGR.

6. Preserve fractional cash values in Power Query. In powerbi/POWER_QUERY.md:50,55, both cash fields are converted to Int64.Type. This rounds fractional amounts before reconciliation. Use a suitable numeric type that preserves the source amounts and include a fractional-cash example in the checks.

Minimum reproducible handoff

Please keep this bounded to the original issue; no additional dashboard redesign or live-database integration is needed.

• Provide the SQLite schema/import steps and exact command(s) to run the checks from a fresh environment.
• Include a small, entirely synthetic dataset with expected outputs. Cover a valid case, a broken relationship, duplicate fills, missing required values, zero initial cash, fractional cash, and an annualized-return mismatch. Separate informational queries (such as candidates without runs) from failures, and label query results so a reader can identify each check.
• Export all measures needed for the described prototype and document active relationships/filter direction, drill-through fields, and how validation exceptions are surfaced. Text instructions or synthetic-only screenshots are sufficient; do not publish the private PBIX or CSVs.
• Complete the selected metric catalogue with formula, unit, source fields, grain, NULL/edge-case behavior, fee treatment, and reproducibility class. For unresolved definitions, state the gap explicitly.
• Update the findings with actual check results and distinguish checks executed on the private snapshot from those reproducible using the public synthetic example.

The value here is a reusable check that reported backtest results reconcile with their underlying records. Internal consistency alone does not establish strategy profitability or independently prove the engine's equity calculations; retaining that distinction will make the contribution stronger.

Please continue to keep private rows, account identifiers, credentials, and production connection details out of the PR. Once these changes are ready, reply here with the reproduction commands and a short results summary, and we can review the same focused scope again.

[Shubhyam-f]

Thanks Miro, really appreciate the detailed review and the time you put into testing the queries against synthetic cases.

I’ve gone through the points and will work through the requested corrections and reproducibility pieces before the next review. I’ll keep the scope focused on the original issue and leave the PR as Draft for now.

[Shubhyam-f]

Hey Miro, I've incorporated the requested corrections and pushed the updated version to Draft PR #176.

The package now includes the synthetic SQLite reproduction flow, expected synthetic results, expanded metric catalogue, required-field/denominator and annualization validation, and updated Power BI documentation.

I also restored the repository README and kept the analytical material under examples/backtest-validation/. Private CSV/PBIX data remains excluded.

The PR is still kept as Draft for your review.