Update to .Net 11 RC 1 with EFCore 11 RC 1 - #299
Merged
Conversation
A corpus of 27 .accdb files downloaded from the web turned up one table LibRed could not read: fctSales in AdventureWorks_Learn_To_Write_DAX, whose OrderDate is a calculated DateTime. The decoder was handed the whole 31-byte slot and refused it as a DateTime of the wrong width. ACE does not store a bare value there. It wraps the result it last computed in an envelope -- 16 reserved bytes, a 4-byte little-endian payload length, the payload, 3 bytes of padding -- and the payload is the value in its ordinary on-disk encoding, so it goes straight to the normal type codec once unwrapped. The descriptor's type is a STORAGE type, not the result type: ACE widens it to the next type in its family and keeps the payload at the result's natural width, so the payload length is what says how to read it. The (declared, width) pair is unambiguous -- Boolean and Byte both store one byte but promote to different declared types. A zero-length payload is Null, and the null-bitmap bit stays set regardless, which makes a calculated Boolean a trap: an ordinary Boolean IS its bitmap bit, but a calculated one carries a real payload and its bit is set for a stored False too, so the old short-circuit reported True for every row. A calculated Memo arrives through the long-value machinery instead: ACE declares it Text with length 0, gives it a long-value map entry, and stores a descriptor that resolves to the same envelope. The guard added in b0ef976 required long-value map entries to name a Memo/OLE column and rejected that -- and because the catalog reads every table definition, one such column made the whole database unopenable. Writing is refused rather than attempted. Only ACE can evaluate the expression, so encoding the decoded value back would put a bare value where an envelope belongs; the check sits on AssembleRow, which both row writers pass through. DELETE stays allowed, and now frees a calculated Memo's pages -- it keyed the free off the declared type, walked past a Memo declared Text, and orphaned them. Measured against ACE-authored columns of every type DAO will create; the corpus goes from 342 tables read with one failure to 343 with none. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…d back LibRed could read a calculated column's cached value; it now creates, evaluates, writes, recomputes and alters them, with a SQL surface of its own since Access SQL has none. The measured format and semantics go in docs/format/page-02e-calculated-columns.md and §3.4a of page-02b-columns.md, added here; the notes below are only what changed. Two latent bugs, both invisible from LibRed's own side: - Text compression was applied to a calculated Memo as well as a calculated Text, because StoredType folds Memo onto Text. That shrank a 36-character result from 95 bytes to 61 and so back under the inline limit, meaning a value ACE puts on a long-value page never left the row. - RenameColumn left every expression that READ the renamed column naming the old one, after which ACE failed every read of the calculated column -- a table broken by an operation that named a different column entirely. New refusals. Reading a cached ERROR now throws rather than reporting Null, the envelope's leading field being a status rather than padding. Writing is refused for an explicit value, for CDbl over Null, for dropping a column an expression reads, and for indexing one. Only the first matches ACE; page-02e records why each of the others is worth diverging for, the short version being that ACE accepts them and produces a file it cannot then use, twice in a way nothing can repair. Indexing is the odd one out -- refusing matches Access's designer and storage engine, and diverges only from its SQL layer, which is the only place the state is reachable from. The SQL surface is `name type AS (expression)`. The parentheses are required, unlike SQL Server's, because the grammar does not parse the expression -- it captures raw text for a separate parser, so the delimiter is what says where it ends. Backtick quoting is translated to brackets on the way in, and the same translation now covers CHECK constraints, which carried the identical latent bug. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
0x0D, 0x0E and 0x11 become placeholders in JetDataType so a TDEF carrying one still parses: the column decodes to its raw bytes, and only writing one is refused. The catalog reads every table's definition, so a single unrecognised column previously made the entire file unopenable -- three of four real Jet 4 .mdb files taken off the web would not open at all, and read cleanly afterwards (69 tables, ~204k rows across the four). Same failure shape as the long-value guard that once rejected a calculated Memo. 0x11 is the only one met in the wild, and never on a user column; what it holds, and which files carry it, is in data-types.md. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CreateEmpty refused anything non-ACCDB because page 0 was once stamped "Standard ACE DB" unconditionally, and that identifier paired with a Jet 4 version byte is exactly the mismatch Detect rejects -- a file it would write and then fail to reopen. BuildDefinitionPage has taken the identifier from the version for a while, so only the guard was left. Nothing else differs: DAO's own dbVersion40 database contains exactly the four core system tables this builds, and Access adds MSysAccessStorage, the nav-pane tables and the MSysDb properties on first open, for a Jet 4 file as for an ACCDB. So the creator writes none of those, and does not write AccessVersion either -- the engine never does. Verified through ACE rather than by reopening alone: it reads a row LibRed wrote into a Jet 4 file, inserts one of its own, and both survive. That exercises the usage maps and free-space accounting on a format LibRed had never written. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The RC1 upgrade moved the TPC/TPT/TPH classes into Query.Inheritance and folded the AdHoc complex-type and many-to-many tests into the classes that used to inherit them. Their old names stayed in the pass-lists, and CI reads a listed test that no longer exists as a regression -- 3360 entries on OLE DB, 1318 on ODBC. Pruned against `dotnet test --list-tests`, keeping only what it discovers. Two traps if this is ever redone, both of which silently delete live tests: the lists are CRLF while --list-tests emits LF, so a naive comparison matches nothing and every entry looks stale; and a long theory name is truncated in the lists with three U+00B7 dots, the logger closing the quote it cut through before appending them, so it can never match the full name exactly. Where even a prefix cannot match -- the two renderings disagree on non-BMP characters -- an entry whose Class.Method still exists is kept, because wrongly keeping one costs a visible CI failure and wrongly removing one loses cover in silence. Nothing is added: CI appends newly-passing tests itself, so the moved classes return under their new names on the next green run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Shard 3 carried everything outside Query, including CompiledModelJetTest, whose 16 tests generate and compile a model each and are the slowest and most crash-prone thing in the suite. Splitting them into shard 4 keeps a crash there from taking the rest of the non-Query tests down with it, and lets the retry loop re-run just those. Shard 3 gains the matching exclusion so they do not run twice. Nothing else needed changing: the result merge and the artifact upload both glob TestResults recursively, so shard 4's trx joins the green-test comparison on its own. The filter matches CompiledModel rather than the full type name, so a second such class would land in the right shard without another edit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The MSysQueries Attribute=1 row is the query KIND, and SELECT is one of its
values -- it is not a marker that the query is an action query. Routing on the
row's mere presence sent every SELECT carrying one into ActionQueries, where the
flag matched no known action and came back as "this stored action query kind is
not supported". Across a corpus of 19 real-world databases that was 221 of 592
stored queries; one file surfaced 1 of its 44.
Read the kind instead. The full vocabulary is now in StoredQueryFormat, measured
by cross-tabulating the flag against DAO's QueryDef.Type over every stored query
in that corpus: 1 SELECT, 2 make-table, 3 append, 4 update, 5 delete, 6 crosstab,
7 DDL, 8 pass-through, 9 union. Append/update/delete confirm what was already
there; DDL is confirmed by our own round-trip; pass-through has no sample here
and is marked unverified in the spec.
Three more things Access encodes that we were dropping or misreading:
- A query with no Attribute=6 rows is SELECT *. That is the shape of every
auto-generated form/report record source, and returning null for it dropped
those queries silently -- neither dictionary, no reason recorded.
- DISTINCTROW is its own option bit (0x08), distinct from DISTINCT (0x02).
They are different keywords: one dedupes output rows, the other dedupes by
contributing base rows.
- PERCENT is a bit (0x20) riding alongside TOP (0x10). Masking on TOP alone
turned "TOP 10 PERCENT" into "TOP 10" -- a silently different row count.
Checking the rebuilt SQL against ACE's own rendering also caught a parameter
type code of 0 being mapped to TEXT when it is Access's untyped Value keyword,
and a lone WHERE predicate being wrapped in a paren pair Access never wrote.
479 of the 592 now come back as views, of which 337 reproduce ACE's SQL exactly
once bracketing and whitespace are normalised. The 87 still refused are all
genuinely non-SELECT kinds, and each refusal now names which. What remains
unread is honest: 26 shapes we cannot rebuild, and 64 that rebuild into SQL our
own parser rejects, nearly all of it bang notation referencing a form control.
ACE will not author the SELECT-with-operation-row shape on demand -- DAO's
CreateQueryDef omits the row and DAO refuses to open MSysQueries at all -- so
the test writes the row with LibRed and has ACE run the resulting query, which
it does. An earlier draft asserted the flag only as a premise check, which is
the one reason it was caught testing nothing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The "not yet" list had become a changelog: a "done since" paragraph, a preamble listing what used to be on it, and six entries carrying "this part is now done" asides. It holds open work only, and everything closed is described in its own area instead. Checking the entries against the code rather than trusting the list found two stale ones -- calculated columns are complete, and UnicodeCompression is modelled, leaving only AllowZeroLength. The package readme gets the same at its own altitude, plus the caveat that reader needs: Access-authored validation rules are read but not enforced. Also drops the green-list entries the new xunit renamed. xunit v3 4.0.0 stopped truncating long theory display names, so every entry carrying the three U+00B7 dots named a test that no longer exists -- nothing failed, and the check step's own append will write the current names back. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Incremented PreReleaseVersionIteration in Version.props from 1 to 2 to begin the next pre-release cycle (alpha.2) after the previous release. This ensures daily builds are correctly labeled for the new iteration.
Updated test output files to reflect many new passing tests for complex types, JSON, inheritance (TPC/TPT), and many-to-many relationships. Added explicit skips for computed column tests due to Jet limitations. Stubbed collation retrieval in MigrationsJetTest.cs to remove SQL Server-specific logic.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tracks the RC 1 release of .NET 11 and EF Core 11, and lands calculated-column support in LibRed along with several fixes to reading databases LibRed did not create.
.NET 11 RC 1 / EF Core 11 RC 1
Updates the SDK pin, the package references and the adapted tests. RC 1 also fixes the upstream ordering regression that appeared during the preview cycle, so the defensive ORDER BY tie-breaker both query postprocessors carried is removed — it was deliberately over-broad when it went in, because the shape that exposed it only really surfaced on a real Jet engine. Baselines are regenerated accordingly.
Calculated columns (ACE 14) — LibRed
Previously LibRed could read the value ACE had cached but not author or maintain one. Now the whole surface works:
The 0xC0 descriptor flag and the Expression / ResultType LvProp properties are written on CREATE TABLE and ALTER, with the SQL AS (…) form added to the grammar.
Values are evaluated on insert and recomputed when a dependency changes; Memo results spill to LVAL pages.
EF Core scaffolding materialises the expression back into ComputedColumnSql.
The declared type is authoritative and the on-disk descriptor type is its promotion — the two genuinely disagree, which is why both are stored.
It also refuses what ACE cannot read back rather than writing a file Access would reject: explicit values for a calculated column, indexes over one, dropping a column another column's expression references, and CDbl(Null). The whitelist and the refusals were measured against ACE rather than inferred, and the format work is written up in docs/format/page-02e-calculated-columns.md.
Reading other people's databases
Three independent fixes, each found by pointing LibRed at real-world files:
Stored queries. The MSysQueries Attribute=1 row is the query kind, not a marker that the query is an action query — Access writes it on plain SELECTs too. Reading it as the former made a large share of any real application's saved queries unreadable. With the kind decoded, plus SELECT * (no column rows) and the DISTINCTROW / TOP … PERCENT option bits, 479 of 592 stored queries across a 19-file corpus now come back as views, 337 of them reproducing ACE's own SQL exactly.
Unmodelled column type codes read as an opaque placeholder instead of failing the open, so one unknown column no longer costs the whole database.
Jet 4 creation. The creation path carried an ACCDB-only guard with nothing behind it; CreateDatabase now covers Jet 4 through ACE 17, verified by opening the result in ACE.
CI
CompiledModelJetTest moves to a fourth shard of its own, with shard 3 gaining the matching exclusion.
Verification
EFCore.LibRed.Extended.FunctionalTests, the broadest suite: 37,885 passed / 201 failed / 390 skipped of 38,476. The LibRed ACE cross-check suites (LibRed.Core.AccessTests 923, LibRed.Engine.AccessTests 261) and the cross-platform suites (LibRed.Engine.Tests 1,243, LibRed.Core.Tests 608) are green, and the solution builds with no warnings. Most of the 302 changed files are regenerated baselines and green-list entries rather than code.