feat: added tree-shaken bundler-friendly scripts

[jurerotar]

Continuation of #168, which was accidentally closed.

[tomayac]

Just let me know whenever you want to merge this.

We still have this settings bug where apparently no one can grant access anymore. I don't get it.

[tomayac]

I looked a bit more into it, @sqlite to my surprise is not an organization, but a regular user account. @sgbeal, do you or Richard have the keys to that? Right now, these accounts have access, but only @sqlite can actually grant new users like @jurerotar access, and, as we're at it, should convert the account into an organization, so it can properly be managed.

Collaborators:

• @sqlite: admin
• @tomayac: write
• @sgbeal: write
• @schickling: write
• @mmomtchev: write

[jurerotar]

Just let me know whenever you want to merge this.

Will do! I still need to expand the test suites to cover all of this. I also want to expand the docs, the README is about to get very long with these new exports added, so I want to clean it up a bit 😄

[sgbeal]

I looked a bit more into it, @sqlite to my surprise is not an organization, but a regular user account

Right - Richard set that up ages ago for the fossil-to-git export. He's the only one with access to it. i'm now wondering whether we should create a new account, like sqlite.org, as an organization? (Edit: or is it possible to convert the account to an organization?)

All @-mentions of sqlite end up in the sqlite support mailbox but we invariably ignore them because everyone and their dog makes commits like bump @sqlite to version X and they don't realize that they're notifying us every time that happens. i'll point drh here to get his input on it - he doesn't appear to have an account here beyond the sqlite one. (edit: will wait until i've got more info about our options.)

[tomayac]

If there's a way to convert the user account sqlite into an organization and rename it to @sqlite.org on the way, this would be ideal, as it means all links would be redirected at the GitHub level. If this isn't possible, then I'd likely keep the old name, just converted to an organization.

[jurerotar]

GitHub now also gives the option to disable PRs on a repository, so you can do that to on upstream repo to remove some more of the noise 😄

[sgbeal]

i'll point drh here to get his input on it - he doesn't appear to have an account here beyond the sqlite one. (edit: will wait until i've got more info about our options.)

drh is currently swamped with higher priorities, so i won't bother him with this just now but am flagging it in my calendar to bring up when his schedule opens up (unlikely to happen before calendar week 40).

Re. disabling PRs - we just ignore them but i'll point out this option to him.

[jurerotar]

Alrighty, I created new examples, there's now a link to each of them from the README.

I'm still unsold on the size of each script. The issue is lack of modularity. Emscripten boilerplate + OO1 API is included in every file, meaning there's a lot of repetition. I can split it up on our side, but that's fragile at best. It's fine to keep as is for now, it wouldn't break anything, but I'd love to find a solution in the future 😄

[sgbeal]

i don't know a way around that duplication. When Emscripten compiles it has to create the JS and the WASM files in pairs. The JS file embeds the "imports" which the WASM file needs, and that list of imports is unknown until Emscripten figures them out. There is no guaranty that any two builds of the library would generate the same imports (but in practice they do for the same set of C-relevant compilation flags, as distinct from JS-relevant flags).

You could hypothetically create one dummy/skeleton build, and then use sed/awk/perl to stuff the individual pieces into it, but it would be extremely fragile because any given Emscripten version can (and does) change their code generator. i've added several hacks to our build over the years to do things like strip out the Emscripten-generated bindings for the sqlite3 functions because we don't use them - they're discarded as soon as the module loads. But those hacks have broken several times by changes in Emscripten's code generator. (That's not their fault -it's mine for editing their code after it's been generated.)

From a slightly different angle...

It's hypothetically possible to create one minimal/skeleton build which includes only the pieces which the library specifically needs to bind to WASM, specifically:

• api/extern-*.js, api/pre-*.js, and api/post-*.js (16kb)
• api/sqlite3-api-prologue.c-pp.js (89kb)
• api/sqlite3-api-glue.c-pp.js (81kb)
• common/whwasmutil.js (100kb)
• jaccwabyt/jaccwabyt.js (53kb)

There's no getting around having those. The extern/pre/post are how we hook into Emscripten in the first place. The bottom two files are the "glue for the glue" - they provide the function argument/result conversions and the C-struct proxies (needed for all APIs which use non-opaque structs: VFSes and virtual tables). The prologue part is what strips out the few pieces we need from Emscripten (exports and WASM heap) and discards the rest of Emscripten immediately after the module is loaded. The glue part binds the prologue's pieces using the bottom two JS files.

That's 339kb right there, probably 60% of which is docs:

[stephan@nuc:~/f/s/lite/ext/wasm]$ wc api/sqlite3-api-prologue.js api/sqlite3-api-glue.c-pp.js common/whwasmutil.js jaccwabyt/jaccwabyt.js 
  2250  10787  89067 api/sqlite3-api-prologue.js
  1971   7376  81153 api/sqlite3-api-glue.c-pp.js
  2547  12777 101035 common/whwasmutil.js
  1198   4465  42271 jaccwabyt/jaccwabyt.js
  7966  35405 313526 total

[stephan@nuc:~/f/s/lite/ext/wasm]$ cat api/sqlite3-api-prologue.js api/sqlite3-api-glue.c-pp.js common/whwasmutil.js jaccwabyt/jaccwabyt.js  | stripccomments | wc
   4592   11115  140533

(Okay, 57% docs - pretty close.)

Then there's still the 877kb WASM file, which we cannot shrink by any appreciable amount and which grows with each release.

So it's still almost 1mb stripped/pre-gzipped and any reductions beyond that would require surgical cuts to the above files and would likely not amount, in total, to more than 10-20kb.

[jurerotar]

Sorry, my previous message was a bit misleading! What I was thinking about was figuring out a way to replace literal copy-paste of the required scripts with imports of said scripts:

import './common/whwasmutil.js';
...

This way VFS-specific scripts still keep all the required functionality, but also remain very slim themselves. We also improve caching, since core and related utils can remain cached independently of each other. That being said, this is a separate topic and would require upstream changes and may not provide enough benefit to be worth the effort, so I don't want to push it right now 😄

In any case, if there's no complaints about the /bundler/{opfs-wl|sah-pool|...} export paths and the new docs, we can merge this later today. I'm going to add another commit, adding publint and attw to make sure the distribution files are in correct shapes between releases 😄

[jurerotar]

Alrighty, I added VFS-specific type exports as well, along with attw and publint 😄