Repository navigation
Carry a MEOS value as its memory image on the typed SQL surfaces - #177
Merged
estebanzimanyi merged 1 commit intoOct 11, 2026
Merged
Conversation
The Flink and the Spark value of a SQL type carry the value's memory image where the catalog states one for its C type, typeEncodings[C].image: the function copying a value from its image, and the function answering the image's length or, for a fixed-size structure, the bytes it is laid out in. decode copies the image into memory malloc allocates, as MEOS allocates, and hands it to the copy function (temporal_copy, set_copy, geo_copy, span_copy, ...), and encode reads the length (temporal_mem_size, geo_mem_size, ..., or the 24 bytes of a Span, the 80 of an STBox) and copies that many bytes out, through MeosSqlRuntime.fromImage and MeosSqlRuntime.image. Over the current catalog 64 of the 69 value types carry their image: every temporal, set, span, span set, box, geometry, geography, circular buffer, pose, network point, point cloud, raster and JSON value. A C type stating no image keeps its codec, as does a type the jar binds no copy function of: box2d and box3d their text, and the H3 index, quadbin and S2 cell travel by value. The user-facing forms are unchanged. A value's string rendering is the form its codec writes, the hex WKB of a temporal, the hex extended WKB of a geometry, the text of a jsonb, rendered from the decoded value through the codec's encoder, and the asText, asEWKB and asHexWKB writers and the FromText and FromHexWKB readers read and write the forms of the catalog codec. The image holds the byte order of the machine that wrote it and the layout of the MEOS version that wrote it, so a Flink checkpoint or savepoint, and a Spark value cached or shuffled, is read back by the same architecture and MEOS version; a value leaving the engine leaves through the text or WKB writers. JvmValueFormTest in the codegen module locks the choice: an image-carried value decodes through its copy function and encodes by its size function or its stated bytes on both engines, renders the form its codec writes, and a value whose C type states no image, or whose copy function the jar does not bind, keeps its codec. Witness: a JFR profile of the dashboards job of chapter 9 (Air Traffic over Europe as a Stream) over one hour of state vectors spends 1974 samples in TGeomPoint.decode, under temporal_from_wkb, called by expandSpace and stboxOverlaps for each pair of its window join. Measured over MobilityDB master b2dcdf7d84, the catalog of MEOS-API 9b337a00fe derived from the installed headers of its libmeos, the JMEOS jar built on them and MobilityFlink main e1e7d2d, its binding generated once by main's generator and once by this one: the dashboards job (its transform spelled geoTransform, the name both surfaces register it under), bounded over the same hour, ran interleaved three rounds in 499.7 s against 105.4 s, 398.4 s against 105.8 s and 420.8 s against 108.1 s, a ratio of 4.74, 3.77 and 3.89, and both wrote the same 129 close pairs in every round. In the profile of this generator's run temporal_from_wkb takes no sample, and TGeomPoint.decode takes 29 samples, MeosSqlRuntime.fromImage 38. Each value of a sampled temporal, span, span set, set, box and geometry reaches the same image from its text and from its WKB under allocation churn, so equality and hashing over the image hold. Why: a value crossing the engine between two MEOS calls is a value of the same process architecture, so it travels in the form MEOS holds it in, and the catalog states that form.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Flink and the Spark value of a SQL type carry the value's memory image where the catalog
states one for its C type, typeEncodings[C].image: the function copying a value from its image,
and the function answering the image's length or, for a fixed-size structure, the bytes it is
laid out in. decode copies the image into memory malloc allocates, as MEOS allocates, and hands it
to the copy function (temporal_copy, set_copy, geo_copy, span_copy, ...), and encode reads the
length (temporal_mem_size, geo_mem_size, ..., or the 24 bytes of a Span, the 80 of an STBox) and
copies that many bytes out, through MeosSqlRuntime.fromImage and MeosSqlRuntime.image. Over the
current catalog 64 of the 69 value types carry their image: every temporal, set, span, span set,
box, geometry, geography, circular buffer, pose, network point, point cloud, raster and JSON
value. A C type stating no image keeps its codec, as does a type the jar binds no copy function
of: box2d and box3d their text, and the H3 index, quadbin and S2 cell travel by value.
The user-facing forms are unchanged. A value's string rendering is the form its codec writes, the
hex WKB of a temporal, the hex extended WKB of a geometry, the text of a jsonb, rendered from the
decoded value through the codec's encoder, and the asText, asEWKB and asHexWKB writers and the
FromText and FromHexWKB readers read and write the forms of the catalog codec. The image holds the byte order of
the machine that wrote it and the layout of the MEOS version that wrote it, so a Flink checkpoint
or savepoint, and a Spark value cached or shuffled, is read back by the same architecture and
MEOS version; a value leaving the engine leaves through the text or WKB writers.
JvmValueFormTest in the codegen module locks the choice: an image-carried value decodes through
its copy function and encodes by its size function or its stated bytes on both engines, renders
the form its codec writes, and a value whose C type states no image, or whose copy function the
jar does not bind, keeps its codec.
Witness: a JFR profile of the dashboards job of chapter 9 (Air Traffic over Europe as a Stream)
over one hour of state vectors spends 1974 samples in TGeomPoint.decode, under
temporal_from_wkb, called by expandSpace and stboxOverlaps for each pair of its window join.
Measured over MobilityDB master b2dcdf7d84, the catalog of MEOS-API 9b337a00fe derived from the
installed headers of its libmeos, the JMEOS jar built on them and MobilityFlink main e1e7d2d, its
binding generated once by main's generator and once by this one: the dashboards job (its transform
spelled geoTransform, the name both surfaces register it under), bounded over the same hour, ran interleaved three rounds in 499.7 s against 105.4 s, 398.4 s against 105.8 s
and 420.8 s against 108.1 s, a ratio of 4.74, 3.77 and 3.89, and both wrote the same 129 close
pairs in every round. In the profile of this generator's run temporal_from_wkb takes no sample, and
TGeomPoint.decode takes 29 samples, MeosSqlRuntime.fromImage 38. Each value of a sampled
temporal, span, span set, set, box and geometry reaches the same image from its text and from its
WKB under allocation churn, so equality and hashing over the image hold.
Why: a value crossing the engine between two MEOS calls is a value of the same process
architecture, so it travels in the form MEOS holds it in, and the catalog states that form.