Skip to content

Carry a MEOS value as its memory image on the typed SQL surfaces - #177

Merged
estebanzimanyi merged 1 commit into
MobilityDB:mainfrom
estebanzimanyi:codegen/meos-image-form
Oct 11, 2026
Merged

estebanzimanyi merged 1 commit into
MobilityDB:mainfrom
estebanzimanyi:codegen/meos-image-form

Conversation

@estebanzimanyi

Copy link
Copy Markdown
Member

The Flink and the Spark value of a SQL type carry the value's memory image where the catalog
states one for its C type, typeEncodings[C].image: the function copying a value from its image,
and the function answering the image's length or, for a fixed-size structure, the bytes it is
laid out in. decode copies the image into memory malloc allocates, as MEOS allocates, and hands it
to the copy function (temporal_copy, set_copy, geo_copy, span_copy, ...), and encode reads the
length (temporal_mem_size, geo_mem_size, ..., or the 24 bytes of a Span, the 80 of an STBox) and
copies that many bytes out, through MeosSqlRuntime.fromImage and MeosSqlRuntime.image. Over the
current catalog 64 of the 69 value types carry their image: every temporal, set, span, span set,
box, geometry, geography, circular buffer, pose, network point, point cloud, raster and JSON
value. A C type stating no image keeps its codec, as does a type the jar binds no copy function
of: box2d and box3d their text, and the H3 index, quadbin and S2 cell travel by value.

The user-facing forms are unchanged. A value's string rendering is the form its codec writes, the
hex WKB of a temporal, the hex extended WKB of a geometry, the text of a jsonb, rendered from the
decoded value through the codec's encoder, and the asText, asEWKB and asHexWKB writers and the
FromText and FromHexWKB readers read and write the forms of the catalog codec. The image holds the byte order of
the machine that wrote it and the layout of the MEOS version that wrote it, so a Flink checkpoint
or savepoint, and a Spark value cached or shuffled, is read back by the same architecture and
MEOS version; a value leaving the engine leaves through the text or WKB writers.

JvmValueFormTest in the codegen module locks the choice: an image-carried value decodes through
its copy function and encodes by its size function or its stated bytes on both engines, renders
the form its codec writes, and a value whose C type states no image, or whose copy function the
jar does not bind, keeps its codec.

Witness: a JFR profile of the dashboards job of chapter 9 (Air Traffic over Europe as a Stream)
over one hour of state vectors spends 1974 samples in TGeomPoint.decode, under
temporal_from_wkb, called by expandSpace and stboxOverlaps for each pair of its window join.

Measured over MobilityDB master b2dcdf7d84, the catalog of MEOS-API 9b337a00fe derived from the
installed headers of its libmeos, the JMEOS jar built on them and MobilityFlink main e1e7d2d, its
binding generated once by main's generator and once by this one: the dashboards job (its transform
spelled geoTransform, the name both surfaces register it under), bounded over the same hour, ran interleaved three rounds in 499.7 s against 105.4 s, 398.4 s against 105.8 s
and 420.8 s against 108.1 s, a ratio of 4.74, 3.77 and 3.89, and both wrote the same 129 close
pairs in every round. In the profile of this generator's run temporal_from_wkb takes no sample, and
TGeomPoint.decode takes 29 samples, MeosSqlRuntime.fromImage 38. Each value of a sampled
temporal, span, span set, set, box and geometry reaches the same image from its text and from its
WKB under allocation churn, so equality and hashing over the image hold.

Why: a value crossing the engine between two MEOS calls is a value of the same process
architecture, so it travels in the form MEOS holds it in, and the catalog states that form.

The Flink and the Spark value of a SQL type carry the value's memory image where the catalog
states one for its C type, typeEncodings[C].image: the function copying a value from its image,
and the function answering the image's length or, for a fixed-size structure, the bytes it is
laid out in. decode copies the image into memory malloc allocates, as MEOS allocates, and hands it
to the copy function (temporal_copy, set_copy, geo_copy, span_copy, ...), and encode reads the
length (temporal_mem_size, geo_mem_size, ..., or the 24 bytes of a Span, the 80 of an STBox) and
copies that many bytes out, through MeosSqlRuntime.fromImage and MeosSqlRuntime.image. Over the
current catalog 64 of the 69 value types carry their image: every temporal, set, span, span set,
box, geometry, geography, circular buffer, pose, network point, point cloud, raster and JSON
value. A C type stating no image keeps its codec, as does a type the jar binds no copy function
of: box2d and box3d their text, and the H3 index, quadbin and S2 cell travel by value.

The user-facing forms are unchanged. A value's string rendering is the form its codec writes, the
hex WKB of a temporal, the hex extended WKB of a geometry, the text of a jsonb, rendered from the
decoded value through the codec's encoder, and the asText, asEWKB and asHexWKB writers and the
FromText and FromHexWKB readers read and write the forms of the catalog codec. The image holds the byte order of
the machine that wrote it and the layout of the MEOS version that wrote it, so a Flink checkpoint
or savepoint, and a Spark value cached or shuffled, is read back by the same architecture and
MEOS version; a value leaving the engine leaves through the text or WKB writers.

JvmValueFormTest in the codegen module locks the choice: an image-carried value decodes through
its copy function and encodes by its size function or its stated bytes on both engines, renders
the form its codec writes, and a value whose C type states no image, or whose copy function the
jar does not bind, keeps its codec.

Witness: a JFR profile of the dashboards job of chapter 9 (Air Traffic over Europe as a Stream)
over one hour of state vectors spends 1974 samples in TGeomPoint.decode, under
temporal_from_wkb, called by expandSpace and stboxOverlaps for each pair of its window join.

Measured over MobilityDB master b2dcdf7d84, the catalog of MEOS-API 9b337a00fe derived from the
installed headers of its libmeos, the JMEOS jar built on them and MobilityFlink main e1e7d2d, its
binding generated once by main's generator and once by this one: the dashboards job (its transform
spelled geoTransform, the name both surfaces register it under), bounded over the same hour, ran interleaved three rounds in 499.7 s against 105.4 s, 398.4 s against 105.8 s
and 420.8 s against 108.1 s, a ratio of 4.74, 3.77 and 3.89, and both wrote the same 129 close
pairs in every round. In the profile of this generator's run temporal_from_wkb takes no sample, and
TGeomPoint.decode takes 29 samples, MeosSqlRuntime.fromImage 38. Each value of a sampled
temporal, span, span set, set, box and geometry reaches the same image from its text and from its
WKB under allocation churn, so equality and hashing over the image hold.

Why: a value crossing the engine between two MEOS calls is a value of the same process
architecture, so it travels in the form MEOS holds it in, and the catalog states that form.
@estebanzimanyi
estebanzimanyi merged commit 67294a8 into MobilityDB:main Oct 11, 2026
2 checks passed
@estebanzimanyi
estebanzimanyi deleted the codegen/meos-image-form branch October 11, 2026 23:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant