From 25583e9838717a460a9664770a48d0a71f4f21c1 Mon Sep 17 00:00:00 2001 From: Lee Rhodes Date: Sat, 5 Sep 2026 13:38:00 -0700 Subject: [PATCH 1/3] Finish repairing the deferred KxQ rebuild, and pin the estimator constants Three independent defects in the HLL estimator state. 1. The deferred KxQ rebuild from #364 still leaves curMin/numAtCurMin merge order dependent. #512 repaired the visible damage, but not the representation the rebuild writes: check_rebuild_kxq_cur_min() stores the true minimum register value and the count at that minimum, while the rest of the HLL_8 code maintains curMin == 0 with numAtCurMin counting the zero registers. When the merged array has no zero register the rebuild leaves curMin > 0, and from then on numAtCurMin -= (curVal == 0) never fires, so the stored pair freezes and drifts away from the registers. Its value depends on when the rebuild fired, hence on merge order. Over 4800 randomized merge-order comparisons 92 differed only in these two fields; the same sweep before #364 gives 0. Estimates and bounds are unaffected because both consumers branch on curMin == 0, so this is a serialization determinism defect. Emit the canonical form instead, and add the HLL_8 / HLL-mode guard that datasketches-java has, so the rebuild can never rewrite curMin on an HLL_4 array whose nibbles are stored relative to it. 2. HLL_HIP_RSE_FACTOR and HLL_NON_HIP_RSE_FACTOR were rounded to seven digits. For lg_k > 12 the bounds use the closed form rather than the interpolation table, so this is observable: the HIP bounds differ from Java by ~4.3e-11 relative and the non-HIP factor is off by 1.7e-6 relative, skewing the bounds of every union result above lg_k 12. Use full precision literals taken from Java's Double.toString output, rather than computing sqrt(log(2.0)) at runtime, so the value cannot vary with the platform libm. 3. harmonicNumber() and getHllBitMapEstimate() called the platform std::log. getBitMapEstimate computes K * (H(K) - H(K - numHit)), whose cancellation amplifies a 1 ULP log difference by about 17x. Apple libm differs from fdlibm on 0.62% of a 400000 sample sweep. Java's StrictMath.log is specified to be fdlibm, so add common/include/fdlibm_log.hpp (FDLIBM 5.3 __ieee754_log, Sun notice preserved) and use it in both places. fdlibm needs strict IEEE evaluation, and the clang pragma is overridden by an explicit -ffp-contract=fast, so the contraction sensitive expressions go through a volatile round trip; verified bit-identical to Java under -O2, -O3, -ffp-contract=on and =fast. LICENSE gains an FDLIBM entry alongside the existing xxhash64, MurmurHash3 and bithacks entries. This changes serialized bytes, unlike the preceding PR. Union results carry the canonical curMin/numAtCurMin, linear-counting estimates shift ~1e-15, and bounds above lg_k 12 shift as described. Reading is unaffected: older images still deserialize and yield identical estimates and bounds. Adds HllKxqRebuildTest.cpp. Three of the four assertions that also compile against the parent branch fail there and pass here; the fourth already held and is kept as a guard. With the companion datasketches-java change, a 1041 record corpus spanning lg_k 4..21, all three target types, 17 sizes across LIST/SET/HLL, round trips and 80 union scenarios goes from 428 differing records to 0, comparing every serialized byte, estimate, composite estimate and bound as raw IEEE bits. Co-Authored-By: Claude Opus 5 --- LICENSE | 14 +++ common/CMakeLists.txt | 1 + common/include/fdlibm_log.hpp | 101 +++++++++++++++ hll/include/HarmonicNumbers-internal.hpp | 3 +- hll/include/HllArray-internal.hpp | 30 +++-- hll/include/HllUtil.hpp | 4 +- hll/test/CMakeLists.txt | 1 + hll/test/HllKxqRebuildTest.cpp | 150 +++++++++++++++++++++++ 8 files changed, 288 insertions(+), 16 deletions(-) create mode 100644 common/include/fdlibm_log.hpp create mode 100644 hll/test/HllKxqRebuildTest.cpp diff --git a/LICENSE b/LICENSE index 2a30395d..3965c5eb 100644 --- a/LICENSE +++ b/LICENSE @@ -296,3 +296,17 @@ APPENDIX B: Additional licenses relevant to this product. Code Locations: * common/include/ceiling_power_of_2.hpp that is adapted from the above. + + ============================================================= + FDLIBM + ============================================================= + Original source code: + Copyright (C) 1993 by Sun Microsystems, Inc. All rights reserved. + Developed at SunSoft, a Sun Microsystems, Inc. business. + + Permission to use, copy, modify, and distribute this software is freely + granted, provided that this notice is preserved. + + Code Locations: + common/include/fdlibm_log.hpp + that is adapted from __ieee754_log in FDLIBM 5.3. diff --git a/common/CMakeLists.txt b/common/CMakeLists.txt index 8514433b..bb316d09 100644 --- a/common/CMakeLists.txt +++ b/common/CMakeLists.txt @@ -40,6 +40,7 @@ install(FILES include/conditional_back_inserter.hpp include/conditional_forward.hpp include/count_zeros.hpp + include/fdlibm_log.hpp include/inv_pow2_table.hpp include/kolmogorov_smirnov_impl.hpp include/kolmogorov_smirnov.hpp diff --git a/common/include/fdlibm_log.hpp b/common/include/fdlibm_log.hpp new file mode 100644 index 00000000..86654806 --- /dev/null +++ b/common/include/fdlibm_log.hpp @@ -0,0 +1,101 @@ +// fdlibm __ieee754_log, used by Java's StrictMath.log (and by Math.log on most JVMs). +// Derived from FDLIBM 5.3, Copyright (C) 1993 by Sun Microsystems, Inc. +// "Permission to use, copy, modify, and distribute this software is freely granted, +// provided that this notice is preserved." +#ifndef FDLIBM_LOG_HPP +#define FDLIBM_LOG_HPP +#include +#include +#include + +namespace fdlibm { + +inline int32_t hi_word(double x){ uint64_t u; std::memcpy(&u,&x,8); return (int32_t)(uint32_t)(u>>32); } +inline uint32_t lo_word(double x){ uint64_t u; std::memcpy(&u,&x,8); return (uint32_t)u; } +inline void set_hi_word(double& x, uint32_t hi){ uint64_t u; std::memcpy(&u,&x,8); + u = (u & 0x00000000ffffffffULL) | ((uint64_t)hi<<32); std::memcpy(&x,&u,8); } + +// Forces a value to be rounded to a double before it is used again. fdlibm needs strict +// IEEE-754 evaluation: a fused multiply-add anywhere in the polynomial below changes the +// result. Pragmas are overridden by an explicit -ffp-contract=fast, so use a volatile +// round-trip, which the standard requires the compiler to honour. +inline double rnd(double v) { volatile double t = v; return t; } + +inline double log(double x) { +// fdlibm depends on strict IEEE-754 evaluation: a fused multiply-add would change the result +// of the polynomial evaluation below, so contraction must be off for this function. +#if defined(__clang__) +#pragma clang fp contract(off) +#endif + static const double + ln2_hi = 6.93147180369123816490e-01, /* 3fe62e42 fee00000 */ + ln2_lo = 1.90821492927058770002e-10, /* 3dea39ef 35793c76 */ + two54 = 1.80143985094819840000e+16, /* 43500000 00000000 */ + Lg1 = 6.666666666666735130e-01, /* 3FE55555 55555593 */ + Lg2 = 3.999999999940941908e-01, /* 3FD99999 9997FA04 */ + Lg3 = 2.857142874366239149e-01, /* 3FD24924 94229359 */ + Lg4 = 2.222219843214978396e-01, /* 3FCC71C5 1D8E78AF */ + Lg5 = 1.818357216161805012e-01, /* 3FC74664 96CB03DE */ + Lg6 = 1.531383769920937332e-01, /* 3FC39A09 D078C69F */ + Lg7 = 1.479819860511658591e-01, /* 3FC2F112 DF3E5244 */ + zero = 0.0; + + double hfsq,f,s,z,R,w,t1,t2,dk; + int32_t k,hx,i,j; + uint32_t lx; + + hx = hi_word(x); + lx = lo_word(x); + + k = 0; + if (hx < 0x00100000) { /* x < 2**-1022 */ + // fdlibm writes these as -two54/zero and (x-x)/zero, which also raise the divide-by-zero + // and invalid flags. MSVC rejects a compile-time division by a zero constant (C2124), so + // return the same values directly. The estimators never call log() with these inputs. + if (((hx & 0x7fffffff) | lx) == 0) { /* log(+-0) = -inf */ + return -std::numeric_limits::infinity(); + } + if (hx < 0) { /* log(-#) = NaN */ + return std::numeric_limits::quiet_NaN(); + } + k -= 54; x *= two54; /* subnormal: scale up */ + hx = hi_word(x); + } + if (hx >= 0x7ff00000) { return x+x; } + k += (hx>>20) - 1023; + hx &= 0x000fffff; + i = (hx + 0x95f64) & 0x100000; + set_hi_word(x, (uint32_t)(hx | (i ^ 0x3ff00000))); /* normalize x or x/2 */ + k += (i>>20); + f = x - 1.0; + if ((0x000fffff & (2+hx)) < 3) { /* |f| < 2**-20 */ + if (f == zero) { + if (k == 0) { return zero; } + dk = (double)k; return rnd(dk*ln2_hi) + rnd(dk*ln2_lo); + } + R = rnd(rnd(f*f)*rnd(0.5 - rnd(0.33333333333333333*f))); + if (k == 0) { return f-R; } + dk = (double)k; return rnd(dk*ln2_hi) - (rnd(R - rnd(dk*ln2_lo)) - f); + } + s = f/(2.0+f); + dk = (double)k; + z = s*s; + i = hx - 0x6147a; + w = z*z; + j = 0x6b851 - hx; + t1 = rnd(w*rnd(Lg2 + rnd(w*rnd(Lg4 + rnd(w*Lg6))))); + t2 = rnd(z*rnd(Lg1 + rnd(w*rnd(Lg3 + rnd(w*rnd(Lg5 + rnd(w*Lg7))))))); + i |= j; + R = t2 + t1; + if (i > 0) { + hfsq = rnd(0.5*f)*f; + if (k == 0) { return f - rnd(hfsq - rnd(s*(hfsq+R))); } + return rnd(dk*ln2_hi) - (rnd(hfsq - rnd(rnd(s*(hfsq+R)) + rnd(dk*ln2_lo))) - f); + } else { + if (k == 0) { return f - rnd(s*(f-R)); } + return rnd(dk*ln2_hi) - (rnd(rnd(s*(f-R)) - rnd(dk*ln2_lo)) - f); + } +} + +} // namespace fdlibm +#endif diff --git a/hll/include/HarmonicNumbers-internal.hpp b/hll/include/HarmonicNumbers-internal.hpp index 4ac1e726..b79e0988 100644 --- a/hll/include/HarmonicNumbers-internal.hpp +++ b/hll/include/HarmonicNumbers-internal.hpp @@ -21,6 +21,7 @@ #define _HARMONICNUMBERS_INTERNAL_HPP_ #include "HarmonicNumbers.hpp" +#include "fdlibm_log.hpp" #include @@ -70,7 +71,7 @@ double HarmonicNumbers::harmonicNumber(const uint64_t x_i) { } else { double x = static_cast(x_i); double invSq = 1.0 / (x * x); - double sum = log(x) + EULER_MASCHERONI_CONSTANT + (1.0 / (2.0 * x)); + double sum = fdlibm::log(x) + EULER_MASCHERONI_CONSTANT + (1.0 / (2.0 * x)); /* note: the number of terms included from this series expansion is appropriate for the size of the exact table (25) and the precision of doubles */ double pow = invSq; // now n^-2 diff --git a/hll/include/HllArray-internal.hpp b/hll/include/HllArray-internal.hpp index 8a081a2e..1da49cab 100644 --- a/hll/include/HllArray-internal.hpp +++ b/hll/include/HllArray-internal.hpp @@ -21,6 +21,7 @@ #define _HLLARRAY_INTERNAL_HPP_ #include "HllArray.hpp" +#include "fdlibm_log.hpp" #include "HllUtil.hpp" #include "HarmonicNumbers.hpp" #include "CubicInterpolation.hpp" @@ -568,7 +569,7 @@ double HllArray::getHllBitMapEstimate() const { //This will eventually go away. if (numUnhitBuckets == 0) { - return configK * log(configK / 0.5); + return configK * fdlibm::log(configK / 0.5); } const uint32_t numHitBuckets = configK - numUnhitBuckets; @@ -601,34 +602,37 @@ bool HllArray::isRebuildKxqCurminFlag() const { template void HllArray::check_rebuild_kxq_cur_min() { if (!rebuild_kxq_curmin_) { return; } + // the deferred rebuild is only ever set on an HLL_8 union gadget. Guarding here also keeps + // this from ever rewriting curMin_ on an HLL_4 array, whose nibbles are stored relative to it. + if (this->getCurMode() != hll_mode::HLL || this->getTgtHllType() != target_hll_type::HLL_8) { + rebuild_kxq_curmin_ = false; + return; + } - uint8_t cur_min = 64; - uint32_t num_at_cur_min = 0; + uint32_t num_zeros = 0; double kxq0 = 1 << this->lgConfigK_; double kxq1 = 0; - auto it = this->begin(true); // want all points to adjust cur_min + auto it = this->begin(true); // want all slots, including the empty ones const auto end = this->end(); while (it != end) { uint8_t v = HllUtil::getValue(*it); if (v > 0) { if (v < 32) { kxq0 += INVERSE_POWERS_OF_2[v] - 1.0; } else { kxq1 += INVERSE_POWERS_OF_2[v] - 1.0; } - } - if (v > cur_min) { ++it; continue; } - if (v < cur_min) { - cur_min = v; - num_at_cur_min = 1; } else { - ++num_at_cur_min; - } + ++num_zeros; + } ++it; } kxq0_ = kxq0; kxq1_ = kxq1; - curMin_ = cur_min; - numAtCurMin_ = num_at_cur_min; + // HLL_8 convention: curMin is always 0 and numAtCurMin is the number of zero registers. + // That is the representation the incremental update path maintains, so the rebuilt state is + // indistinguishable from it and the timing of this rebuild is not observable in the image. + curMin_ = 0; + numAtCurMin_ = num_zeros; rebuild_kxq_curmin_ = false; // HipAccum is not affected diff --git a/hll/include/HllUtil.hpp b/hll/include/HllUtil.hpp index 844d2824..1e207d73 100644 --- a/hll/include/HllUtil.hpp +++ b/hll/include/HllUtil.hpp @@ -87,8 +87,8 @@ static const uint32_t EMPTY = 0; static const uint8_t MIN_LOG_K = 4; static const uint8_t MAX_LOG_K = 21; -static const double HLL_HIP_RSE_FACTOR = 0.8325546; // sqrt(ln(2)) -static const double HLL_NON_HIP_RSE_FACTOR = 1.03896; // sqrt((3 * ln(2)) - 1) +static const double HLL_HIP_RSE_FACTOR = 0.8325546111576977; // sqrt(ln(2)) +static const double HLL_NON_HIP_RSE_FACTOR = 1.0389617614136892; // sqrt((3 * ln(2)) - 1) static const double COUPON_RSE_FACTOR = 0.409; // at transition point not the asymptote static const double COUPON_RSE = COUPON_RSE_FACTOR / (1 << 13); diff --git a/hll/test/CMakeLists.txt b/hll/test/CMakeLists.txt index 2bfaef05..f63d1a50 100644 --- a/hll/test/CMakeLists.txt +++ b/hll/test/CMakeLists.txt @@ -43,6 +43,7 @@ target_sources(hll_test CrossCountingTest.cpp HllArrayTest.cpp HllFullSizeTest.cpp + HllKxqRebuildTest.cpp HllSketchTest.cpp HllUnionTest.cpp TablesTest.cpp diff --git a/hll/test/HllKxqRebuildTest.cpp b/hll/test/HllKxqRebuildTest.cpp new file mode 100644 index 00000000..b5ae1c7e --- /dev/null +++ b/hll/test/HllKxqRebuildTest.cpp @@ -0,0 +1,150 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +#include +#include +#include +#include + +#include "hll.hpp" +#include "fdlibm_log.hpp" + +namespace datasketches { + +// offsets into the HLL updatable image +static const size_t CUR_MIN_BYTE = 6; +static const size_t NUM_AT_CUR_MIN_INT = 32; +static const size_t HLL_BYTE_ARR_START = 40; + +static hll_sketch make(uint8_t lg_k, target_hll_type type, uint64_t lo, uint64_t hi) { + hll_sketch sk(lg_k, type); + for (uint64_t i = lo; i < hi; ++i) sk.update(i); + return sk; +} +static uint32_t num_at_cur_min_of(const hll_sketch::vector_bytes& img) { + uint32_t v; std::memcpy(&v, img.data() + NUM_AT_CUR_MIN_INT, sizeof(v)); return v; +} + +TEST_CASE("hll kxq rebuild: union result is merge-order independent", "[hll_kxq]") { + // C stays in SET mode, so this exercises the coupon-update path into a gadget with a + // deferred rebuild pending, which is where the stored curMin/numAtCurMin used to drift + const hll_sketch a = make(12, HLL_4, 20000, 30364); + const hll_sketch b = make(10, HLL_8, 5000, 14699); + const hll_sketch c = make(17, HLL_4, 70000, 72598); + const hll_sketch* in[3] = {&a, &b, &c}; + + auto run = [&](int i, int j, int k) { + hll_union u(7); + u.update(*in[i]); u.update(*in[j]); u.update(*in[k]); + return u.get_result(HLL_8).serialize_updatable(); + }; + const auto ref = run(0, 1, 2); + REQUIRE(run(0, 2, 1) == ref); + REQUIRE(run(1, 0, 2) == ref); + REQUIRE(run(1, 2, 0) == ref); + REQUIRE(run(2, 0, 1) == ref); + REQUIRE(run(2, 1, 0) == ref); +} + +TEST_CASE("hll kxq rebuild: reading an estimate does not change a later result", + "[hll_kxq]") { + // the rebuild is lazy; when it fires must not be observable in the serialized image + const hll_sketch p = make(13, HLL_8, 0, 50000); + const hll_sketch q = make(13, HLL_8, 50000, 100000); + + for (uint8_t lg_max_k: {uint8_t(7), uint8_t(8), uint8_t(9)}) { + hll_union peeked(lg_max_k); + peeked.update(p); peeked.update(q); + (void) peeked.get_estimate(); // forces the rebuild here + for (uint64_t v = 9000000; v < 9400000; ++v) peeked.update(v); + + hll_union plain(lg_max_k); + plain.update(p); plain.update(q); + for (uint64_t v = 9000000; v < 9400000; ++v) plain.update(v); + + REQUIRE(peeked.get_result(HLL_8).serialize_updatable() + == plain.get_result(HLL_8).serialize_updatable()); + } +} + +TEST_CASE("hll kxq rebuild: stored curMin and numAtCurMin agree with the registers", + "[hll_kxq]") { + // HLL_8 convention: curMin is always 0 and numAtCurMin counts the zero registers + const hll_sketch a = make(15, HLL_8, 0, 100000); + const hll_sketch b = make(8, HLL_8, 100000, 200000); + hll_union u(8); + u.update(a); u.update(b); + const auto img = u.get_result(HLL_8).serialize_updatable(); + + uint32_t zeros = 0; + for (size_t i = HLL_BYTE_ARR_START; i < img.size(); ++i) if (img[i] == 0) ++zeros; + + REQUIRE(img[CUR_MIN_BYTE] == 0); + REQUIRE(num_at_cur_min_of(img) == zeros); +} + +TEST_CASE("hll kxq rebuild: relative error constants are full precision", "[hll_kxq]") { + // lg_k > 12 uses the closed form rather than the interpolation table, so a constant + // truncated to seven digits is directly observable in the bounds + const double hip = std::sqrt(std::log(2.0)); // sqrt(ln 2) + const double non_hip = std::sqrt((3.0 * std::log(2.0)) - 1.0); // sqrt(3 ln 2 - 1) + + for (uint8_t lg_k: {uint8_t(13), uint8_t(16), uint8_t(21)}) { + const double k = static_cast(1 << lg_k); + for (uint8_t sd = 1; sd <= 3; ++sd) { + const double got_hip = hll_union::get_rel_err(false, false, lg_k, sd); + const double got_non = hll_union::get_rel_err(false, true, lg_k, sd); + REQUIRE(got_hip == Approx(sd * hip / std::sqrt(k)).epsilon(1e-15)); + REQUIRE(got_non == Approx(sd * non_hip / std::sqrt(k)).epsilon(1e-15)); + } + } +} + +TEST_CASE("hll kxq rebuild: log matches the fdlibm reference", "[hll_kxq]") { + // the linear counting estimator subtracts two nearby harmonic numbers, which amplifies a + // 1 ULP difference in log() by more than an order of magnitude. Pin log() to fdlibm, the + // function datasketches-java's StrictMath.log is specified to be. These expected values + // were taken from Java; the platform libm differs from every one of them. + struct { int x; uint64_t bits; } expected[] = { + { 48, 0x400ef8383c50bb74ULL}, + { 74, 0x4011375cd6fcab1cULL}, + { 185, 0x4014e1a4f518c72cULL}, + { 196, 0x40151cca16d7bba8ULL}, + { 299, 0x4016cd411481a020ULL}, + { 308, 0x4016eb9f470ac0b8ULL}, + { 334, 0x40173e9bbe951e9cULL}, + { 343, 0x401759d602a5c3c2ULL}, + { 1261, 0x401c8f031e7e1220ULL}, + }; + for (const auto& e: expected) { + const double got = fdlibm::log(static_cast(e.x)); + uint64_t bits; std::memcpy(&bits, &got, sizeof(bits)); + REQUIRE(bits == e.bits); + } + + // edge cases: fdlibm produces these by dividing by a zero constant, which MSVC rejects at + // compile time, so they are returned directly. Pin the values. + REQUIRE(fdlibm::log(0.0) == -std::numeric_limits::infinity()); + REQUIRE(fdlibm::log(-0.0) == -std::numeric_limits::infinity()); + REQUIRE(std::isnan(fdlibm::log(-1.0))); + REQUIRE(fdlibm::log(1.0) == 0.0); + REQUIRE(fdlibm::log(5e-320) == Approx(-735.2178).epsilon(1e-6)); // subnormal scaling path +} + +} /* namespace datasketches */ From 0d954d6fa2db044ffbfcd067b8bf57606c9a0b63 Mon Sep 17 00:00:00 2001 From: Lee Rhodes Date: Wed, 9 Sep 2026 11:59:29 -0700 Subject: [PATCH 2/3] Exclude fdlibm_log.hpp from RAT and date the deprecated alias Two follow-ups now that #523 (Apache RAT) and #521 have merged. RAT flags common/include/fdlibm_log.hpp, which carries the Sun FDLIBM notice rather than the ASF header, so add it to .rat-excludes. It is already recorded in LICENSE, which is the other half of what the check requires for third-party source. Record when the FULL_SIZE_FLAG_MASK alias was added, so a future reader can decide when it is safe to drop without having to reconstruct the history. A date and PR number are facts; a predicted release number is not, since the release following 5.2.0 has not been numbered yet. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01BKM9nrFVMhH2gmFaag2JuC --- .rat-excludes | 1 + hll/include/HllUtil.hpp | 3 ++- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/.rat-excludes b/.rat-excludes index 5762f0af..3c5bcd39 100644 --- a/.rat-excludes +++ b/.rat-excludes @@ -7,6 +7,7 @@ # the matching entry to LICENSE in the same commit. MurmurHash3\.h xxhash64\.h +fdlibm_log\.hpp # # Dotfiles and repository metadata with no comment convention. \..* diff --git a/hll/include/HllUtil.hpp b/hll/include/HllUtil.hpp index 21ff2ee3..5a9e0069 100644 --- a/hll/include/HllUtil.hpp +++ b/hll/include/HllUtil.hpp @@ -50,7 +50,8 @@ static const uint8_t OUT_OF_ORDER_FLAG_MASK = 16; // are free. static const uint8_t RESERVED_FLAG_MASK_32 = 32; // Deprecated alias for the bit above, retained for source compatibility. Do not use. -// (No [[deprecated]] attribute: this library targets C++11, where it is unavailable.) +// Added 2026-09-07 (#521), first shipping in the release after 5.2.0. No [[deprecated]]: +// this library targets C++11, so the alias warns no one. static const uint8_t FULL_SIZE_FLAG_MASK = RESERVED_FLAG_MASK_32; static const uint32_t PREAMBLE_INTS_BYTE = 0; From 27e096a4450499c6f3d0f48434c721703810abd0 Mon Sep 17 00:00:00 2001 From: Lee Rhodes Date: Wed, 9 Sep 2026 15:22:50 -0700 Subject: [PATCH 3/3] Treat .sk sketch fixtures as binary, not text .gitattributes declared "*.sk text eol=lf", which subjects serialized sketch images to line-ending conversion. A .sk file is arbitrary bytes, so any stray CRLF pair in one gets rewritten to a lone LF on check-in, silently truncating the fixture by a byte and corrupting it. Two of the fifteen tracked fixtures contain a CRLF pair today: theta/test/theta_compact_estimation_from_java_v1.sk theta/test/theta_compact_estimation_from_java_v2.sk Neither is damaged in the repository, but git reports both as modified in any fresh worktree, which blocks merges and rebases until worked around, and a routine "git add" would have committed the truncation. The remaining thirteen are safe only because they happen to contain no CRLF pair; any regenerated sketch could acquire one. Move *.sk to the block that already covers genuinely binary files and record the reason. .gitattributes is export-ignore'd, so it is not part of a source release and this cannot affect released artifacts. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01BKM9nrFVMhH2gmFaag2JuC --- .gitattributes | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/.gitattributes b/.gitattributes index 43b37dbc..3739a952 100644 --- a/.gitattributes +++ b/.gitattributes @@ -11,7 +11,6 @@ NOTICE text eol=lf *.html text eol=lf *.hpp text eol=lf *.cpp text eol=lf -*.sk text eol=lf *.md text eol=lf *.properties text eol=lf *.sh text eol=lf @@ -27,6 +26,10 @@ NOTICE text eol=lf *.cmd text eol=crlf # Explicitly denote all files that are truly binary and should not be modified. +# .sk files are serialized sketch images: arbitrary bytes that must never be +# line-ending converted. A stray CRLF pair would be rewritten to LF on check-in, +# silently truncating the fixture by one byte. +*.sk binary *.jpg binary *.png binary *.svg binary