GH-50944: [C++] Replace RapidJSON with simdjson in JSON chunker by Reranko05 · Pull Request #50945 · apache/arrow

Reranko05 · 2026-08-21T13:39:30Z

Rationale for this change

This PR continues the simdjson migration by replacing the RapidJSON-based JSON boundary detection used by the JSON chunker.

The existing implementation uses RapidJSON's streaming parser to identify complete JSON values. This change replaces that logic with structural boundary detection and simdjson validation.

Changes

Replace the RapidJSON custom stream and boundary detection.
Detect complete JSON values with structural scanning.
Validate complete candidates with simdjson::dom::parser.
Preserve incomplete-value, block-boundary, and error handling behavior.
Update the affected error expectation.

Fixes: #50944

GitHub Issue: [C++] Replace RapidJSON with simdjson in JSON chunker #50944

Reranko05 · 2026-08-21T14:05:22Z

While working on this migration, @rok's earlier implementation rok#47 of the simdjson-based JSON chunker. It was very helpful. Thanks :)

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

pitrou

I don't understand why this is parsing JSON by hand?

It seems that we might be able to use simdjson::ondemand::parser::iterate_many.

Reranko05 · 2026-08-24T10:05:59Z

I don't understand why this is parsing JSON by hand?
It seems that we might be able to use simdjson::ondemand::parser::iterate_many.

I initially tried using simdjson::ondemand::parser::iterate_many, but after a few attempts I ran into boundary-handling issues with the chunker semantics. I then referred to an earlier implementation by @rok as a reference and followed that approach.

That said, I agree that parsing JSON manually here is not ideal. I'll revisit this using iterate_many and address the other review comments as well.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Reranko05 · 2026-08-25T12:46:28Z

@pitrou I tried using simdjson::ondemand::parser::iterate_many(). Most tests pass, but PropagateErrorsNonLinewiseChunker behaves differently: malformed input is split into a separate document and the error is then reported by the JSON parser, whereas the current implementation reports a chunker error.

Manual structural parsing approach preserved the existing test behavior. @rok, since your earlier implementation was helpful here, do you have any suggestions on how to preserve the current error behavior with iterate_many()?

pitrou · 2026-08-25T12:51:17Z

Most tests pass, but PropagateErrorsNonLinewiseChunker behaves differently: malformed input is split into a separate document and the error is then reported by the JSON parser, whereas the current implementation reports a chunker error.

As long as an error is reported while reading the JSON stream, I don't think we care if it's reported by the chunker or the parser.

Reranko05 · 2026-08-26T11:33:30Z

@pitrou I gave iterate_many() more that a few attempts, but I am running into semantic differences with the previous RapidJSON implementation, especially around stopping after a complete value when it is followed by a partial or malformed value. At this point, I think manually finding the boundary of the first complete object/array and then validating that slice with simdjson may be simpler and closer to the existing behavior. Do you think that would be a reasonable approach?

pitrou · 2026-08-26T11:43:13Z

@pitrou I gave iterate_many() more that a few attempts, but I am running into semantic differences with the previous RapidJSON implementation, especially around stopping after a complete value when it is followed by a partial or malformed value.

Hmm, I see. Thanks for trying anyway :-)

At this point, I think manually finding the boundary of the first complete object/array and then validating that slice with simdjson may be simpler and closer to the existing behavior. Do you think that would be a reasonable approach?

Well, as a last resort, yes. The problem:

We're writing our own JSON parser, which means we must careful test it.
We're losing performance unless we implement our own SIMD optimizations.

Thoughts @HuaHuaY @cyb70289 ?

cyb70289 · 2026-08-27T02:47:23Z

Try to understand the issue. Is it that json strings legal for rapidjson may fail on simdjson, makes future Arrow release potentially incompatible to old version?

Writing our own optimized version looks not ideal. Can we just use simdjson? It's state-of-the-art, and even with self written object delimiter, there's still incompatibility risk I'm afraid.

Reranko05 · 2026-08-27T08:18:07Z

@cyb70289 I don't think the issue is JSON compatibility between RapidJSON and simdjson. The main issue I ran into is the boundary/streaming semantics.

The previous RapidJSON implementation uses kParseStopWhenDoneFlag, so it stops as soon as the first complete top-level value is parsed, even if the remaining input contains a partial or malformed value.

With iterate_many(), I wasn't able to reproduce that behavior: errors in subsequent/incomplete data can affect whether we successfully identify the preceding complete value.

I agree that writing our own optimized JSON parser would not be ideal. The manual approach I used, and which @rok also implemented in rok#47, only scans for the boundary of the first complete object or array while respecting strings and escapes, and then lets simdjson perform the actual JSON validation. But I agree this still introduces complexity and needs careful testing.

If there is a way to use simdjson directly while preserving the old stop-after-one-value semantics, that would definitely be preferable.

pitrou · 2026-08-27T09:06:21Z

With iterate_many(), I wasn't able to reproduce that behavior: errors in subsequent/incomplete data can affect whether we successfully identify the preceding complete value.

Is that a problem? We want to keep compatibility when parsing valid JSON streams. The failure mode for an invalid JSON stream can change.

cyb70289 · 2026-08-27T09:27:33Z

A discussion about ignoring trailing garbage in simdjson. Looks there're real use cases lenient parsing can be useful.
simdjson/simdjson#2502

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

                parse_options.newlines_in_values = newlines_in_values
                read_options.block_size = 4
-                with pytest.raises(ValueError,
-                                   match="try to increase block size"):
+                with pytest.raises(ValueError):
                    self.read_bytes(data, read_options=read_options,
                                    parse_options=parse_options)


Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (3)

Previously missed (1) — in code that hasn't changed since the last review.

cpp/src/arrow/json/chunker.cc:134

FindLast() pre-validates the first non-whitespace character and errors unless it is '{' or '['. That contradicts the later logic that explicitly allows a lone '}' or ']' to appear at the start of a block (as the remainder of a straddled value). As written, a block containing only '}'/']' will be rejected before reaching that special-case handling.

    if (block_length > 0) {
      const size_t start = ConsumeWhitespace(block);
      if (start < block.size() && block[start] != '{' && block[start] != '[') {
        return Status::Invalid("JSON parse error: Invalid value");
      }

cpp/src/arrow/json/chunker.cc:62

ConsumeWholeObject() relies on simdjson::ondemand::parser::iterate_many() over the entire buffer. simdjson can reject a stream if there are invalid trailing bytes after an otherwise-complete JSON value, which prevents emitting a valid prefix (this is also reflected by the updated ChunkerTest.Errors expectations). This appears to conflict with the issue/PR goal of preserving previous boundary/error-propagation behavior; consider switching to true structural boundary scanning to find a candidate end position, then validating only that candidate with simdjson so trailing invalid data doesn’t suppress earlier complete records.

  simdjson::ondemand::parser parser;
  simdjson::ondemand::document_stream stream;

  if (parser.iterate_many(input).get(stream) != simdjson::SUCCESS) {
    return std::string_view::npos;

python/pyarrow/tests/test_json.py:153

This test no longer checks the error message when block_size is too small. The newline-delimited path still surfaces the specific "try to increase block size" hint, so it’s useful to keep asserting it for newlines_in_values=False while allowing message variance for newlines_in_values=True.

                with pytest.raises(ValueError):

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

Suppressed comments (1)

python/pyarrow/tests/test_json.py:155

test_block_sizes previously asserted the failure was specifically the “straddling … (try to increase block size?)” condition. Dropping the match= makes this test much less specific and could allow unrelated ValueErrors to satisfy it (especially since other tests in this file still validate the exact message). Consider restoring a message match to keep coverage of the intended failure mode.

                with pytest.raises(ValueError):
                    self.read_bytes(data, read_options=read_options,
                                    parse_options=parse_options)

+  // simdjson rejects the malformed stream as a whole, so no complete chunk
+  // is emitted before the trailing invalid data.
+  ASSERT_TRUE(whole);
+  ASSERT_EQ(std::string_view(*whole), "");
+  ASSERT_EQ(std::string_view(*rest), parts[0] + parts[1]);


+    simdjson::padded_string input;
+
+    if (partial.empty()) {
+      input = simdjson::padded_string(block);
+    } else if (block.empty()) {
+      input = simdjson::padded_string(partial);
+    } else {
+      simdjson::padded_string_builder builder(partial.size() + block.size());
+      builder.append(partial);
+      builder.append(block);
+      input = builder.convert();
+    }


pitrou · 2026-09-01T08:52:01Z

+    }
+
+    while (it != stream.end()) {
+      if (!ConsumeDocument(it).ok()) {


If ConsumeDocument returns an error, then a document is invalid, right? Why not return with -1 here?

I also tried returning -1 immediately when ConsumeDocument() fails, but it causes the same streaming error-propagation tests to fail. I will keep the current break behavior.

pitrou · 2026-09-01T08:54:34Z

+      if (start < block.size()) {
+        const char first_char = block[start];
+
+        // An incomplete object/array is valid here because it may continue


I still don't understand why we're doing this. It will error out in the next chunking call, right?

I removed this check, but the existing streaming tests failed, so it looks like we need to keep it.

iterate_many() can treat invalid/incomplete data as the next document when no complete document was found. This check prevents clearly invalid data from being carried into the next chunk.

pitrou · 2026-09-01T08:55:32Z

      *out_pos = -1;
    } else {
-      consumed_length += ConsumeWhitespace(block);
+      // Check the suffix after the last complete document. This is the part


Similarly, I don't understand why this is useful.

Same here, removing this check caused the streaming tests to fail

Similarly, I don't understand why this is useful.

iterate_many() doesn't give us the chunking behavior we need for trailing data here. This check ensures that non-whitespace data after the last complete document is a valid start of another JSON record, so the error is detected at the current chunk boundary.

It seems we are having this discussion again and again.

It doesn't matter when exactly the error is detected, as long as an error is detected somehow (either by the JSON chunker, or the JSON parser that runs on the chunks cut out by the JSON chunker).
The JSON chunker is not a public API, it's used to distribute work to the JSON parser(s) running in parallel threads.

github-actions Bot added Component: C++ awaiting review Awaiting review labels Aug 21, 2026

Reranko05 added CI: Extra: C++ Run extra C++ CI and removed awaiting review Awaiting review labels Aug 21, 2026

Reranko05 marked this pull request as ready for review August 21, 2026 16:33

Reranko05 requested a review from pitrou as a code owner August 21, 2026 16:33

Copilot AI lite review requested due to automatic review settings August 21, 2026 16:33

Copilot AI reviewed Aug 21, 2026

Reranko05 requested review from kou and rok August 21, 2026 16:33

github-actions Bot added the awaiting review Awaiting review label Aug 21, 2026

Replace RapidJSON with simdjson in JSON chunker

be4c2a1

Reranko05 force-pushed the gh-35460-chunker branch from e9fc9fe to be4c2a1 Compare August 24, 2026 07:52

pitrou requested changes Aug 24, 2026

View reviewed changes

Comment thread cpp/src/arrow/json/chunker.cc Outdated

Comment thread cpp/src/arrow/json/chunker.cc Outdated

Comment thread cpp/src/arrow/json/chunker.cc Outdated

Comment thread cpp/src/arrow/json/chunker.cc Outdated

Comment thread cpp/src/arrow/json/chunker.cc Outdated

github-actions Bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Aug 24, 2026

WIP: preserve chunker behavior before iterate_many

aa6e20d

Copilot AI review requested due to automatic review settings August 24, 2026 20:06

Copilot AI reviewed Aug 24, 2026

WIP: Iterate Many

a5b89a7

Reranko05 added 2 commits August 31, 2026 22:16

Add ASSERT_RAISES

1ae1290

Lint fix

4772832

Copilot AI review requested due to automatic review settings August 31, 2026 16:49

Copilot AI reviewed Aug 31, 2026

Reranko05 requested a review from pitrou August 31, 2026 17:41

kou reviewed Sep 1, 2026

View reviewed changes

Comment thread cpp/src/arrow/json/chunker.cc Outdated

Comment thread cpp/src/arrow/json/chunker.cc Outdated

Comment thread cpp/src/arrow/json/chunker.cc Outdated

github-actions Bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Sep 1, 2026

Use ARROW_RETURN_NOT_OK

c0b00da

Copilot AI review requested due to automatic review settings September 1, 2026 01:53

github-actions Bot removed the awaiting changes Awaiting changes label Sep 1, 2026

Copilot started reviewing on behalf of Reranko05 September 1, 2026 01:54 View session

github-actions Bot added the awaiting change review Awaiting change review label Sep 1, 2026

Copilot AI reviewed Sep 1, 2026

View reviewed changes

kou reviewed Sep 1, 2026

View reviewed changes

Comment thread cpp/src/arrow/json/chunker.cc Outdated

github-actions Bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Sep 1, 2026

Use ARROW_ASSIGN_OR_RAISE

5251f39

Copilot AI review requested due to automatic review settings September 1, 2026 07:36

Copilot started reviewing on behalf of Reranko05 September 1, 2026 07:37 View session

github-actions Bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Sep 1, 2026

Copilot AI reviewed Sep 1, 2026

View reviewed changes

Address Copilot Feedback

6369695

Copilot AI review requested due to automatic review settings September 1, 2026 07:51

Copilot started reviewing on behalf of Reranko05 September 1, 2026 07:53 View session

Copilot AI reviewed Sep 1, 2026

View reviewed changes

Reranko05 requested a review from kou September 1, 2026 08:01

pitrou reviewed Sep 1, 2026

View reviewed changes

Reranko05 requested a review from pitrou September 1, 2026 11:29

Uh oh!

Conversation

Reranko05 commented Aug 21, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

Changes

Uh oh!

Reranko05 commented Aug 21, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

pitrou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reranko05 commented Aug 24, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Reranko05 commented Aug 25, 2026

Uh oh!

pitrou commented Aug 25, 2026

Uh oh!

Reranko05 commented Aug 26, 2026

Uh oh!

pitrou commented Aug 26, 2026

Uh oh!

cyb70289 commented Aug 27, 2026

Uh oh!

Reranko05 commented Aug 27, 2026

Uh oh!

pitrou commented Aug 27, 2026

Uh oh!

cyb70289 commented Aug 27, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

pitrou Sep 1, 2026

Choose a reason for hiding this comment

Uh oh!

Reranko05 Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pitrou Sep 1, 2026

Choose a reason for hiding this comment

Uh oh!

Reranko05 Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pitrou Sep 1, 2026

Choose a reason for hiding this comment

Uh oh!

Reranko05 Sep 1, 2026

Choose a reason for hiding this comment

Reranko05 commented Aug 21, 2026 •

edited by github-actions Bot

Loading

Reranko05 Sep 1, 2026 •

edited

Loading

Reranko05 Sep 1, 2026 •

edited

Loading