Update Structured Streaming page to describe Real-time Mode

[lisancao]

The Structured Streaming product page at spark.apache.org/streaming/ predates
Real-time Mode and only described micro-batch streaming. This updates the page to
cover both execution modes.

Changes:

• Describe Real-time Mode and micro-batch mode, and note that both use the same
  APIs and codebase and differ only by the trigger.
• Fix the code examples to use the real trigger APIs: Trigger.ProcessingTime("1 minute")
  for micro-batch and Trigger.RealTime("5 minutes") for Real-time Mode, with the
  update output mode that Real-time Mode requires.
• Update the latency and performance wording for the two modes.

This is the product page only. The programming guide under docs/ is built and
published with each release, so it is out of scope here.

site/streaming/index.html was regenerated with bundle exec jekyll build
(Jekyll 4.4.1) and is included.

[HyukjinKwon]

Review summary

Thanks for updating the Structured Streaming page for Real-time Mode. Both code samples now use real trigger APIs that match the 4.2.0 API reference: Trigger.ProcessingTime(String), and Trigger.RealTime(String batchDuration), whose default is 5 minutes.

The main issue is that the page repeatedly says switching to Real-time Mode needs only a trigger change (lines 17, 21, 37 and 60). Spark rejects real-time queries that don't use Update output mode, or that use sources, operators or sinks outside the Real-time Mode allowlists (STREAMING_REAL_TIME_MODE in the 4.2.0 error-condition reference). The page's own sample has to add .outputMode("update").

Two presentation issues are smaller. The 692 px incremental-execution diagram is no longer sized to the content column, so it overflows on viewports narrower than about 992 px. The download button's caption "Includes Spark Structured Streaming" has become a second link to the downloads page.

Review-level concerns

• Author input: Several new Real-time Mode claims can't be traced to Apache Spark's documentation or release notes. They are the up-to-100x speedup, the few-millisecond and roughly 300 ms P99 latency figures, the 'pipelined and streaming shuffles' capability, and 'without sacrificing exactly-once guarantees'. The release notes describe only stateless Real-time Mode support in 4.1 and PySpark trigger support in 4.2, and the documented candidate sinks are at-least-once. Before these sentences can be corrected, the author needs to confirm whether they describe released Apache Spark or a vendor runtime, and for which Spark version. One answer decides whether each claim should be cited, scoped to released behavior, or softened to the project's release-note wording. This question doesn't block landing by itself; it decides how these sentences should read.

Cross-cutting observations

• Both presentation regressions come from converting the page from raw HTML to Markdown. That conversion dropped two conventions every sibling landing page (sql, mllib, graphx, spark-connect) still uses: inline-styled responsive images and the captioned btn-cta download button. Keeping raw HTML for these two elements, as the page already does for its code samples, restores both.
  Separation: The two fixes touch different elements with different invariants and can land independently. The shared conversion cause explains where they came from, but it creates no shared owner, coordination requirement, or change in repair scope.

Findings

3 total: 0 P0, 0 P1, 1 P2, 2 P3.

Non-blocking (P2)

• Real-time Mode isn't a trigger-only switch — streaming/index.md:21 — see inline.

Nit (P3)

• Diagram images lost their responsive sizing — streaming/index.md:69 — see inline.
• Download caption became a second link — streaming/index.md:91 — see inline.

Verification

• Both new code samples use APIs documented in the Spark 4.2.0 Java API reference for org.apache.spark.sql.streaming.Trigger (site/docs/4.2.0/api/java/org/apache/spark/sql/streaming/Trigger.html): ProcessingTime(String), and RealTime(String batchDuration), described as 'A trigger for real time mode, with batch at the specified duration', with a 5-minute default.

PR description suggestions

• The description's first bullet says the two modes "differ only by the trigger", but the next bullet (and the page's Real-time Mode sample) says Real-time Mode also requires the update output mode. Consider rewording that bullet to match the corrected page text.

Decision challenges

How Real-time Mode relates to the existing Continuous Processing mode

The page says Structured Streaming has "two different modes" and describes Real-time Mode as "continuous processing" (lines 14, 62 and 67). The programming guide this page links to still documents a separate, experimental mode named Continuous Processing: Trigger.Continuous, "end-to-end latencies as low as 1 millisecond with at-least-once guarantees" (docs/4.2.0/streaming/index.html). Nothing on the site deprecates it, and the 4.1.0 release notes also call RTM "continuous, sub-second latency processing", so this is a naming and positioning decision rather than a clear error. Is Real-time Mode meant to supersede the experimental Continuous Processing trigger? If so, it may be worth saying that on the page. If not, avoiding the phrase "continuous processing" for RTM would keep readers who follow the guide link from mixing the two up, especially next to the exactly-once claim on line 67.

Is "without sacrificing exactly-once guarantees" accurate for Real-time Mode?

What end-to-end guarantee does Real-time Mode provide, and for which sinks?

What I could verify: the linked 4.2.0 guide documents exactly-once for the micro-batch engine. It documents at-least-once for Continuous Processing and for foreach/foreachBatch writes by default, and the Kafka integration guide says "Apache Kafka only supports at least once write semantics". Real-time Mode also restricts sinks to an allowlist: STREAMING_REAL_TIME_MODE.SINK_NOT_SUPPORTED points to a "Real-Time Mode User Guide" for the supported sinks, and that guide isn't published on the site.

What I couldn't verify is the actual RTM guarantee per sink. For example, it might be exactly-once state with at-least-once output. If RTM output into Kafka can duplicate records after a failure, readers building the "mission-critical" fraud-detection or alerting pipelines this page describes might skip deduplication based on this sentence. Could you confirm the guarantee and qualify the sentence to match?

Source for the Real-time Mode performance claims

Could you share the source for these figures and capabilities, and the Apache Spark version they apply to?

• "up to 100x faster than micro-batch" (lines 14 and 62)
• "as low as a few milliseconds" (line 62)
• "P99 latencies ranging from a few milliseconds to around 300 ms depending on the workload" (this line)
• "pipelined and streaming shuffles" (this line)

What I could verify: none of these appears in the Apache Spark docs or release notes on this site. The 4.1.0 release notes describe the feature as "Real-time Mode in Structured Streaming (Scala stateless support)" (SPARK-53736) and say only "For stateless tasks, latency can even drop to single-digit milliseconds". The 4.2.0 notes add PySpark trigger support (SPARK-54660). RTM operators are also gated by an allowlist (spark.sql.streaming.realTimeMode.allowlistCheck).

What I couldn't verify is whether these numbers and the shuffle capability come from an Apache Spark benchmark or SPIP, or from a vendor runtime with additional capabilities. If they don't reflect what released Apache Spark ships, consider citing the source or scoping them to 4.1/4.2 behavior, for example the release-note wording.