From 24f403c3ac0a81b88f5de41d0db92a66b9beb93d Mon Sep 17 00:00:00 2001 From: lisa-cao_data Date: Sat, 10 Oct 2026 09:39:35 +0100 Subject: [PATCH 1/2] Update Structured Streaming page to cover Real-time Mode Describe the Real-time Mode and micro-batch execution modes on the Structured Streaming product page, with corrected trigger examples (Trigger.RealTime / Trigger.ProcessingTime) and the required update output mode for Real-time Mode. --- site/streaming/index.html | 156 ++++++++++++++++++-------------------- streaming/index.md | 154 +++++++++++++++++-------------------- 2 files changed, 141 insertions(+), 169 deletions(-) diff --git a/site/streaming/index.html b/site/streaming/index.html index 00a32102cb..af94982044 100644 --- a/site/streaming/index.html +++ b/site/streaming/index.html @@ -158,102 +158,90 @@

Spark Structured Streaming

-
- Spark Structured Streaming makes it easy to build streaming applications and pipelines with the same and familiar Spark APIs. -
+

Spark Structured Streaming makes it easy to build streaming applications and pipelines with the same and familiar Spark APIs.

-
-
-

Easy to use

-

- Spark Structured Streaming abstracts away complex streaming concepts such as incremental processing, checkpointing, and watermarks - so that you can build streaming applications and pipelines without learning any new concepts or tools. -

-
-
- -
-
- spark
-   .readStream
-   .select($"value".cast("string").alias("jsonData"))
-   .select(from_json($"jsonData",jsonSchema).alias("payload"))
-   .writeStream
-   .trigger("1 seconds")
-   .start() -
-
-
-
+

Spark Structured Streaming has two different modes:

-
-
-

Unified batch and streaming APIs

-

- Spark Structured Streaming provides the same structured APIs (DataFrames and Datasets) as Spark so that you don’t need to develop on or maintain two different technology stacks for batch and streaming. - In addition, unified APIs make it easy to migrate your existing batch Spark jobs to streaming jobs. -

-
-
-
- -
+
    +
  • Spark Real-time Mode (RTM): Designed for sub-second, operational workloads. RTM provides continuous, millisecond-level processing that is up to 100x faster than micro-batch mode, eliminating the need to maintain a separate real-time engine.
    • +
  • Spark Micro-batch Mode: Designed for analytical pipelines and ETL. Micro-batch mode processes data streams as a series of small batch jobs, a highly cost-effective option for second- or minute-level latencies.
    • +
+ +

Both modes share the exact same codebase, so you can shift between cost-effective ETL and sub-second operational streaming with a single configuration change.

+ +

Easy to use

+ +

Spark Structured Streaming abstracts away complex streaming concepts such as incremental processing, checkpointing, and watermarks so you can build streaming applications without learning new tools. Shifting from minute-level ETL to millisecond-scale operational streaming requires only a change to your trigger.

+ +

Micro-batch mode — second- or minute-level latency for analytical ETL:

+ +
+
+ spark
+   .readStream
+   .select($"value".cast("string").alias("jsonData"))
+   .select(from_json($"jsonData", jsonSchema).alias("payload"))
+   .writeStream
+   .trigger(Trigger.ProcessingTime("1 minute"))
+   .start()
-
-
-

Low latency and cost effective

-

- Spark Structured Streaming uses the same underlying architecture as Spark so that you can take advantage of all the performance and cost optimizations built into the Spark engine. - With Spark Structured Streaming, you can build low latency streaming applications and pipelines cost effectively. -

-
-
-
- -
+

Real-time Mode — millisecond-scale latency for operational workloads. Switch the trigger (the output mode must be update); the rest of the query is unchanged:

+ +
+
+ spark
+   .readStream
+   .select($"value".cast("string").alias("jsonData"))
+   .select(from_json($"jsonData", jsonSchema).alias("payload"))
+   .writeStream
+   .outputMode("update")
+   .trigger(Trigger.RealTime("5 minutes"))
+   .start()
-
+

Unified batch and streaming APIs

-
-

Getting started

-

- To get started with Spark Structured Streaming: -

- -
+

Spark Structured Streaming provides the same structured APIs (DataFrames and Datasets) as Spark, so you don’t need to develop on or maintain two different technology stacks for batch and streaming. The same codebase scales from historical batch processing to real-time inference, and unified APIs make it easy to migrate existing batch Spark jobs to streaming.

-
-

Community

-

- Spark Structured Streaming is developed as part of Apache Spark. It thus gets tested and updated with each Spark release. -

-

- If you have questions about the system, ask on the - Spark mailing lists. -

-

- The Spark Structured Streaming developers welcome contributions. If you'd like to help out, - read how to - contribute to Spark, and send us a patch! -

-
+

-
+

Built for both analytical and operational workloads

-
-
- - Download Apache Spark
Includes Spark Structured Streaming - -
-
+

Run cost-effective ETL and sub-second operational applications on the same engine. Spark Structured Streaming supports both micro-batch and Real-time execution, so you can change latency targets with a single-line trigger configuration.

+ +
    +
  • Operational streaming (Spark Real-time Mode): A continuous processing architecture that executes events as they arrive. Real-time Mode is up to 100x faster than micro-batch for operational streaming workloads, dropping end-to-end latency to as low as a few milliseconds, enabling inline fraud detection, live personalization, and real-time alerting without a separate streaming engine.
    • +
  • Analytical streaming (Spark Micro-batch Mode): This model balances high throughput with cost-efficiency. Use it for traditional ETL, Medallion architectures, and ingestion pipelines where latency is measured in seconds or minutes.
    • +
+ +

Low latency and cost-effective

+ +

Spark Structured Streaming uses the same underlying architecture as Spark, so you can take advantage of all the performance and cost optimizations built into the Spark engine. While micro-batch execution provides highly cost-effective processing for analytical ETL, Real-time Mode uses continuous processing with pipelined and streaming shuffles to reach P99 latencies ranging from a few milliseconds to around 300 ms depending on the workload. Run mission-critical operational workloads with ultra-low latency without sacrificing exactly-once guarantees.

+ +

+ +

Getting started

+ +

To get started with Spark Structured Streaming:

+ + + +

Community

+ +

Spark Structured Streaming is developed as part of Apache Spark. It thus gets tested and updated with each Spark release.

+ +

If you have questions about the system, ask on the Spark mailing lists.

+ +

The Spark Structured Streaming developers welcome contributions. If you’d like to help out, read how to contribute to Spark, and send us a patch!

+ +

Download Apache Spark
+Includes Spark Structured Streaming

diff --git a/streaming/index.md b/streaming/index.md index 10e6a7f59d..bcb6b6bf90 100644 --- a/streaming/index.md +++ b/streaming/index.md @@ -7,101 +7,85 @@ subproject: Streaming --- -
- Spark Structured Streaming makes it easy to build streaming applications and pipelines with the same and familiar Spark APIs. -
+[Spark Structured Streaming]({{site.baseurl}}/docs/latest/structured-streaming-programming-guide.html) makes it easy to build streaming applications and pipelines with the same and familiar Spark APIs. +Spark Structured Streaming has two different modes: +- **Spark Real-time Mode (RTM):** Designed for sub-second, operational workloads. RTM provides continuous, millisecond-level processing that is up to 100x faster than micro-batch mode, eliminating the need to maintain a separate real-time engine. +- **Spark Micro-batch Mode:** Designed for analytical pipelines and ETL. Micro-batch mode processes data streams as a series of small batch jobs, a highly cost-effective option for second- or minute-level latencies. -
-
-

Easy to use

-

- Spark Structured Streaming abstracts away complex streaming concepts such as incremental processing, checkpointing, and watermarks - so that you can build streaming applications and pipelines without learning any new concepts or tools. -

-
-
- -
-
- spark
-   .readStream
-   .select($"value".cast("string").alias("jsonData"))
-   .select(from_json($"jsonData",jsonSchema).alias("payload"))
-   .writeStream
-   .trigger("1 seconds")
-   .start() -
-
-
-
+Both modes share the exact same codebase, so you can shift between cost-effective ETL and sub-second operational streaming with a single configuration change. -
-
-

Unified batch and streaming APIs

-

- Spark Structured Streaming provides the same structured APIs (DataFrames and Datasets) as Spark so that you don’t need to develop on or maintain two different technology stacks for batch and streaming. - In addition, unified APIs make it easy to migrate your existing batch Spark jobs to streaming jobs. -

-
-
-
- -
+## Easy to use + +Spark Structured Streaming abstracts away complex streaming concepts such as incremental processing, checkpointing, and watermarks so you can build streaming applications without learning new tools. Shifting from minute-level ETL to millisecond-scale operational streaming requires only a change to your trigger. + +**Micro-batch mode** — second- or minute-level latency for analytical ETL: + +
+
+ spark
+   .readStream
+   .select($"value".cast("string").alias("jsonData"))
+   .select(from_json($"jsonData", jsonSchema).alias("payload"))
+   .writeStream
+   .trigger(Trigger.ProcessingTime("1 minute"))
+   .start()
-
-
-

Low latency and cost effective

-

- Spark Structured Streaming uses the same underlying architecture as Spark so that you can take advantage of all the performance and cost optimizations built into the Spark engine. - With Spark Structured Streaming, you can build low latency streaming applications and pipelines cost effectively. -

-
-
-
- -
+**Real-time Mode** — millisecond-scale latency for operational workloads. Switch the trigger (the output mode must be `update`); the rest of the query is unchanged: + +
+
+ spark
+   .readStream
+   .select($"value".cast("string").alias("jsonData"))
+   .select(from_json($"jsonData", jsonSchema).alias("payload"))
+   .writeStream
+   .outputMode("update")
+   .trigger(Trigger.RealTime("5 minutes"))
+   .start()
-
- -
-

Getting started

-

- To get started with Spark Structured Streaming: -

- -
+## Unified batch and streaming APIs -
-

Community

-

- Spark Structured Streaming is developed as part of Apache Spark. It thus gets tested and updated with each Spark release. -

-

- If you have questions about the system, ask on the - Spark mailing lists. -

-

- The Spark Structured Streaming developers welcome contributions. If you'd like to help out, - read how to - contribute to Spark, and send us a patch! -

-
+Spark Structured Streaming provides the same structured APIs (DataFrames and Datasets) as Spark, so you don't need to develop on or maintain two different technology stacks for batch and streaming. The same codebase scales from historical batch processing to real-time inference, and unified APIs make it easy to migrate existing batch Spark jobs to streaming. -
+![]({{site.baseurl}}/images/spark-unified-batch-and-streaming.png) -
-
- - Download Apache Spark
Includes Spark Structured Streaming - -
-
+## Built for both analytical and operational workloads + +Run cost-effective ETL and sub-second operational applications on the same engine. Spark Structured Streaming supports both micro-batch and Real-time execution, so you can change latency targets with a single-line trigger configuration. + +- **Operational streaming (Spark Real-time Mode):** A continuous processing architecture that executes events as they arrive. Real-time Mode is up to 100x faster than micro-batch for operational streaming workloads, dropping end-to-end latency to as low as a few milliseconds, enabling inline fraud detection, live personalization, and real-time alerting without a separate streaming engine. +- **Analytical streaming (Spark Micro-batch Mode):** This model balances high throughput with cost-efficiency. Use it for traditional ETL, Medallion architectures, and ingestion pipelines where latency is measured in seconds or minutes. + +## Low latency and cost-effective + +Spark Structured Streaming uses the same underlying architecture as Spark, so you can take advantage of all the performance and cost optimizations built into the Spark engine. While micro-batch execution provides highly cost-effective processing for analytical ETL, Real-time Mode uses continuous processing with pipelined and streaming shuffles to reach P99 latencies ranging from a few milliseconds to around 300 ms depending on the workload. Run mission-critical operational workloads with ultra-low latency without sacrificing exactly-once guarantees. + +![]({{site.baseurl}}/images/spark-structured-streaming-incremental-execution.png) + + + +### Getting started + +To get started with Spark Structured Streaming: + +- [Download Spark]({{site.baseurl}}/downloads.html). It includes Structured Streaming as a module. +- Read the [Spark Structured Streaming programming guide]({{site.baseurl}}/docs/latest/structured-streaming-programming-guide.html), which includes programming models, tutorials, configurations, etc. + +### Community + +Spark Structured Streaming is developed as part of Apache Spark. It thus gets tested and updated with each Spark release. + +If you have questions about the system, ask on the [Spark mailing lists]({{site.baseurl}}/community.html#mailing-lists). + +The Spark Structured Streaming developers welcome contributions. If you'd like to help out, read [how to contribute to Spark]({{site.baseurl}}/contributing.html), and send us a patch! + + + +[Download Apache Spark]({{site.baseurl}}/downloads.html) +[Includes Spark Structured Streaming]({{site.baseurl}}/downloads.html) From 45a8ee835afb127957bb2f2da63c26008abc9243 Mon Sep 17 00:00:00 2001 From: lisancao Date: Tue, 13 Oct 2026 01:05:18 +0100 Subject: [PATCH 2/2] Address review feedback on Real-time Mode streaming page - Clarify that RTM also requires the Update output mode and an allowlisted subset of sources, operators, and sinks, rather than only a trigger change - Scope performance claims to released OSS behavior: single-digit millisecond latency for stateless tasks (4.1 release-note wording); drop the unverified 100x-vs-micro-batch, ~300 ms P99, and exactly-once statements - Restore responsive image sizing and the captioned download button that were lost in the HTML-to-Markdown conversion - Regenerate site/streaming/index.html --- site/streaming/index.html | 27 ++++++++++++++++----------- streaming/index.md | 27 ++++++++++++++++----------- 2 files changed, 32 insertions(+), 22 deletions(-) diff --git a/site/streaming/index.html b/site/streaming/index.html index af94982044..9efb495f82 100644 --- a/site/streaming/index.html +++ b/site/streaming/index.html @@ -163,15 +163,15 @@

Spark Structured Streaming

Spark Structured Streaming has two different modes:

    -
  • Spark Real-time Mode (RTM): Designed for sub-second, operational workloads. RTM provides continuous, millisecond-level processing that is up to 100x faster than micro-batch mode, eliminating the need to maintain a separate real-time engine.
    • +
  • Spark Real-time Mode (RTM): Designed for sub-second, operational workloads. RTM provides continuous processing that can bring latency for stateless tasks down to single-digit milliseconds, eliminating the need to maintain a separate real-time engine.
  • Spark Micro-batch Mode: Designed for analytical pipelines and ETL. Micro-batch mode processes data streams as a series of small batch jobs, a highly cost-effective option for second- or minute-level latencies.
-

Both modes share the exact same codebase, so you can shift between cost-effective ETL and sub-second operational streaming with a single configuration change.

+

Both modes share the exact same codebase, so you can move between cost-effective ETL and sub-second operational streaming with only small changes to your query.

Easy to use

-

Spark Structured Streaming abstracts away complex streaming concepts such as incremental processing, checkpointing, and watermarks so you can build streaming applications without learning new tools. Shifting from minute-level ETL to millisecond-scale operational streaming requires only a change to your trigger.

+

Spark Structured Streaming abstracts away complex streaming concepts such as incremental processing, checkpointing, and watermarks so you can build streaming applications without learning new tools. Shifting from minute-level ETL to millisecond-scale operational streaming is mostly a trigger change. Real-time Mode also requires the Update output mode and supports a defined subset of sources, operators, and sinks.

Micro-batch mode — second- or minute-level latency for analytical ETL:

@@ -187,7 +187,7 @@

Easy to use

-

Real-time Mode — millisecond-scale latency for operational workloads. Switch the trigger (the output mode must be update); the rest of the query is unchanged:

+

Real-time Mode — millisecond-scale latency for operational workloads. Switch the trigger and set the output mode to update. Real-time Mode supports a defined subset of sources, operators, and sinks, so queries that stay within it need no other changes:

@@ -206,22 +206,22 @@

Unified batch and streaming APIs

Spark Structured Streaming provides the same structured APIs (DataFrames and Datasets) as Spark, so you don’t need to develop on or maintain two different technology stacks for batch and streaming. The same codebase scales from historical batch processing to real-time inference, and unified APIs make it easy to migrate existing batch Spark jobs to streaming.

-

+

Unified batch and streaming APIs in Spark Structured Streaming

Built for both analytical and operational workloads

-

Run cost-effective ETL and sub-second operational applications on the same engine. Spark Structured Streaming supports both micro-batch and Real-time execution, so you can change latency targets with a single-line trigger configuration.

+

Run cost-effective ETL and sub-second operational applications on the same engine. Spark Structured Streaming supports both micro-batch and Real-time execution, so you can change latency targets largely through the trigger, within Real-time Mode’s supported set of sources, operators, and sinks.

    -
  • Operational streaming (Spark Real-time Mode): A continuous processing architecture that executes events as they arrive. Real-time Mode is up to 100x faster than micro-batch for operational streaming workloads, dropping end-to-end latency to as low as a few milliseconds, enabling inline fraud detection, live personalization, and real-time alerting without a separate streaming engine.
    • +
  • Operational streaming (Spark Real-time Mode): A continuous processing architecture that executes events as they arrive, dropping end-to-end latency for stateless tasks to the single-digit millisecond range, enabling inline fraud detection, live personalization, and real-time alerting without a separate streaming engine.
  • Analytical streaming (Spark Micro-batch Mode): This model balances high throughput with cost-efficiency. Use it for traditional ETL, Medallion architectures, and ingestion pipelines where latency is measured in seconds or minutes.

Low latency and cost-effective

-

Spark Structured Streaming uses the same underlying architecture as Spark, so you can take advantage of all the performance and cost optimizations built into the Spark engine. While micro-batch execution provides highly cost-effective processing for analytical ETL, Real-time Mode uses continuous processing with pipelined and streaming shuffles to reach P99 latencies ranging from a few milliseconds to around 300 ms depending on the workload. Run mission-critical operational workloads with ultra-low latency without sacrificing exactly-once guarantees.

+

Spark Structured Streaming uses the same underlying architecture as Spark, so you can take advantage of all the performance and cost optimizations built into the Spark engine. While micro-batch execution provides highly cost-effective processing for analytical ETL, Real-time Mode uses continuous processing with pipelined and streaming shuffles to reach single-digit millisecond latencies for stateless workloads. Run mission-critical operational workloads with ultra-low latency.

-

+

Structured Streaming incremental execution model

Getting started

@@ -240,8 +240,13 @@

Community

The Spark Structured Streaming developers welcome contributions. If you’d like to help out, read how to contribute to Spark, and send us a patch!

-

Download Apache Spark
-Includes Spark Structured Streaming

+
+
+ + Download Apache Spark
Includes Spark Structured Streaming + +
+
diff --git a/streaming/index.md b/streaming/index.md index bcb6b6bf90..44c4e74056 100644 --- a/streaming/index.md +++ b/streaming/index.md @@ -11,14 +11,14 @@ subproject: Streaming Spark Structured Streaming has two different modes: -- **Spark Real-time Mode (RTM):** Designed for sub-second, operational workloads. RTM provides continuous, millisecond-level processing that is up to 100x faster than micro-batch mode, eliminating the need to maintain a separate real-time engine. +- **Spark Real-time Mode (RTM):** Designed for sub-second, operational workloads. RTM provides continuous processing that can bring latency for stateless tasks down to single-digit milliseconds, eliminating the need to maintain a separate real-time engine. - **Spark Micro-batch Mode:** Designed for analytical pipelines and ETL. Micro-batch mode processes data streams as a series of small batch jobs, a highly cost-effective option for second- or minute-level latencies. -Both modes share the exact same codebase, so you can shift between cost-effective ETL and sub-second operational streaming with a single configuration change. +Both modes share the exact same codebase, so you can move between cost-effective ETL and sub-second operational streaming with only small changes to your query. ## Easy to use -Spark Structured Streaming abstracts away complex streaming concepts such as incremental processing, checkpointing, and watermarks so you can build streaming applications without learning new tools. Shifting from minute-level ETL to millisecond-scale operational streaming requires only a change to your trigger. +Spark Structured Streaming abstracts away complex streaming concepts such as incremental processing, checkpointing, and watermarks so you can build streaming applications without learning new tools. Shifting from minute-level ETL to millisecond-scale operational streaming is mostly a trigger change. Real-time Mode also requires the Update output mode and supports a defined subset of sources, operators, and sinks. **Micro-batch mode** — second- or minute-level latency for analytical ETL: @@ -34,7 +34,7 @@ Spark Structured Streaming abstracts away complex streaming concepts such as inc
-**Real-time Mode** — millisecond-scale latency for operational workloads. Switch the trigger (the output mode must be `update`); the rest of the query is unchanged: +**Real-time Mode** — millisecond-scale latency for operational workloads. Switch the trigger and set the output mode to `update`. Real-time Mode supports a defined subset of sources, operators, and sinks, so queries that stay within it need no other changes:
@@ -53,20 +53,20 @@ Spark Structured Streaming abstracts away complex streaming concepts such as inc Spark Structured Streaming provides the same structured APIs (DataFrames and Datasets) as Spark, so you don't need to develop on or maintain two different technology stacks for batch and streaming. The same codebase scales from historical batch processing to real-time inference, and unified APIs make it easy to migrate existing batch Spark jobs to streaming. -![]({{site.baseurl}}/images/spark-unified-batch-and-streaming.png) +Unified batch and streaming APIs in Spark Structured Streaming ## Built for both analytical and operational workloads -Run cost-effective ETL and sub-second operational applications on the same engine. Spark Structured Streaming supports both micro-batch and Real-time execution, so you can change latency targets with a single-line trigger configuration. +Run cost-effective ETL and sub-second operational applications on the same engine. Spark Structured Streaming supports both micro-batch and Real-time execution, so you can change latency targets largely through the trigger, within Real-time Mode's supported set of sources, operators, and sinks. -- **Operational streaming (Spark Real-time Mode):** A continuous processing architecture that executes events as they arrive. Real-time Mode is up to 100x faster than micro-batch for operational streaming workloads, dropping end-to-end latency to as low as a few milliseconds, enabling inline fraud detection, live personalization, and real-time alerting without a separate streaming engine. +- **Operational streaming (Spark Real-time Mode):** A continuous processing architecture that executes events as they arrive, dropping end-to-end latency for stateless tasks to the single-digit millisecond range, enabling inline fraud detection, live personalization, and real-time alerting without a separate streaming engine. - **Analytical streaming (Spark Micro-batch Mode):** This model balances high throughput with cost-efficiency. Use it for traditional ETL, Medallion architectures, and ingestion pipelines where latency is measured in seconds or minutes. ## Low latency and cost-effective -Spark Structured Streaming uses the same underlying architecture as Spark, so you can take advantage of all the performance and cost optimizations built into the Spark engine. While micro-batch execution provides highly cost-effective processing for analytical ETL, Real-time Mode uses continuous processing with pipelined and streaming shuffles to reach P99 latencies ranging from a few milliseconds to around 300 ms depending on the workload. Run mission-critical operational workloads with ultra-low latency without sacrificing exactly-once guarantees. +Spark Structured Streaming uses the same underlying architecture as Spark, so you can take advantage of all the performance and cost optimizations built into the Spark engine. While micro-batch execution provides highly cost-effective processing for analytical ETL, Real-time Mode uses continuous processing with pipelined and streaming shuffles to reach single-digit millisecond latencies for stateless workloads. Run mission-critical operational workloads with ultra-low latency. -![]({{site.baseurl}}/images/spark-structured-streaming-incremental-execution.png) +Structured Streaming incremental execution model @@ -87,5 +87,10 @@ The Spark Structured Streaming developers welcome contributions. If you'd like t -[Download Apache Spark]({{site.baseurl}}/downloads.html) -[Includes Spark Structured Streaming]({{site.baseurl}}/downloads.html) +
+
+ + Download Apache Spark
Includes Spark Structured Streaming + +
+