Read ahead: cutting shuffle-read stalls with the Linux page cache

July 28, 2025 · Performance · 7 min read

Riffle's read-ahead path moves upcoming disk extents into the Linux page cache before the executor asks for them. The first microbenchmark cut total read time from 1.823 seconds to 0.270 seconds; later validation reported more than 15% faster shuffle reads.

85.2% lower total read time in the initial microbenchmark
90.2% lower median batch latency in the same run
>15% faster shuffle reads reported by later validation

Why reads stall

Shuffle readers usually consume a partition in ascending file-offset order. Without prefetching, each RPC can arrive before its pages are resident, leaving the request path waiting on disk I/O. The effect is most visible in cold-cache reads and in the long tail.

Issue #370 framed the optimization: use the page cache to bring the next extent into memory before it is requested, reducing long-tail read latency without changing the data format or the Apache Uniffle client protocol.

The first implementation

PR #411 added a read-ahead layer around Riffle’s local-file handler. On Linux, the layer calls posix_fadvise(..., POSIX_FADV_WILLNEED) for the next file range. The kernel can then start I/O while the application consumes the current range.

The initial implementation used a 14 MB batch. It prefetched four batches on the first read and scheduled another two batches as the reader approached the end of the loaded window. Prefetch work stayed below the existing local-file interface, so the normal read path did not need a separate storage format.

What the benchmark showed

The PR description records a cold-cache test over 1 GB with a 14 MB batch. The captured run below shows one worker reading 20 batches, with a 400 ms pause between batches; the page cache was cleared before each run.

Terminal benchmark comparing Riffle disk reads without and with read ahead. Total read time falls from 1823 milliseconds to 270 milliseconds, and median batch latency falls from 92.119 milliseconds to 9.019 milliseconds.
The original benchmark attached to PR #411. The upper run disables read ahead; the lower run enables it. Open the full-size image.
Measurement Without read ahead With read ahead Change
Total read time 1,823 ms 270 ms −85.2%
Median batch latency 92.119 ms 9.019 ms −90.2%
Maximum batch latency 109.361 ms 108.966 ms −0.4%

The total and median numbers move sharply because most reads can consume pages loaded by the previous prefetch. The maximum barely changes, which is a useful reminder: read ahead overlaps recurring sequential I/O; it does not eliminate every cold or first-touch stall.

This is a focused microbenchmark, not a promise for every Spark workload. PR #488 later reports the broader validation result separately: more than 15% improvement in shuffle-read speed for the first-phase sequential strategy.

Activating it only when it helps

Blindly prefetching every read would waste bandwidth and evict useful pages. Riffle therefore evolved the first mechanism into two activation paths.

Sequential partitions

Issue #454 proposed using client read information to distinguish sequential access. PR #456 implemented that detection and added the per-application switch:

spark.rss.riffle.readAheadEnabled=true

The server applies read ahead to applications identified as sequential instead of enabling it indiscriminately.

AQE skew partitions

Adaptive Query Execution can split a large skewed partition into non-sequential ranges. The server can no longer infer the next range from the current offset. PR #488 introduced a second strategy: the client sends a read plan containing upcoming segments.

Because these segments are smaller and more numerous, Riffle processes them in a shared, dedicated thread pool and uses a semaphore to cap concurrency. That keeps prefetch overhead from overwhelming the normal RPC path.

Sequential strategy The server predicts the next contiguous file window.
Read-plan strategy The client reports upcoming ranges for AQE skew reads.

Enabling read-plan prefetch

The read-plan path requires the Apache Uniffle client change from apache/uniffle#2603. The Spark application reports upcoming segments with:

spark.rss.riffle.readAheadEnabled=true
spark.rss.client.read.nextReadSegmentsReportEnabled=true

The configuration used in PR #488 was:

[localfile_store.read_ahead_options]
batch_size = "15M"
batch_number = 4
read_plan_enable = true
read_plan_concurrency = 1000

[urpc_config]
get_index_rpc_version = "V2"

Treat these values as a starting point. Batch size and concurrency should be tuned against disk bandwidth, page-cache capacity, partition size, and the number of simultaneous readers.

Measuring effectiveness

The follow-up work made read ahead observable rather than opaque:

  • PR #461 added hit, miss, bytes, wasted bytes, and operation-duration metrics.
  • PR #463 scoped the latency comparison to data-file reads.
  • PR #484 corrected the hit metric and added an end-to-end sequential-read latency measurement.
  • PR #493 added hit and miss metrics for read-plan tasks.

The operational question is not simply whether prefetch is enabled. Watch hit rate, hit latency versus the no-ahead baseline, and the proportion of prefetched bytes that are never consumed. A high hit rate with excessive wasted bytes can still increase disk pressure and hurt neighboring workloads.

The result

Read ahead turns a predictable access pattern into useful overlap: while Spark processes one shuffle block, Riffle asks Linux to fetch the next. Sequential detection keeps the optimization selective; client-supplied read plans extend it to AQE skew; dedicated concurrency control keeps the background work bounded.

The mechanism is intentionally small, but it removes disk wait from the hottest part of the shuffle-read path—and the original benchmark makes that effect visible.

Sources