CDN log delivery usually arrives in one of two shapes: a continuous stream of records with low delay, or files and batches delivered after events accumulate. Both can contain similar request data, but they create very different operational systems. Real-time logs optimize for fast reaction. Batch delivery optimizes for efficient, complete, and durable movement of large datasets. Choosing between them requires more than asking which is faster.

The useful question is what decision must be made, how soon it must be made, and what happens if some records are late or missing. A security control that must detect a credential-stuffing surge within minutes has different needs from a monthly cache-efficiency review. Treating every use case as real time can create unnecessary cost and complexity, while treating every use case as batch can delay incident response.

What “real time” means in practice

Real-time CDN logs are usually delivered within seconds or a small number of minutes to a stream, event bus, collector, or HTTP endpoint. They are intended to be processed continuously. Consumers parse each event, apply routing or enrichment, and write to a search platform, time-series system, security pipeline, or short-lived buffer. The path must tolerate bursts because edge traffic can change much faster than an average daily volume suggests.

Real time does not mean instantaneous or perfectly complete. Providers can sample, retry, aggregate, or omit records according to the product’s guarantees. Network disruptions can delay delivery. A consumer can fall behind. Clocks can differ. The architecture should therefore measure end-to-end lag, record counts, and parse failures rather than assuming that “streaming” automatically means fresh and correct.

How batch delivery behaves

Batch systems collect records and write them as compressed objects or periodic exports. A file may represent a time window, a size threshold, or both. Files are efficient to transfer and archive, especially in columnar formats or partitioned object storage. They are also easy to reprocess: a failed transformation can be rerun against immutable source files without asking the CDN to resend the original traffic.

The tradeoff is delay. Files may arrive several minutes or much longer after the request. Individual events are less convenient for immediate alerting, and downstream systems must handle late files and overlapping windows. Batch pipelines also need a manifest, object naming convention, or checkpoint strategy so that reprocessing does not create silent duplication.

Compare the design dimensions

Latency

Streaming wins when a human or automated system must act quickly. Examples include an incident dashboard, a sudden origin-error alert, an abuse detector, or a routing safeguard. Batch is sufficient when the decision horizon is hours or days, such as retention analysis, cost allocation, capacity modeling, or a historical investigation.

Completeness and replay

Batch exports often make completeness easier to reason about because source objects can be counted, checksummed, retained, and replayed. Streaming can also be durable, but only if the event transport, consumer offsets, dead-letter handling, and idempotent writes are designed correctly. A stream without a replay plan becomes fragile during schema changes or downstream outages.

Cost

Real-time pipelines keep infrastructure active and may write many small events into expensive indexing systems. Batch pipelines compress well and can use lower-cost object storage and scheduled compute. The largest cost difference often comes from the destination, not the CDN export itself. Indexing every header and high-cardinality field for instant search can cost far more than retaining the same data in compressed files.

Operational complexity

A reliable stream needs backpressure control, retries, checkpoints, consumer scaling, ordering expectations, duplicate handling, and observability. A batch path needs file discovery, late-arrival handling, partition management, and reprocessing controls. Neither is “simple” by default; they fail in different ways.

Use cases that favor streaming

  • Live error-rate and latency monitoring for customer-facing routes.
  • Security detections that must trigger during an active event.
  • Rapid validation after a CDN, cache, routing, or origin configuration change.
  • Operational traffic steering based on fresh delivery signals.
  • Short feedback loops for launches, live events, or high-risk migrations.

For these use cases, reduce the schema to the fields needed for the decision. A lean stream is easier to process reliably. The complete event can still travel through a slower archival path.

Use cases that favor batch

  • Long-term retention and compliance archives.
  • Large historical scans across months of traffic.
  • Cache-key analysis and content-popularity modeling.
  • Provider billing reconciliation and traffic allocation.
  • Backfills after a parser, enrichment rule, or data model changes.

Batch is especially strong when queries can be expressed over partitions such as date, provider, account, distribution, or service. Good partitioning limits how much data must be scanned and avoids turning every historical question into a full-table operation.

The hybrid pattern

Many teams should not choose one path. A hybrid architecture sends a compact operational schema to a real-time destination while writing a complete record set to object storage. The streaming path powers dashboards, detections, and alerts. The batch path is the source of truth for replay, long-term analysis, and audits. Shared event identifiers and a common schema version allow the two paths to be reconciled.

This pattern also supports progressive enrichment. Time-sensitive processing can attach only service ownership and route groups. A later batch job can perform expensive geolocation, user-agent parsing, threat-context joins, or sessionization. Separating urgent and non-urgent work keeps the real-time consumer predictable during traffic spikes.

Delivery semantics to document

Before implementation, write down whether delivery is at-most-once, at-least-once, or effectively once after deduplication. Determine whether ordering is guaranteed globally, per partition, or not at all. Define the timestamp used for freshness, the maximum tolerated delay, how retries are represented, and how the consumer detects gaps. These details influence storage keys, alert thresholds, and incident runbooks.

Idempotency is usually more practical than trying to force exactly-once behavior across every component. A deterministic event key can combine provider, distribution, request ID, and event time. Consumers can then retry safely, and batch reprocessing can avoid creating multiple copies of the same request.

Common mistakes

  • Sending every available field into a premium real-time index without a query plan.
  • Alerting on raw event counts without accounting for sampling or delivery lag.
  • Assuming the stream is complete because it is fast.
  • Building no archival path, making schema fixes and historical reprocessing impossible.
  • Ignoring burst capacity and sizing only for average requests per second.
  • Using export time as event time, which distorts latency and incident timelines.

A decision framework

Start with the response deadline. If the data must influence a decision in less time than the normal batch interval, use a stream. Next, evaluate whether incomplete or duplicated records are acceptable and what safeguards are required. Estimate destination cost using peak volume and field size, not just daily averages. Finally, decide whether the data must be replayable after code or schema changes. These answers often point to a hybrid design.

CDNLog.com’s developer schema guide and log streaming architecture page provide reference patterns for event envelopes, retries, and normalization. The goal is not to maximize speed everywhere. It is to deliver each class of edge data at the latency, durability, and cost appropriate to the decision it supports.

Related field guides