STORAGE COST OPTIMIZATION
Why does accessing data you already own keep generating new charges?
Cloud object storage is priced for capacity. It bills for access.
S3 storage costs per gigabyte are low. S3 API call costs, egress fees, and data transfer charges are not, and they compound every time a compute node fetches a dataset it already fetched in the last training run, the last batch job, the last pipeline execution. For AI clusters, HPC workloads, and data-intensive analytics pipelines, the access pattern is inherently repetitive. The billing model treats every repetition as a new event.
Why S3 access costs compound at scale
Three patterns turn routine compute workloads into runaway storage bills:
Repeated reads across parallel compute nodes
When thousands of parallel GPU threads or HPC nodes execute simultaneously, each queries the S3 bucket independently. The parallelism that makes the compute efficient makes the storage billing expensive.
High-frequency training runs re-fetch identical datasets
Each training epoch pulls the same data as the last. Each experiment re-fetches the baseline dataset. Without a caching layer, every run pays the full access cost regardless of how recently the same data was read.
API transaction costs accumulate independently of data volume
S3 charges per GET request as well as per gigabyte transferred. Large numbers of small reads generate transaction costs that scale with request volume rather than data size.
What uncached object storage costs your teams
The impact lands differently depending on where you sit:
AI infrastructure leader
Compute budgets sized for active training are burning on idle time. A local caching layer that keeps GPUs fed reduces effective cost per training run without purchasing additional compute.
HPC and SRE lead
Standard S3 client mounts that lack peer-to-peer caching logic force every compute node to query the bucket independently. Research teams work around the problem by manually partitioning and staging datasets onto local scratch disks.
FinOps practitioner
Hyperscaler API transaction costs and egress fees expand faster than storage capacity because they scale with access frequency rather than data volume. A dataset that grows by 10% can generate 10x the API costs if the number of concurrent readers increases.
Why standard storage approaches don't control costs
Challenge 1
Parallel filesystems force expensive replication
Platforms like GPFS or Lustre require specialised network fabrics and permanent data duplication across regions, creating storage tier costs that offset any egress savings and introducing data consistency risks across distributed environments.
Challenge 2
Premium cloud storage tiers don't scale affordably
Fully managed cloud filesystems carry per-gigabyte monthly pricing that becomes cost-prohibitive at multi-petabyte scale or under high-frequency access patterns.
Challenge 3
Basic client-side mounts multiply API calls
Standard mounting utilities without horizontal cache peering query the S3 bucket independently on every node. Thousands of parallel compute instances generate thousands of redundant API calls and egress charges for identical data.
Challenge 4
Cloud-native accelerators create zone lock-in
Solutions like S3 Express One Zone reduce latency within a single availability zone but offer no hybrid-cloud capability, no cross-region support, and no native POSIX semantics for ML frameworks.
"Our purpose-built storage could not keep our GPUs fed. Varnish AI Accelerator fixed that bottleneck, and cut our storage bill while doing it."
Global high-frequency trading firm
Two pathways to storage cost optimization
Two pathways to storage cost optimization depending on your workload requirements, both built on the core Varnish caching engine:
Varnish AI Accelerator
Custom-engineered for AI/ML training, GPU farms, autonomous vehicle data pipelines, genomics, and HPC workloads. Mounts cloud buckets directly to compute nodes as read-only local directories via high-performance FUSE mounts, feeding data-hungry GPUs natively without pipeline modification.
01
Regional S3 cache shield |
A dedicated caching layer positioned adjacent to compute clusters intercepts inbound read queries, serving hot data locally and blocking redundant outbound calls to the primary bucket. API charges and egress fees stop accumulating on data the cluster has already fetched. |
02
Object storage mapped to a local POSIX filesystem |
The cache mounts directly onto compute instances as a read-only local directory. S3 objects are translated to standard POSIX files on the fly — PyTorch, TensorFlow, and custom Python pipelines run entirely unchanged. |
03
Version-pinned consistency across distributed clusters |
Cache lookups bind to unique S3 object version IDs, ensuring training runs ingest consistent, immutable data across all nodes without redundant credential-checking round trips to cloud IAM. |
04
Byte-range segmentation for large objects |
Massive multi-gigabyte objects are automatically segmented into byte-range chunks, fetched and stored in parallel. Only the requested parts of a file consume local cache space; memory buffers never get choked by a single large asset. |
Varnish Enterprise
Software-defined S3 shield for general enterprise workloads. Best for big-data analytics, VOD platforms, and large cloud-backed data lakes that need a high-density, software-defined origin shield without the AI/ML-specific tooling. Deploys on commodity bare metal, cloud VMs, or private infrastructure.
The numbers
API transaction costs stop scaling with access frequency. Egress fees stop compounding across parallel compute nodes.
99.8%
<0.5ms
3x
Varnish AI Accelerator and Varnish Enterprise
Two deployment models, one principle: serve data from local cache and stop paying the hyperscaler for reads you've already made.
Varnish AI Accelerator
|
Custom-engineered specifically for AI/ML training, GPU farms, autonomous vehicle data pipelines, genomics, and HPC workloads. It mounts cloud buckets directly to compute nodes as read-only local directories via high-performance FUSE mounts to feed data-hungry GPUs natively. |
Varnish EnterpriseS3 optimization
|
Acts as a high-density, software-defined origin shield for general enterprise workflows, big-data analytics, VOD platforms, and massive cloud-backed data lakes. |
Resources and media
-
The secret to reducing cloud storage fees while improving performance
With Varnish we can build an S3 shield: a caching layer that will preserve the benefits of cloud storage while also solving the locality problem.
-
Overcoming the challenges of scaling object & cloud storage
Watch the webinar to learn how to maximize cloud performance and reduce costs using cutting-edge techniques, including S3 shielding.
-
Improve performance and reduce costs by shielding Your S3 buckets with Varnish
Join Varnish Technical Evangelist, Thijs Feryn, to learn about S3, why you should shield your S3 buckets, and how to get started.
Next steps
Talk to our team to learn more about Varnish Virtual Registry or try it for yourself for free today.
