Google Cloud Storage Fuse

Performance & Cost Diagnosis

Telemetry-driven root-causing for "training through the mount is slow" and "the Cloud Storage bill spiked after we mounted the bucket". The core insight: the default mount is coherency-tuned, not performance-tuned — most regressions on untuned mounts are the defaults doing exactly what they promise.

Never hypothesize before reading telemetry. If client metrics are not enabled, enabling them is the first remediation step; while waiting for a remount window, the server-side storage.googleapis.com/api/request_count metric is always available and needs no mount change.

The coherency-tuned defaults (the usual suspects)

A ~5M-small-file dataset re-read each epoch against these defaults produces exactly the classic symptom pair: throughput far below local disk, and an operations-dominated bill.

Step 1 — Establish telemetry

# Self-managed: remount with metrics, reproduce the workload, then inspect
gcsfuse --prometheus-port=9920 BUCKET /mnt/data
curl -s localhost:9920/metrics | grep -E "file_cache_read_count|gcs_request_count|fs_ops_count"

Client metrics only accumulate from mount time — reproduce at least one epoch (or a representative slice) after enabling before reading the numbers.

Also capture the effective mount configuration (the misconfig surface): the running gcsfuse command line and --config-file contents on self-managed hosts; on GKE, kubectl get pv PV_NAME -o yaml (check spec.mountOptions and spec.csi.volumeAttributes) plus the pod's gke-gcsfuse/* annotations.

Step 2 — Read the signals

On GKE the same client metrics appear prefixed and underscored: gcsfusecsi/file_cache_read_count, gcsfusecsi/gcs_request_count, gcsfusecsi/fs_ops_count, etc.

Step 3 — Map the signature to the misconfiguration

Attribution logic for the bill: no file cache → every epoch re-reads all objects (objects.get, Class B, billed per operation); expiring metadata + evicting stat cache → re-stats (objects.get metadata, Class B); list cache off / implicit-dirs → repeated objects.list (Class A, ~12.5× the per-op price of Class B on standard buckets). Deletes are free.

Step 4 — Apply the fix

# tuned.yaml - read-heavy, immutable dataset re-read across epochs
cache-dir: /mnt/local-ssd/gcsfuse-cache
file-cache:
  max-size-mb: -1              # or size to the dataset
metadata-cache:
  ttl-secs: -1                 # immutable data only
  stat-cache-max-size-mb: -1   # ~1.5 GiB RAM per 1M files
file-system:
  kernel-list-cache-ttl-secs: -1   # read-only mounts only
# only-dir: dataset/train      # optional: scope namespace on 10M+ object buckets

gcsfuse --config-file=tuned.yaml BUCKET /mnt/data

[!WARNING]

Staleness trade-off: -1 TTLs mean this mount never sees changes made to the bucket by other clients. Apply only to data that is immutable for the life of the mount, and prefer mounting read-only (-o ro). If the bucket is mutable, size TTLs to the acceptable staleness window instead.

Config hygiene while editing: metadata-cache: type-cache-max-size-mb is a no-op since v3.8.0 (type cache merged into stat cache) — delete it from any config that still sets it. On GKE, apply the same keys as PV mountOptions entries (file-cache:max-size-mb:... colon syntax) — see GKE Training Deployment. On Rapid (zonal) buckets the file cache and buffered reads are no-ops by default (the kernel read path replaces them) — this runbook's cache fixes apply to standard buckets.

Step 5 — Verify and state the expected billing delta

Re-run one epoch, then check:

• file_cache_read_count{cache_hit="true"} ≈ 100% of reads from epoch 2 onward (first epoch populates the cache).

• gcs/request_count drops to roughly one GET per object per mount lifetime (plus the initial listing) instead of per epoch.

• On the bill: the Class B line (objects.get — data reads and metadata stats) drops roughly in proportion to the former epoch count; the Class A line (objects.list) drops if list caching or HNS addressed a list-heavy signature. Confirm server-side with api/request_count by method before/after.

Documentation

• Cloud Storage FUSE metrics

• Config file reference

• File caching

• Performance tuning

• Cloud Storage pricing (operation classes)

• Google Cloud metrics list (api/request_count)