Menu

Feeding GPUs from Object Storage: A Self-Hosted S3 Pipeline for Kubernetes MLOps
πŸ“°

Feeding GPUs from Object Storage: A Self-Hosted S3 Pipeline for Kubernetes MLOps

DEV Community: kubernetesΒ·Ethan CarterΒ·17 days ago
#lFbelewJ
#devtokubernetes#devto#dev#rustfs#chart
Reading 0:00
15s threshold

Most AI training pipelines waste GPU cycles waiting on data. The fix is to serve datasets, checkpoints, and model artifacts from an S3-compatible object store that runs inside your own Kubernetes cluster. RustFS gives you that S3 tier under Apache 2.0, with no per-GB egress bill from the store and no vendor lock. This applies to any CNCF-conformant cluster, including OpenShift, EKS, GKE, or a local kind cluster.

Key Stats

Dimension RustFS

License Apache 2.0

API surface S3-compatible (HTTP REST), built in Rust

Kubernetes deployment K8s Helm Charts Available β€” install per the official Helm docs, credentials via values or an existing Secret

Credentials Docker quick start: rustfsadmin / rustfsadmin (rotate); the Helm chart rejects well-known default credentials

File access No POSIX/FUSE mount in the Feature & Status table; built-in FTPS and WebDAV (default build), SFTP (opt-in build feature), or third-party s3fs-fuse

Preview features S3 Tables (Iceberg REST), MinIO on-disk compatibility

What does…

Continue reading β€” create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.