bucket‑as‑a‑service (BaaS) topology

At a glance#

• Purpose: Reference architecture for offering S3-compatible bucket storage as an internal service.
• Applies to: MinIO-based object storage platforms.
• Risk: None - design reference, no changes applied.
• Time: Reference article.

• At least 3–4 worker nodes dedicated to storage (plus your control plane).
• Spread across failure domains (different racks/AZs) so a single node/zone loss doesn’t take you down.
• MinIO Tenants are the unit you sell/operate; each tenant is its own MinIO cluster with its own pods, PVCs, console, and S3 endpoints. The Operator is designed for this model. (AIStor Object Store Documentation, MinIO)
• Don’t run prod as single‑node; MinIO’s docs explicitly call that out. (AIStor Object Store Documentation)

MinIO protects data with Erasure Coding (EC) across the “drives” (PVCs) you give it. More drives per pool → better efficiency & more failure tolerance. Typical prod sets are 8, 12, or 16 drives per pool. (MinIO)

Two practical patterns:

A) Direct‑attached local disks (recommended for perf)

• Use Local Persistent Volumes or MinIO DirectPV to expose node disks as PVs; MinIO handles redundancy via EC. Great performance, clean failure domains. (operator)

B) Replicated CSI (Ceph/Longhorn/etc.)

• Easier day‑2 ops and snapshots, but you pay double redundancy (CSI replication plus MinIO EC). Choose this if ops simplicity beats raw efficiency for you.

Capacity math you’ll use with customers

• Raw per tenant = servers × volumesPerServer × pvcSize
• Usable depends on EC layout. Rule of thumb:
  • 4 drives → ~50% usable (2+2)
  • 8 drives → ~75% usable (6+2)
  • 16 drives → ~87.5% usable (14+2)

MinIO requires quorum per erasure set—keep enough drives online to write/read. (MinIO, AIStor Object Store Documentation)

Expanding later: add pools to a tenant (don’t reshape existing pools). The Operator supports this growth model. (GitHub, MinIO Blog)

Goal: 100 GiB usable per customer with decent efficiency & fault‑tolerance.

Option 1 (simple): 4 pods × 1 PVC each @ 50 Gi → 200 Gi raw → ~100 Gi usable (2+2 EC).

Option 2 (nicer): 4 pods × 2 PVCs each @ 20 Gi → 160 Gi raw → ~120 Gi usable (6+2 EC), then quota the bucket to 100 Gi for safety.

Either way the Tenant CR (sketch) looks like:

apiVersion: minio.min.io/v2
kind: Tenant
metadata:
  name: cust-xyz
  namespace: cust-xyz
spec:
  requestAutoCert: true
  pools:
  - name: pool-0
    servers: 4                # 4 pods
    volumesPerServer: 2       # 8 drives total (6+2 EC typical)
    volumeClaimTemplate:
      spec:
        storageClassName: <your-storageclass>
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 20Gi      # adjust per SKU
 # keep pods apart (per-zone/host) to survive node loss
    nodeAffinity: {...}
    podAntiAffinity: {...}

The Operator/CRD supports node/pod affinity & anti‑affinity—use it to spread pods across nodes and AZs. (AIStor Object Store Documentation)

• One namespace per customer; one Tenant per namespace.
• Give each tenant its own console and S3 endpoint (the Traefik pattern you already used is perfect).
• Enforce hard quotas on customer buckets so they can’t overrun shared capacity, e.g.:

```bash mc quota set <alias>/<bucket> --size 100Gi

```

(Hard bucket quotas prevent growth past the limit.) (AIStor Object Store Documentation)

• Consider ResourceQuotas at the namespace level (K8s) to cap total requested storage.

• TLS everywhere (Operator can mint certs; you can also use your wildcard via Ingress). (AIStor Object Store Documentation)
• KMS + SSE‑S3/SSE‑KMS for encryption at rest (MinIO KES with your HSM/KMS).
• Versioning + Object Lock (WORM) where customers need immutability/retention. (MinIO, MinIO Blog)
• Tenant isolation with NetworkPolicies; per‑tenant IAM (users/groups/policies).
• Optional: integrate an IDP (Keycloak, etc.) for STS and SSO to the console.

• Design tenants to survive node loss via EC (spread pods/drives across nodes). (MinIO)
• For site‑level failures, use multi‑site (active‑active) bucket/site replication to a second cluster in another AZ/region/datacenter. Keep IDP configs aligned across sites. (MinIO, AIStor Object Store Documentation)
• Storage‑level snapshots/backups (from your CSI) if you’re on a replicated backend.

• Monitoring: scrape MinIO/tenant metrics with Prometheus + alerts; track per‑tenant capacity and quota utilization.
• Logging/audit: ship console & server logs to your SIEM.
• Upgrades: Operator‑driven rolling updates; practice in staging first.
• Growth: when a customer needs more, add a pool (keeps IO smooth). (GitHub)

Example SKUs you can sell#

• Bronze (dev/test): 50 Gi usable, 4 pods × 1 PVC @ 25 Gi (100 Gi raw), no DR.
• Silver (prod): 100 Gi usable, 4 pods × 2 PVC @ 20 Gi (160 Gi raw), cross‑node spread, daily snapshots.
• Gold (mission‑critical): 1 TiB usable, 4 pods × 4 PVC @ 85 Gi (~1.36 TiB raw ~6+2 EC), active‑active replication to second site, Object Lock.

Pricing note: always price on usable capacity but plan/budget for the raw you must provision given EC + DR overhead.

Migration path from your single‑node lab#

1. Stand up a new multi‑node cluster with the Operator.
2. Create production Tenants with the layouts above.
3. Replicate buckets from the lab into prod tenants (server‑side bucket replication) and cut DNS over. (MinIO)