Deploy Scalable Loki Logging on Amazon EKS with S3 Storage

This guide describes the current design boundary for running Loki on Amazon EKS with S3 object storage. It is a deployment foundation, not a universal production sizing template: ingestion volume, retention, query concurrency, availability targets, and failure tolerance determine the final topology and resource requests.

For a new Loki installation, use the TSDB index with schema v13. Grafana recommends TSDB for Loki 2.8 and newer; the older BoltDB shipper pattern should be treated as migration context rather than the default for a new deployment.

Architecture

The core data path is:

Kubernetes workloads
    -> Grafana Alloy or another supported collector
    -> Loki gateway/write path
    -> TSDB index and compressed chunks in Amazon S3
    -> Loki read path
    -> Grafana queries

Grafana recommends Alloy as the primary collector for new Loki pipelines. Promtail reached end of life, so a new deployment should not introduce it as a long-lived dependency.

Decisions to make before installation

Define these values before writing Helm configuration:

• expected log bytes and active streams per second
• retention period and deletion requirements
• acceptable loss window during component or Availability Zone failure
• peak query concurrency and time range
• single-tenant or multi-tenant authentication boundary
• encryption, bucket policy, and data residency requirements
• whether the cluster needs monolithic or distributed mode

The Loki Helm chart supports monolithic, simple scalable, and distributed deployment modes. Grafana documents simple scalable mode as deprecated ahead of Loki 4.0. For production at scale, its current chart guidance points to distributed mode; smaller installations should still choose based on measured workload rather than copying a large topology by default.

Create dedicated S3 buckets

Use globally unique names. Do not use the chart's generic default bucket names. Open-source Loki normally needs a chunks bucket and a ruler bucket; the admin bucket is associated with Grafana Enterprise Logs rather than a typical open-source deployment.

Apply the controls required by your environment:

• block public access
• enable server-side encryption
• restrict bucket policy to the Loki workload role
• record access and configuration changes
• use lifecycle rules only as an additional safeguard, not as a replacement for Loki retention

Loki needs bucket-level list access and object-level read/write access. Retention-driven deletion also requires s3:DeleteObject.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ListLokiBuckets",
      "Effect": "Allow",
      "Action": ["s3:ListBucket"],
      "Resource": [
        "arn:aws:s3:::example-loki-chunks",
        "arn:aws:s3:::example-loki-ruler"
      ]
    },
    {
      "Sid": "ManageLokiObjects",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Resource": [
        "arn:aws:s3:::example-loki-chunks/*",
        "arn:aws:s3:::example-loki-ruler/*"
      ]
    }
  ]
}

If the buckets use a customer-managed KMS key, add only the required KMS permissions to the workload role and allow that role in the key policy.

Use EKS workload identity

Do not put static AWS access keys in Helm values. Bind the Loki Kubernetes service account to a scoped IAM role using EKS Pod Identity or IRSA, depending on the cluster standard.

For IRSA, the role trust policy must restrict both the audience and service-account subject. The Helm values then annotate the exact service account used by Loki:

serviceAccount:
  create: true
  name: loki
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/loki-s3

Confirm the rendered chart uses this service account for every component that accesses S3. Do not assume the name without inspecting the chart version you plan to deploy.

Configure TSDB and S3

The following fragment shows the important storage choices for a new installation. It deliberately omits topology-specific replicas and resources because those must be derived from the selected chart version and workload measurements.

deploymentMode: Distributed

loki:
  schemaConfig:
    configs:
      - from: "2026-01-01"
        store: tsdb
        object_store: s3
        schema: v13
        index:
          prefix: loki_index_
          period: 24h

  storage:
    type: s3
    bucketNames:
      chunks: example-loki-chunks
      ruler: example-loki-ruler
    s3:
      region: ap-southeast-3
      s3ForcePathStyle: false

  limits_config:
    retention_period: 672h

  compactor:
    retention_enabled: true
    delete_request_store: s3

For a brand-new deployment, the schema from date must be in the past so a schema is active when Loki starts. Do not copy that rule into a schema migration. When adding a new schema to a deployment that already contains data, Grafana requires a future cutover date; an incorrect change can make data unreadable, and schema changes cannot simply be rolled back.

Verify the exact key names against the pinned chart before applying:

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

helm search repo grafana/loki --versions
helm show values grafana/loki --version <PINNED_CHART_VERSION> > loki-default-values.yaml

Pin the chart version in Git after testing it. Render and review the Kubernetes objects before deployment:

helm template loki grafana/loki \
  --namespace loki \
  --version <PINNED_CHART_VERSION> \
  --values values.yaml > rendered-loki.yaml

helm upgrade --install loki grafana/loki \
  --namespace loki \
  --create-namespace \
  --version <PINNED_CHART_VERSION> \
  --values values.yaml \
  --atomic \
  --timeout 15m

Collect Kubernetes logs with Alloy

Collector configuration is a separate deployment concern. Keep it independently versioned so Loki storage changes do not silently alter collection behavior.

For a Kubernetes log pipeline, validate:

• node log paths for the runtime in use
• Kubernetes metadata discovery and relabeling
• multiline handling before production rollout
• label cardinality and dropped-label policy
• tenant and authentication headers
• buffering and retry behavior during Loki outages
• memory and CPU limits under backlog conditions

Avoid promoting high-cardinality values such as request IDs, trace IDs, user IDs, or pod-generated filenames into Loki labels. Keep those values in the log body and query them at search time.

Retention is a Loki operation

With TSDB, the Compactor performs retention by removing index references and deleting chunks asynchronously. Configure retention in Loki and ensure the Compactor can delete objects. An S3 lifecycle policy may provide defense in depth, but an earlier lifecycle expiration can remove chunks while Loki still expects them to exist.

Retention also has compliance implications. Confirm whether the environment requires immutable archives, legal holds, deletion requests, or different periods by tenant or stream before choosing one global value.

Validation

Do not stop at helm install succeeding. Validate the full write and read path:

kubectl -n loki get pods
kubectl -n loki get events --sort-by=.lastTimestamp
kubectl -n loki logs deploy/loki-gateway --tail=100

Then:

1. Generate a uniquely identifiable test log.
2. Confirm the collector accepts and forwards it.
3. Query that exact value through the same gateway used by Grafana.
4. Confirm new objects appear in the chunks bucket.
5. Exercise a component restart and verify ingestion recovers.
6. Test an unauthorized request against the gateway.
7. Verify retention in a non-production environment with a short test period.

Component names vary by chart version and deployment mode, so inspect kubectl -n loki get deploy,statefulset,daemonset before copying a resource name into a runbook.

Operational limits

S3 durability does not make the whole logging service highly available. Availability also depends on ingester replication, zone placement, query components, gateway capacity, memberlist or object-store connectivity, and the collector's ability to buffer during an outage.

Monitor at least:

• rejected samples and ingestion errors
• distributor and ingester request latency
• unhealthy rings or unavailable replicas
• compactor failures and retention backlog
• query latency, timeouts, and queue depth
• S3 request errors and throttling
• collector retries, dropped entries, and buffer usage
• active-series and label-cardinality growth

Load-test with representative log volume and queries before assigning production-ready capacity claims.

References

• Grafana Loki: Deploy the Helm chart on AWS
• Grafana Loki: Storage schema
• Grafana Loki: Storage configuration
• Grafana Loki: Helm chart components
• Grafana Loki: Ingest logs with Alloy