Kubernetes 1.29 Native Sidecars: Jobs That Finish, Ordered Shutdown and the 1.28 Trap

On this page

Kubernetes 1.29, released on 13 December 2023, turned on native sidecar containers by default. The feature arrived as alpha in 1.28 behind the SidecarContainers feature gate and is beta in 1.29. Amazon EKS added 1.29 on 23 January 2024, and AKS has had 1.29.0 in preview since its 7 February release, so many teams can now use it in a managed cluster for the first time.

The change is small in YAML and large in behaviour. A sidecar is now an init container with restartPolicy: Always. Kubernetes starts it before the main containers, keeps it running, stops it after them, and no longer lets it hold a Job open. This article runs seven scenarios on kind clusters and reports what actually happened, including two cases where the new behaviour can still lose data, and one where the same manifest on a 1.28 cluster leaves the Pod stuck forever.

The problem sidecars had before 1.29

A sidecar used to be an ordinary second container in the Pod: a log shipper, a service mesh proxy, a Cloud SQL or database proxy, a secrets agent. Kubernetes treated it as an equal of the application container, which caused three long-standing problems:

  • Jobs never finished. A Job’s Pod completes only when all its containers have exited. The main container finishes; the log shipper does not; the Job hangs. Teams worked around it with shared files, curl calls to a proxy’s quit endpoint, or wrapper scripts.
  • No start order. The application could start before its proxy was ready and fail its first outbound calls.
  • No stop order. On deletion every container got SIGTERM at once, so a proxy could exit while the application was still draining requests through it.

The first problem is easy to reproduce. This Job runs a five-second task next to a log shipper written as a normal container:

YAML17 lines
# The pre-1.29 pattern: the log shipper is just another container.
apiVersion: batch/v1
kind: Job
metadata:
  name: classic-sidecar
spec:
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: main
          image: busybox:1.36.1
          command: ["sh", "-c", "echo working; sleep 5; echo done"]
        - name: log-shipper
          image: busybox:1.36.1
          command: ["sh", "-c", "while true; do sleep 1; done"]

After 60 seconds the Job had no condition at all. The main container had completed, but the Pod was still Running because the log shipper was:

Text6 lines
[v1.29/job-classic] server v1.29.2
  job_condition: none after 60 s
  seconds_to_condition: None
  pod_phase: Running
  log-shipper     main reason=None exit=None restarts=0
  main            main reason=Completed exit=0 restarts=0

The 1.29 model

A native sidecar is declared in initContainers, with one new field. Init containers normally run one after another and must exit before the next starts. With restartPolicy: Always, the kubelet starts the container, considers it done as soon as it has started (or, if it has a startup probe, once the probe passes) and moves on, but keeps it running and restarts it if it exits:

YAML19 lines
# Kubernetes 1.29: the log shipper is an init container with restartPolicy: Always.
apiVersion: batch/v1
kind: Job
metadata:
  name: native-sidecar
spec:
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      initContainers:
        - name: log-shipper
          image: busybox:1.36.1
          restartPolicy: Always
          command: ["sh", "-c", "while true; do sleep 1; done"]
      containers:
        - name: main
          image: busybox:1.36.1
          command: ["sh", "-c", "echo working; sleep 5; echo done"]
%%{init: {'theme': 'base', 'themeVariables': {'signalColor': '#0073BB', 'signalTextColor': '#0F172A', 'actorBkg': '#FFF3E0', 'actorBorder': '#FF9900', 'actorTextColor': '#232F3E', 'noteBkgColor': '#FFF8E7', 'noteBorderColor': '#FF9900', 'noteTextColor': '#232F3E'}}}%%
sequenceDiagram
    participant K as kubelet
    participant S as Sidecar (init, restartPolicy Always)
    participant M as Main container
    K->>S: start
    Note over K,S: wait for the startup probe, if any
    K->>M: start
    M-->>K: exits (Job finished)
    K->>S: SIGTERM
    S-->>K: exits, or SIGKILL when the grace period runs out
    Note over K: Pod Succeeded, Job Complete

You can check whether the gate is on from the API server’s metrics, with kubectl get --raw /metrics. The 1.28.7 test cluster reported the alpha gate off:

Text1 line
kubernetes_feature_enabled{name="SidecarContainers",stage="ALPHA"} 0

and the 1.29.2 cluster reported the beta gate on:

Text1 line
kubernetes_feature_enabled{name="SidecarContainers",stage="BETA"} 1

Test setup

Every scenario ran on kind v0.22.0, released on 14 February 2024, with its default node image kindest/node:v1.29.2 pinned by digest. The 1.28 scenario used kindest/node:v1.28.7. kubectl was v1.29.2 and every container used busybox:1.36.1, loaded into the cluster before the tests so image pulls did not affect timings. A Python harness applied each manifest, waited, read container states from the API and deleted what it created.

Status timestamps from the API have one-second resolution, which is too coarse to see a shutdown order. So in the two shutdown scenarios each container appends a line to a file on the node when it gets SIGTERM and when it exits. The timestamp is the node’s /proc/uptime, one clock for every container on the node. This is the second sidecar’s definition; the others differ only in name and in how long they take to stop:

YAML5 lines
    - name: second-sidecar
      image: busybox:1.36.1
      restartPolicy: Always
      volumeMounts: [{name: events, mountPath: /events}]
      command: ["sh", "-c", "log() { echo \"$(cut -d' ' -f1 /proc/uptime) second-sidecar $1\" >> /events/log; }; trap 'log TERM; sleep 1; log exit; exit 0' TERM; log start; while true; do sleep 0.2; done"]

Test results

Table of seven native sidecar scenarios and their outcomes on Kubernetes 1.29 and 1.28
Seven scenarios, one cluster per version.

Jobs complete, after the sidecar stops. With the log shipper as a native sidecar the Job reached Complete. It took 40 seconds for a five-second task, and the sidecar’s exit code shows why:

Text6 lines
[v1.29/job-native] server v1.29.2
  job_condition: Complete
  seconds_to_condition: 40.3
  pod_phase: Succeeded
  log-shipper     init reason=Error exit=137 restarts=0
  main            main reason=Completed exit=0 restarts=0

When the main container finished, the kubelet sent the sidecar SIGTERM. The sidecar was a shell loop running as process 1 with no SIGTERM handler, and Linux does not deliver default-action signals to process 1 in a container, so it ignored the signal. The kubelet waited out the 30-second default grace period and then killed it, exit code 137. The Job still counted as Complete; only the main container’s result matters. But every run of that Job now takes 30 seconds longer than it should. Real log shippers and proxies handle SIGTERM; check yours does, or set a shorter terminationGracePeriodSeconds on Jobs.

Start order works, with a startup probe. The proxy in this scenario creates a ready file after eight seconds, and its startup probe checks for it every second:

YAML21 lines
# The proxy takes 8 seconds to become ready. Does main wait for it?
apiVersion: v1
kind: Pod
metadata:
  name: startup-order
spec:
  restartPolicy: Never
  initContainers:
    - name: proxy
      image: busybox:1.36.1
      restartPolicy: Always
      command: ["sh", "-c", "sleep 8; touch /tmp/ready; while true; do sleep 1; done"]
      startupProbe:
        exec:
          command: ["test", "-f", "/tmp/ready"]
        periodSeconds: 1
        failureThreshold: 30
  containers:
    - name: main
      image: busybox:1.36.1
      command: ["sh", "-c", "sleep 3600"]

The main container started 9 seconds after the proxy, so the kubelet waited for the probe. Without a startup probe the kubelet moves on as soon as the sidecar’s process has started, which is not the same as ready to serve.

Shutdown is ordered. When the Pod was deleted, the main container got SIGTERM first. Only after it had exited did the second sidecar get SIGTERM, and only after that one had exited did the first sidecar get it:

Text11 lines
[v1.29/termination-order] server v1.29.2
  delete_returned_after_s: 6.0
    0.00s  main            TERM
    3.01s  main            exit
    3.05s  second-sidecar  TERM
    4.05s  second-sidecar  exit
    4.09s  first-sidecar   TERM
    5.09s  first-sidecar   exit
  final first-sidecar   exit=0 reason=Completed
  final second-sidecar  exit=0 reason=Completed
  final main            exit=0 reason=Completed
Timeline of Pod deletion: main gets SIGTERM at 0 seconds and exits at 3.01, the second sidecar gets SIGTERM at 3.05, the first sidecar at 4.09; below, a sidecar killed before it can flush when main uses the whole grace period
Reverse order of declaration, one at a time.

The order is the reverse of declaration, so declare sidecars in dependency order: something every other container needs, such as a network proxy, goes first; a log shipper that should outlive the proxy’s own logs goes after it.

The grace period is shared, and sidecars come last. In the fifth scenario the main container ignores SIGTERM, the grace period is 10 seconds, and the log shipper needs five seconds to flush:

Text6 lines
[v1.29/grace-squeeze] server v1.29.2
  delete_returned_after_s: 13.0
    0.00s  main            TERM-ignored
   10.00s  log-shipper     TERM
  final log-shipper     exit=137 reason=Error
  final main            exit=137 reason=Error

The sidecar received SIGTERM only at 10 seconds, when the kubelet killed the main container at the end of the grace period. It never logged “flushed” and ended with exit code 137. The grace period covers the whole Pod, not each container, and sidecars get whatever is left. A main container that drains slowly can use up the time a log shipper needs to send the last lines, which are usually the ones that explain why the Pod was being stopped.

Size the grace period as a sum, not a maximum. Take the main container’s drain time, including any preStop hook, because hooks run inside the same grace period. Then add each sidecar’s own shutdown time, because the sidecars stop one after another. A web container that drains for 20 seconds, a proxy that needs 2 and a log shipper that needs 5 want at least 27 seconds, so the 30-second default leaves only three to spare. Keeping the main container’s drain short is usually cheaper than raising the grace period for every Pod.

Sidecars restart even when the Pod does not. The sixth scenario’s sidecar exits with code 1 after three seconds, in a Job whose Pod has restartPolicy: Never. The kubelet restarted it 2 times while the main container ran, and the Job completed:

Text6 lines
[v1.29/sidecar-crash] server v1.29.2
  job_condition: Complete
  seconds_to_condition: 29.7
  pod_phase: Succeeded
  log-shipper     init reason=Error exit=1 restarts=2
  main            main reason=Completed exit=0 restarts=0

That is the point of restartPolicy: Always on the container, and it is better behaviour than before, when a crashed log shipper stayed dead for the rest of the Job. It also means a broken sidecar does not fail the Job, so alert on sidecar restart counts.

The 1.28 trap

The last scenario applied the native sidecar Job to a 1.28.7 cluster, where the SidecarContainers gate is off by default. The API server accepted the manifest without an error or a warning, dropped the restartPolicy field, and stored an ordinary init container:

Text6 lines
[v1.28/native-on-old] server v1.28.7
  pod_phase: Pending
  stored_restartPolicy: (field dropped)
  job_condition_after_45s: None
  log-shipper     init reason=None exit=None restarts=0
  main            main reason=PodInitializing exit=None restarts=0

An ordinary init container must exit before the main containers start. A log shipper never exits, so the Pod stayed Pending with the main container waiting in PodInitializing. A Job like this sits there until someone notices. This is the standard Kubernetes rule for fields behind a disabled feature gate: they are dropped, not rejected. In practice it means a Helm chart or manifest that uses native sidecars must not reach a cluster older than 1.29, or a 1.28 cluster without the gate. Pin the chart’s kubeVersion to >=1.29.0-0, or check kubectl version in the pipeline, so the deploy fails loudly instead of hanging.

⚠️
CHECK EVERY CLUSTER, NOT JUST ONE

The gate has to be on in the API server and in the kubelet on the node that runs the Pod. During an upgrade, the control plane is usually upgraded before the nodes. Until every node pool is on 1.29, schedule sidecar workloads only onto upgraded nodes, or wait for the upgrade to finish.

Resource requests and service meshes

Native sidecars also change how the scheduler counts a Pod’s resources. An ordinary init container’s request only matters during initialisation, so the Pod’s effective request is the larger of the biggest init container and the sum of the app containers. A native sidecar runs alongside the app containers, so its request is added to that sum. During initialisation, each later init container’s request is counted together with the sidecars declared before it. In practice: a proxy that requested 100m CPU as an ordinary container still costs 100m as a native sidecar, but an init container placed after it now needs room for both.

Most sidecars in production are not written by hand; a service mesh or an agent’s mutating webhook injects them. Those keep the old behaviour until the injector puts the container in initContainers. Istio has had experimental support since the 1.19 release in 2023, behind the ENABLE_NATIVE_SIDECARS setting on istiod, which is off by default. Check what your mesh and agents do before expecting the start and stop order described here; a Pod can mix both kinds, and only the native ones get the new lifecycle.

Migrating an existing sidecar

  • Move the container from containers to initContainers and add restartPolicy: Always. Put it after any ordinary init containers that it depends on, and before any it serves.
  • Add a startup probe if the main container must not start before the sidecar is ready, such as a proxy that must accept connections first.
  • Delete the workarounds: shared-file signals, quit endpoints called from the main container, wrapper scripts that kill the sidecar.
  • Check the sidecar handles SIGTERM and exits promptly. If it runs as process 1 through a shell, use exec or an init such as tini so the signal reaches it.
  • Review terminationGracePeriodSeconds: it must cover the main container’s drain time plus every sidecar’s shutdown time, one after another.
  • Sidecar injectors from service meshes and agents decide where the container goes. Until yours supports native sidecars, injected sidecars keep the old behaviour.

Decision framework

SituationRecommendation
Jobs or CronJobs with a log shipper or proxyMove to native sidecars on 1.29; this is the clearest win
Long-running Deployments with a mesh proxyWorth it for start and stop order; wait for your mesh’s support if it injects the proxy
Clusters still on 1.28Do not ship native sidecar manifests there; the Pod hangs in Init
Sidecar that needs time to flush on shutdownRaise terminationGracePeriodSeconds to cover main plus sidecars, and keep main’s drain short
Sidecar that must be ready before the app startsGive it a startup probe
Sidecar that crashes now and thenNative sidecars restart it; alert on its restart count, because the Job will not fail
📝
HOW THIS WAS CHECKED

All seven scenarios ran in one GitHub Actions run (run) on kind v0.22.0 with node images v1.29.2 and v1.28.7, kubectl v1.29.2 and busybox 1.36.1. The outputs above are copied from that run’s summary file and metric files. kind only, one node: no managed service, no service mesh, and no cluster with mixed control-plane and node versions was tested. Release dates come from the Kubernetes, AWS and AKS announcements.

Key takeaways

  • In Kubernetes 1.29 a sidecar is an init container with restartPolicy: Always; the feature is beta and on by default.
  • Jobs with native sidecars complete, but only after the sidecar exits, so a sidecar that ignores SIGTERM adds the full grace period to every run.
  • Start order needs a startup probe; stop order is automatic, main containers first and then sidecars in reverse order of declaration.
  • The grace period is shared and sidecars come last; a slow main container can leave a log shipper no time to flush.
  • On 1.28 with the gate off, the API silently drops restartPolicy and the Pod waits in Init forever.

If you have moved a mesh proxy or log shipper to a native sidecar and hit something not covered here, let me know in the comments below.

Glossary

Sidecar container
A helper container that runs alongside the application container in the same Pod, such as a log shipper or proxy.
Native sidecar
An init container with restartPolicy: Always, supported from Kubernetes 1.28 (alpha) and on by default from 1.29 (beta).
Init container
A container that runs before the Pod’s main containers; ordinary init containers must exit successfully first.
SidecarContainers
The feature gate that controls native sidecars; it must be on in the API server and the kubelet.
Feature gate
A named switch that turns a Kubernetes feature on or off; alpha gates are off by default, beta gates usually on.
Startup probe
A check the kubelet runs until it passes; for a native sidecar it decides when the next container may start.
terminationGracePeriodSeconds
How long the kubelet waits after SIGTERM before killing the Pod’s containers; 30 seconds by default.
SIGTERM
The signal asking a process to stop; SIGKILL follows if it has not stopped when the grace period ends.
Exit code 137
128 plus 9: the process was killed with SIGKILL.
kind
Kubernetes in Docker, a tool that runs a cluster’s nodes as Docker containers.
Job
A Kubernetes workload that runs Pods to completion and records whether they succeeded.

References & Further Reading


Discover more from C4: Container, Code, Cloud & Context

Subscribe to get the latest posts sent to your email.

Written by

Nithin Mohan TK

Hello Namaste! 👋🏻 I'm an accomplished Enterprise Solution Architect & Engineering Leader with 20+ years of progressive experience architecting, building, and scaling transformative enterprise solutions. ⚜️ Solution Architect / AI & Cloud 📌 Galway, Ireland

Leave a comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.