Why We Switched Argo Rollouts Canary Analysis from Datadog KSM to HTTP

Introduction#

When you adopt Argo Rollouts for progressive delivery, one of the most important decisions is what signal should block promotion of a canary.

For a while, our default was a shared Datadog-based check: global-pod-ready-check. It queries kubernetes_state.pod.ready and looks for tags like service, env, and version.

That sounds reasonable on paper. In practice, a staging drill on a sample app (demo-api) showed a different story:

• Healthy canaries were aborted because Datadog had no data yet
• Config-only rollouts could pass without validating the canary
• Even a ReplicaSet-based Datadog query fixed isolation but not speed
• Switching to HTTP/web analysis against the canary preview service passed in ~45 seconds

This post documents what we tested, what failed, why, and what we recommend going forward.

The setup#

We run canary rollouts on a staging Kubernetes cluster using:

• Argo Rollouts with a shared Helm chart (shared-rollout-chart)
• ClusterAnalysisTemplate resources deployed as a platform utility
• Datadog KSM (kubeStateMetricsCore) for Kubernetes state metrics
• A simple demo web app (demo-api) for the drill

Typical canary steps:

strategy:
  canary:
    steps:
      - setWeight: 50
      - pause: { duration: 2m }
      - analysis:
          templates:
            - templateName: global-pod-ready-check
              clusterScope: true
      - setWeight: 100

The chart auto-injects analysis args:

• service-name
• env
• canary-version
• canary-hash

Apps must also set Datadog unified service tags:

extraLabels:
  tags.datadoghq.com/env: staging
  tags.datadoghq.com/service: demo-api
  tags.datadoghq.com/version: demo-api-v3   # must match image.tag

The original Datadog query:

min:kubernetes_state.pod.ready{
  kube_namespace:{{args.namespace}},
  service:{{args.service-name}},
  env:{{args.env}},
  version:{{args.canary-version}},
  condition:true
}

Analysis timing:

initialDelay: 180s
interval: 30s
count: 5
failureLimit: 2

What we tested#

We ran six scenarios on the demo app in staging:

The important observation: during Datadog failures, kubectl showed the canary pod as Ready with correct labels. The app was fine. Kubernetes was fine. The metrics pipeline was not ready yet.

Problem 1: Datadog indexing lag#

New tag combinations do not show up in Datadog immediately after a canary pod becomes Ready.

Observed timeline:

T+0m     Canary pod Running + Ready (kubectl ✓)
T+3m     initialDelay ends → 1st Datadog query → [] or 0  ✗
T+3.5m   2nd query fails → failureLimit=2 → rollout ABORT
T+5–7m   Tag appears in Datadog facets
T+7m+    Retry AnalysisRun → passes

This explains the pattern many teams see:

First analysis run fails. Second attempt works. Nothing changed in the app.

It feels flaky, but it is often timing, not application health.

Increasing initialDelay to 420s helps avoid false aborts, but it also means every rollout waits 7+ minutes even when tags appear quickly. That is a poor tradeoff for a simple "is the canary up?" gate.

Problem 2: False pass on config-only rollouts#

When only config changes and image.tag / tags.datadoghq.com/version stay the same, the version-based query matches both stable and canary pods.

Stable alone satisfies:

min(...) == 1

So analysis passes without validating the canary ReplicaSet in isolation.

We proved this with a config-only drill: only an environment variable changed (e.g. LOG_LEVEL=debug), image stayed at v2, and the version query returned 5/5 successes because stable v2 pods were already in Datadog.

Problem 3: ReplicaSet query fixed isolation, not speed#

We updated global-pod-ready-check to use a ReplicaSet-scoped query:

min:kubernetes_state.pod.ready{
  kube_namespace:{{args.namespace}},
  kube_replica_set:{{args.service-name}}-{{args.canary-hash}},
  condition:true
}

This is the right fix for isolation:

• Works for image rollouts
• Works for config-only rollouts
• Targets only the canary ReplicaSet

But in staging, new ReplicaSet names still took ~4–5 minutes to appear in Datadog. RS query is more correct, not faster.

Problem 4: Label drift makes version queries worse#

The chart prefers extraLabels["tags.datadoghq.com/version"] over image.tag when building canary-version for analysis.

If you bump the image but forget the label:

image:
  tag: "demo-api-v3"
extraLabels:
  tags.datadoghq.com/version: demo-api-v2  # oops

Then:

• Canary runs v3 image
• Pods are labeled v2
• Analysis queries v3
• Datadog returns no series forever

That is a config mistake, but version-based analysis makes it painful.

Rule: always update image.tag and tags.datadoghq.com/version together.

The fix: HTTP/web analysis#

We added global-http-health-check:

apiVersion: argoproj.io/v1alpha1
kind: ClusterAnalysisTemplate
metadata:
  name: global-http-health-check
spec:
  args:
    - name: namespace
    - name: service-name
    - name: health-path
      value: "/health"
  metrics:
    - name: http-health
      initialDelay: 15s
      interval: 10s
      count: 3
      successCondition: "true"
      failureLimit: 2
      provider:
        web:
          url: "http://{{args.service-name}}-preview.{{args.namespace}}.svc.cluster.local{{args.health-path}}"
          timeoutSeconds: 5

How it works#

1. The chart creates a preview service ({release-name}-preview) when canary steps exist
2. Argo Rollouts points the preview selector at the canary ReplicaSet only
3. During analysis, the Rollouts controller performs an HTTP GET inside the cluster
4. 2xx → pass. Non-2xx or timeout → fail

No Datadog. No KSM indexing. No waiting for tag facets.

Example app wiring#

readinessProbe:
  httpGet:
    path: /health
    port: http
  initialDelaySeconds: 5
  periodSeconds: 10

strategy:
  canary:
    steps:
      - setWeight: 50
      - pause: { duration: 2m }
      - analysis:
          templates:
            - templateName: global-http-health-check
              clusterScope: true
      - setWeight: 100
    analysisArgs:
      - name: namespace
        value: demo-api
      - name: health-path
        value: /health

For the demo app, the preview URL resolved to something like:

http://demo-api-shared-rollout-chart-preview.demo-api.svc.cluster.local/health

Measured result#

Datadog vs HTTP: when to use what#

Do not use Datadog KSM pod-ready as the sole promotion gate unless the team explicitly accepts 5–7 minute analysis windows and retry behavior.

Datadog is still valuable. We are only changing what blocks promote.

What each app needs for HTTP analysis#

Required#

Strongly recommended#

Example for a typical backend service#

readinessProbe:
  httpGet:
    path: /health/ready
    port: http

strategy:
  canary:
    analysisArgs:
      - name: namespace
        value: orders-api
      - name: health-path
        value: /health/ready

Rule: health-path should match readinessProbe.httpGet.path unless you intentionally check a different endpoint.

Slow app startup: the remaining concern#

HTTP analysis is fast once the app responds. It does not wait for Datadog. But it will fail if the app is not serving 2xx when analysis runs.

Demo app timing budget#

Our demo app (simple HTTP server on /health) boots in seconds. Plenty of margin.

What happens if boot is slower#

1. Canary pod starts
2. Without startupProbe, kubelet may restart the container if readiness fails too long
3. If the app needs > ~2m 45s to serve 2xx, analysis fails
4. Rollout aborts → stable unchanged

That is correct safety behavior. You tune timing to match reality.

Tuning knobs#

Rule of thumb:

pause + initialDelay + (count × interval) > p95 cold-start time + buffer

Unlike Datadog lag, slow-start tuning is per-app and predictable.

Practical checklist before migrating an app#

• Confirm canary steps and preview service exist
• Identify health HTTP path (match readinessProbe or add one)
• Add health-path to strategy.canary.analysisArgs
• Switch template to global-http-health-check
• If boot time > 60s, add startupProbe and tune pause/analysis timing
• Dry-run one staging promote and watch AnalysisRun
• Keep tags.datadoghq.com/version synced with image.tag for APM/Datadog elsewhere

Key commands for debugging#

Watch the rollout:

kubectl argo rollouts get rollout demo-api -n demo-api --watch

Inspect the latest AnalysisRun:

kubectl get analysisrun -n demo-api --sort-by=.metadata.creationTimestamp | tail -1
kubectl get analysisrun <name> -n demo-api -o yaml

For HTTP analysis, check the URL:

kubectl get analysisrun <name> -n demo-api \
  -o jsonpath='{.spec.metrics[0].provider.web.url}{"\n"}'

Manual curl from inside the cluster:

kubectl run curl-test --rm -it --restart=Never --image=curlimages/curl -- \
  curl -sf http://demo-api-shared-rollout-chart-preview.demo-api.svc.cluster.local/health

Verify pod version label matches image:

kubectl get pod -n demo-api -l rollouts-pod-template-hash=<hash> \
  -o jsonpath='{.items[0].metadata.labels.tags\.datadoghq\.com/version}{"\n"}'

Conclusion#

Our staging drill did not fail because Argo Rollouts or Datadog were "broken." It failed because we asked the wrong signal to make a fast, blocking decision.

Datadog KSM pod-ready is a slow, indirect metric path:

• New tags lag minutes behind Kubernetes reality
• Version queries can false-pass on config-only rollouts
• RS queries fix isolation but not speed

HTTP analysis against the canary preview service is:

• Direct
• Fast (~45s in our demo)
• Canary-isolated by design
• Aligned with how we already think about app health

For teams rolling out Argo Rollouts, my recommendation is clear:

Use HTTP/web analysis as the default canary promotion gate when your app exposes a health endpoint. Keep Datadog for observability and higher-level signals like error rate — not for "did the pod show up in metrics yet?"

The only real concern left is app boot time. That is a solvable, per-app tuning problem — not an infrastructure indexing lottery.

References#

• Argo Rollouts Web Metrics docs
• Argo Rollouts Analysis / ClusterAnalysisTemplate