Error occurs when deploying canary in istio environment using gateway.

[oliverpark999]

Checklist:

• I've included steps to reproduce the bug.
• I've included the version of argo rollouts.
• I've included the version of the gateway API plugin
• I've included my Rollout Spec (anonymized if needed)

Describe the bug

To Reproduce

Expected behavior

Read Document
https://rollouts-plugin-trafficrouter-gatewayapi.readthedocs.io/en/latest/quick-start/

Environment

• Istio
• GatewayAPI
• Check the normal configuration of the GatewayAPI Plugin

Contents

• When deploying Rollout, the Pod does not come up normally.
• If you check the Rollout log of the application, it is as follows.
  ** Message: Rollout does not have minimum availability

│ Spec:                                                                                                                                   │
│   Replicas:                3                                                                                                            │
│   Revision History Limit:  2                                                                                                            │
│   Selector:                                                                                                                             │
│     Match Labels:                                                                                                                       │
│       App:  rollout-canary-demo                                                                                                         │
│   Strategy:                                                                                                                             │
│     Canary:                                                                                                                             │
│       Canary Service:  rollout-canary-demo-canary                                                                                       │
│       Stable Service:  rollout-canary-demo-stable                                                                                       │
│       Steps:                                                                                                                            │
│         Set Weight:  30                                                                                                                 │
│         Pause:                                                                                                                          │
│         Set Weight:  60                                                                                                                 │
│         Pause:                                                                                                                          │
│         Set Weight:  100                                                                                                                │
│         Pause:                                                                                                                          │
│       Traffic Routing:                                                                                                                  │
│         Plugins:                                                                                                                        │
│           argoproj-labs/gatewayAPI:                                                                                                     │
│             Http Route:  rollout-canary-demo-httproute                                                                                  │
│             Namespace:   foobar                                                                                                          │
│   Template:                                                                                                                             │
│     Metadata:                                                                                                                           │
│       Labels:                                                                                                                           │
│         App:  rollout-canary-demo                                                                                                       │
│     Spec:                                                                                                                               │
│       Containers:                                                                                                                       │
│         Image:  argoproj/rollouts-demo:yellow                                                                                           │
│         Name:   rollout-canary-demo                                                                                                     │
│         Ports:                                                                                                                          │
│           Container Port:  8080                                                                                                         │
│           Name:            http                                                                                                         │
│           Protocol:        TCP                                                                                                          │
│         Resources:                                                                                                                      │
│           Requests:                                                                                                                     │
│             Cpu:     5m                                                                                                                 │
│             Memory:  32Mi                                                                                                               │
│ Status:                                                                                                                                 │
│   Blue Green:                                                                                                                           │
│   Canary:                                                                                                                               │
│   Conditions:                                                                                                                           │
│     Last Transition Time:  2024-10-29T00:17:23Z                                                                                         │
│     Last Update Time:      2024-10-29T00:17:23Z                                                                                         │
│     Message:               Rollout does not have minimum availability                                                                   │
│     Reason:                AvailableReason                                                                                              │
│     Status:                False                                                                                                        │
│     Type:                  Available                                                                                                    │
│     Last Transition Time:  2024-10-29T00:17:23Z                                                                                         │
│     Last Update Time:      2024-10-29T00:17:23Z                                                                                         │
│     Message:               RolloutCompleted                                                                                             │
│     Reason:                RolloutCompleted                                                                                             │
│     Status:                False                                                                                                        │
│     Type:                  Completed                                                                                                    │
│     Last Transition Time:  2024-10-29T00:17:23Z                                                                                         │
│     Last Update Time:      2024-10-29T00:17:23Z                                                                                         │
│     Message:               Created new replica set "rollout-canary-demo-bb7dc4b6c"                                                      │
│     Reason:                NewReplicaSetCreated                                                                                         │
│     Status:                True                                                                                                         │
│     Type:                  Progressing                                                                                                  │
│   Current Pod Hash:        bb7dc4b6c                                                                                                    │
│   Current Step Hash:       fb9f5677  

Name:         rollouts-demo
Namespace:    foobar
Labels:       <none>
Annotations:  rollout.argoproj.io/revision: 1
API Version:  argoproj.io/v1alpha1
Kind:         Rollout
Metadata:
  Creation Timestamp:  2024-10-29T01:42:55Z
Name:            rollouts-demo
Namespace:       foobar
Status:          ◌ Progressing
Message:         waiting for rollout spec update to be observed
Strategy:        Canary
  Step:          0/4
  SetWeight:     50
  ActualWeight:  0
Replicas:
  Desired:       3
  Current:       0
  Updated:       0
  Ready:         0
  Available:     0

NAME                              KIND        STATUS         AGE    INFO
⟳ rollouts-demo                   Rollout     ◌ Progressing  5m46s
└──# revision:1
   └──⧉ rollouts-demo-abc  ReplicaSet  • ScaledDown   5m46s  canary

Screenshots

Version
plugin version => 0.4.0
argo rollouts controller version => 1.7.2

Logs

# Paste the logs from the rollout controller

# Logs for the entire controller:
kubectl logs -n argo-rollouts deployment/argo-rollouts

│ {"level":"info","msg":"delaying service switch from  to abc: ReplicaSet not fully available","namespace":"foobar","rollout":"rollout-canary-demo","service":"rollout-canary-demo-canary","time":"2024-10-29T00:20:40Z"}                                                           │
│ {"generation":1,"level":"error","msg":"roCtx.reconcile err failed to get traffic router plugin argoproj-labs/gatewayAPI: unable to start plugin system: unable to get plugin client (argoproj-labs/gatewayAPI) for ping: exec: Stdout already set","namespace":"foobar","resourceVersio │
│ {"generation":1,"level":"info","msg":"Reconciliation completed","namespace":"foobar","resourceVersion":"44704875","rollout":"rollout-canary-demo","time":"2024-10-29T00:20:40Z","time_ms":1.3126}                                                                                       │
│ {"level":"error","msg":"rollout syncHandler error: failed to get traffic router plugin argoproj-labs/gatewayAPI: unable to start plugin system: unable to get plugin client (argoproj-labs/gatewayAPI) for ping: exec: Stdout already set","namespace":"foobar","rollout":"rollout-cana │
│ {"level":"info","msg":"rollout syncHandler queue retries: 33 : key \"foobar/rollout-canary-demo\"","namespace":"foobar","rollout":"rollout-canary-demo","time":"2024-10-29T00:20:40Z"}                                                                                                   │
│ {"error":null,"level":"error","msg":"failed to get traffic router plugin argoproj-labs/gatewayAPI: unable to start plugin system: unable to get plugin client (argoproj-labs/gatewayAPI) for ping: exec: Stdout already set","time":"2024-10-29T00:20:40Z"}                            │
│ {"generation":1,"level":"info","msg":"Started syncing rollout","namespace":"foobar","resourceVersion":"44704875","rollout":"rollout-canary-demo","time":"2024-10-29T00:20:50Z"}                                                                                                         │
│ {"level":"info","msg":"delaying service switch from  to abc: ReplicaSet not fully available","namespace":"foobar","rollout":"rollout-canary-demo","service":"rollout-canary-demo-canary","time":"2024-10-29T00:20:50Z"}                                                           │
│ {"generation":1,"level":"error","msg":"roCtx.reconcile err failed to get traffic router plugin argoproj-labs/gatewayAPI: unable to start plugin system: unable to get plugin client (argoproj-labs/gatewayAPI) for ping: exec: Stdout already set","namespace":"foobar","resourceVersio │
│ {"generation":1,"level":"info","msg":"Reconciliation completed","namespace":"foobar","resourceVersion":"44704875","rollout":"rollout-canary-demo","time":"2024-10-29T00:20:50Z","time_ms":1.721427}                                                                                     │
│ {"level":"error","msg":"rollout syncHandler error: failed to get traffic router plugin argoproj-labs/gatewayAPI: unable to start plugin system: unable to get plugin client (argoproj-labs/gatewayAPI) for ping: exec: Stdout already set","namespace":"foobar","rollout":"rollout-cana │
│ {"level":"info","msg":"rollout syncHandler queue retries: 34 : key \"foobar/rollout-canary-demo\"","namespace":"foobar","rollout":"rollout-canary-demo","time":"2024-10-29T00:20:50Z"}                                                                                                   │
│ {"error":null,"level":"error","msg":"failed to get traffic router plugin argoproj-labs/gatewayAPI: unable to start plugin system: unable to get plugin client (argoproj-labs/gatewayAPI) for ping: exec: Stdout already set","time":"2024-10-29T00:20:50Z"}                            │
│                                                                                                                                                          

# Logs for a specific rollout:
kubectl logs -n argo-rollouts deployment/argo-rollouts | grep rollout=<ROLLOUTNAME

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.

[oliverpark999]

The problem was the line below.
When I removed trafficRouting, the pod was deployed.
Why is that?

trafficRouting:
  plugins:
    argoproj-labs/gatewayAPI:
      httpRoute: argo-rollouts-http-route # our created httproute
      namespace: foobar

[kostis-codefresh]

Hello

The part that you mention is the part that loads the plugin. And the error you posted shows that the plugin is not loaded.

Possible reasons:

1. The clusters doesn't have access to the URL you have defined for the plugin
2. You are trying to load the plugin of a different architecture.

Have you also followed https://rollouts-plugin-trafficrouter-gatewayapi.readthedocs.io/en/latest/installation/ ?

Can you post the argo-rollouts-config please and also mention what type of machine is running Argo Rollouts?

Also the plugin you are trying to load is the latest one for Kubernetes API gateway 1.0+ Are you sure the version of Istio you are running also supports this version?

[oliverpark999]

Hi!

When I configured the plugin with v0.3.0, the plugin was loaded normally and Replicas also worked.

But here's my question.

My k8s Gateway API is version 1.0.

So all resources are already using gateway.networking.k8s.io/v1.

However, in version v0.4.0, an error message was printed and it did not work properly,

but in version v0.3.0, it worked normally.

Do you know anything about this?

[oliverpark999]

I read the release notes and it seems that k8s Gateway API v1.0 is supported starting from v0.3.0. Then, is this a bug in v0.4.0?

[Philipp-Plotnikov]

Can you say machine architecture and the plugin that you are trying to download, please ?

[oliverpark999]

Yes,
The hardware architecture of my Kubernetes Node server is x86, and the plugin that caused the error message is gatewayapi-plugin-linux-amd64(v0.4.0).

[kostis-codefresh]

My k8s Gateway API is version 1.0.

What is your Istio version?

When I configured the plugin with v0.3.0, the plugin was loaded normally and Replicas also worked.

But did you enable again the plugin in the Rollout? Because previously you said you removed that part

trafficRouting:
  plugins:
    argoproj-labs/gatewayAPI:
      httpRoute: argo-rollouts-http-route # our created httproute
      namespace: foobar

To the best of my knowledge there are no major API changes between plugin 0.3.0 and 0.4.0

[oliverpark999]

Istio is version 1.22.3.
As you said, I removed trafficRouting and checked that it works.
However, I tried several things without stopping the game.
That was v0.3.0, and I added trafficRouting again.
I also restarted Argo-Rollout Deployment.
The result was normal operation as below.

Normal  RolloutAddedToInformer  10s   rollouts-controller  Rollout resource added to informer: foo/bar
Normal  RolloutNotCompleted     10s   rollouts-controller  Rollout not completed, started update to revision 1
Normal  RolloutUpdated          10s   rollouts-controller  Rollout updated to revision 1
Normal  NewReplicaSetCreated    10s   rollouts-controller  Created ReplicaSet bar-xxx (revision 1)
Normal  TrafficWeightUpdated    9s    rollouts-controller  Traffic weight updated to 0
Normal  ScalingReplicaSet       9s    rollouts-controller  Scaled up ReplicaSet bar-xxx (revision 1) from 0 to 1
Normal  RolloutCompleted        9s    rollouts-controller  Rollout completed update to revision 1 : Initial deploy
Normal  TrafficWeightUpdated    9s    rollouts-controller  Traffic weight updated
Normal  SwitchService           1s    rollouts-controller  Switched selector for service 'bar' from '' to 'xx'
Normal  SwitchService           1s    rollouts-controller  Switched selector for service 'bar-canary' from '' to 'xx'