Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

route weights don't get reset to original values. #14

Open
nsalfos opened this issue Oct 24, 2024 · 1 comment
Open

route weights don't get reset to original values. #14

nsalfos opened this issue Oct 24, 2024 · 1 comment

Comments

@nsalfos
Copy link

nsalfos commented Oct 24, 2024

Hi Folks,
i was able to get the plugin running after building it myself (the latest release on your repo doesn't work properly apparently)
now, when testing the rollout the first time it worked great, and i was able to see traffic shifting and also the weights being changed as i promoted the rollout.
but when deleting the rollout and creating one again the route weights remain whatever the last value was before completing the promotion.

even after deleting the route and creating it again with stable: 100 / canary: 0 now is not changing the weights. so this only worked once for me and not sure why.

I saw another issue similar to this one, i which you tried steps to reproduce but it worked fine for you.
Is it strictly necesary to have "Red Hat OpenShift GitOps" operator installed for this to work as expected??

last time i saw "new weights" being set on was on the first try with working plugin as metnioned 
> k logs -n argo-rollouts argo-rollouts-754c744cf6-m9tfh | grep "New weights"
time="2024-10-22T17:06:38Z" level=info msg="New weights: &TrafficWeights{Canary:WeightDestination{Weight:0,ServiceName:argo-rollouts-canary-service,PodTemplateHash:687d76d795,},Stable:WeightDestination{Weight:100,ServiceName:argo-rollouts-stable-service,PodTemplateHash:,},Additional:[]WeightDestination{},Verified:nil,}" namespace=nsalfos-triage rollout=rollouts-demo
time="2024-10-22T17:06:39Z" level=info msg="New weights: &TrafficWeights{Canary:WeightDestination{Weight:0,ServiceName:argo-rollouts-canary-service,PodTemplateHash:687d76d795,},Stable:WeightDestination{Weight:100,ServiceName:argo-rollouts-stable-service,PodTemplateHash:687d76d795,},Additional:[]WeightDestination{},Verified:nil,}" namespace=nsalfos-triage rollout=rollouts-demo
time="2024-10-22T17:07:01Z" level=info msg="New weights: &TrafficWeights{Canary:WeightDestination{Weight:0,ServiceName:argo-rollouts-canary-service,PodTemplateHash:6cf78c66c5,},Stable:WeightDestination{Weight:100,ServiceName:argo-rollouts-stable-service,PodTemplateHash:687d76d795,},Additional:[]WeightDestination{},Verified:nil,}" namespace=nsalfos-triage rollout=rollouts-demo
time="2024-10-22T17:07:02Z" level=info msg="New weights: &TrafficWeights{Canary:WeightDestination{Weight:25,ServiceName:argo-rollouts-canary-service,PodTemplateHash:6cf78c66c5,},Stable:WeightDestination{Weight:75,ServiceName:argo-rollouts-stable-service,PodTemplateHash:687d76d795,},Additional:[]WeightDestination{},Verified:nil,}" namespace=nsalfos-triage rollout=rollouts-demo
time="2024-10-22T17:09:30Z" level=info msg="New weights: &TrafficWeights{Canary:WeightDestination{Weight:75,ServiceName:argo-rollouts-canary-service,PodTemplateHash:6cf78c66c5,},Stable:WeightDestination{Weight:25,ServiceName:argo-rollouts-stable-service,PodTemplateHash:687d76d795,},Additional:[]WeightDestination{},Verified:nil,}" namespace=nsalfos-triage rollout=rollouts-demo
Full scenario showing steps
Step 1 - no rollout, route state after deletion of previous rollouts 
> k argo rollouts get rollout rollouts-demo --watch
Error: rollout.argoproj.io "rollouts-demo" not found

> k describe route rollouts-demo-route 
Name:         rollouts-demo-route
Namespace:    nsalfos-triage
Labels:       <none>
Annotations:  openshift.io/host.generated: true
API Version:  route.openshift.io/v1
Kind:         Route
Metadata:
  Creation Timestamp:  2024-10-22T09:06:48Z
  Resource Version:    1310740253
  UID:                 ca882b51-823a-4cd3-9b7a-1bf1762a7879
Spec:
  Alternate Backends:
    Kind:    Service
    Name:    argo-rollouts-canary-service
    Weight:  75
  Host:      <REDACTED>
  Port:
    Target Port:  http
  Tls:
    Insecure Edge Termination Policy:  Redirect
    Termination:                       edge
  To:
    Kind:           Service
    Name:           argo-rollouts-stable-service
    Weight:         25
  Wildcard Policy:  None
Status:
  Ingress:
    Conditions:
      Last Transition Time:  2024-10-22T09:06:48Z
      Status:                True
      Type:                  Admitted
    Host:                    <REDACTED>
    Router Name:             router
    Wildcard Policy:         None
    Conditions:
      Last Transition Time:     2024-10-22T09:06:48Z
      Status:                   True
      Type:                     Admitted
    Host:                       <REDACTED>
    Router Canonical Hostname:  <REDACTED>
    Router Name:                default
    Wildcard Policy:            None
Events:                         <none>
Step 2- new rollout 
> oc create -f ocp-route-plugin/rollout-cr.yaml 
rollout.argoproj.io/rollouts-demo created

> k argo rollouts get rollout rollouts-demo && k get route rollouts-demo-route
Name:            rollouts-demo
Namespace:       nsalfos-triage
Status:          ✔ Healthy
Strategy:        Canary
  Step:          4/4
  SetWeight:     100
  ActualWeight:  100
Images:          argoproj/rollouts-demo:blue (stable)
Replicas:
  Desired:       4
  Current:       4
  Updated:       4
  Ready:         4
  Available:     4

NAME                                       KIND        STATUS     AGE  INFO
⟳ rollouts-demo                            Rollout     ✔ Healthy  57s  
└──# revision:1                                                        
   └──⧉ rollouts-demo-687d76d795           ReplicaSet  ✔ Healthy  57s  stable
      ├──□ rollouts-demo-687d76d795-2xrbs  Pod         ✔ Running  57s  ready:1/1
      ├──□ rollouts-demo-687d76d795-4zfkg  Pod         ✔ Running  57s  ready:1/1
      ├──□ rollouts-demo-687d76d795-kd5nr  Pod         ✔ Running  57s  ready:1/1
      └──□ rollouts-demo-687d76d795-tpfgd  Pod         ✔ Running  57s  ready:1/1
NAME                  HOST/PORT                                                                            PATH   SERVICES                                                              PORT   TERMINATION     WILDCARD
rollouts-demo-route   <REDACTED>          argo-rollouts-stable-service(25%),argo-rollouts-canary-service(75%)   http   edge/Redirect   None
Step 3 - changing image no changes to route. 
> k argo rollouts get rollout rollouts-demo && k get route rollouts-demo-route
Name:            rollouts-demo
Namespace:       nsalfos-triage
Status:          ॥ Paused
Message:         CanaryPauseStep
Strategy:        Canary
  Step:          1/4
  SetWeight:     25
  ActualWeight:  25
Images:          argoproj/rollouts-demo:blue (stable)
                 argoproj/rollouts-demo:yellow (canary)
Replicas:
  Desired:       4
  Current:       4
  Updated:       1
  Ready:         4
  Available:     4

NAME                                       KIND        STATUS         AGE    INFO
⟳ rollouts-demo                            Rollout     ॥ Paused       2m56s  
├──# revision:2                                                              
│  └──⧉ rollouts-demo-6cf78c66c5           ReplicaSet  ✔ Healthy      3s     canary
│     └──□ rollouts-demo-6cf78c66c5-tfnsv  Pod         ✔ Running      3s     ready:1/1
└──# revision:1                                                              
   └──⧉ rollouts-demo-687d76d795           ReplicaSet  ✔ Healthy      2m56s  stable
      ├──□ rollouts-demo-687d76d795-2xrbs  Pod         ✔ Running      2m56s  ready:1/1
      ├──□ rollouts-demo-687d76d795-4zfkg  Pod         ◌ Terminating  2m56s  ready:1/1
      ├──□ rollouts-demo-687d76d795-kd5nr  Pod         ✔ Running      2m56s  ready:1/1
      └──□ rollouts-demo-687d76d795-tpfgd  Pod         ✔ Running      2m56s  ready:1/1
NAME                  HOST/PORT                                                                            PATH   SERVICES                                                              PORT   TERMINATION     WILDCARD
rollouts-demo-route   <REDACTED>          argo-rollouts-stable-service(25%),argo-rollouts-canary-service(75%)   http   edge/Redirect   None
Step 4 - fully promoting 
> k argo rollouts get rollout rollouts-demo && k get route rollouts-demo-route
Name:            rollouts-demo
Namespace:       nsalfos-triage
Status:          ✔ Healthy
Strategy:        Canary
  Step:          4/4
  SetWeight:     100
  ActualWeight:  100
Images:          argoproj/rollouts-demo:yellow (stable)
Replicas:
  Desired:       4
  Current:       4
  Updated:       4
  Ready:         4
  Available:     4

NAME                                       KIND        STATUS         AGE  INFO
⟳ rollouts-demo                            Rollout     ✔ Healthy      17m  
├──# revision:2                                                            
│  └──⧉ rollouts-demo-6cf78c66c5           ReplicaSet  ✔ Healthy      14m  stable
│     ├──□ rollouts-demo-6cf78c66c5-tfnsv  Pod         ✔ Running      14m  ready:1/1
│     ├──□ rollouts-demo-6cf78c66c5-cfswh  Pod         ✔ Running      12m  ready:1/1
│     ├──□ rollouts-demo-6cf78c66c5-xhqv8  Pod         ✔ Running      12m  ready:1/1
│     └──□ rollouts-demo-6cf78c66c5-pb2rc  Pod         ✔ Running      7s   ready:1/1
└──# revision:1                                                            
   └──⧉ rollouts-demo-687d76d795           ReplicaSet  • ScaledDown   17m  
      └──□ rollouts-demo-687d76d795-kd5nr  Pod         ◌ Terminating  17m  ready:1/1
NAME                  HOST/PORT                                                                            PATH   SERVICES                                                              PORT   TERMINATION     WILDCARD
rollouts-demo-route    <REDACTED>          argo-rollouts-stable-service(25%),argo-rollouts-canary-service(75%)   http   edge/Redirect   None
New rollout, new route 
> oc create -f ocp-route-plugin/route.yaml 
route.route.openshift.io/rollouts-demo-route created

> k argo rollouts get rollout rollouts-demo; k get route rollouts-demo-route
Error: rollout.argoproj.io "rollouts-demo" not found
NAME                  HOST/PORT                                                                            PATH   SERVICES                                                              PORT   TERMINATION     WILDCARD
rollouts-demo-route   <REDACTED>           argo-rollouts-stable-service(100%),argo-rollouts-canary-service(0%)   http   edge/Redirect   None

> oc create -f ocp-route-plugin/rollout-cr.yaml 
rollout.argoproj.io/rollouts-demo created

> k argo rollouts get rollout rollouts-demo; k get route rollouts-demo-route
Name:            rollouts-demo
Namespace:       nsalfos-triage
Status:          ✔ Healthy
Strategy:        Canary
  Step:          4/4
  SetWeight:     100
  ActualWeight:  100
Images:          argoproj/rollouts-demo:blue (stable)
Replicas:
  Desired:       4
  Current:       4
  Updated:       4
  Ready:         4
  Available:     4

NAME                                       KIND        STATUS     AGE  INFO
⟳ rollouts-demo                            Rollout     ✔ Healthy  3s   
└──# revision:1                                                        
   └──⧉ rollouts-demo-687d76d795           ReplicaSet  ✔ Healthy  3s   stable
      ├──□ rollouts-demo-687d76d795-cbhrh  Pod         ✔ Running  3s   ready:1/1
      ├──□ rollouts-demo-687d76d795-kh4g6  Pod         ✔ Running  3s   ready:1/1
      ├──□ rollouts-demo-687d76d795-v5hwb  Pod         ✔ Running  3s   ready:1/1
      └──□ rollouts-demo-687d76d795-v8k5x  Pod         ✔ Running  3s   ready:1/1
NAME                  HOST/PORT                                                                            PATH   SERVICES                                                              PORT   TERMINATION     WILDCARD
rollouts-demo-route   <REDACTED>           argo-rollouts-stable-service(100%),argo-rollouts-canary-service(0%)   http   edge/Redirect   None

> kubectl argo rollouts set image rollouts-demo \
  rollouts-demo=argoproj/rollouts-demo:yellow
rollout "rollouts-demo" image updated

> k argo rollouts get rollout rollouts-demo; k get route rollouts-demo-route
Name:            rollouts-demo
Namespace:       nsalfos-triage
Status:          ॥ Paused
Message:         CanaryPauseStep
Strategy:        Canary
  Step:          1/4
  SetWeight:     25
  ActualWeight:  25
Images:          argoproj/rollouts-demo:blue (stable)
                 argoproj/rollouts-demo:yellow (canary)
Replicas:
  Desired:       4
  Current:       4
  Updated:       1
  Ready:         4
  Available:     4

NAME                                       KIND        STATUS         AGE  INFO
⟳ rollouts-demo                            Rollout     ॥ Paused       18s  
├──# revision:2                                                            
│  └──⧉ rollouts-demo-6cf78c66c5           ReplicaSet  ✔ Healthy      4s   canary
│     └──□ rollouts-demo-6cf78c66c5-k2pxz  Pod         ✔ Running      3s   ready:1/1
└──# revision:1                                                            
   └──⧉ rollouts-demo-687d76d795           ReplicaSet  ✔ Healthy      18s  stable
      ├──□ rollouts-demo-687d76d795-cbhrh  Pod         ✔ Running      18s  ready:1/1
      ├──□ rollouts-demo-687d76d795-kh4g6  Pod         ✔ Running      18s  ready:1/1
      ├──□ rollouts-demo-687d76d795-v5hwb  Pod         ✔ Running      18s  ready:1/1
      └──□ rollouts-demo-687d76d795-v8k5x  Pod         ◌ Terminating  18s  ready:1/1
NAME                  HOST/PORT                                                                            PATH   SERVICES                                                              PORT   TERMINATION     WILDCARD
rollouts-demo-route   <REDACTED>           argo-rollouts-stable-service(100%),argo-rollouts-canary-service(0%)   http   edge/Redirect   None

> k argo rollouts promote rollouts-demo                                       
rollout 'rollouts-demo' promoted

> k argo rollouts get rollout rollouts-demo; k get route rollouts-demo-route
Name:            rollouts-demo
Namespace:       nsalfos-triage
Status:          ॥ Paused
Message:         CanaryPauseStep
Strategy:        Canary
  Step:          3/4
  SetWeight:     75
  ActualWeight:  75
Images:          argoproj/rollouts-demo:blue (stable)
                 argoproj/rollouts-demo:yellow (canary)
Replicas:
  Desired:       4
  Current:       4
  Updated:       3
  Ready:         4
  Available:     4

NAME                                       KIND        STATUS         AGE    INFO
⟳ rollouts-demo                            Rollout     ॥ Paused       6m42s  
├──# revision:2                                                              
│  └──⧉ rollouts-demo-6cf78c66c5           ReplicaSet  ✔ Healthy      6m28s  canary
│     ├──□ rollouts-demo-6cf78c66c5-k2pxz  Pod         ✔ Running      6m27s  ready:1/1
│     ├──□ rollouts-demo-6cf78c66c5-bwpmq  Pod         ✔ Running      3s     ready:1/1
│     └──□ rollouts-demo-6cf78c66c5-rvt2f  Pod         ✔ Running      3s     ready:1/1
└──# revision:1                                                              
   └──⧉ rollouts-demo-687d76d795           ReplicaSet  ✔ Healthy      6m42s  stable
      ├──□ rollouts-demo-687d76d795-cbhrh  Pod         ✔ Running      6m42s  ready:1/1
      ├──□ rollouts-demo-687d76d795-kh4g6  Pod         ◌ Terminating  6m42s  ready:1/1
      └──□ rollouts-demo-687d76d795-v5hwb  Pod         ◌ Terminating  6m42s  ready:1/1
NAME                  HOST/PORT                                                                            PATH   SERVICES                                                              PORT   TERMINATION     WILDCARD
rollouts-demo-route   <REDACTED>           argo-rollouts-stable-service(100%),argo-rollouts-canary-service(0%)   http   edge/Redirect   None

> k argo rollouts get rollout rollouts-demo; k get route rollouts-demo-route
Name:            rollouts-demo
Namespace:       nsalfos-triage
Status:          ✔ Healthy
Strategy:        Canary
  Step:          4/4
  SetWeight:     100
  ActualWeight:  100
Images:          argoproj/rollouts-demo:yellow (stable)
Replicas:
  Desired:       4
  Current:       4
  Updated:       4
  Ready:         4
  Available:     4

NAME                                       KIND        STATUS         AGE    INFO
⟳ rollouts-demo                            Rollout     ✔ Healthy      6m52s  
├──# revision:2                                                              
│  └──⧉ rollouts-demo-6cf78c66c5           ReplicaSet  ✔ Healthy      6m38s  stable
│     ├──□ rollouts-demo-6cf78c66c5-k2pxz  Pod         ✔ Running      6m37s  ready:1/1
│     ├──□ rollouts-demo-6cf78c66c5-bwpmq  Pod         ✔ Running      13s    ready:1/1
│     ├──□ rollouts-demo-6cf78c66c5-rvt2f  Pod         ✔ Running      13s    ready:1/1
│     └──□ rollouts-demo-6cf78c66c5-pp5gp  Pod         ✔ Running      2s     ready:1/1
└──# revision:1                                                              
   └──⧉ rollouts-demo-687d76d795           ReplicaSet  • ScaledDown   6m52s  
      └──□ rollouts-demo-687d76d795-cbhrh  Pod         ◌ Terminating  6m52s  ready:1/1
NAME                  HOST/PORT                                                                            PATH   SERVICES                                                              PORT   TERMINATION     WILDCARD
rollouts-demo-route   <REDACTED>           argo-rollouts-stable-service(100%),argo-rollouts-canary-service(0%)   http   edge/Redirect   None
@nsalfos
Copy link
Author

nsalfos commented Nov 6, 2024

anyone around?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@nsalfos