Diego v2.37.0

Resources

• Download release v2.37.0 from bosh.io.
• Verified with cloudfoundry/cf-deployment @ 5d9063c5b93b6e928fab7e58c0efecc9b369bf81.

Changes from v2.36.0 to v2.37.0

Significant changes

Component Coordination

• As a CF operator, I expect my diego-cell deployment to succeed consistently through various startup orders of the services on my cell (bosh-dns/route-emitter) and invalid secondary DNS servers

Cell Capacity Reporting & LRP Placement

• As a CF operator, I do not want the advertised disk capacity of my foundation's cells to be greater than what is actually usable by garden on those cells so I can be sure apps pushed to the foundation do not fail due to the cell running out of actual disk space

Local Route Emitters

• As a Diego operator, I expect that the communication between the route emitter and routing API to be over mTLS so I can be sure of the security of my foundation

Volume Support

• remove CSI support from Diego
• cloudfoundry/diego-upgrade-stability-tests #1: remove CSI support from Diego
• cloudfoundry/executor #47: remove CSI support from Diego
• cloudfoundry/inigo #21: remove CSI support from Diego
• cloudfoundry/rep #29: remove CSI support from Diego
• cloudfoundry/diego-release #437: remove CSI support from Diego
• update troubleshooting guide for volume services

Per-Instance Proxy

• Envoy proxy binary bumped to 882a30677619856446f7e1b9d28c6ab319b21d1b

App Logging and Metrics

• cloudfoundry/executor #30: fix unexpected container CPU usage figure

Component Logging and Metrics

• As a CF operator, I expect not to see the auctioneer handle an empty placement request when a BBS API client creates a 0-instance DesiredLRP so that I am not confused by extraneous Diego component activity
• As a Diego operator, I expect to be able to consume key performance indicator metric visualizations for the BBS so I do not have to build my own dashboards and can leverage the Diego team's operational expertise
• As a Diego operator, I expect to be able to consume key performance indicator metric visualizations for the rep so I do not have to build my own dashboards and can leverage the Diego team's operational expertise
• As a Diego operator, I expect the indicator document for the auctioneer to include a metric that indicates if the auctioneer lock is held so I can be sure my foundation is healthy
• Audit runtime metrics being emitted by our components and clean up Diego release docs for those that no longer exist

Dependencies

• As a CF operator, I expect Diego components to use Golang 1.12.9 so that they are up to date with this important dependency
• Bump grpc go to v1.23.0

Test Suites and Tooling

• Remove error ignoring in vizzini that was bypassing GCP load balancer error
• cloudfoundry/inigo #21: remove CSI support from Diego

Security

• As a Diego operator, I expect that the communication between the route emitter and routing API to be over mTLS so I can be sure of the security of my foundation

BOSH property changes

rep and rep_windows

• The following properties have been deleted from both rep and rep_windows to remove CSI support from Diego
  • diego.executor.volman.csi_paths
  • diego.executor.volman.csi_mount_root_dir
• The following property has been added to both rep and rep_windows
  • diego.executor.use_schedulable_disk_size

Diego v2.36.0

Resources

• Download release v2.36.0 from bosh.io.
• Verified with cloudfoundry/cf-deployment @ 244c6ef40dbab44d2e98ad17dd5256a311e224a9.

Changes from v2.35.0 to v2.36.0

Significant changes

cfdot

• As a CF operator, I expect to understand that the ALG-based commands and events will be removed from cfdot no earlier than Diego v3.0.0

Routing

• repeating route deregistration messages causing errors for cf push/delete/push routine

Garden OCI Image Support (Experimental)

• As a CF Operator, I expect to be able to enable OCI mode on cells in my foundation and see that app instance disk usage includes the app droplet, so I can be confident my platform resources are being correctly accounted for

Per-Instance Proxy

• Envoy proxy binary bumped to 7b0ce0d32a9b584626e8c16b5ae07817eade322d

Buildpack Support

• Lifecycle can accept process types in launch.yml and bin/supply
• cloudfoundry/buildpackapplifecycle #38: Large credhub interpolation value causes app to not stage or start

Windows Support

• Refactor inigo and develop setup scripts to get instance identity tests working on Windows

Component Logging and Metrics

• As a CF Operator, I want to see a metric on how many app instances are exceeding the 10 second graceful shutdown interval when they are being stopped so that I can understand how the current 10 second global value is impacting app consumer experience
• As a Diego operator, I expect to be able to consume key performance indicator metric visualizations for the auctioneer so I do not have to build my own dashboards and can leverage the Diego team's operational expertise
• As a Diego operator, I expect to be able to consume key performance indicator metric visualizations for the route-emitter so I do not have to build my own dashboards and can leverage the Diego team's operational expertise

Dependencies

• Bump routing-release dependencies in diego-release to stay up to date

Test Suites and Tooling

• Fix electron-cannon vizzini panic failure
• investigate why warp drive windows canary app experienced downtime during deploy
• investigate vizzini StartTimeout test failures

Cleanup

• As a CF operator, I expect the ssh-proxy and any other Diego components to use the new "flattened" ActualLRP API endpoints and associated instance-specific event types instead of ActualLRPGroup-based ones so that they will continue to function after their future removal

Sidecar Support

• As a BBS client, I expect to be able to create and fetch DesiredLRPs with a "sidecar" definition
• As a Diego operator, I expect when I desire an LRP with a sidecar definition, the sidecar action is run in my LRP container

BOSH job changes

BOSH property changes

None.

BOSH link changes

None.

Diego v2.35.0

Resources

• Download release v2.35.0 from bosh.io.
• Verified with cloudfoundry/cf-deployment @ 89a2bf0bdf53c9a862ac489ec361215bb8e40ce6.

Changes from v2.34.0 to v2.35.0

Significant changes

Component Coordination

• cloudfoundry/diego-release #432: rep prestart hook failed: /proc/sys/net/netfilter/nf_conntrack_max: No such file or directory
  • Previously we relied on pre-start scripts from other jobs to load the nf_conntrack linux module.
    Now we enable nf_conntrack module to assure its available before we update its value.

Routing

• As a CF operator, when I enable route integrity (on Windows and Linux) I expect that when I update my cells with a new compiled package, the update completes successfully within a reasonable amount of time

Per-Instance Proxy

• Envoy proxy binary bumped to c0a1ded969095e2989bac70f0f4637ed4c42ffb1

Test Suites and Tooling

• As a Diego contributor, I expect to be able to configure the vizzini errand with a preloaded rootfs that is included in my Diego deployment so that I do not need to change the code in diego-release to run vizzini against environments with different configurations
• Investigate bbs migrations tests data race

Security

• As a CF operator, I expect that cloud controller generates https urls to file-server assets via a BOSH link from the file-server so I can be sure communication paths in my foundation are secure
• cloudfoundry/locket #9: Cert expiry or misconfiguration can cause useless error messages

Miscellany

• cloudfoundry/bytefmt #23: Add support for Petabyte and Exabyte
• Add deprecation notice for cfhttp v1 package

BOSH property changes

file_server

+ https_url:
+ description: "The URL provided in file_server link"
+ default: "https://file-server.service.cf.internal:8443"

vizzini

+ default_rootfs:
+ description: "Default preloaded rootfs to target for running Tasks and LRPs"
+ default: "preloaded:cflinuxfs3"

Diego v2.34.0

Resources

• Download release v2.34.0 from bosh.io.
• Verified with cloudfoundry/cf-deployment @ 2311d5e1311a5b28884f50b6e68e2104a3fabcd1.

Changes from v2.33.0 to v2.34.0

Significant changes

BBS API

• cloudfoundry/bbs #40: marking checkdefinition not experimental

Local Route Emitters

• As an app developer on a foundation with route integrity enabled, I expect that if I un-map a route to my app, my app is no longer accessible at that route, even in the case of NATS routing tier instability, so I can be confident requests to my app are not misrouted

Per-Instance Proxy

• Envoy proxy binary bumped to ad57ed8511b636869afb3eef3c21b52890d71890

Windows Support

• As a CF Operator, I would like diego-cell evacuation cleanup to delete containers instead of only stopping them so that my bosh deploy updates to foundations with only one windows-cell and >= 1 windows app running will not fail.

Component Logging and Metrics

• As a CF operator, I expect the BBS stderr output not to contain "http: multiple response.WriteHeader calls" lines so that I am not concerned about these BBS errors
• As a CF operator, I expect each cell rep's capacity metrics to be tagged with its availability zone so that I can easily aggregate CF workload capacity and allocations by AZ

Diego v2.33.0

Changes from v2.32.0 to v2.33.0

Per-Instance Proxy

• Envoy proxy binary bumped to 54e22406c38987d5b0401f18b614501bc184f41f

App Logging and Metrics

• As an App Developer I want my app logs to include information identifying the org and space to which my app belongs so I can filter/analyze the logs for my app by org/space in downstream systems
• As an app operator, I want to be able to distinguish the http stop/start metrics associated with different versions of my app so I can make scaling decisions

Test Suites and Tooling

• cloudfoundry/diego-release #430: Introduce a ifrit startcheck timeout configuration

Security

• As a CF operator, I expect that assets (app lifecycle bundles) from the Diego file-server that are put into application/task containers are downloaded with TLS so I can be confident about the integrity of those assets

BOSH property changes

file_server

• https_server_enabled (NEW)
  • description: "Use HTTPS for serving file_server assets"
  • default: false
• https_listen_addr (NEW)
  • description: "Address at which HTTPS server is listening"
  • default: "0.0.0.0:8443"
• tls.cert (NEW)
  • description: "PEM-encoded tls certificate that can be used for server auth"
• tls.key (NEW)
  • description: "PEM-encoded tls key"

Diego v2.32.0

Resources

• Download release v2.32.0 from bosh.io.
• Verified with cloudfoundry/cf-deployment @ b8e2976f7340db0bfa5180cc1382c9e968f7d431.

Changes from v2.31.0 to v2.32.0

Significant changes

BBS API

• cloudfoundry/diego-release #428: BBS Crashing and Failing over
  • updated how bbs handles context cancellations to eliminate concurrent map read/writes panic's which could cause the BBS to crash/failover.

LRP Convergence

• As a Diego operator, I expect the BBS to be able to run convergence successfully on desired LRPs that have > 283 instances when my backing store is MySQL so that I do not have extra error messages in my BBS logs and do not have undesired ActualLRPs running

SSH

• cloudfoundry/route-emitter #14: Enable cf ssh for Windows+Linux with 3rd Party Network Plugins + bbs #166247777 + diego-ssh #166247920 + executor #166247994 + rep #166248025
  • A new property on the Rep job, advertise_preference_for_instance_address allows the ability to cf ssh to apps on heterogenous foundations (Linux and Windows Diego cells) when third-party container network interface (CNI) plugins are present. Previously on such environments, cf ssh would work to only Windows OR Linux apps, but not both.
  • During your upgrade deployment to a heterogeneous foundation which includes a third-party container network interface (CNI) plugin:
    • cf ssh to Linux apps should continue to work although there may be short outage as the instance groups containing the ssh_proxy and rep jobs begin to roll
    • cf ssh to Windows apps should start working once the windows diego-cells begin to roll
    • There should be no app availability downtime for either Windows or Linux apps during the upgrade
  • During your upgrade deployment to a foundation which doesn't include a third party container network interface (CNI) plugin:
    • cf ssh to both Linux AND Windows apps should continue to work although there may be short outage as the instance groups containing the ssh_proxy and rep jobs begin to roll
    • There should be no app availability downtime for either Windows or Linux apps during the upgrade

Routing

• Bump routing-api in diego-release (currently ref's a SHA from 2017)

Per-Instance Proxy

• Envoy proxy binary bumped to 619dcbe469d17114ab5c9b079f46866f0f5ce296

Windows Support

• As a CF operator, I expect that Windows Diego cell reps deployed to Azure configure their zone based on the assigned Azure Fault Domain or Azure Availability zone when I opt-in to doing so, so that the auctioneer can make appropriate LRP instance placements

App Logging and Metrics

• As a Diego operator, I expect to be able to consume key performance indicator metric visualizations for Locket so I do not have to build my own dashboards and can leverage the Diego team's operational expertise

Component Logging and Metrics

• Investigate high log volume being sent to logit.io
  • In an effort to minimize the number of logs under Info, we removed and re-arranged a few places where we believe it's unneccessary executor / 0dc5df0

Test Suites and Tooling

• cloudfoundry/diego-release #430: Introduce a ifrit startcheck timeout configuration

BOSH property changes

rep and rep_windows

• Added advertise_preference_for_instance_address in developing against the "Enable cf ssh for Windows+Linux with 3rd Party Network Plugins" feature stories described in the SSH section of these release notes (default value: false)

Diego v2.31.0

Resources

• Download release v2.31.0 from bosh.io.
• Verified with cloudfoundry/cf-deployment @ 6c760d8720a18c30946df7f8cbef5c150455edfd.

Changes from v2.30.0 to v2.31.0

Significant changes

BBS API

• As a BBS API client, I expect the SHA-512 digest algorithm to be deprecated in the image-layer spec so that I avoid adopting it accidentally before its future removal
• As a BBS API client, I expect to understand that the ActualLRPGroup message and associated BBS API endpoints are deprecated and will be removed no earlier than Diego v4.0.0

BBS Benchmarks

• Fixed failing BBS Benchmark Build

Performance Tuning

• As a CF operator, I expect that locket should release idle database connections so that it does not take an outsize share of the shared CF database resources

Local Route Emitters

• As a CF operator, I expect the Diego route-emitter to use the new "flattened" ActualLRP API endpoints and associated instance-specific event types instead of ActualLRPGroup-based ones so that it will continue to function after their future removal
  • Investigate and resolve an increase in 502s that was introduced by the above change (prior to release) from canary app on warp-drive and electron-cannon

Container Networking Support

• As a Diego operator, I would like the executor's garden healthcheck containers to have shorter distinct container handles so that their egress-rule iptables chains are unlikely to collide (take 2)

Per-Instance Proxy

• As a CF operator, I expect not to see spurious error messages about proxy config directory deletions in the rep logs so that I am not distracted or concerned about behavior that is actually normal
• Bumps Envoy proxy binary to 78ad883b70764c27f8b391ee3a5056a64b403426

Component Logging and Metrics

• Investigate benchmark bbs failure
  • Add DBWaitCount and DBWaitDuration metrics for BBS and Locket
• As a Diego operator, I expect to observe a metric that indicates when a Diego cell has stopped accepting work specifically because Garden is not functioning correctly so that I can diagnose the Garden malfunction
  • UnhealthyCell metric deprecated in favor of GardenHealthCheckFailed.

Test Suites and Tooling

• Remove vizzini rootfs tests because it is redundant
• Change references to cflinuxfs2 in diego-release to cflinuxfs3
• validate that DUSTs can be run from a workstation locally
• rewrite vizzini download-cancellation test because it is not testing what it should be

Documentation

• clean up diego-release contributing docs, remove obsolete references (to cf-release, go versions, etc.)

BOSH job changes

• NONE

BOSH property changes

• NONE

BOSH link changes

• NONE

Diego v2.30.0

Resources

• Download release v2.30.0 from bosh.io.
• Verified with cloudfoundry/cf-deployment @ 8c741136d25727eb5d3fd34a034d67b84e1b7f42.

Changes from v2.29.0 to v2.30.0

Significant changes

Executor Cache

• look into relint's Windows cell logs for cell that filled up its disk
  • Fixed bug to ensure temporary files are removed when any part of a file upload step fails

Per-Instance Proxy

• cloudfoundry/executor #44: Temporarily disable stats logging in envoy to enable scaling goals
• Bumps Envoy proxy binary to 8b713b324c02f31784d4b3603c68b77c9e62b371

Azure Support

• As a CF operator on Azure, I expect that when I configure the rep to use Azure fault domains and I configure my BOSH AZs to point to Azure availability zones so that I do not experience application downtime when an AZ goes down

Dependencies

• As a CF operator, I expect Diego components to use Golang 1.12.1 so that they are up to date with this important dependency
• cloudfoundry/diego-release #421: license of yaml.v2 contained in Diego V2.1.0
  • Bumps docker/distribution to v2.7.1

Test Suites and Tooling

• DUSTs upgrade-acceptance-tests-from-v1.0.0 continually flaky on: Crashes with no monitor action when running several actions codependently when lot of subprocesses fail the crash count is incremented

Cleanup

• eliminate unused code in diego-release

BOSH job changes

None.

BOSH property changes

None.

BOSH link changes

None.

Diego v2.29.0

Resources

• Download release v2.29.0 from bosh.io.
• Verified with cloudfoundry/cf-deployment @ fa34f6a4c787dbaedada7f9623cc5121aefd0b9f.

Changes from v2.28.0 to v2.29.0

Significant changes

BBS API

• As a BBS API client, I expect the BBS API protobuf message definitions to migrate to v3 syntax so that I can more easily generate language-specific client code

Executor Cache

• look at electron-cannon download cache orphaned temp files
  • temporary files from failed cacheddownloader downloads and transformations are now properly deleted

SSH

• As a CF operator, I expect to be able to disable the ssh-proxy HTTP healthcheck server

Per-Instance Proxy

• cloudfoundry/executor #43: deprecated --v2-config-only flag breaks Envoy on Windows
• Bump Envoy proxy blob to 0526e80244406b30727ab6b303d17899ad36d86f

Docker Support

• As a CF Docker-image app developer, I expect Docker image staging to request schema version 2 image manifests so that I can stage images from Docker registries that support only this version of manifests

Component Logging and Metrics

• As a Diego operator, I expect to see request count and latency metrics for the cell rep workload API endpoints so that I can distinguish which cells are behaving anomalously

Dependencies

• cloudfoundry/cfhttp #5: user can customize dial timeout
• Diego team should take over ownership of cloudfoundry/tlsconfig GitHub repo
• Use cfhttp v2 in all diego components and backfill tests to assert HTTP client timeout configuration is respected

Test Suites and Tooling

• Regenerate all CI certs

Cleanup

• eliminate unused code in diego-release

BOSH job changes

None.

BOSH property changes

ssh_proxy

• Added diego.ssh_proxy.disable_healthcheck_server: Whether to disable the ssh_proxy HTTP healthcheck server. Defaults to false.

BOSH link changes

None.

Diego v2.28.0

Resources

• Download release v2.28.0 from bosh.io.
• Verified with cloudfoundry/cf-deployment @ 80a3786c19489efe4feb16af2fe65ae84a73b48a.

Changes from v2.27.0 to v2.28.0

Significant changes

Dependencies

• As a CF operator, I expect Diego components to use Golang 1.11.5 so that they are up to date with this important dependency

Test Suites and Tooling

• fix route-emitter main test flake

BOSH job changes

None.

BOSH property changes

None.

BOSH link changes

None.