Diego v2.40.0

Resources

• Download release v2.40.0 from bosh.io.
• Verified with cloudfoundry/cf-deployment @ bf96d52f8823c48ef3124cda4a8a93e86e27fa33.

Changes from v2.39.0 to v2.40.0

Significant changes

Component Coordination

• cloudfoundry/diego-release #429: Make BBS Timeout values configurable in the REP

CF Tasks

• As a Diego user, I expect the rep to actually stream out the full task result file when completing a task so I get accurate task results

Local Route Emitters

• Remove the cfroutes package from route-emitter

Per-Instance Proxy

• Envoy proxy binary bumped to 1e3e792a45236b8dc29db5f463a08f9a9928ab2c

Dependencies

• As a CF operator, I expect Diego components to use Golang 1.13.3 so that they are up to date with this important dependency

BOSH property changes

rep and rep_windows

• diego.rep*.bbs.request_timeout adds ability to configure the request timeout from rep to bbs (default: 10s)

vizzini

• vizzini.file_server.address adds ability to configure address of the file_server (default: file-server.service.cf.internal:8080)

Diego v2.39.0

Resources

• Download release v2.39.0 from bosh.io.
• Verified with cloudfoundry/cf-deployment @ f9c0a5fd2eaa8ec2d06e8cf533f469015262c7d5.

Changes from v2.38.0 to v2.39.0

Significant changes

Local Route Emitters

• As a Diego operator, I expect the route-emitter to use the preferred address from ActualLRPs over the configured mode for advertising instance addresses

Per-Instance Proxy

• Envoy proxy binary bumped to 4478c1984d17146b1ff78d0babedae2a4752b027

App Logging and Metrics

• cloudfoundry/diego-logging-client #7: Send container CPU usage spike metric
• cloudfoundry/executor #49: Let's Emit a CPU Usage Spike Metric

Documentation

• clean up/update envoy configuration doc and properly describe envoy sidecar isolation/resources

BOSH property changes

auctioneer

• bpm.enabled is no longer experimental.

bbs

• The following properties are no longer experimental:
  • bpm.enabled
  • tasks.max_retries

file_server

• bpm.enabled is no longer experimental.

locket

• bpm.enabled is no longer experimental.

rep and rep_windows

• The following spec properties are no longer experimental:
  • bpm.enabled
  • diego.executor.volman.driver_paths - (property removed from rep_windows)
  • containers.graceful_shutdown_interval_in_seconds
  • containers.proxy.require_and_verify_client_certificates
  • containers.proxy.trusted_ca_certificates
  • containers.proxy.verify_subject_alt_name

route_emitter and route_emitter_windows

• bpm.enabled is no longer experimental.

ssh_proxy

• bpm.enabled is no longer experimental.

vizzini

• The following properties are no longer experimental:
  • enable_declarative_healthcheck
  • max_task_retries
  • enable_container_proxy_tests
  • vizzini.container_proxy.ca
  • vizzini.container_proxy.client_cert
  • vizzini.container_proxy.client_key

Diego v2.38.0

Changes from v2.37.0 to v2.38.0

Significant changes

Component Coordination

• As a application developer I want the system to handle the transfer of staging results larger than 10k to the cloud controller so my applications which generate larger staging results can successfully stage and subsequently start/run on the foundation
  • A recent bump to Buildpack Application Lifecycle can cause task result files to be larger than 10K
    (e.g. java apps). We are bumping the MAX_RESULT_SIZE to 20K to address
    the recent change.

Per-Instance Proxy

• Envoy proxy binary bumped to 4478c1984d17146b1ff78d0babedae2a4752b027

Docker/Image Support

• As a CF app developer, I expect to be able to push Docker apps that are hosted on AWS ECR and that they continue to run when restarted, crashed, or evacuated after the typical AWS ECR credential expiration period
  • App developers who wish to push apps based on images from AWS ECR should set their CF_DOCKER_PASSWORD env variable to the AWS Secret Access Key for the IAM user and pass the AWS Access Key ID for the IAM user as the docker-username in their cf push... command:
    • cf push [appname] --docker-image [repo/ECR-container-image-name] --docker-username [aws-access-key-id]

Component Logging and Metrics

• As a Diego Operator, I can observe the auctioneer logs and see what the auctioneer was trying to place when there was a placement failure so I can better diagnose the root cause of that placement failure
• As a 3rd party network plugin author, I expect my component to be able to tell what containers are internal system containers and don't require networking setup
• Log an error for slow readers on BBS

Test Suites and Tooling

• regenerate-certs.sh under ./rep/cmd/rep/fixtures should regenerate all of the fixture certs
• Flaky Test Crashes with a monitor action when the monitor never succeeds when the process dies with exit code 1 [It] gets marked as crashed (immediately)
• Failing Benchmark BBS Build - mitigate test flakiness
• Failing Benchmark BBS Build - Set the buffer on receiving channel for BBS events

Diego v2.37.0

Resources

• Download release v2.37.0 from bosh.io.
• Verified with cloudfoundry/cf-deployment @ 5d9063c5b93b6e928fab7e58c0efecc9b369bf81.

Changes from v2.36.0 to v2.37.0

Significant changes

Component Coordination

• As a CF operator, I expect my diego-cell deployment to succeed consistently through various startup orders of the services on my cell (bosh-dns/route-emitter) and invalid secondary DNS servers

Cell Capacity Reporting & LRP Placement

• As a CF operator, I do not want the advertised disk capacity of my foundation's cells to be greater than what is actually usable by garden on those cells so I can be sure apps pushed to the foundation do not fail due to the cell running out of actual disk space

Local Route Emitters

• As a Diego operator, I expect that the communication between the route emitter and routing API to be over mTLS so I can be sure of the security of my foundation

Volume Support

• remove CSI support from Diego
• cloudfoundry/diego-upgrade-stability-tests #1: remove CSI support from Diego
• cloudfoundry/executor #47: remove CSI support from Diego
• cloudfoundry/inigo #21: remove CSI support from Diego
• cloudfoundry/rep #29: remove CSI support from Diego
• cloudfoundry/diego-release #437: remove CSI support from Diego
• update troubleshooting guide for volume services

Per-Instance Proxy

• Envoy proxy binary bumped to 882a30677619856446f7e1b9d28c6ab319b21d1b

App Logging and Metrics

• cloudfoundry/executor #30: fix unexpected container CPU usage figure

Component Logging and Metrics

• As a CF operator, I expect not to see the auctioneer handle an empty placement request when a BBS API client creates a 0-instance DesiredLRP so that I am not confused by extraneous Diego component activity
• As a Diego operator, I expect to be able to consume key performance indicator metric visualizations for the BBS so I do not have to build my own dashboards and can leverage the Diego team's operational expertise
• As a Diego operator, I expect to be able to consume key performance indicator metric visualizations for the rep so I do not have to build my own dashboards and can leverage the Diego team's operational expertise
• As a Diego operator, I expect the indicator document for the auctioneer to include a metric that indicates if the auctioneer lock is held so I can be sure my foundation is healthy
• Audit runtime metrics being emitted by our components and clean up Diego release docs for those that no longer exist

Dependencies

• As a CF operator, I expect Diego components to use Golang 1.12.9 so that they are up to date with this important dependency
• Bump grpc go to v1.23.0

Test Suites and Tooling

• Remove error ignoring in vizzini that was bypassing GCP load balancer error
• cloudfoundry/inigo #21: remove CSI support from Diego

Security

• As a Diego operator, I expect that the communication between the route emitter and routing API to be over mTLS so I can be sure of the security of my foundation

BOSH property changes

rep and rep_windows

• The following properties have been deleted from both rep and rep_windows to remove CSI support from Diego
  • diego.executor.volman.csi_paths
  • diego.executor.volman.csi_mount_root_dir
• The following property has been added to both rep and rep_windows
  • diego.executor.use_schedulable_disk_size

Diego v2.36.0

Resources

• Download release v2.36.0 from bosh.io.
• Verified with cloudfoundry/cf-deployment @ 244c6ef40dbab44d2e98ad17dd5256a311e224a9.

Changes from v2.35.0 to v2.36.0

Significant changes

cfdot

• As a CF operator, I expect to understand that the ALG-based commands and events will be removed from cfdot no earlier than Diego v3.0.0

Routing

• repeating route deregistration messages causing errors for cf push/delete/push routine

Garden OCI Image Support (Experimental)

• As a CF Operator, I expect to be able to enable OCI mode on cells in my foundation and see that app instance disk usage includes the app droplet, so I can be confident my platform resources are being correctly accounted for

Per-Instance Proxy

• Envoy proxy binary bumped to 7b0ce0d32a9b584626e8c16b5ae07817eade322d

Buildpack Support

• Lifecycle can accept process types in launch.yml and bin/supply
• cloudfoundry/buildpackapplifecycle #38: Large credhub interpolation value causes app to not stage or start

Windows Support

• Refactor inigo and develop setup scripts to get instance identity tests working on Windows

Component Logging and Metrics

• As a CF Operator, I want to see a metric on how many app instances are exceeding the 10 second graceful shutdown interval when they are being stopped so that I can understand how the current 10 second global value is impacting app consumer experience
• As a Diego operator, I expect to be able to consume key performance indicator metric visualizations for the auctioneer so I do not have to build my own dashboards and can leverage the Diego team's operational expertise
• As a Diego operator, I expect to be able to consume key performance indicator metric visualizations for the route-emitter so I do not have to build my own dashboards and can leverage the Diego team's operational expertise

Dependencies

• Bump routing-release dependencies in diego-release to stay up to date

Test Suites and Tooling

• Fix electron-cannon vizzini panic failure
• investigate why warp drive windows canary app experienced downtime during deploy
• investigate vizzini StartTimeout test failures

Cleanup

• As a CF operator, I expect the ssh-proxy and any other Diego components to use the new "flattened" ActualLRP API endpoints and associated instance-specific event types instead of ActualLRPGroup-based ones so that they will continue to function after their future removal

Sidecar Support

• As a BBS client, I expect to be able to create and fetch DesiredLRPs with a "sidecar" definition
• As a Diego operator, I expect when I desire an LRP with a sidecar definition, the sidecar action is run in my LRP container

BOSH job changes

BOSH property changes

None.

BOSH link changes

None.

Diego v2.35.0

Resources

• Download release v2.35.0 from bosh.io.
• Verified with cloudfoundry/cf-deployment @ 89a2bf0bdf53c9a862ac489ec361215bb8e40ce6.

Changes from v2.34.0 to v2.35.0

Significant changes

Component Coordination

• cloudfoundry/diego-release #432: rep prestart hook failed: /proc/sys/net/netfilter/nf_conntrack_max: No such file or directory
  • Previously we relied on pre-start scripts from other jobs to load the nf_conntrack linux module.
    Now we enable nf_conntrack module to assure its available before we update its value.

Routing

• As a CF operator, when I enable route integrity (on Windows and Linux) I expect that when I update my cells with a new compiled package, the update completes successfully within a reasonable amount of time

Per-Instance Proxy

• Envoy proxy binary bumped to c0a1ded969095e2989bac70f0f4637ed4c42ffb1

Test Suites and Tooling

• As a Diego contributor, I expect to be able to configure the vizzini errand with a preloaded rootfs that is included in my Diego deployment so that I do not need to change the code in diego-release to run vizzini against environments with different configurations
• Investigate bbs migrations tests data race

Security

• As a CF operator, I expect that cloud controller generates https urls to file-server assets via a BOSH link from the file-server so I can be sure communication paths in my foundation are secure
• cloudfoundry/locket #9: Cert expiry or misconfiguration can cause useless error messages

Miscellany

• cloudfoundry/bytefmt #23: Add support for Petabyte and Exabyte
• Add deprecation notice for cfhttp v1 package

BOSH property changes

file_server

+ https_url:
+ description: "The URL provided in file_server link"
+ default: "https://file-server.service.cf.internal:8443"

vizzini

+ default_rootfs:
+ description: "Default preloaded rootfs to target for running Tasks and LRPs"
+ default: "preloaded:cflinuxfs3"

Diego v2.34.0

Resources

• Download release v2.34.0 from bosh.io.
• Verified with cloudfoundry/cf-deployment @ 2311d5e1311a5b28884f50b6e68e2104a3fabcd1.

Changes from v2.33.0 to v2.34.0

Significant changes

BBS API

• cloudfoundry/bbs #40: marking checkdefinition not experimental

Local Route Emitters

• As an app developer on a foundation with route integrity enabled, I expect that if I un-map a route to my app, my app is no longer accessible at that route, even in the case of NATS routing tier instability, so I can be confident requests to my app are not misrouted

Per-Instance Proxy

• Envoy proxy binary bumped to ad57ed8511b636869afb3eef3c21b52890d71890

Windows Support

• As a CF Operator, I would like diego-cell evacuation cleanup to delete containers instead of only stopping them so that my bosh deploy updates to foundations with only one windows-cell and >= 1 windows app running will not fail.

Component Logging and Metrics

• As a CF operator, I expect the BBS stderr output not to contain "http: multiple response.WriteHeader calls" lines so that I am not concerned about these BBS errors
• As a CF operator, I expect each cell rep's capacity metrics to be tagged with its availability zone so that I can easily aggregate CF workload capacity and allocations by AZ

Diego v2.33.0

Changes from v2.32.0 to v2.33.0

Per-Instance Proxy

• Envoy proxy binary bumped to 54e22406c38987d5b0401f18b614501bc184f41f

App Logging and Metrics

• As an App Developer I want my app logs to include information identifying the org and space to which my app belongs so I can filter/analyze the logs for my app by org/space in downstream systems
• As an app operator, I want to be able to distinguish the http stop/start metrics associated with different versions of my app so I can make scaling decisions

Test Suites and Tooling

• cloudfoundry/diego-release #430: Introduce a ifrit startcheck timeout configuration

Security

• As a CF operator, I expect that assets (app lifecycle bundles) from the Diego file-server that are put into application/task containers are downloaded with TLS so I can be confident about the integrity of those assets

BOSH property changes

file_server

• https_server_enabled (NEW)
  • description: "Use HTTPS for serving file_server assets"
  • default: false
• https_listen_addr (NEW)
  • description: "Address at which HTTPS server is listening"
  • default: "0.0.0.0:8443"
• tls.cert (NEW)
  • description: "PEM-encoded tls certificate that can be used for server auth"
• tls.key (NEW)
  • description: "PEM-encoded tls key"

Diego v2.32.0

Resources

• Download release v2.32.0 from bosh.io.
• Verified with cloudfoundry/cf-deployment @ b8e2976f7340db0bfa5180cc1382c9e968f7d431.

Changes from v2.31.0 to v2.32.0

Significant changes

BBS API

• cloudfoundry/diego-release #428: BBS Crashing and Failing over
  • updated how bbs handles context cancellations to eliminate concurrent map read/writes panic's which could cause the BBS to crash/failover.

LRP Convergence

• As a Diego operator, I expect the BBS to be able to run convergence successfully on desired LRPs that have > 283 instances when my backing store is MySQL so that I do not have extra error messages in my BBS logs and do not have undesired ActualLRPs running

SSH

• cloudfoundry/route-emitter #14: Enable cf ssh for Windows+Linux with 3rd Party Network Plugins + bbs #166247777 + diego-ssh #166247920 + executor #166247994 + rep #166248025
  • A new property on the Rep job, advertise_preference_for_instance_address allows the ability to cf ssh to apps on heterogenous foundations (Linux and Windows Diego cells) when third-party container network interface (CNI) plugins are present. Previously on such environments, cf ssh would work to only Windows OR Linux apps, but not both.
  • During your upgrade deployment to a heterogeneous foundation which includes a third-party container network interface (CNI) plugin:
    • cf ssh to Linux apps should continue to work although there may be short outage as the instance groups containing the ssh_proxy and rep jobs begin to roll
    • cf ssh to Windows apps should start working once the windows diego-cells begin to roll
    • There should be no app availability downtime for either Windows or Linux apps during the upgrade
  • During your upgrade deployment to a foundation which doesn't include a third party container network interface (CNI) plugin:
    • cf ssh to both Linux AND Windows apps should continue to work although there may be short outage as the instance groups containing the ssh_proxy and rep jobs begin to roll
    • There should be no app availability downtime for either Windows or Linux apps during the upgrade

Routing

• Bump routing-api in diego-release (currently ref's a SHA from 2017)

Per-Instance Proxy

• Envoy proxy binary bumped to 619dcbe469d17114ab5c9b079f46866f0f5ce296

Windows Support

• As a CF operator, I expect that Windows Diego cell reps deployed to Azure configure their zone based on the assigned Azure Fault Domain or Azure Availability zone when I opt-in to doing so, so that the auctioneer can make appropriate LRP instance placements

App Logging and Metrics

• As a Diego operator, I expect to be able to consume key performance indicator metric visualizations for Locket so I do not have to build my own dashboards and can leverage the Diego team's operational expertise

Component Logging and Metrics

• Investigate high log volume being sent to logit.io
  • In an effort to minimize the number of logs under Info, we removed and re-arranged a few places where we believe it's unneccessary executor / 0dc5df0

Test Suites and Tooling

• cloudfoundry/diego-release #430: Introduce a ifrit startcheck timeout configuration

BOSH property changes

rep and rep_windows

• Added advertise_preference_for_instance_address in developing against the "Enable cf ssh for Windows+Linux with 3rd Party Network Plugins" feature stories described in the SSH section of these release notes (default value: false)

Diego v2.31.0

Resources

• Download release v2.31.0 from bosh.io.
• Verified with cloudfoundry/cf-deployment @ 6c760d8720a18c30946df7f8cbef5c150455edfd.

Changes from v2.30.0 to v2.31.0

Significant changes

BBS API

• As a BBS API client, I expect the SHA-512 digest algorithm to be deprecated in the image-layer spec so that I avoid adopting it accidentally before its future removal
• As a BBS API client, I expect to understand that the ActualLRPGroup message and associated BBS API endpoints are deprecated and will be removed no earlier than Diego v4.0.0

BBS Benchmarks

• Fixed failing BBS Benchmark Build

Performance Tuning

• As a CF operator, I expect that locket should release idle database connections so that it does not take an outsize share of the shared CF database resources

Local Route Emitters

• As a CF operator, I expect the Diego route-emitter to use the new "flattened" ActualLRP API endpoints and associated instance-specific event types instead of ActualLRPGroup-based ones so that it will continue to function after their future removal
  • Investigate and resolve an increase in 502s that was introduced by the above change (prior to release) from canary app on warp-drive and electron-cannon

Container Networking Support

• As a Diego operator, I would like the executor's garden healthcheck containers to have shorter distinct container handles so that their egress-rule iptables chains are unlikely to collide (take 2)

Per-Instance Proxy

• As a CF operator, I expect not to see spurious error messages about proxy config directory deletions in the rep logs so that I am not distracted or concerned about behavior that is actually normal
• Bumps Envoy proxy binary to 78ad883b70764c27f8b391ee3a5056a64b403426

Component Logging and Metrics

• Investigate benchmark bbs failure
  • Add DBWaitCount and DBWaitDuration metrics for BBS and Locket
• As a Diego operator, I expect to observe a metric that indicates when a Diego cell has stopped accepting work specifically because Garden is not functioning correctly so that I can diagnose the Garden malfunction
  • UnhealthyCell metric deprecated in favor of GardenHealthCheckFailed.

Test Suites and Tooling

• Remove vizzini rootfs tests because it is redundant
• Change references to cflinuxfs2 in diego-release to cflinuxfs3
• validate that DUSTs can be run from a workstation locally
• rewrite vizzini download-cancellation test because it is not testing what it should be

Documentation

• clean up diego-release contributing docs, remove obsolete references (to cf-release, go versions, etc.)

BOSH job changes

• NONE

BOSH property changes

• NONE

BOSH link changes

• NONE