Diego v2.9.0

Changes from v2.8.0 to v2.9.0

• Verified with cloudfoundry/cf-deployment @ 402208fabd058d77dcc8a808fa17afb48be8b3fb.
• Verified with garden-runc-release v1.14.0.
• Verified with garden-windows-bosh-release v0.14.0.
• Verified with cflinuxfs2-release v1.216.0.

IMPORTANT: This version of Diego does not shut down application processes gracefully with a TERM signal, and instead typically sends them only a KILL signal shortly after stopping the workload container. The Diego team recommends that platform operators use Diego v2.10.0 or later instead of this release.

Significant changes

Container Execution

• As a BBS API client, I expect that a Diego cell that fails to create a Task container to submit the Task for re-placement
• As a CF app end user, I expect my requests not to be directed to an app instance that crashed during startup

Component Coordination

• cloudfoundry/bbs #31: Fixed issue #30.

De-Consuling Service Discovery (Experimental)

• As a CF operator, I expect to be able to create a CF deployment with several diego-api instances regularly when both BOSH DNS and a networking add-on such as IPsec is enabled so that I can deploy reliably

Rootless Garden Integration

• As a Diego contributor, I expect to be able to skip privileged-container tests when I run the vizzini test errand so that I can run vizzini against an environment that does not support privileged containers (in flight)

Component Logging and Metrics

• cloudfoundry/route-emitter #13: fix logger session traceability in routingtable

Dependencies

• Fix diego-release NATS dependency

Cleanup

• Ignored Errors in bbs/db/sqldb/task_convergence.go
• refactor AllocateContainers in executor.client to not return an error

BOSH job changes

None.

BOSH property changes

vizzini

Added enable_privileged_container_tests: Whether to run tests that make privileged containers. Defaults to true.

BOSH link changes

None.

Diego v2.8.0

Changes from v2.7.1 to v2.8.0

• Verified with cloudfoundry/cf-deployment @ 3b8c31223030ee1878ff5090eb36b37164a3cff4.
• Verified with garden-runc-release v1.13.3.
• Verified with garden-windows-bosh-release v0.14.0.
• Verified with cflinuxfs2-release v1.211.0.

SECURITY NOTE: This version of Diego fixes CVE-2018-1265. The Diego team recommends all operators upgrade to version v2.8.0 or later of Diego. CF operators using cf-deployment for manifest generation should upgrade to version v1.37.0 or later.

Significant changes

BBS Relational Datastore

• cloudfoundry/bbs #31: Fixed issue #30. (in flight)

Container Execution

• As a CF app end user, I expect my requests not to be directed to an app instance that crashed during startup

Windows Support

• Explore destroying Windows 2016 app instances with instance-identity credentials enabled on Windows 1803 candidate stemcells to discover failure cases

BOSH job changes

None.

BOSH property changes

None.

BOSH link changes

None.

Diego v2.7.1

Changes from v2.7.0 to v2.7.1

• Verified with cloudfoundry/cf-deployment @ 2c8ec659a80f4735de3b59b426b9a6b0ea0a3d32.
• Verified with garden-runc-release v1.13.3.
• Verified with garden-windows-bosh-release v0.14.0.
• Verified with cflinuxfs2-release v1.209.0.

NOTE: This release fixes the cell rep evacuation issue observed with Diego v2.7.0.

Significant changes

Container Placement

• fix rep panic during evacuation

Dependencies

• Investigate garden failures in inigo

Cleanup

• Prevent resolution of an ActualLRPGroup from panicking

BOSH job changes

None.

BOSH property changes

None.

BOSH link changes

None.

Diego v2.7.0 - DO NOT USE

DO NOT USE

IMPORTANT – DO NOT USE: The Diego team has identified an issue with this release in which the Diego cell reps may crash when in evacuation mode during cell VM draining. This crash may result in reduced availability of application instances during rolling deploys, as replacement application instances may not being scheduled and started for instances on draining cell VMs.

The Diego team will have a resolution for this issue in the next final Diego release, which we will produce as soon as possible. In the meantime, if you have already deployed Diego v2.7.0, we recommend that the next time you update your Diego cells you downgrade to Diego v2.6.0, which does not have this issue with cell rep evacuation.

For CF environments with route integrity enabled, this issue may also result in the gorouters returning 502 errors to clients if they accumulate a large enough number of stale application route registrations. In this case, we also recommend that operators restart their gorouters after any rolling deploy in which Diego cells running v2.7.0 have been drained in order to remove these stale route registrations.

UPDATE: Diego v2.7.1 resolves this issue.

Changes from v2.6.0 to v2.7.0

• Verified with cloudfoundry/cf-deployment @ 94e1d26898a93058160227071d47587f646f35d5.
• Verified with garden-runc-release v1.12.1.
• Verified with garden-windows-bosh-release v0.14.0.
• Verified with cflinuxfs2-release v1.208.0.

Significant changes

Container Execution

• cloudfoundry/executor #31: Why do non-cacheable blobs fail?

App Logging and Metrics

• As a CF app developer, I expect to see log messages that indicate when CF is replacing an app instance on an evacuating cell so that I can understand that my app instance has not itself failed

Component Logging and Metrics

• As a Diego operator, I expect each Locket instance to emit its LocksExpired and PresencesExpired metrics periodically so that I can view them continually and derive rates of change from them
• As a lager library user, I expect the lager/chug package to process the improved lager log format so that I can continue to use it to relog lager-formatted output
• As a CF operator, I expect the route-emitter to log an appropriate volume without redundancies so that I do not have to aggregate logs of little to no value

Dependencies

• As a CF operator, I expect to downgrade to Golang 1.9.6+ so that the cell rep can continue to download assets from external sources over TLS
• Switch to using pxc-release instead of cf-mysql-release in some Diego CI environments

Documentation

• cloudfoundry/diego-release #390: Contributing docs
• cloudfoundry/bbs #29: fix typo

BOSH job changes

None.

BOSH property changes

None.

BOSH link changes

None.

Diego v2.6.0

Changes from v2.5.0 to v2.6.0

• Verified with cloudfoundry/cf-deployment @ d62bf37da92c085a23c002de172369790e6a0ab8.
• Verified with garden-runc-release v1.13.1.
• Verified with garden-windows-bosh-release v0.14.0.
• Verified with cflinuxfs2-release v1.203.0.

Significant changes

Container Placement

• As a BBS API client, I expect to see task_changed events in the Task-event stream when Tasks fail placement

Instance Identity Credentials

• As a CF operator, I expect to find information about instance-identity credential configuration in the CF docs website so that I can enable it correctly
• As a CF app developer, I expect to have documentation about the properties of instance-identity credentials so that I can use them appropriately for app communication
• As a CF app developer, I expect the instance-identity credentials certificate also to include the org and space guids so that I can use them to satisfy broader CF authorization predicates (in flight)

Declarative Health Checks

• As a CF operator, I expect my Diego cells to opt into preferring declarative healthchecks by default so that I can automatically benefit from this stability improvement

Per-Instance Proxy (Experimental)

• As a CF operator, I expect to understand how the Envoy proxies consume memory resources under concurrent request load so that I can provision resources appropriately to support route-integrity improvements
• As a CF app developer, I expect my app instances to be able to handle more than 1024 concurrent connections even if the Envoy proxy is enabled on its host cells

Component Logging and Metrics

• As a CF operator, I expect the RFC3339-formatted lager timestamps always to have exactly 9 decimal places so that I can use sort to sort them chronologically

Dependencies

• As a Diego operator, I expect the release to use Golang v1.10.2+ so that I am up to date with this important dependency

Security

• As a CF operator, I expect the cell rep job not to affect the kernel configuration when running in a container itself so that it does not affect other jobs or processes on the host

Cleanup

• Remove event filtering logic from route-emitters

BOSH job changes

None.

BOSH property changes

rep and rep_windows

• Changed containers.proxy.additional_memory_allocation_mb default value from 5 to 32.
• Removed experimental status of enable_declarative_healthcheck.
• Removed experimental status of declarative_healthcheck_path.

BOSH link changes

None.

Diego v2.5.0

Changes from v2.4.0 to v2.5.0

• Verified with cloudfoundry/cf-deployment @ b70090f9237b5e24ab86410842b0288bcf6862f1.
• Verified with garden-runc-release v1.13.1.
• Verified with garden-windows-bosh-release v0.13.0.
• Verified with cflinuxfs2-release v1.203.0.

Significant changes

BBS API

• As a CF operator, I expect the BBS not to crash because it cannot find a valid ActualLRP

Container Placement

• As a CF operator, I expect to be able experimentally to opt Diego into retrying placement of a Task a limited number of times so that Task placement can be more resilient to transient drops in addressable capacity
• As a BBS API client, I expect to see placement errors and placement failure counts on Tasks so that I can understand why they are still Pending

SSH

• As a CF app developer, I expect to be able to create remote SSH tunnels to my app instances so that I can connect them to network resources local to my workstation
• As a CF operator, I expect the ssh_proxy's diego.ssh_proxy.uaa.port property to default to 8443 so that I do not have to supply it in my CF manifest
• As a CF Java developer, I expect the default CF SSH cipher suites, MACs, and key exchange algorithms to be compatible with the SSHJ client library so that I can use existing Java developer tooling against my CF deployment

Per-Instance Proxy (Experimental)

• As a CF operator, I expect to understand how the Envoy proxies consume memory resources under concurrent request load so that I can provision resources appropriately to support route-integrity improvements (in flight)
• As a CF operator, I expect to be able to run at least 250 container instances on my proxy-enabled Linux Diego cells so that my container density is not artificially limited

App Logging and Metrics

• As a Diego operator, I expect the cell rep API to report container metrics so that I can inspect these properties on-demand

BOSH job changes

None.

BOSH property changes

bbs

• Added tasks.max_retries. Experimental: number of times to retry placement of a Task. Defaults to 0.

ssh_proxy

• Updated diego.ssh_proxy.uaa.port to have a default value of 8443.

vizzini

• Added max_task_retries: Whether to run acceptance tests for task placement retries.

BOSH link changes

None.

Diego v2.4.0

Changes from v2.3.0 to v2.4.0

• Verified with cloudfoundry/cf-deployment @ fca6dced5e7c094de57ae48c26505ca72d0bb22b.
• Verified with garden-runc-release v1.12.1.
• Verified with garden-windows-bosh-release v0.13.0.
• Verified with cflinuxfs2-release v1.196.0.

Significant changes

Per-Instance Proxy (Experimental)

• As a BBS API client, I expect the cells to run LRP instances without memory limits as specified even if the cells supply Envoy proxies with an additional memory allocation
• As a CF app developer, when my app instance crashes because the sidecar Envoy proxy process exits, I expect to see information about that reason in the app instance logs and crash reason so that I can distinguish it as a platform issue and not an issue with my application

Windows Support

• cloudfoundry/diego-release #388: Add getenv binary to BAL and WAL.
• apps with newlines in environment variables should be pushable to windows cells

App Logging and Metrics

• As a CF app operator, I expect the disk usage for my Docker-image-based app instances to include the size of the image layers so that I can correctly understand the amount of disk space remaining in each instance

Component Logging and Metrics

• cacheddownloader should not mask the original error encountered in the download if the step is cancelled
• As a CF operator, I expect to be able to determine which goroutines in my Diego components prevented a shutdown within the time monit allows so that I can provide useful details to resolve this behavior in the future

Documentation

• As a CF operator, I expect to know what horizontal and/or vertical scaling options are appropriate for individual Diego BOSH jobs so that I can provision my environment for redundancy efficiently
• As a CF operator, I expect to have better guidance about how to adjust the resources that I provide to the BBS and Locket services so that I can maintain a stable CF control plane

BOSH job changes

None.

BOSH property changes

None.

BOSH link changes

None.

Diego v2.3.0

Changes from v2.2.0 to v2.3.0

• Verified with cloudfoundry/cf-deployment@59fe9d1f04b31d4e03ec252d91d5e2e46e67ee3d.
• Verified with garden-runc-release v1.12.1.
• Verified with garden-windows-bosh-release v0.13.0.
• Verified with cflinuxfs2-release v1.195.0.

Significant changes

cfdot

• cloudfoundry/diego-release #387: cfdot broken on stemcell series 3541

Component Logging and Metrics

• As a CF operator, I expect to be able to opt all the Diego components into the improved lager timestamp format so that I can understand their logs more easily
• BBS should log detected-missing-cells only if it actually detected at least one missing cell

Documentation

• As a CF operator, I expect to know what horizontal and/or vertical scaling options are appropriate for individual Diego BOSH jobs so that I can provision my environment for redundancy efficiently (in flight)
• As a CF operator, I expect to have better guidance about how to adjust the resources that I provide to the BBS and Locket services so that I can maintain a stable CF control plane (in flight)

Cleanup

• cloudfoundry/rep #19: use the waitgroup instead, and return early to save some time.

BOSH job changes

None.

BOSH property changes

auctioneer

• Added logging.format.timestamp: Controls the formatting style of the component log timestamp.

bbs

• Added logging.format.timestamp: Controls the formatting style of the component log timestamp.

locket

• Added logging.format.timestamp: Controls the formatting style of the component log timestamp.

rep and rep_windows

• Added logging.format.timestamp: Controls the formatting style of the component log timestamp.

route_emitter and route_emitter_windows

• Added logging.format.timestamp: Controls the formatting style of the component log timestamp.

ssh_proxy

• Added logging.format.timestamp: Controls the formatting style of the component log timestamp.

BOSH link changes

None.

Diego v2.2.0

Changes from v2.1.0 to v2.2.0

• Verified with cloudfoundry/cf-deployment @ 5c97d2f85481e23b953965b471fb2a137547edf5.
• Verified with garden-runc-release v1.12.1.
• Verified with garden-windows-bosh-release v0.13.0.
• Verified with cflinuxfs2-release v1.194.0.

Significant changes

SSH

• cloudfoundry/diego-ssh #41: Insecure default settings in sshConfig

Per-Instance Proxy (Experimental)

• As a Diego operator, I expect the rep to clean up proxy-manager config directories so that I do not run out of inodes on my cell VMs
• As a CF app developer, I expect that the envoy sidecar process does not have a limit on the number of files it can have open so that I am not artificially limited in the number of connections my app instances can handle

Component Logging and Metrics

• Spike: Explore approaches to changing Diego component logging to allow configuration for more human-readable output
• As a CF operator, I expect to be able experimentally to opt the Diego file-server into more human-readable lager output so that I can understand its timestamps and log severity more easily

Dependencies

• bump garden and guardian in diego-release

Test Suites and Tooling

• re-enable inigo and CATs private docker registry tests using our existing Azure private docker registry
• start-inigo-container should not start a docker container if the grub configuration isn't valid according to the comment that is printed in the script
• [Flake] inigo certs rotation test
• warp-drive-vizzini flakiness with lrps that are not supposed to crash when stopped

Documentation

• As a CF contributor, I expect cloudfoundry/systemcerts to be deprecated in favor of x509.SystemCertPool
• As a CF operator, I would like to know when diego-release properties were deprecated and when they were removed so that I can be certain about my manifest changes

Cleanup

• cloudfoundry/executor #29: No need to call gardenContainer.Info() twice between two pure memory operations

BOSH job changes

None.

BOSH property changes

file_server

• Added logging.format.timestamp: Controls the formatting style of the component log timestamp.

rep_windows

• Added diego.executor.volman.driver_paths: Paths for volman to inspect for voldriver plugins.
• Added diego.executor.volman.csi_paths: Paths for volman to inspect for CSI plugins.
• Added diego.executor.volman.csi_mount_root_dir: Path under which volman will mount CSI volumes.

BOSH link changes

None.

Diego v2.1.0

Changes from v2.0.0 to v2.1.0

• Verified with cloudfoundry/cf-deployment @ 01fe35738e8b3e1f719b66bd0b79f1a535beeadf
• Verified with garden-runc-release v1.11.1.
• Verified with garden-windows-bosh-release v0.13.0.
• Verified with cflinuxfs2-release v1.189.0.

Significant changes

BBS API

• As a BBS API client, I expect BBS API endpoints and fields deprecated before Diego v1.0.0 to be removed so that I can have a simpler API against which to integrate

Container Execution

• As a CF operator, I expect to see cell and instance identifiers in the container lifecycle messages so that I can easily identify from app log-stream output the cells on which instances were placed
• As a Diego operator, I expect the state endpoint on the cell rep API to report container state so that I can assess instance state and health on a particular cell

Custom CAs

• As a CF contributor, I expect cloudfoundry/systemcerts to be deprecated in favor of x509.SystemCertPool (in flight, to be reversed)

SSH

• Fix the racy use of Waitgroups in diego-ssh server

De-Consuling Locks

• As a Diego operator, I expect locket to report errors with its config file gracefully instead of panicking (in flight)

v2 Loggregator API Adoption

• As a Diego operator, I expect Diego components no longer to support emitting log and metric data to the v1 Loggregator ingress API so that I can standardize on the v2 API

Per-Instance Proxy (Experimental)

• As a Diego operator, I expect the Envoy and long-running health-check processes to run in Garden sidecars for privileged containers so that I can understand their operation consistently

Component Logging and Metrics

• As a Diego operator, I expect the Diego components no longer to emit metrics deprecated before Diego v2 so that I can reduce the number of metrics I aggregate

Dependencies

• As a Diego operator, I expect diego-release to update to Golang 1.10.0+ so that I am up-to-date with the Golang dependency
• cloudfoundry/systemcerts #1: Fix compilation on Darwin with go1.10

Test Suites and Tooling

• Refactor cfdot tests to avoid global Config data race
• Flaky lager chug test
• Decide what to do about vizzini assertions that are marked as no longer necessary after #89463754
• cfdot aftereach can sometimes fail if locket doesn't exit within 1 second

Security

• As a Diego operator, I expect the BBS, auctioneer, and rep components all to require mutual TLS to secure their API servers so that I can ensure their security against remote clients
• As a Diego operator, I expect no longer to be able to configure the cell rep API separation properties so that I can simplify my manifest configuration

Documentation

• As a Diego operator, I expect to know that the properties that opt auctioneer and rep clients into using TLS are deprecated so that I can simplify my manifest after fully upgrading to Diego v2.0+

Cleanup

• Remove unused versioner/format code from bbs.
• cloudfoundry/bbs #27: Remove formatter versioner
• As a Diego operator, I expect no longer to be able to configure the BOSH properties deprecated before Diego v2 so that I can simplify my deployment manifest

BOSH job changes

None.

BOSH property changes

auctioneer

• Removed diego.auctioneer.dropsonde_port.
• Deprecated diego.auctioneer.rep.require_tls.

bbs

• Deprecated diego.bbs.auctioneer.require_tls.
• Deprecated diego.bbs.rep.require_tls.
• Removed diego.bbs.auctioneer.api_url in favor of diego.bbs.auctioneer.api_location.
• Removed diego.bbs.dropsonde_port.
• Removed diego.bbs.sql.db_connection_string.

cfdot

• Removed diego.cfdot.bbs.ca_cert in favor of tls.ca_certificate.
• Removed diego.cfdot.bbs.client_cert in favor of tls.certificate.
• Removed diego.cfdot.bbs.client_key in favor of tls.private_key.

file_server

• Removed diego.file_server.dropsonde_port.

locket

• Removed dropsonde_port.

rep and rep_windows

• Removed diego.executor.ca_certs_for_downloads.
• Removed diego.executor.export_network_env_vars.
• Removed diego.rep.dropsonde_port.
• Removed diego.rep.enable_legacy_api_endpoints.
• Removed diego.rep.listen_addr.
• Removed diego.rep.trusted_certs.

route_emitter and route_emitter_windows

• Removed diego.route_emitter.dropsonde_port.

ssh_proxy

• Removed diego.ssh_proxy.dropsonde_port.
• Removed diego.ssh_proxy.uaa_token_url in favor of diego.ssh_proxy.uaa.url and diego.ssh_proxy.uaa.port.

BOSH link changes

None.