Diego v0.1467.0

Changes from v0.1466.0 to v0.1467.0

• Verified with garden-linux-release v0.337.0.
• Verified with etcd-release v45.
• Verified with cflinuxfs2-rootfs-release v0.2.0.

Significant changes

BBS Relational Datastore (Experimental)

• As a Diego operator, I can run a CF+Diego deployment backed by a MySQL DB instance
• As a Diego developer, I expect to run Diego BBS benchmarks against an AWS environment with the BBS backed by an RDS MySQL instance

Container Execution

• ClaimActualLRP should set the Since field on the ActualLRP and should be a no-op if nothing important changes (in flight)

Container Networking Support (Experimental)

• add a properties field to desiredlrp and task, flow them through to garden

Cleanup

• cloudfoundry-incubator/cacheddownloader #8: replace_windows is now obsolete in go 1.5

BOSH job changes

None.

BOSH property changes

• Add diego.bbs.sql.max_open_connections: Maximum number of connections for the BBS to hold to the SQL database.
• Add benchmark-bbs.sql.db_connection_string: Connection string to use for SQL backend in the BBS benchmark test suite errand.

Diego v0.1466.0

Changes from v0.1465.0 to v0.1466.0

• Verified with garden-linux-release v0.337.0.
• Verified with etcd-release v45.
• Verified with cflinuxfs2-rootfs-release v0.2.0.

Significant changes

NOTE: This version of diego-release adds _experimental_ support for a SQL relational datastore. Opting into this support via the diego.bbs.sql.db_connection_string BOSH property is not yet recommended for existing deployments or for new production deployments, as existing data will not yet be migrated from the etcd datastore and the SQL schema is subject to change. The relational store is also not included in the Diego deployment manifest that the manifest-generation scripts produce.

This version of diego-release also updates the bundled Golang package to version 1.6.1 to address CVE-2016-3958 and CVE-2016-3959. More details are in the Golang announement.

BBS Relational Datastore (Experimental)

• As a Diego developer, I can run BBS unit tests against MySQL on my local workstation
• As a Diego operator, I can run a CF+Diego deployment backed by a MySQL DB instance (in flight)

Component Logging and Metrics

• As a Diego operator, I would like to see a higher signal-to-noise ratio in the BBS logs

Dependencies

• Update Golang in diego-release to 1.6.1+

BOSH job changes

None.

BOSH property changes

• Add diego.bbs.sql.db_connection_string: Connection string to use for SQL backend.

Diego v0.1465.0

Changes from v0.1464.0 to v0.1465.0

• Verified with garden-linux-release v0.336.0.
• Verified with etcd-release v44.
• Verified with cflinuxfs2-rootfs-release v0.2.0

Significant changes

Note: The Diego team fixed an incorrect SHA checksum for the license blob in the 0.1463.0 final release manifest. To ensure that the tags on the diego-release repository correspond to valid versions, the v0.1463.0 and v0.1464.0 tags were moved to commits with the updated release manifest. If these tags are already checked out in a clone of this repository, running git fetch --tags will update them. We apologize for any inconvenience this may have caused.

BBS Relational Datastore (Experimental)

• As a Diego developer, I can run BBS unit tests against MySQL on my local workstation (in flight)

Performance Tuning

• Increase file-descriptor limit on tps-listener process

SSH

• When I invoke a command through the Diego SSH proxy, I should receive its stderr output as stderr output of the ssh client

Guardian Integration

• As a Diego team member, I expect to have a CF+Diego environment for CI against guardian
• As a Diego operator, I would like to be able to opt into using guardian as the Garden implementation on all my cells

Manifest Generation

• Diego manifest generation should take the syslog_daemon_config properties from the CF manifest
• release version overriding is not working in diego-release/scripts/generate-deployment-manifest
• cloudfoundry-incubator/diego-release #153: Make the BBS advertise address configurable
• cloudfoundry-incubator/diego-release #157: Add iaas-settings for vsphere

App Logging

• As a space developer, I expect the 'source' for logging for processes to be [APP/PROC/PROCESS_TYPE/INDEX]

Dependencies

• Update/Validate the behavior of pid_utils in diego-release.
• Update dropsonde library in diego-release
• Update version of noaa library in diego-release and use new consumer
• Bump crypto

Test Suites and Tooling

• I should be able to run the entire vizzini suite against BOSH-Lite from the host machine

Cleanup

• Fix license file on Diego 0.1463.0 release YML file

BOSH job changes

None.

BOSH property changes

• Add diego.bbs.advertisement_base_hostname: Suffix for the BBS advertised hostname. Defaults to bbs.service.cf.internal.

Diego v0.1456.0

Changes from v0.1455.0 to v0.1456.0

• Depends on garden-linux-release v0.334.0.
• Depends on etcd-release v36.

Significant changes

This version of diego-release completely removes the Diego Smoke Tests suite, as it is redundant with the CF Smoke Tests. If you are using the Diego Smoke Tests to monitor a CF deployment backed by Diego, please switch to using the CF Smoke Tests as soon as possible.

Routing

• Investigate reports of slowness in route-mapping propagation time

Dependencies

• Upgrade cflinuxfs2 rootfs in diego-release to 1.40.0+
• Upgrade cflinuxfs2 rootfs in diego-release to 1.41.0+

Test Suites and Tooling

• Deprecate the Diego smoke tests in favor of the CF smoke tests

Documentation

• Document the Tasks API (public and internal) and Actions on the BBS (in flight)

Licensing

• Update LICENSE and NOTICE files for all Diego repos for 2016

BOSH job changes

Removed the smoke-tests job.

BOSH property changes

Removed all BOSH properties under diego.smoke_tests.

Diego v0.1464.0

Changes from v0.1463.0 to v0.1464.0

• Verified with garden-linux-release v0.335.0.
• Verified with etcd-release v44.
• Verified with cflinuxfs2-rootfs-release v0.1.0

Significant changes

BBS Relational Datastore (Experimental)

• As a Diego developer, I can run BBS unit tests against MySQL on my local workstation

Performance Tuning

• Increase the rep's download idle timeout to 10 seconds

Custom CAs

• cloudfoundry-incubator/diego-release #155: Bump the timeout for certification updates to a full minute, match BOSH.
• As a Diego operator, I would like to be able to configure the rep to trust additional CAs only for downloads

Routing

• When the rep is shutting down after finishing evacuation, it should remove its evacuating ActualLRPs
• as a space developer, I can specify multiple ports on a process type

Volume Support (Experimental)

• cephfs driver can be colocated on the Cell
• Volume Drivers have an Unix Socket transport
• move fakedriver acceptance from cmd/volman to fakedriver/acceptance

Rootfs Release Extraction

• As a Diego team developer, I expect all Diego CI environments to consume final versions of the cflinuxfs2-rootfs release

Manifest Generation

• cloudfoundry-incubator/diego-release #151: Require consul certs and keys from cf manifest
• prepare to remove non-encrypted support from Consul by updating cf and diego manifest templates
• Make compilation.workers manifest configurable and decrease the number of workers for bosh-lite.

App Logging

• Emit log lines correctly for failed cached-dependency downloads

Component Logging and Metrics

• As a Diego operator, I would like to see a higher signal-to-noise ratio in the cell rep logs

Dependencies

• cloudfoundry-incubator/candiedyaml #19: Add quotes around strings containing a colon followed by whitespace
• cloudfoundry-incubator/candiedyaml #20: Allow strings to start with a colon

BOSH job changes

None.

BOSH property changes

• Added diego.executor.ca_certs_for_downloads: Bundle of additional CAs for the executor to trust when downloading assets.

Diego v0.1463.0

Changes from v0.1462.0 to v0.1463.0

• Verified with garden-linux-release v0.335.0.
• Verified with etcd-release v43.

Significant changes

Note: We decided to remove the consul-agent port properties that were added to the diego-release BOSH jobs in Diego v0.1462.0. The HTTP API port on the consul-agent job in consul-release cannot itself be configured, so the port properties would not be immediately useful, and if it is made configurable in the future it may make more sense for the entire address or URL to be configurable instead. We hope the removal of these properties has not inconvenienced anyone consuming the release.

BBS Relational Datastore (Experimental)

• As a Diego developer, I can run BBS unit tests against MySQL on my local workstation (in flight)

Routing

• When the rep is shutting down after finishing evacuation, it should remove its evacuating ActualLRPs

Volume Support (Experimental)

• CI runs VolMan integration tests using CephFS driver and Ceph cluster
• tasks can create and mount a volume
• LRPs can create and mount a volume
• volman can remove volumes
• Auctioneer filters cells by volume driver
• executor and rep function correctly with volman disabled

CC-Bridge Transfer

• CAPI Release - CC Bridge components (in flight)

Manifest Generation

• Ensure all diego-release job defaults are in the job specs, instead of in ERB or spiff templates

Dependencies

• Upgrade Golang in diego-release to 1.6+

Test Suites and Tooling

• Remove BBS measurements suite
• don't run ssh tests in persi CI

BOSH job changes

None.

BOSH property changes

• Removed diego.auctioneer.consul_agent_port.
• Removed diego.bbs.consul_agent_port.
• Removed diego.cc_uploader.consul_agent_port.
• Removed diego.converger.consul_agent_port.
• Removed diego.file_server.consul_agent_port.
• Removed diego.nsync.consul_agent_port.
• Removed diego.rep.consul_agent_port.
• Removed diego.route_emitter.consul_agent_port.
• Removed diego.ssh_proxy.consul_agent_port.
• Removed diego.stager.consul_agent_port.
• Removed diego.tps.consul_agent_port.

Diego v0.1462.0

Changes from v0.1461.0 to v0.1462.0

• Verified with garden-linux-release v0.335.0.
• Verified with etcd-release v43.

Significant changes

The Diego team is proceeding with experimental support for the BBS to use a relational database as its backing datastore, starting with support for MySQL. Work is currently proceeding in the BBS codebase, but we expect it to be exposed through BOSH configuration and manifest-generation in the near future. Please note that this work is strictly experimental and hence is not yet supported for production deployments.

This version of the Diego BOSH release also uses Golang 1.6 throughout. If you are building Diego components locally, please upgrade your Golang runtime to 1.6.

The Diego and Persi teams have been working on experimental support for volume mounts in the executor and the BBS. Any additions to the Diego APIs to enable this work are completely experimental and may change at any time.

The Buildpacks and Diego teams have started work to extract the 'rootfses' job in the Diego BOSH release into its own independent release. For now, Diego manifests can be used unchanged. If you are using the manifest-generation scripts and templates in diego-release, you can opt-in to using the new release with the -r flag (don't forget to create and upload the rootfs release before deploying!).

The CAPI and Diego teams have also started work to transfer the CC-Bridge jobs (stager, cc-uploader, nsync, and tps) from Diego to the new CAPI release. This transfer requires no manifest changes at present. Since the manifest-generation scripts already rely on the presence of a CF release and deployment, we expect to be able to make this transition transparent, with the option to opt-in early via a flag on the manifest-generation script.

BBS Relational Datastore (Experimental)

• As a Diego developer, I can run BBS unit tests against MySQL on my local workstation (in flight)

Performance Tuning

• cloudfoundry-incubator/diego-release #146: Add a switch to control writing to /proc/sys

SSH

• cloudfoundry-incubator/diego-release #144: add ssh-proxy properties for specifying allowed cipher,mac,kex algorithms

Volume Support (Experimental)

• tasks can create and mount a volume (in flight)
• LRPs can create and mount a volume (in flight)
• Volume Drivers have an HTTP transport using .json file (in flight)
• executor advertises available volume drivers (in flight)

Rootfs Release Extraction

• Extract diego-release 'rootfses' job and related packages and blobs into a cflinuxfs2-rootfs-release (in flight)
• cloudfoundry-incubator/diego-release #148: This is a work in progress to separate the rootfs bosh release

CC-Bridge Transfer to CAPI

• CAPI Release - CC Bridge components (in flight)
• fix app and task freshness bumping in nsync bulker
• refactor processor

Manifest Generation

• Ensure all diego-release job defaults are in the job specs, instead of in ERB or spiff templates (in flight)
• Change order of jobs in job template lists to start consul first

Dependencies

• Upgrade Golang in diego-release to 1.6+ (in flight)
• Upgrade cflinuxfs2 rootfs in diego-release to 1.47.0+ 1.48.0+

Documentation

• cloudfoundry-incubator/diego-release #147: Feedback on AWS deployment instructions

Licensing

• Update LICENSE and NOTICE files on diego repos

BOSH job changes

None.

BOSH property changes

• Added diego.auctioneer.consul_agent_port: Port on which the Auctioneer connects to the HTTP API of the local consul agent. Defaults to 8500.
• Added diego.bbs.consul_agent_port: Port on which the BBS connects to the HTTP API of the local consul agent. Defaults to 8500.
• Added diego.cc_uploader.consul_agent_port: Port on which the CC-Uploader connects to the HTTP API of the local consul agent. Defaults to 8500.
• Added diego.converger.consul_agent_port: Port on which the Converger connects to the HTTP API of the local consul agent. Defaults to 8500.
• Added diego.file_server.consul_agent_port: Port on which the File-Server connects to the HTTP API of the local consul agent. Defaults to 8500.
• Added diego.nsync.consul_agent_port: Port on which the Nsync-Bulker and Nsync-Listener connect to the HTTP API of the local consul agent. Defaults to 8500.
• Added diego.rep.consul_agent_port: Port on which the Cell Rep connects to the HTTP API of the local consul agent. Defaults to 8500.
• Added diego.route_emitter.consul_agent_port: Port on which the Route-Emitter connects to the HTTP API of the local consul agent. Defaults to 8500.
• Added diego.ssh_proxy.consul_agent_port: Port on which the SSH-Proxy connects to the HTTP API of the local consul agent. Defaults to 8500.
• Added diego.ssh_proxy.allowed_ciphers: Allowed cipher algorithms for connections to the SSH-Proxy.
• Added diego.ssh_proxy.allowed_macs: Allowed MAC algorithms for connections to the SSH-Proxy.
• Added diego.ssh_proxy.allowed_keyexchanges: Allowed key-exchange algorithms for connections to the SSH-Proxy.
• Added diego.stager.consul_agent_port: Port on which the Stager connects to the HTTP API of the local consul agent. Defaults to 8500.
• Added diego.tps.consul_agent_port: Port on which the TPS-Listener and TPS-Watcher connect to the HTTP API of the local consul agent. Defaults to 8500.

Diego v0.1461.0

Changes from v0.1460.0 to v0.1461.0

• Depends on garden-linux-release v0.334.0.
• Depends on etcd-release v38.

Significant changes

Volume Support (Experimental)

• executor can mount multiple volumes on a container
• executor unmounts volumes when deleting containers
• Garden can read/write to a mounted FS

Manifest Generation

• cloudfoundry-incubator/diego-release #142: Set consul.agent.domain property to cf.internal
• cloudfoundry-incubator/diego-release #143: Update manifest template to have explicit etcd.advertise_urls_dns_suffix

Dependencies

• Upgrade cflinuxfs2 rootfs in diego-release to 1.44.0+

Test Suites and Tooling

• remove route checks from the DUSTs

Cleanup

• Review bridge code and XTP with CAPI to make it better

BOSH job changes

None.

BOSH property changes

None.

Diego v0.1460.0

Changes from v0.1459.0 to v0.1460.0

• Depends on garden-linux-release v0.334.0.
• Depends on etcd-release v38.

Significant changes

Component Logging and Metrics

• pivotal-golang/lager #17: Debug logging does JSON marshaling regardless of whether debug logging is enabled
• cloudfoundry-incubator/cf-debug-server #3: Enable pprof block profiling (parameterized)

Documentation

• Document the Tasks API (public and internal) and Actions on the BBS

BOSH job changes

None.

BOSH property changes

None.

Diego v0.1459.0

Changes from v0.1458.0 to v0.1459.0

• Depends on garden-linux-release v0.334.0.
• Depends on etcd-release v37.

Significant changes

NOTE: Although Diego v0.1455.0 re-introduced the feature that SSH sessions and commands in a CF app container have the same environment variables set as the app's main process, we have rolled that functionality back in Diego v0.1459.0. For buildpack-based apps, the mechanism by which this feature was implemented would execute the buildpack-installed .profile.d scripts twice. While those scripts often only set environment variables for the app process, in general they can have arbitrary effects, and may not be safe to call repeatedly and concurrently. The team is considering safer ways to share that environment, but for now we will leave it unimplemented.

To support SSH users who wish to have the same environment as their start commands, we have submitted documentation of the manual steps to set up that environment to the developer-guide Cloud Foundry documentation repo. Those changes will appear on the "Accessing Apps with SSH" page shortly.

Custom CAs

• cflinuxfs2 rootfs pre-start script should retry certificate installation

SSH

• Roll back SSH environment setup changes (again)
• cloudfoundry-incubator/diego-ssh #21: add flags to ssh-proxy for specifying allowed cipher,mac,kex algorithms

CF Tasks

• As a space developer, I expect the 'source' for task logging to be [App/TASK/TASK_NAME]

Component Logging and Metrics

• log cell id when cell is missing

Documentation

• Document the Tasks API (public and internal) and Actions on the BBS

BOSH job changes

None.

BOSH property changes

None.