- Feed has been updated to cope with an nginx reload that fails because the nginx master process has been killed. If this occurs, feed will now log a fatal error and the process will exit. This will cause the pod to be restarted by kubernetes (which in turn restarts the nginx master in the new replica).
- Feed is updated to use the
networking/v1
kubernetes ingress API, instead of the previously-usedextensions/v1beta1
API. This means that Feed v6 is only compatible with kubernetes >= 1.19 (when thenetworking/v1
API was introduced). Feed v5 was only compatible with k8s <= 1.21 (theextensions/v1beta1
API was removed in k8s 1.22). Feed v6 in contrast is compatible with the most recent kubernetes versions (as of this writing, 1.24). - The
merlin
integration is dropped from feed. Merlin was an IPVS-based load balancer developed by Sky; it has been EOL since 2019.
- A readiness endpoint is added to feed. This is useful for kubernetes rolling deployments, to prevent proceeding with deployments before a feed replica has registered itslef with an AWS LB target group. Itʼs particularly useful for NLB type load balancers, where there is a delay of several minutes between the registration API call to AWS and the initial traffic sent to the feed replica.
- The nginx VTS module is updated to the most recent upstream version.
The new version exposes prometheus histograms for the latency of
requests transiting feed. This behavior can be controlled with the
--nginx-vhost-stats-request-buckets
command line option.
-
[SECURITY] Previous versions of feed did not validate that the format of the path field on an ingress object (as received from the k8s API) was formatted in a way that feed/nginx would expect. This would allow an attacker to inject nginx configuration constructs into the path field of the ingress object. This could lead to a denial of service (by injecting constructs that make the configuration invalid, rendering nginx unable to process future configuration updates). Itʼs theoretically possible that an attacker could inject config to allow themselves to steal information from the nginx pod or seize control of nginx itself, although we currently do not see an avenue to exploit the vulnerability in this way.
The fix is to enforce that the path field only contains legal characters for the path of a URL, according to RFC 3986. Because this narrows the definition of what is acceptable in the path field of an ingress, it is technically a breaking change.
It is recommended that all users upgrade to this new version immediately to benefit from the security fix. (It is worth noting that this vulnerability is only exploitable by actors with the permission to create kubernetes ingresses – itʼs not exploitable merely by hitting the ingress with traffic from outside the cluster.)
- New static mode which sets ingress status to a static hostname #238
- Upgrade to nginx version 1.21.3 from 1.15.7
- Configure and add a dynamic module which will suppress the
Server
response header. This will add to the existing feature of suppressing the nginx version and build information in the response headers.
- Refine the docker build process to pull in newer packages from Alpine and eliminate trivy security warnings for the resulting docker images
- Bump the version of the go
crypto
library to pull in the fix for a trivy-reported security warning.
- [BUGFIX] Fix issue with feed not waiting for the drain duration despite having at least one successful call to de-register from the list of registered target groups.
- Feed instance can attach to multiple target groups belonging to the load balancer having a tag matching the value in the flag
ingress-class
. The fix is for feed instance to wait for the drain duration even if the de-register call succeeds to at least one of those target groups.
- Feed instance can attach to multiple target groups belonging to the load balancer having a tag matching the value in the flag
- Provide support for multiple labels for namespace selection
- Remove flag
--ingress-controller-namespace-selector
- Add flag
--ingress-controller-namespace-selectors
- which can accept comma separated or repeated inputs - Add flag
--match-all-namespace-selectors
- for how to match the above provided labels
- Remove flag
- Add flag
set-real-ip-from-header
to specify the name of the request header for the real ip module to use- The name of the header will be used by the real ip module in the
set_real_ip_from
directive.
- The name of the header will be used by the real ip module in the
- Switch feed-ingress base image to alpine to reduce the number of vulnerabilities
- Support latest backend config for nginx upstream module
- Support setting keepalive_requests and keepalive_timeout for nginx upstream module
- Support annotations on ingress to be able to set the values in nginx config for each upstream
- Build upstream id including ingress name (This might increase the number of virtual host entries)
- Needed to be able to configure an upstream for ingresses which has same hostname and backend service but with a different path. This is so that the configuration won't be overwritten in this scenario.
- [BUGFIX] Do not generate Prometheus metrics for invalid nginx-vts entries
- [BUGFIX] Reduce the number of DescribeTargetGroups to avoid reaching the API limits
- [BUGFIX] Do not set the nlb updater as initialised if it failed to describe the target groups
- [BUGFIX] Generate the root path location for servers without the root ingress (#225)
- [BUGFIX] Account for concurrent status updates (#219)
- [BUGFIX] Trim whitespace from sky.uk/allow entries (#223)
- [BUGFIX] Validate
sky.uk/allow
annotation entries (#220)
- [BUGFIX] Wait for informer cache to be fully populated before interrogating it (#218)
- Fix X-Forwarded-Host to correctly forward client Host header (#217)
- Breaking change
Feed will not bind port http 8080 by default, unless
--ingress-port=XXXX
is provided- Remove http binding by default (#216)
- Support IP and Instance TargetGroup attachments (#214)
- [BUGFIX] Handle receiving 0 ingresses from the k8s client by not reloading the Nginx config
- [BUGFIX] Improve sorting when choosing which duplicate ingresses to keep
- Added a new metric for Nginx reloads
- [BUGFIX] feed-ingress metrics now have correct prefix of
feed_ingress_
(wasfeed_dns_
)
- [BUG] feed-ingress metrics are prefixed with
feed_dns_
instead offeed_ingress_
. Fixed in v2.2.2 - Attach to NLBs using the instance's private IP address rather than the instance ID to allow services to route to a feed instance on the same host. More information can be found here: https://aws.amazon.com/premiumsupport/knowledge-center/target-connection-fails-load-balancer/.
- [BUG] feed-ingress metrics are prefixed with
feed_dns_
instead offeed_ingress_
. Fixed in v2.2.2 - Reintroduce support for deprecated ingress resource annotations
sky.uk/frontend-elb-scheme
andsky.uk/backend-keepalive-seconds
- [BUG] feed-ingress metrics are prefixed with
feed_dns_
instead offeed_ingress_
. Fixed in v2.2.2 - Added support for AWS Network Load Balancers with the
nlb
subcommand
- Added support for multiple feed-ingress controllers per cluster
- Feed-ingress invocation split into subcommands, using double-dashed arguments
- Remove support for deprecated ingress resource annotations:
sky.uk/frontend-elb-scheme
(replacementsky.uk/frontend-scheme
),sky.uk/backend-keepalive-seconds
(replacementsky.uk/backend-timeout-seconds
) - Remove deprecated feed-ingress command-line argument
--nginx-default-backend-keepalive-seconds
(replacement--nginx-default-backend-timeout-seconds
)
This is a breaking change. Follow the instructions to upgrade from v1 to v2
- Skip ingress when http and/or path are not defined
- Set max_conns default to 0
- Upgrade Nginx from 1.12.2 to 1.15.7
- Add support for exact paths when specified as locations #197
- New flag to set the worker shutdown timeout
- Formatting changes #190
- Fix duplicate path when path is not specified #193
- Remove unnecessary config reload after start up #194
- Fix bug in v1.12.0 with the nginx.conf template #188
- Enable overriding proxy buffer values. Defaults to
proxy_buffer_size 16k
andproxy_buffers 4 16k
Can be overridden with relevant annotations - The values overridden by the annotations are capped at a permissible max
proxy_buffer_size 32k
andproxy_buffers 8 32k
- Supports handling of large client requests (header and body). Refer https://github.com/sky-uk/feed#handling-large-client-requests
- DO NOT USE THIS VERSION
- Update nginx-opentracing version for bug fix to proxy headers
- Manually compile OpenTracing modules to avoid binary incompatibilities
- Add OpenTracing support
- Expose status updater error logs rather than printing a list of failed ingresses
- Bug fix for k8s/status updater where feed would exit the update loop if any ingress was 'unchanged'
- Added
merlin-internal-hostname
andmerlin-internet-facing-hostname
flags for setting Merlin ingress status, replacing themerlin-internet-facing-vip
flag - Included extra testing around ingress validation
- Added
k8s/status
library for setting ingress status - ELB and Merlin updaters set relevant ingress status
- Attach to https in addition to http for merlin.
- Fix merlin deregistration, which was failing due to long lived connections getting killed.
- Bug fix for merlin attacher - fix netlink and capabilities for feed-ingress.
- Introduced flag to set the amount of memory allocated to the vhost statistics module (default: 1 MiB)
- Add ability to specify health checks for merlin frontends.
- Upgraded to Nginx 1.12.2 and VTS 0.1.15
- Introduced
sky.uk/backend-max-connections
annotation that sets upstream.max_conns (http://nginx.org/en/docs/http/ngx_http_upstream_module.html#max_conns) - Introduced flag to set global value for upstream.max_conns (default: 1024)
- Add support for merlin frontend
- Swap to dep from govendor
- Moved to using pester as an http client
- Implemented retries on calls to the gorb API
- Enable SSL termination Set default ssl path + name file without extension. Feed expects two files: one ending in .crt (the CA) and the other in .key (the private key), for example: -ssl-path=/etc/ssl/default-ssl/default-ssl
- Add
gorb-backend-healthcheck-type
that can be either 'tcp', 'http' or 'none' - Remove deprecated
elb-drain-delay
feed-ingress flag
- Reduce logging each ingress in the controller from Info to Debug, introduced in v1.3.0
- Fix wrong output direction when managing loopback interface using sudo
- Add support for configuring IPVS via gorb with Direct Server Return packet-forwarding method.
Various flags prefixed with
gorb-
have been added to feed-ingress to customise gorb configuration. - Add
registration-frontend-type
feed-ingress flag to specify either elb, alb or gorb. - Deprecate
elb-drain-delay
feed-ingress flag in favour of the more genericdrain-delay
.
- Add support for non-AWS load balancers, which are referenced by static hostnames.
- Stop logging out the entire Route53 record set on update at Info in feed-dns: reduce this to Debug, and instead emit the number of records currently in the record set at Info
- Aggressively rotate access logs to avoid excessive file cache usage. This can lead to kernel allocation failures when running feed inside a container with a memory limit.
- Rename annotation
sky.uk/backend-keepalive-seconds
tosky.uk/backend-timeout-seconds
to make it clear that this value only affects request timeouts. The old annotation is preserved for backwards compatibility. - Update to golang 1.9.1.
- Fix bug where no ELB updater would be created if the
elb-label-value
is provided.
- Fix bug where feed-ingress would wait for
elb-drain-delay
andalb-target-group-deregistration-delay
even if no instances where attached. - Do not create ELB or ALB updater when
elb-label-value
oralb-target-group-names
, respectively, are empty. - Note: this image is broken, it does not create an ELB updater if the
elb-label-value
is provided.
Make deduping ingress entries deterministic.
The previous approach tried to order ingress by CreationTimestamp before picking the most recent ingress. This did not work because multiple duplicate ingresses could be created at the same time.
This fix orders ingress entries by Namespace,Name,Host,Path and only uses the first ingress 'Host/Path' encountered to dedupe. Kubernetes guarentees unique ingress for a given 'Namespace/Name' which will make this deduping deterministic.
- Do not delete unassociated resource record sets (http://github.com/sky-uk/feed/pull/144)
- Fix bug where feed-ingress could return 404s for a brief period upon startup.
- Fix bug in feed that causes unhealthy status at startup (#141)
- First official release with our production-ready ingress controllers.