Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Getting 400 Bad Request #6

Open
muvaf opened this issue Dec 27, 2020 · 13 comments
Open

Getting 400 Bad Request #6

muvaf opened this issue Dec 27, 2020 · 13 comments

Comments

@muvaf
Copy link

muvaf commented Dec 27, 2020

I've completed the installation with the following commands:

kubectl apply -f https://raw.githubusercontent.com/compumike/hairpin-proxy/v0.1.2/deploy.yml
kubectl -n hairpin-proxy set env deployment/hairpin-proxy-haproxy TARGET_SERVER=ingress-nginx-controller.kube-system.svc.cluster.local

Before installation, the request would time out but now I'm getting 400 Bad Request.

I couldn't find much so I'm sharing all I have right now. Here is the log form my debug pod:

$ dig mysub.domain.com
; <<>> DiG 9.12.4-P2 <<>> mysub.domain.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 12261
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: e77c2f9dd19d168d (echoed)
;; QUESTION SECTION:
;mysub.domain.com.		IN	A

;; ANSWER SECTION:
mysub.domain.com.	25	IN	A	10.106.209.95

;; Query time: 1 msec
;; SERVER: 10.96.0.10#53(10.96.0.10)
;; WHEN: Sun Dec 27 22:40:58 UTC 2020
;; MSG SIZE  rcvd: 97

$ dig hairpin-proxy.hairpin-proxy.svc.cluster.local

; <<>> DiG 9.12.4-P2 <<>> hairpin-proxy.hairpin-proxy.svc.cluster.local
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 62161
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 5b49c52bd3a873ea (echoed)
;; QUESTION SECTION:
;hairpin-proxy.hairpin-proxy.svc.cluster.local. IN A

;; ANSWER SECTION:
hairpin-proxy.hairpin-proxy.svc.cluster.local. 30 IN A 10.106.209.95

;; Query time: 1 msec
;; SERVER: 10.96.0.10#53(10.96.0.10)
;; WHEN: Sun Dec 27 22:41:16 UTC 2020
;; MSG SIZE  rcvd: 147

$ curl mysub.domain.com
<html>
<head><title>400 Bad Request</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<hr><center>nginx</center>
</body>
</html>

Hairpin proxy pods show no logs. Here is the log line I see in nginx-controller when I send the curl request:

10.44.0.0 - - [27/Dec/2020:22:50:35 +0000] "PROXY TCP4 10.44.0.0 10.44.0.21 1886 80" 400 150 "-" "-" 0 0.001 [] [] - - - - ed328050593d01bbd17e87c3eb3a255d

Here is the raw curl and I've confirmed that 10.106.209.95 belongs to hairpin-proxy service:

$ curl -iv --raw mysub.domain.com

* Rebuilt URL to: mysub.domain.com/
*   Trying 10.106.209.95...
* TCP_NODELAY set
* Connected to mysub.domain.com (10.106.209.95) port 80 (#0)
> GET / HTTP/1.1
> Host: mysub.domain.com
> User-Agent: curl/7.61.1
> Accept: */*
>
< HTTP/1.1 400 Bad Request
HTTP/1.1 400 Bad Request
< Date: Sun, 27 Dec 2020 22:51:47 GMT
Date: Sun, 27 Dec 2020 22:51:47 GMT
< Content-Type: text/html
Content-Type: text/html
< Content-Length: 150
Content-Length: 150
< Connection: close
Connection: close

<
<html>
<head><title>400 Bad Request</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<hr><center>nginx</center>
</body>
</html>
* Closing connection 0

FWIW, I'm able to acces the URL from the public and getting the following in response to kubectl describe challenge:

Status:
  Presented:   true
  Processing:  true
  Reason:      Waiting for HTTP-01 challenge propagation: wrong status code '400', expected '200'
  State:       pending
Events:
  Type    Reason     Age   From          Message
  ----    ------     ----  ----          -------
  Normal  Started    21m   cert-manager  Challenge scheduled for processing
  Normal  Presented  20m   cert-manager  Presented challenge using HTTP-01 challenge mechanism
@muvaf
Copy link
Author

muvaf commented Dec 28, 2020 •
edited
Loading

I've just realized that curl mysub.domain.com --haproxy-protocol doesn't work either, which README says it should in the step 0.

$ curl mattermost.muvaf.com --haproxy-protocol
curl: (28) Failed to connect to mysub.domain.com port 80: Operation timed out

My setup is nginx-ingress + MetalLB with one external IP used by many ingress resources.

@muvaf
Copy link
Author

muvaf commented Dec 28, 2020

I ran cert-manager in my laptop and it all worked since it accessed from outside the cluster. I'm trying to set up a dev/test instance so expiration is not an issue. But that's not viable normally for majority of use cases, so I'll keep this issue open and I'd be more than happy to help you debug by reproducing it.

@shibumi
Copy link

shibumi commented Jan 4, 2021

Hi @muvaf I encountered the exact same problem with ingress-nginx and metallb. @compumike could you please have a look on this?

@shibumi
Copy link

shibumi commented Jan 4, 2021

@muvaf just an idea, but do we need to configure proxy protocol for our ingress-nginx?

@muvaf
Copy link
Author

muvaf commented Jan 5, 2021

@shibumi I'm not sure. Here is my values.yaml for ingress-nginx helm installation if that helps:

controller:
  externalTrafficPolicy: "Local"
  extraArgs:
    enable-ssl-passthrough: ""
  kind: DaemonSet
  type: LoadBalancer
  tolerations:
  - effect: NoSchedule
    operator: Exists

@shibumi
Copy link

shibumi commented Jan 5, 2021

@muvaf does it help if you set externalTrafficPolicy: Cluster? The whole purpose of the hairpin proxy is to have working Cluster TrafficPolicy, or am I wrong?

Btw I fixed the whole problem with another solution. Instead of using using the hairpin proxy, I just defined hostAliases for the coreDNS deployment and then ipropagated the /etc/hosts file

@reesericci
Copy link

reesericci commented Jan 23, 2021 •
edited
Loading

@compumike Getting the same error, any solution?

@reesericci
Copy link

Using metallb + traefik

@shibumi
Copy link

shibumi commented Jan 24, 2021

@reesericci I uninstalled the project. Instead I am adding host/IP tuples to the /etc/hosts file of my coredns inside of the cluster.
This manual work is tedious, but it fixes my problem :)

@reesericci
Copy link

I'm looking currently at k8s_gateway external plugin for coredns, trying to get it to work.

@muvaf
Copy link
Author

muvaf commented Jan 25, 2021

does it help if you set externalTrafficPolicy: Cluster? The whole purpose of the hairpin proxy is to have working Cluster TrafficPolicy, or am I wrong?

@shibumi My goal was to just get the let's encrypt query to work and I did that manually by running the cert-manager in my laptop since it was only a one-time operation for me (test cluster). I don't really know if that'd have worked

@shibumi
Copy link

shibumi commented Jan 25, 2021

@muvaf This was my intention, too. I just added the local ip address + hostname to the /etc/hosts file from the coredns inside of the cluster. This way coredns distributes their /etc/hosts file to every other service inside of the cluster.

@natehouk
Copy link

natehouk commented Jan 21, 2023 •
edited
Loading

@shibumi @muvaf @reesericci

You can solve the 400 status issue by enabling proxy-protocol on your load balancer by annotating your nginx-ingress and enabling use-proxy-protocl in the controller.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@natehouk @shibumi @muvaf @reesericci