Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to scrape Postgres cert metrics #801

Open
atmanohar opened this issue Jun 18, 2021 · 4 comments · May be fixed by #1112
Open

Unable to scrape Postgres cert metrics #801

atmanohar opened this issue Jun 18, 2021 · 4 comments · May be fixed by #1112

Comments

@atmanohar
Copy link

atmanohar commented Jun 18, 2021 •
edited
Loading

Host operating system: output of uname -a

blackbox_exporter version: output of blackbox_exporter --version

What is the blackbox.yml module config.

modules:
  tls_connect:
    prober: tcp
    timeout: 60s
    tcp:
      preferred_ip_protocol: "ip4"
      ip_protocol_fallback: false
      query_response:
        - expect: "OK.*STARTTLS"
        - send: ". STARTTLS"
        - expect: "OK"
        - starttls: true
        - send: ". capability"
        - expect: "CAPABILITY IMAP4rev1"
      tls: true
      tls_config:
        insecure_skip_verify: true

  ldap:
    prober: tcp
    timeout: 5s
    tcp:
      preferred_ip_protocol: "ip4"
      query_response:
        - send: !!binary "MCICAQFgHQIBAwQIbm90YXVzZXKADm5vdHRoZXBhc3N3b3Jk"
        - expect: !!binary "MAwCAQFhBw=="

  postgres:
    prober: tcp
    tcp:
      query_response:
        - send: !!binary AAAACATSFi8=
        - starttls: true

What is the prometheus.yml scrape config.

- job_name: postgres-blackbox
    honor_timestamps: true
    scrape_interval: 5m
    scrape_timeout: 5m
    metrics_path: /probe
    params:
      module: [tcp_connect] ##also tried with modules ldap and postgres

    kubernetes_sd_configs:
    - api_server: null
      role: service
      namespaces:
        names: []
    relabel_configs:
    - source_labels: [__meta_kubernetes_namespace]
      target_label: kubernetes_namespace
    - source_labels: [__meta_kubernetes_service_name]
      target_label: kubernetes_name
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: __address__
      replacement: prometheus-blackbox-exporter.do-prom-athira.svc.cluster.local:9115

What logging output did you get from adding &debug=true to the probe URL?

What did you do that produced an error?

What did you expect to see?

# HELP probe_ssl_earliest_cert_expiry Returns earliest SSL cert expiry date
# TYPE probe_ssl_earliest_cert_expiry gauge
probe_ssl_earliest_cert_expiry 1.655444464e+09
# HELP probe_ssl_last_chain_expiry_timestamp_seconds Returns last SSL chain expiry in unixtime
# TYPE probe_ssl_last_chain_expiry_timestamp_seconds gauge
probe_ssl_last_chain_expiry_timestamp_seconds -6.21355968e+10
# HELP probe_ssl_last_chain_info Contains SSL leaf certificate information
# TYPE probe_ssl_last_chain_info gauge

What did you see instead?

Logs for the probe:
ts=2021-06-18T11:49:24.361029282Z caller=main.go:304 module=tls_connect target=blackblock-hippo-svc.do-blackbox-postgres.svc:8009 level=info msg="Beginning probe" probe=tcp timeout_seconds=60
ts=2021-06-18T11:49:24.361147413Z caller=tcp.go:41 module=tls_connect target=blackblock-hippo-svc.do-blackbox-postgres.svc:8009 level=info msg="Resolving target address" ip_protocol=ip4
ts=2021-06-18T11:49:24.364193802Z caller=tcp.go:41 module=tls_connect target=blackblock-hippo-svc.do-blackbox-postgres.svc:8009 level=info msg="Resolved target address" ip=240.224.66.208
ts=2021-06-18T11:49:24.364219851Z caller=tcp.go:122 module=tls_connect target=blackblock-hippo-svc.do-blackbox-postgres.svc:8009 level=info msg="Dialing TCP with TLS"
ts=2021-06-18T11:49:24.366345309Z caller=main.go:119 module=tls_connect target=blackblock-hippo-svc.do-blackbox-postgres.svc:8009 level=error msg="Error dialing TCP" err="tls: first record does not look like a TLS handshake"
ts=2021-06-18T11:49:24.366385816Z caller=main.go:304 module=tls_connect target=blackblock-hippo-svc.do-blackbox-postgres.svc:8009 level=error msg="Probe failed" duration_seconds=0.00527054
@tobiasbp
Copy link

I'm also wondering if blackbox_exporter supports checking of certs used by PostgreSQL?

openssl supports the postgreqsl protocol nowadays: https://www.feistyduck.com/library/openssl-cookbook/online/ch-testing-with-openssl.html#testing-protocols-that-upgrade-to-ssl

@ful09003
Copy link

👋 just adding my experience with this problem. At my workplace, we have need to monitor an internal SSL/TLS-enabled Postgres. After a lot of unsuccessful attempts to coerce blackbox_exporter into doing that for us (as we use it everywhere else), we decided to instead add functionality into an in-house SSL/TLS management daemon to monitor this for us (with Postgres in mind). It would be great to some day remove that functionality and leverage blackbox_exporter for this need, too!

For the blackbox_exporter maintainers or others looking to do this with an established Golang ecosystem: our functionality for monitoring Postgres began with investigating how this worked, and improving it for our platform: https://github.com/chr4/pg-check-cert. I hope it can serve as a good starting point for how to do this solely in Go (or maybe there are more recent/idiomatic examples somewhere out there 🤷)

@tarvip
Copy link

tarvip commented Aug 24, 2021

Alternatively you can now use ssl_exporter to scrape postgres cert metrics. Postgres support is available since version v2.3.1.

@tacho tacho linked a pull request Aug 18, 2023 that will close this issue
Open
@tacho
Copy link

tacho commented Aug 18, 2023

I also encountered this, and the issue is that currently the blackbox exporter query response expect matches operate in line mode, whereas PostgreSQL sends a single byte as a response to the SSLRequest, so it never matches and the exporter times out. The linked PR fixes this and adds a working example. Using it, I have been able to successfully scrape Postgres certificate metrics.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat: Implement byte matching in TCP query responses tacho/blackbox_exporter
5 participants
@tacho @tarvip @tobiasbp @ful09003 @atmanohar