Fixed SSL IP address verification

[hikui]

Problem description

Our server connects to MongoDB via an IP address. We created an SSL certificate and added the ip address in. However the MongoDB driver fails to verify the host name.

After digging deep into the code, I found that the driver sets server_name_indication for IP addresses, which leads to wrong ReferenceID being tagged as :dns_id. According to https://github.com/erlang/otp/blob/OTP-27.0/lib/public_key/src/public_key.erl#L2771, this will not match any IP address in the certificate.

This PR added IP address detection and skip setting SNI when IP address is presented.

Experiments

Experiment 1:

defmodule SSLConn do
  def pkix_verify_hostname(fqdn, host) do
    IO.inspect("#{inspect(fqdn)}, #{inspect(host)}")
    :default
  end

  def connect() do
    host = ~c"127.0.0.1"
    port = 27018

    {:ok, socket} = :gen_tcp.connect(host, port, [:binary, active: false, packet: :raw, nodelay: true])

    :ssl.connect(socket,
      cacertfile: "/home/hmiao/Workspace/mongo-ca.pem",
      verify: :verify_peer,
      customize_hostname_check: [match_fun: &pkix_verify_hostname/2],
      server_name_indication: ~c"127.0.0.1"
    )
  end
end

Output:

"{:dns_id, ~c\"127.0.0.1\"}, {:iPAddress, [127, 0, 0, 1]}"
{:error,
 {:tls_alert,
  {:handshake_failure,
   ~c"TLS client: In state wait_cert at ssl_handshake.erl:2140 generated CLIENT ALERT: Fatal - Handshake Failure\n {bad_cert,hostname_check_failed}"}}}

Experiment 2

defmodule SSLConn do
  def pkix_verify_hostname(fqdn, host) do
    IO.inspect("#{inspect(fqdn)}, #{inspect(host)}")
    :default
  end

  def connect() do
    host = ~c"127.0.0.1"
    port = 27018

    {:ok, socket} = :gen_tcp.connect(host, port, [:binary, active: false, packet: :raw, nodelay: true])

    :ssl.connect(socket,
      cacertfile: "/home/hmiao/Workspace/mongo-ca.pem",
      verify: :verify_peer,
      customize_hostname_check: [match_fun: &pkix_verify_hostname/2],
    )
  end
end

Output

"{:ip, {127, 0, 0, 1}}, {:iPAddress,  [127, 0, 0, 1]}"
{:ok,
 {:sslsocket, {:gen_tcp, #Port<0.11>, :tls_connection, :undefined},
  [#PID<0.372.0>, #PID<0.371.0>]}}

[zookzook]

Thank you for your clear and understandable description.