Diagnosing LDAP authentication issues

[dylannorthrup]

Summary

I am continuing this discussion per the suggestion from @lukebakken and migrating it from a Discord Thread to a discussion here. I'm trying to configure LDAP authentication for RabbitMQ containers against our Active Directory infrastructure, but am encountering errors. I am able to successfully query the LDAP server using ldapsearch and ldapwhoami.

I do not know and have never directly used erlang, so the stack traces shown in the error logs are not actionable. Also, I was unable to find any standalone LDAP clients using the erlang-eldap library to do validation outside the context of RabbitMQ and solely focused on LDAP.

I'd like to get suggestions on next steps for identifying the cause of the issue and, hopefully, correcting the fault.

Caveat

The configs, scripts, and logs are sanitized¹. If something looks not quite right, I'm happy to double check the source to verify if the sanitizing went awry.

Overview

RabbitMQ Config

I have a RabbitMQ cluster running in containers using the rabbitmq:4.0-management image. The config section is as follows:

auth_backends.1 = internal
auth_backends.2.authn = ldap
auth_backends.2.authz = internal

auth_ldap.servers.1 = ldap.ad.example.com
auth_ldap.use_ssl = true
auth_ldap.port = 636
auth_ldap.dn_lookup_bind.user_dn = CN=mqadmin-lab,OU=ServiceAccounts,OU=LAB,OU=RabbitMQ,OU=Services,OU=ExampleOrg,DC=ad,DC=example,DC=com
auth_ldap.dn_lookup_bind.password = RedactedPassword
auth_ldap.dn_lookup_attribute = sAMAccountName
auth_ldap.dn_lookup_base = DC=ad,DC=example,DC=com
auth_ldap.log = network_unsafe

Errors when accessing Management UI

With the configuration above, I get the following error when attempting to connect to the Web UI:

[debug] <0.1021.0> User 'mqadmin-lab' failed authentication by backend rabbit_auth_backend_internal
[info] <0.1021.0> LDAP CHECK: login for mqadmin-lab
[info] <0.1021.0>     LDAP connecting to servers: ["ldap.ad.example.com"]
[debug] <0.1021.0> User 'mqadmin-lab' authentication failed with exit:{noproc,
[debug] <0.1021.0>                                                          {gen_server2,call,
[debug] <0.1021.0>                                                           [ldap_pool,
[debug] <0.1021.0>                                                            {next_free,
[debug] <0.1021.0>                                                             <0.1021.0>},
[debug] <0.1021.0>                                                            infinity]}}:
[debug] <0.1021.0> [{gen_server2,call,3,[{file,"gen_server2.erl"},{line,346}]},
[debug] <0.1021.0>  {worker_pool,submit,3,[{file,"worker_pool.erl"},{line,91}]},
[debug] <0.1021.0>  {rabbit_auth_backend_ldap,user_login_authentication,2,
[debug] <0.1021.0>      [{file,"rabbit_auth_backend_ldap.erl"},{line,72}]},
[debug] <0.1021.0>  {rabbit_access_control,try_authenticate,3,
[debug] <0.1021.0>      [{file,"rabbit_access_control.erl"},{line,87}]},
[debug] <0.1021.0>  {rabbit_access_control,try_authenticate_and_try_authorize,4,
[debug] <0.1021.0>      [{file,"rabbit_access_control.erl"},{line,78}]},
[debug] <0.1021.0>  {lists,foldl_1,3,[{file,"lists.erl"},{line,2151}]},
[debug] <0.1021.0>  {rabbit_access_control,check_user_login,2,
[debug] <0.1021.0>      [{file,"rabbit_access_control.erl"},{line,37}]},
[debug] <0.1021.0>  {rabbit_web_dispatch_access_control,is_authorized,8,
[debug] <0.1021.0>      [{file,"rabbit_web_dispatch_access_control.erl"},{line,144}]}]
[warning] <0.1021.0> HTTP access denied: User 'mqadmin-lab' authentication failed with internal error. Enable debug logs to see the real error.

Errors when running rabbitmqctl authenticate_user

Trying to use rabbitmqctl to trigger user authentication outside the Management UI I get the following error:

[debug] <0.1986.0> User 'mqadmin-lab' authentication failed with exit:{noproc,
[debug] <0.1986.0>                                                          {gen_server2,call,
[debug] <0.1986.0>                                                           [ldap_pool,
[debug] <0.1986.0>                                                            {next_free,
[debug] <0.1986.0>                                                             <0.1986.0>},
[debug] <0.1986.0>                                                            infinity]}}:
[debug] <0.1986.0> [{gen_server2,call,3,[{file,"gen_server2.erl"},{line,346}]},
[debug] <0.1986.0>  {worker_pool,submit,3,[{file,"worker_pool.erl"},{line,91}]},
[debug] <0.1986.0>  {rabbit_auth_backend_ldap,user_login_authentication,2,
[debug] <0.1986.0>                            [{file,"rabbit_auth_backend_ldap.erl"},{line,72}]},
[debug] <0.1986.0>  {rabbit_access_control,try_authenticate,3,
[debug] <0.1986.0>                         [{file,"rabbit_access_control.erl"},{line,87}]},
[debug] <0.1986.0>  {rabbit_access_control,try_authenticate_and_try_authorize,4,
[debug] <0.1986.0>                         [{file,"rabbit_access_control.erl"},{line,78}]},
[debug] <0.1986.0>  {lists,foldl_1,3,[{file,"lists.erl"},{line,2151}]},
[debug] <0.1986.0>  {rabbit_access_control,check_user_login,2,
[debug] <0.1986.0>                         [{file,"rabbit_access_control.erl"},{line,37}]},
[debug] <0.1986.0>  {erpc,execute_call,4,[{file,"erpc.erl"},{line,1250}]}]

Testing using OpenLDAP tooling

The following script uses the openldap clients ldapsearch and ldapwhoami (from the ldap-utils dpkg) with the bindDN and password defined above.

#!/bin/bash

pw="RedactedPassword"
bindDN="CN=mqadmin-lab,OU=ServiceAccounts,OU=LAB,OU=RabbitMQ,OU=Services,OU=ExampleOrg,DC=ad,DC=example,DC=com"

doSearch() {
  echo "Running dn search for '(&(samaccountname=${1}))'"
    /usr/bin/ldapsearch -H ldaps://ldap.ad.example.com:636 -D "${bindDN}" -w "${pw}" -b "DC=ad,DC=example,DC=com" -o ldif-wrap=no -E pr=1000/noprompt -s sub "(&(samaccountname=${1}))" dn | grep '^dn:'
}

echo "== Searching for Dylan"
doSearch "dylan"

echo "== Attempting to get info for bind DN:"
echo "    ${bindDN}"
/usr/bin/ldapwhoami -v -H ldaps://ldap.ad.example.com:636 -D "${bindDN}" -x -w "${pw}" -d 16384

Here's the output from that script running inside the RabbitMQ container:

== Searching for Dylan
Running dn search for '(&(samaccountname=dylan))'
dn: CN=dylan,OU=Geeks,OU=People,DC=ad,DC=example,DC=com
== Attempting to get info for bind DN:
    CN=mqadmin-lab,OU=ServiceAccounts,OU=LAB,OU=RabbitMQ,OU=Services,OU=ExampleOrg,DC=ad,DC=example,DC=com
ldap_initialize( ldaps://ldap.ad.example.com:636/??base )
u:EXAMPLEAD\mqadmin-lab
Result: Success (0)

Conclusion

I'm happy to run additional commands, modify configs, provide logs, etc. Please be aware that, if I'm asked to run erlang scripts directly, it's likely you'll want to "explain like i'm 5" . . . or at least like I'm a college freshman in the first week of "Intro to Erlang". I expect I'll be making use of the erlang environment inside the RabbitMQ container since that will be a common base of reference.

Thank you in advance for any information, assistance, guidance, etc you are able to provide.

Footnotes:

Footnotes

1. Names of users and hosts; LDAP/AD hierarchy paths; and passwords have been modified, but should have been modified in a consistent fashion. Nothing else should have been modified or redacted. The modifications are done semi-manually, so they should be consistent, but there is always the chance I've fat-fingered something during the copy-paste-edit-copy-paste workflow. If something is inconsistent and within the scope mentioned, I'm happy to double check and verify the correctness of what I'm providing. ↩

[lukebakken]

Hello! What is the version of all of the software you're using?

• RabbitMQ
• Erlang
• OpenLDAP
• Windows / Active Directory

If you restart your RabbitMQ nodes, does the problem resolve itself?

[lukebakken]

In both cases, I got errors consistent with my original post above.

Do you immediately get the errors, or after some period of time?

[dylannorthrup]

The error shows up in the logs almost immediately...

#  time rabbitmqctl authenticate_user snap-mqadmin-lab RedactedPassword
Authenticating user "mqadmin-lab" ...
Error:
Error: failed to authenticate user "mqadmin-lab"
User 'mqadmin-lab' authentication failed with internal error. Enable debug logs to see the real error.

real    0m0.724s
user    0m0.621s
sys     0m0.420s

[lukebakken]

Sorry, I may not have been clear.

After a restart, if you immediately try to authenticate a user using LDAP, do you see the noproc errors in the logs?

I'm trying to determine if this erroneous behavior exists right after a full restart, or takes some time to manifest. Thanks for investigating, because I have never seen this issue nor can I reproduce it.

[dylannorthrup]

I will test this ASAP. Out of curiosity, should there be a difference if I an restarting a single node, or should I be restarting all nodes in the cluster?

[lukebakken]

Restart all nodes. Can you reproduce this in a non-production environment? If so, stop all applications, restart all nodes, then re-try your authentication command above.

[dylannorthrup]

The RabbitMQ instance runs in a container, so it took a second to log in and run rabbitmqctl authenticate_user, but here's the logs showing the tail part of the rabbitmq startup followed by the running of the rabbitmqctl command.

2025-04-23 20:10:50.103394+00:00 [debug] <0.615.0> Will reschedule virtual host process reconciliation after 10 seconds
2025-04-23 20:10:50.185165+00:00 [info] <0.10.0> Time to start RabbitMQ: 3818 ms
2025-04-23 20:10:51.096520+00:00 [debug] <0.1051.0> User 'mqadmin-lab' failed authentication by backend rabbit_auth_backend_internal
2025-04-23 20:10:51.106069+00:00 [info] <0.1051.0> LDAP CHECK: login for mqadmin-lab
2025-04-23 20:10:51.106172+00:00 [info] <0.1051.0>     LDAP connecting to servers: ["ldap.ad.ufl.edu"]
2025-04-23 20:10:51.106306+00:00 [debug] <0.1051.0> User 'mqadmin-lab' authentication failed with exit:{noproc,
2025-04-23 20:10:51.106306+00:00 [debug] <0.1051.0>                                                          {gen_server2,call,
2025-04-23 20:10:51.106306+00:00 [debug] <0.1051.0>                                                           [ldap_pool,
2025-04-23 20:10:51.106306+00:00 [debug] <0.1051.0>                                                            {next_free,
2025-04-23 20:10:51.106306+00:00 [debug] <0.1051.0>                                                             <0.1051.0>},
2025-04-23 20:10:51.106306+00:00 [debug] <0.1051.0>                                                            infinity]}}:
2025-04-23 20:10:51.106306+00:00 [debug] <0.1051.0> [{gen_server2,call,3,[{file,"gen_server2.erl"},{line,346}]},
2025-04-23 20:10:51.106306+00:00 [debug] <0.1051.0>  {worker_pool,submit,3,[{file,"worker_pool.erl"},{line,91}]},
2025-04-23 20:10:51.106306+00:00 [debug] <0.1051.0>  {rabbit_auth_backend_ldap,user_login_authentication,2,
2025-04-23 20:10:51.106306+00:00 [debug] <0.1051.0>                            [{file,"rabbit_auth_backend_ldap.erl"},{line,72}]},
2025-04-23 20:10:51.106306+00:00 [debug] <0.1051.0>  {rabbit_access_control,try_authenticate,3,
2025-04-23 20:10:51.106306+00:00 [debug] <0.1051.0>                         [{file,"rabbit_access_control.erl"},{line,87}]},
2025-04-23 20:10:51.106306+00:00 [debug] <0.1051.0>  {rabbit_access_control,try_authenticate_and_try_authorize,4,
2025-04-23 20:10:51.106306+00:00 [debug] <0.1051.0>                         [{file,"rabbit_access_control.erl"},{line,78}]},
2025-04-23 20:10:51.106306+00:00 [debug] <0.1051.0>  {lists,foldl_1,3,[{file,"lists.erl"},{line,2151}]},
2025-04-23 20:10:51.106306+00:00 [debug] <0.1051.0>  {rabbit_access_control,check_user_login,2,
2025-04-23 20:10:51.106306+00:00 [debug] <0.1051.0>                         [{file,"rabbit_access_control.erl"},{line,37}]},
2025-04-23 20:10:51.106306+00:00 [debug] <0.1051.0>  {erpc,execute_call,4,[{file,"erpc.erl"},{line,1250}]}]
2025-04-23 20:10:54.701652+00:00 [debug] <0.1061.0> User 'mqadmin-lab' failed authentication by backend rabbit_auth_backend_internal

[lukebakken]

So weird! Just for curiosity's sake, could you please run rabbitmq-plugins list on all nodes to verify that rabbitmq_auth_backend_ldap is enabled on all nodes?

[dylannorthrup]

rabbitmq_auth_backend_ldap is not enabled.... I used rabbitmqctl to enable it, but will make sure the enabled_plugins file on all systems contains it. Once I do that, I'll re-test and provide the results.

[lukebakken]

Ah, yes, that will fix the issue. I would have expected the use of auth_ldap in rabbitmq.conf to keep the broker from booting, if the plugin isn't enabled. I will double-check that behavior.

[dylannorthrup]

Now that I have the plugin enabled (after some Docker shenanigans to get the container to mount the existing enabled_plugins file I thought it was already mounting), it gives a much different and more informative error.

2025-04-23 20:44:23.540081+00:00 [info] <0.1121.0> LDAP CHECK: login for mqadmin-lab
2025-04-23 20:44:23.540134+00:00 [info] <0.1121.0>     LDAP connecting to servers: ["ldap.ad.example.com"]
2025-04-23 20:44:23.547210+00:00 [info] <0.1122.0>     LDAP network traffic: Connect: "ldap.ad.example.com" failed {error,
2025-04-23 20:44:23.547210+00:00 [info] <0.1122.0>                                                              {options,
2025-04-23 20:44:23.547210+00:00 [info] <0.1122.0>                                                               incompatible,
2025-04-23 20:44:23.547210+00:00 [info] <0.1122.0>                                                               [{verify,
2025-04-23 20:44:23.547210+00:00 [info] <0.1122.0>                                                                 verify_peer},
2025-04-23 20:44:23.547210+00:00 [info] <0.1122.0>                                                                {cacerts,
2025-04-23 20:44:23.547210+00:00 [info] <0.1122.0>                                                                 undefined}]}}
2025-04-23 20:44:23.547210+00:00 [info] <0.1122.0>
2025-04-23 20:44:23.547399+00:00 [info] <0.578.0>     LDAP connect error: {error,"connect failed"}
2025-04-23 20:44:23.547781+00:00 [info] <0.1121.0>     LDAP connecting to servers: ["ldap.ad.example.com"]
2025-04-23 20:44:23.547973+00:00 [info] <0.1123.0>     LDAP network traffic: Connect: "ldap.ad.example.com" failed {error,
2025-04-23 20:44:23.547973+00:00 [info] <0.1123.0>                                                              {options,
2025-04-23 20:44:23.547973+00:00 [info] <0.1123.0>                                                               incompatible,
2025-04-23 20:44:23.547973+00:00 [info] <0.1123.0>                                                               [{verify,
2025-04-23 20:44:23.547973+00:00 [info] <0.1123.0>                                                                 verify_peer},
2025-04-23 20:44:23.547973+00:00 [info] <0.1123.0>                                                                {cacerts,
2025-04-23 20:44:23.547973+00:00 [info] <0.1123.0>                                                                 undefined}]}}
2025-04-23 20:44:23.547973+00:00 [info] <0.1123.0>
2025-04-23 20:44:23.548165+00:00 [info] <0.578.0>     LDAP connect error: {error,"connect failed"}
2025-04-23 20:44:23.548444+00:00 [info] <0.1121.0> LDAP DECISION: login for mqadmin-lab: {error,ldap_connect_error}

This definitely gives signal about the issue. Thanks for your help and your patience, @lukebakken!

[lukebakken]

OK, great. This is much better 😸

The problem is that you are using a TLS/SSL connection to your LDAP server, but no CA certificate has been configured to validate the server's identity. You can resolve this in two ways:

• Turn off peer verification:

  auth_ldap.ssl_options.verify = verify_none
  

• Get the CA certificate (or concatenate the entire signing chain into one file) and make that available to RabbitMQ:

  auth_ldap.ssl_options.cacertfile = /abs/path/to/ca_certificate.pem
  

[lukebakken]

rabbitmq/rabbitmq-server#13783