Can't seem to change password of RabbitMQ user

[vkurup]

#182 seems to have been fixed at some point, but now I'm seeing a similar, though silent problem. During the RabbitMQ upgrade from 3.5 -> 3.6.0, multiple people noted that they had to delete and then recreate the RabbitMQ user, which means that our RabbitMQ state which is supposed to force the user to have the new specified password is working. It also doesn't seem to throw an error

vkurup@ip-172-31-17-214:~$ sudo salt -G "roles:salt-master" -ldebug state.sls project.queue
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: ip.eu-west-1.compute.internal
[DEBUG   ] Missing configuration file: /home/vkurup/.saltrc
[DEBUG   ] Configuration file path: /etc/salt/master
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: ip.eu-west-1.compute.internal
[DEBUG   ] Missing configuration file: /home/vkurup/.saltrc
[DEBUG   ] MasterEvent PUB socket URI: ipc:///var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: ipc:///var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Sending event - data = {'_stamp': '2016-01-04T20:58:40.395430'}
[DEBUG   ] LazyLoaded local_cache.get_load
[DEBUG   ] get_iter_returns for jid 20160104205840401321 sent to set(['ip.eu-west-1.compute.internal']) will timeout at 20:59:40.413946
[DEBUG   ] jid 20160104205840401321 return from ip.eu-west-1.compute.internal
[DEBUG   ] LazyLoaded highstate.output
[DEBUG   ] LazyLoaded nested.output
[DEBUG   ] LazyLoaded nested.output
[DEBUG   ] LazyLoaded nested.output
ip.eu-west-1.compute.internal:
  Name: deb http://www.rabbitmq.com/debian/ testing main - Function: pkgrepo.managed - Result: Clean
  Name: rabbitmq-server - Function: pkg.latest - Result: Clean
  Name: /etc/rabbitmq/rabbitmq.config - Function: file.managed - Result: Clean
  Name: rabbitmq-server - Function: service.running - Result: Clean
  Name: guest - Function: rabbitmq_user.absent - Result: Clean
  Name: ufw - Function: pkg.installed - Result: Clean
  Name: ufw - Function: service.running - Result: Clean
  Name: firewall_policy - Function: ufw.default - Result: Changed
  Name: firewall_status - Function: ufw.enabled - Result: Changed
  Name: rescue_id_production - Function: rabbitmq_vhost.present - Result: Clean
  Name: rescue_id_production - Function: rabbitmq_user.present - Result: Changed
  Name: 5672 - Function: ufw.allow - Result: Clean

Summary
-------------
Succeeded: 12 (changed=3)
Failed:     0
-------------
Total states run:     12

After doing that, I restarted the celery processes and continued to see connection errors in the celery logs, with this in the RabbitMQ log:

=ERROR REPORT==== 4-Jan-2016::21:07:51 ===
Error on AMQP connection <0.30040.3> (127.0.0.1:59865 -> 127.0.0.1:5672, state: starting):
AMQPLAIN login refused: user 'rescue_id_production' - invalid credentials

This suggests to me that the salt state is not successfully changing the password, but it's also not throwing an error.

[vkurup]

Just FYI, the workaround for this issue is to delete the rabbitmq project user and redeploy:

vkurup@localhost:~$ dpkg -l | grep rabbit
ii  rabbitmq-server                    3.6.0-1                                all          Multi-protocol messaging broker
vkurup@localhost:~$ sudo rabbitmqctl list_users
Listing users ...
epicallieshq_staging    []
vkurup@localhost:~$ sudo rabbitmqctl delete_user epicallieshq_staging
Deleting user "epicallieshq_staging" ...
vkurup@localhost:~$ logout
Connection to staging.epicallies.com closed.
vkurup@caktus014:~/dev/ea $ fab staging deploy