wait_for_connection to localhost explicitly specified in the inventory and ansible_python_interpreter set to ansible_playbook_python fails with 'timed out waiting for ping module test: 'int' object has no attribute 'template'"'

[pstrzelczak]

Reproduction steps:

$ docker run --rm -it rockylinux:8.9.20231119 bash
$ yum install -y python38
$ pip3 install ansible==6.7.0
$ pip3 install mitogen=0.3.7
$ ANSIBLE_STRATEGY_PLUGINS=/usr/local/lib/python3.8/site-packages/ansible_mitogen/plugins/strategy ANSIBLE_STRATEGY=mitogen_linear ansible -i /tmp/hosts.yml localhost -m ansible.builtin.wait_for_connection -vvv

where

# $ cat /tmp/hosts.yml 
all:
        hosts:
                localhost:
                        ansible_connection: local
                        ansible_python_interpreter: "{{ ansible_playbook_python }}"

Result:

localhost | FAILED! => {
    "changed": false,
    "elapsed": 600,
    "msg": "timed out waiting for ping module test: 'int' object has no attribute 'template'"
}

Suprisingly, with implicit inventory it works fine:

$ ANSIBLE_STRATEGY_PLUGINS=/usr/local/lib/python3.8/site-packages/ansible_mitogen/plugins/strategy ANSIBLE_STRATEGY=mitogen_linear ansible localhost -m ansible.builtin.wait_for_connection
[WARNING]: No inventory was parsed, only implicit localhost is available
localhost | SUCCESS => {
    "changed": false,
    "elapsed": 0
}

• Which version of Ansible are you running?

$ ansible --version
ansible [core 2.13.13]
(...)
  python version = 3.8.17 (default, Aug 10 2023, 18:03:39) [GCC 8.5.0 20210514 (Red Hat 8.5.0-20)]
  jinja version = 3.1.3
  libyaml = True

• Is your version of Ansible patched in any way?
  No

• Are you running with any custom modules, or module_utils loaded?
  No

• Have you tried the latest master version from Git?
  Yes

• Do you have some idea of what the underlying problem may be?
  https://mitogen.networkgenomics.com/ansible_detailed.html#common-problems has
  instructions to help figure out the likely cause and how to gather relevant
  logs.
  No

• Mention your host and target OS and versions
  Rocky 8.8

• Mention your host and target Python versions
  Python 3.8

• If reporting a performance issue, mention the number of targets and a rough
  description of your workload (lots of copies, lots of tiny file edits, etc.)

• If reporting a crash or hang in Ansible, please rerun with -vvv and include
  200 lines of output around the point of the error, along with a full copy of
  any traceback or error text in the log. Beware "-vvv" may include secret
  data! Edit as necessary before posting.

• If reporting any kind of problem with Ansible, please include the Ansible
  version along with output of "ansible-config dump --only-changed".
  Empty

EDIT: 2024-10-07 @moreati Syntax highlighting

[moreati]

Now I've done a lot of digging around in ansible_mitogen/transport_config.py and ansible_mitogen/connection.py I can probably diagnose this. Prime suspect is

mitogen/ansible_mitogen/connection.py

Line 944 in 8e64459

templar=0,

and 833e284 probably contains the building blocks to fix it.

[moreati]

Still occurs with master,

➜  mitogen git:(master) ✗ ANSIBLE_STRATEGY=mitogen_linear ansible -i issue1079_inv.yml localhost -m ansible.builtin.wait_for_connection -a "timeout=10"     
localhost | FAILED! => {
    "changed": false,
    "elapsed": 10,
    "msg": "timed out waiting for ping module test: 'int' object has no attribute 'template'"
}

same -vvv

➜  mitogen git:(master) ✗ ANSIBLE_STRATEGY=mitogen_linear ansible -i issue1079_inv.yml localhost -m ansible.builtin.wait_for_connection -a "timeout=10" -vvv
ansible [core 2.17.5]
  config file = /Users/alex/src/mitogen/ansible.cfg
  configured module search path = ['/Users/alex/src/mitogen/tests/ansible/lib/modules']
  ansible python module location = /Users/alex/.local/share/uv/tools/ansible-core/lib/python3.13/site-packages/ansible
  ansible collection location = /Users/alex/.ansible/collections:/usr/share/ansible/collections
  executable location = /Users/alex/.local/bin/ansible
  python version = 3.13.0 (main, Oct 16 2024, 08:05:40) [Clang 18.1.8 ] (/Users/alex/.local/share/uv/tools/ansible-core/bin/python)
  jinja version = 3.1.4
  libyaml = True
Using /Users/alex/src/mitogen/ansible.cfg as config file
host_list declined parsing /Users/alex/src/mitogen/issue1079_inv.yml as it did not pass its verify_file() method
script declined parsing /Users/alex/src/mitogen/issue1079_inv.yml as it did not pass its verify_file() method
Parsed /Users/alex/src/mitogen/issue1079_inv.yml inventory source with yaml plugin
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.
[top  88106] 16:20:03.219759 D ansible_mitogen.process: inherited open file limits: soft=1048575 hard=(infinity)
[top  88106] 16:20:03.219842 D ansible_mitogen.process: max open files already set to hard limit: 524288
[mux  88108] 16:20:03.224877 D mitogen.service: Pool(81a0, size=32, th='MainThread'): initialized
[mux  88108] 16:20:03.225766 D ansible_mitogen.process: Service pool configured: size=32
wait_for_connection: attempting ping module test
...
wait_for_connection: attempting ping module test
[task 88109] 16:20:13.271395 D ansible_mitogen.mixins: _remove_tmp_path(None)
[task 88109] 16:20:13.273145 D ansible_mitogen.mixins: _remove_tmp_path(None)
localhost | FAILED! => {
    "changed": false,
    "elapsed": 10,
    "msg": "timed out waiting for ping module test: 'int' object has no attribute 'template'"
}
[mux  88108] 16:20:13.280671 D mitogen: Waker(fd=36/38): disconnecting
[mux  88108] 16:20:13.282092 D mitogen.service: thread mitogen.Pool.81a0.0 exiting gracefully
[mux  88108] 16:20:13.283423 D mitogen.service: thread mitogen.Pool.81a0.1 exiting gracefully
[mux  88108] 16:20:13.284447 D mitogen.service: thread mitogen.Pool.81a0.5 exiting gracefully
[mux  88108] 16:20:13.285470 D mitogen: Router(Broker(ee40)): stats: 0 module requests in 0 ms, 0 sent (0 ms minify time), 0 negative responses. Sent 0.0 kb total, 0.0 kb avg.
[mux  88108] 16:20:13.286389 D mitogen.service: thread mitogen.Pool.81a0.6 exiting gracefully
...
[mux  88108] 16:20:13.304953 D mitogen.service: thread mitogen.Pool.81a0.30 exiting gracefully
[mux  88108] 16:20:13.305462 D mitogen.service: thread mitogen.Pool.81a0.31 exiting gracefully
[mux  88108] 16:20:13.306406 D mitogen.service: FileService().on_shutdown()
[top  88106] 16:20:13.308218 D ansible_mitogen.process: multiplexer 0 PID 88108 exited with return code 0

doesn't occur with vanilla Ansible

➜  mitogen git:(master) ✗ ANSIBLE_STRATEGY=linear ansible -i issue1079_inv.yml localhost -m ansible.builtin.wait_for_connection -a "timeout=10"     
[WARNING]: Reset is not implemented for this connection
localhost | SUCCESS => {
    "changed": false,
    "elapsed": 0
}

all:
  hosts:
    localhost:
      ansible_connection: local
      ansible_python_interpreter: "{{ ansible_playbook_python }}"

[moreati]

Untemplated ansible_interpreter_python makes the problem go away

➜  mitogen git:(master) ✗ ANSIBLE_STRATEGY=mitogen_linear ansible -i issue1079_inv.yml localhost -m ansible.builtin.wait_for_connection -a "timeout=20"                               
localhost | SUCCESS => {
    "changed": false,
    "elapsed": 0
}

all:
  hosts:
    localhost:
      ansible_connection: local
      ansible_python_interpreter: python3

[moreati]

I need to ponder this some more. The templar made available in '_mitogen.smuggled.reset_connection' is only available during a meta: reset_connection task.

[moreati]

Note to self: There's already some special handling for wait_for_connection

➜  mitogen git:(issue1079-templated-python-interpreter) ✗ ag wait_for_connection   
ansible_mitogen/mixins.py
367:        # wait_for_connection, the `ping` test from Ansible won't pass because we lost connection
369:        # see https://github.com/dw/mitogen/issues/655 and Ansible's `wait_for_connection` module for more info
370:        if module_name == 'ansible.legacy.ping' and type(self).__name__ == 'wait_for_connection':

ansible_mitogen/services.py
162:        # and then a `wait_for_connection` right afterwards

ansible_mitogen/connection.py
865:        Ansible 2.4 wait_for_connection plug-in.
984:    # Compatibility with Ansible 2.4 wait_for_connection plug-in.

tests/ansible/integration/context_service/reconnection.yml
24:    - wait_for_connection:

tests/ansible/regression/issue_655__wait_for_connection_error.yml
2:# Spins up a Centos8 container and runs the wait_for_connection test inside of it
7:- name: regression/issue_655__wait_for_connection_error.yml
85:              wait_for_connection:

tests/ansible/regression/all.yml
15:- import_playbook: issue_655__wait_for_connection_error.yml

docs/changelog.rst
242:* :gh:issue:`655` wait_for_connection gives errors