Installing commcare-cloud on Ubuntu 22.04 failing on elasticsearch

[alhamdubello]

I making a new install of commcare and its failing on the elasticsearch task.

TASK [elasticsearch : Check that Elasticsearch is up after 20 seconds] ****************************************************************
task path: /home/abello/commcare-cloud/src/commcare_cloud/ansible/roles/elasticsearch/tasks/main.yml:181

fatal: [172.16.2.74]: FAILED! => {
    "changed": false,
    "elapsed": 300,
    "invocation": {
        "module_args": {
            "active_connection_states": [
                "ESTABLISHED",
                "FIN_WAIT1",
                "FIN_WAIT2",
                "SYN_RECV",
                "SYN_SENT",
                "TIME_WAIT"
            ],
            "connect_timeout": 5,
            "delay": 20,
            "exclude_hosts": null,
            "host": "172.16.2.74",
            "msg": null,
            "path": null,
            "port": 9200,
            "search_regex": null,
            "sleep": 1,
            "state": "started",
            "timeout": 300
        }
    },
    "msg": "Timeout when waiting for 172.16.2.74:9200"
}
<172.16.2.75> (1, b'\r\n\r\n{"elapsed": 300, "failed": true, "msg": "Timeout when waiting for 172.16.2.74:9200", "invocation": {"module_args": {"host": "172.16.2.74", "port": 9200, "delay": 20, "timeout": 300, "connect_timeout": 5, "active_connection_states": ["ESTABLISHED", "FIN_WAIT1", "FIN_WAIT2", "SYN_RECV", "SYN_SENT", "TIME_WAIT"], "state": "started", "sleep": 1, "path": null, "search_regex": null, "exclude_hosts": null, "msg": null}}}\r\n', b'Shared connection to 172.16.2.75 closed.\r\n')
<172.16.2.75> Failed to connect to the host via ssh: Shared connection to 172.16.2.75 closed.
<172.16.2.75> ESTABLISH SSH CONNECTION FOR USER: ansible
<172.16.2.75> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="ansible"' -o ConnectTimeout=30 -o UserKnownHostsFile=/home/abello/environments/coronams/known_hosts -o 'ControlPath="/home/abello/.ansible/cp/a3d1d4c7d4"' 172.16.2.75 '/bin/sh -c '"'"'rm -f -r /tmp/ansible-tmp-1742889533.4132552-31561-192082091576929/ > /dev/null 2>&1 && sleep 0'"'"''
<172.16.2.75> (0, b'', b'')
fatal: [172.16.2.75]: FAILED! => {
    "changed": false,
    "elapsed": 300,
    "invocation": {
        "module_args": {
            "active_connection_states": [
                "ESTABLISHED",
                "FIN_WAIT1",
                "FIN_WAIT2",
                "SYN_RECV",
                "SYN_SENT",
                "TIME_WAIT"
            ],
            "connect_timeout": 5,
            "delay": 20,
            "exclude_hosts": null,
            "host": "172.16.2.74",
            "msg": null,
            "path": null,
            "port": 9200,
            "search_regex": null,
            "sleep": 1,
            "state": "started",
            "timeout": 300
        }
    },
    "msg": "Timeout when waiting for 172.16.2.74:9200"
}

The previous task in the install did not start elasticsearch service

curl: (7) Failed to connect to 172.16.2.74 port 9200 after 0 ms: Connection refused```

checking the status of elasticsearch

``` elasticsearch.service - Elasticsearch 5.6.16
     Loaded: loaded (/etc/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2025-03-25 08:11:00 UTC; 466ms ago
       Docs: http://www.elastic.co
   Main PID: 111138 (java)
      Tasks: 15 (limit: 19055)
     Memory: 81.4M
        CPU: 878ms
     CGroup: /system.slice/elasticsearch.service
             └─111138 /bin/java -Xms2048m -Xmx2048m -Djava.io.tmpdir=/opt/data/elasticsearch-5.6.16/tmp -XX:-HeapDumpOnOutOfMemoryErro>

Mar 25 08:11:00 db1 systemd[1]: Started Elasticsearch 5.6.16. ```

checking elasticsearch logs we can see the service fails

ode_name][es0] publish_address {172.16.2.74:9300}, bound_addresses {172.16.2.74:9300}
Mar 25 08:12:32 db1 elasticsearch[111659]: [2025-03-25T08:12:32,214][INFO ][o.e.b.BootstrapChecks ] [
Mar 25 08:12:32 db1 elasticsearch[111659]: ode_name][es0] bound or publishing to a non-loopback address, enforcing bootstrap checks
Mar 25 08:12:32 db1 elasticsearch[111659]: ERROR: [1] bootstrap checks failed
Mar 25 08:12:32 db1 elasticsearch[111659]: [1]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
Mar 25 08:12:32 db1 elasticsearch[111659]: 2025-03-25 08:12:32,228 pool-1-thread-1 ERROR Unable to unregister MBeans java.security.AccessControlException: access denied ("javax.management.MBeanServerPermission" "createMBeanServer")
Mar 25 08:12:32 db1 elasticsearch[111659]: at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
Mar 25 08:12:32 db1 elasticsearch[111659]: at java.security.AccessController.checkPermission(AccessController.java:886)
Mar 25 08:12:32 db1 elasticsearch[111659]: at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
Mar 25 08:12:32 db1 elasticsearch[111659]: at java.lang.management.ManagementFactory.getPlatformMBeanServer(ManagementFactory.java:465)
Mar 25 08:12:32 db1 elasticsearch[111659]: at org.apache.logging.log4j.core.jmx.Server.unregisterLoggerContext(Server.java:248)
Mar 25 08:12:32 db1 elasticsearch[111659]: at org.apache.logging.log4j.core.LoggerContext.stop(LoggerContext.java:340)
Mar 25 08:12:32 db1 elasticsearch[111659]: at org.apache.logging.log4j.core.LoggerContext$1.run(LoggerContext.java:281)
Mar 25 08:12:32 db1 elasticsearch[111659]: at org.apache.logging.log4j.core.util.DefaultShutdownCallbackRegistry$RegisteredCancellable.run(DefaultShutdownCallbackRegistry.java:109)
Mar 25 08:12:32 db1 elasticsearch[111659]: at org.apache.logging.log4j.core.util.DefaultShutdownCallbackRegistry.run(DefaultShutdownCallbackRegistry.java:74)
Mar 25 08:12:32 db1 elasticsearch[111659]: at java.lang.Thread.run(Thread.java:750)
Mar 25 08:12:32 db1 elasticsearch[111659]: [2025-03-25T08:12:32,229][INFO ][o.e.n.Node ] [
Mar 25 08:12:32 db1 elasticsearch[111659]: ode_name][es0] stopping ...
Mar 25 08:12:32 db1 elasticsearch[111659]: Exception: java.security.AccessControlException thrown from the UncaughtExceptionHandler in thread "Thread-3"
Mar 25 08:12:32 db1 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=78/CONFIG
Mar 25 08:12:32 db1 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Mar 25 08:12:32 db1 systemd[1]: elasticsearch.service: Consumed 6.149s CPU time.

This looking like a SecComp failure.