Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dream leaks sockets on macOS #347

Open
polytypic opened this issue Sep 13, 2024 · 6 comments
Open

Dream leaks sockets on macOS #347

polytypic opened this issue Sep 13, 2024 · 6 comments
Labels
🐛 bug

Comments

@polytypic
Copy link

On macOS a simple Dream server,

let () =
  Dream.router
    [
      Dream.get "/" (fun _ ->
          Lwt.return
            (Dream.response
               ~headers:[ ("Content-Type", "text/plain") ]
               "On macOS I leak.\n"));
    ]
  |> Dream.logger |> Dream.run ~port:8080

leaks sockets (does not close them properly) and crashes after about 1k requests:

...
13.09.24 13:37:02.468    dream.logger  INFO REQ 1011 GET / ::1:50350 curl/8.7.1
On macOS I leak.
13.09.24 13:37:02.468    dream.logger  INFO REQ 1011 200 in 53 μs
13.09.24 13:37:02.481    dream.logger  INFO REQ 1012 GET / ::1:50351 curl/8.7.1
13.09.24 13:37:02.481    dream.logger  INFO REQ 1012 200 in 46 μs
On macOS I leak.
Fatal error: exception Unix.Unix_error(Unix.EINVAL, "select", "")
Raised by primitive operation at Lwt_engine.select#select in file "src/unix/lwt_engine.ml", line 407, characters 26-60
Called from Lwt_engine.select_based#iter in file "src/unix/lwt_engine.ml", line 348, characters 8-39
Called from Lwt_main.run.run_loop in file "src/unix/lwt_main.ml", line 36, characters 6-49
Called from Lwt_main.run in file "src/unix/lwt_main.ml", line 106, characters 8-13
Re-raised at Lwt_main.run in file "src/unix/lwt_main.ml", line 112, characters 4-13
Called from Dune__exe__Server in file "bin/server/server.ml", lines 2-10, characters 2-41

See here for a full project that reproduces the issue with GitHub actions.
@aantron
Copy link
Owner

aantron commented Sep 13, 2024

Thank you! This appears to be a very serious bug.

For confirmation, could you try with alpha7 and confirm whether conf-libev is installed or not when you observe this issue, as I am concerned by the usage of select, as opposed to kevent?

@polytypic
Copy link
Author

polytypic commented Sep 14, 2024 •
edited
Loading

Yes, I have those locally and they also seem to be installed on the CI machine.

...
  ∗ conf-libev            4-12         [required by dream]
...
  ∗ dream                 1.0.0~alpha7 [required by dream_leak]
...

The stack trace seems to indicate that Unix.select is being used and the 1024 FD_SETSIZE of select triggers the EINVAL.

Output from lsof shows the accumulation of sockets (the below output is actually from another program using dream, i.e. not the minimalist program I wrote to reproduce the issue):

COMMAND     PID         USER   FD    TYPE             DEVICE SIZE/OFF      NODE NAME
server.ex 59846 vesakarvonen  cwd     DIR               1,15      640 157511577 /Users/vesakarvonen/[SNIP]
server.ex 59846 vesakarvonen  txt     REG               1,15 17363256 169063747 /Users/vesakarvonen/[SNIP]
server.ex 59846 vesakarvonen  txt     REG               1,15    58344 168544184 /Library/Preferences/Logging/.plist-cache.h9P6rqdN
server.ex 59846 vesakarvonen  txt     REG               1,15   838768 167839059 /opt/homebrew/Cellar/openssl@3/3.3.2/lib/libssl.3.dylib
server.ex 59846 vesakarvonen  txt     REG               1,15  4222880 167839058 /opt/homebrew/Cellar/openssl@3/3.3.2/lib/libcrypto.3.dylib
server.ex 59846 vesakarvonen    0u    CHR               16,1  0t53247       937 /dev/ttys001
server.ex 59846 vesakarvonen    1u    CHR               16,1  0t53247       937 /dev/ttys001
server.ex 59846 vesakarvonen    2u    CHR               16,1  0t53247       937 /dev/ttys001
server.ex 59846 vesakarvonen    3    PIPE 0x3b301f34a7a4d0a8    16384           ->0x2ed9d2d7b93491af
server.ex 59846 vesakarvonen    4    PIPE 0x2ed9d2d7b93491af    16384           ->0x3b301f34a7a4d0a8
server.ex 59846 vesakarvonen    5u  systm 0xdb74cc20e24433df      0t0           [ctl com.apple.netsrc id 7 unit 58]
server.ex 59846 vesakarvonen    6u   unix 0x2043d34c4baa7b94      0t0           ->0x9ce578178c31b316
server.ex 59846 vesakarvonen    7u   IPv6 0xd0fff8d49aba59ba      0t0       TCP localhost:http-alt (LISTEN)
server.ex 59846 vesakarvonen    8u   IPv6 0x3672a6174266ceee      0t0       TCP localhost:http-alt->localhost:63111 (CLOSED)
server.ex 59846 vesakarvonen    9u   IPv6 0xcee0bbf6b1aa18de      0t0       TCP localhost:http-alt->localhost:63112 (CLOSED)
server.ex 59846 vesakarvonen   10u   IPv6 0xd42159047e5df058      0t0       TCP localhost:http-alt->localhost:63113 (CLOSED)
server.ex 59846 vesakarvonen   11u   IPv6 0x8b00c6f0a2b16cd5      0t0       TCP localhost:http-alt->localhost:63114 (CLOSED)
server.ex 59846 vesakarvonen   12u   IPv6  0x4096f01ead76f5b      0t0       TCP localhost:http-alt->localhost:63115 (CLOSED)
server.ex 59846 vesakarvonen   13u   IPv6 0x9d0d4ffd6c51fe20      0t0       TCP localhost:http-alt->localhost:63116 (CLOSED)
server.ex 59846 vesakarvonen   14u   IPv6 0xd57ede59ad34b152      0t0       TCP localhost:http-alt->localhost:63117 (CLOSED)
server.ex 59846 vesakarvonen   15u   IPv6 0x510f3e337d99c90a      0t0       TCP localhost:http-alt->localhost:63118 (CLOSED)
server.ex 59846 vesakarvonen   16u   IPv6 0x7044f9e8bfe761ed      0t0       TCP localhost:http-alt->localhost:63119 (CLOSED)
server.ex 59846 vesakarvonen   17u   IPv6 0x2e487eb535fe886c      0t0       TCP localhost:http-alt->localhost:63120 (CLOSED)
server.ex 59846 vesakarvonen   18u   IPv6  0x8db736f8f92f23d      0t0       TCP localhost:http-alt->localhost:63121 (CLOSED)
server.ex 59846 vesakarvonen   19u   IPv6 0x8592f925dee252be      0t0       TCP localhost:http-alt->localhost:63122 (CLOSED)
server.ex 59846 vesakarvonen   20u   IPv6 0x4b6836d10d013582      0t0       TCP localhost:http-alt->localhost:63123 (CLOSED)
server.ex 59846 vesakarvonen   21u   IPv6 0xe5a14959aa3f71eb      0t0       TCP localhost:http-alt->localhost:63124 (CLOSED)
server.ex 59846 vesakarvonen   22u   IPv6  0xdc19500c8393f71      0t0       TCP localhost:http-alt->localhost:63125 (CLOSED)
server.ex 59846 vesakarvonen   23u   IPv6 0xd53f438170051464      0t0       TCP localhost:http-alt->localhost:63126 (CLOSED)
server.ex 59846 vesakarvonen   24u   IPv6 0x6a73660b825d5e73      0t0       TCP localhost:http-alt->localhost:63127 (CLOSED)
server.ex 59846 vesakarvonen   25u   IPv6 0xbc4d818a10fc3a54      0t0       TCP localhost:http-alt->localhost:63128 (CLOSED)
server.ex 59846 vesakarvonen   26u   IPv6 0x63b5393650d01921      0t0       TCP localhost:http-alt->localhost:63129 (CLOSED)
server.ex 59846 vesakarvonen   27u   IPv6 0xf906a2ba02b64eae      0t0       TCP localhost:http-alt->localhost:63130 (CLOSED)
server.ex 59846 vesakarvonen   28u   IPv6 0x707f5e78993cc449      0t0       TCP localhost:http-alt->localhost:63131 (CLOSED)
server.ex 59846 vesakarvonen   29u   IPv6 0x9dd658d094bdb3e9      0t0       TCP localhost:http-alt->localhost:63132 (CLOSED)
server.ex 59846 vesakarvonen   30u   IPv6 0xb45b21ce16bd20b8      0t0       TCP localhost:http-alt->localhost:63133 (CLOSED)
server.ex 59846 vesakarvonen   31u   IPv6 0xffae20ed721db2f5      0t0       TCP localhost:http-alt->localhost:63134 (CLOSED)
server.ex 59846 vesakarvonen   32u   IPv6 0x53ac8f49744a8d86      0t0       TCP localhost:http-alt->localhost:63135 (CLOSED)
server.ex 59846 vesakarvonen   33u   IPv6 0x2a4a2e6e74e8ccca      0t0       TCP localhost:http-alt->localhost:63136 (CLOSED)
server.ex 59846 vesakarvonen   34u   IPv6 0x8b73f7e513e18015      0t0       TCP localhost:http-alt->localhost:63137 (CLOSED)
server.ex 59846 vesakarvonen   35u   IPv6 0xd0e8e18f74491411      0t0       TCP localhost:http-alt->localhost:63138 (CLOSED)
server.ex 59846 vesakarvonen   36u   IPv6 0xca6de780edb10e0b      0t0       TCP localhost:http-alt->localhost:63139 (CLOSED)
server.ex 59846 vesakarvonen   37u   IPv6  0xe75cce2087b81a8      0t0       TCP localhost:http-alt->localhost:63140 (CLOSED)
server.ex 59846 vesakarvonen   38u   IPv6 0xf4f4bd0c7c56b3da      0t0       TCP localhost:http-alt->localhost:63141 (CLOSED)
server.ex 59846 vesakarvonen   39u   IPv6 0x5c838887fe3b937a      0t0       TCP localhost:http-alt->localhost:63142 (ESTABLISHED)
server.ex 59846 vesakarvonen   40u   IPv6 0xf1f0a73f0662d7da      0t0       TCP localhost:http-alt->localhost:63143 (ESTABLISHED)
server.ex 59846 vesakarvonen   41u   IPv6 0x255736770f70ca30      0t0       TCP localhost:http-alt->localhost:63144 (CLOSED)
server.ex 59846 vesakarvonen   42u   IPv6 0xeb3643281b42e5c4      0t0       TCP localhost:http-alt->localhost:63145 (CLOSED)
server.ex 59846 vesakarvonen   43u   IPv6 0xee427aa9f357b367      0t0       TCP localhost:http-alt->localhost:63146 (ESTABLISHED)
server.ex 59846 vesakarvonen   44u   IPv6 0x61f2ed5879cbe868      0t0       TCP localhost:http-alt->localhost:63147 (CLOSED)
server.ex 59846 vesakarvonen   45u   IPv6 0x3928660ec1ff6ea1      0t0       TCP localhost:http-alt->localhost:63148 (CLOSED)
server.ex 59846 vesakarvonen   46u   IPv6 0xf537ef97cddd7211      0t0       TCP localhost:http-alt->localhost:63149 (CLOSED)
server.ex 59846 vesakarvonen   47u   IPv6 0xb53305669cda63f9      0t0       TCP localhost:http-alt->localhost:63150 (CLOSED)
server.ex 59846 vesakarvonen   48u   IPv6  0x6bf3e3b0d3ffb01      0t0       TCP localhost:http-alt->localhost:63152 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   49u   IPv6 0xb6ee54c4457b5fe6      0t0       TCP localhost:http-alt->localhost:63153 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   50u   IPv6 0xd0d81dd6a40a8706      0t0       TCP localhost:http-alt->localhost:63154 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   51u   IPv6 0xe830711b7ec44ad8      0t0       TCP localhost:http-alt->localhost:63155 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   52u   IPv6 0xadedce3935dfb76d      0t0       TCP localhost:http-alt->localhost:63156 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   53u   IPv6 0x1c7d57e59e26d007      0t0       TCP localhost:http-alt->localhost:63157 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   54u   IPv6  0xd6083a1f079c313      0t0       TCP localhost:http-alt->localhost:63158 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   55u   IPv6 0x67053314f87eb302      0t0       TCP localhost:http-alt->localhost:63159 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   56u   IPv6 0xd34b226ce6d15f33      0t0       TCP localhost:http-alt->localhost:63160 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   57u   IPv6 0xa33bb6818ff2eb69      0t0       TCP localhost:http-alt->localhost:63161 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   58u   IPv6 0x7289e3b104857d66      0t0       TCP localhost:http-alt->localhost:63162 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   59u   IPv6 0xaa53ba2be62c1f05      0t0       TCP localhost:http-alt->localhost:63163 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   60u   IPv6 0xf0e5da74421d6e9b      0t0       TCP localhost:http-alt->localhost:63164 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   61u   IPv6 0x9269837e7b080be8      0t0       TCP localhost:http-alt->localhost:63165 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   62u   IPv6 0x29dfd24d59920322      0t0       TCP localhost:http-alt->localhost:63166 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   63u   IPv6 0x7eba9a4caa4655c8      0t0       TCP localhost:http-alt->localhost:63167 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   64u   IPv6 0x54620832402dec62      0t0       TCP localhost:http-alt->localhost:63168 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   65u   IPv6 0x17c80d0e8983fa21      0t0       TCP localhost:http-alt->localhost:63169 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   66u   IPv6 0x4a415e00a85602f9      0t0       TCP localhost:http-alt->localhost:63170 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   67u   IPv6 0x6456f192e6f2a2b5      0t0       TCP localhost:http-alt->localhost:63171 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   68u   IPv6 0x3ec44f92009a3699      0t0       TCP localhost:http-alt->localhost:63172 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   69u   IPv6  0x86279e7995de3b6      0t0       TCP localhost:http-alt->localhost:63173 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   70u   IPv6 0x806a40d971574fc5      0t0       TCP localhost:http-alt->localhost:63174 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   71u   IPv6 0xdd3fdf164250ad5d      0t0       TCP localhost:http-alt->localhost:63175 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   72u   IPv6 0xfb3ca33bcad567f8      0t0       TCP localhost:http-alt->localhost:63176 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   73u   IPv6 0x20ae1dcbdbb2a679      0t0       TCP localhost:http-alt->localhost:63177 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   74u   IPv6 0xa3f387780b8a0fee      0t0       TCP localhost:http-alt->localhost:63178 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   75u   IPv6 0x7f54505c925d1460      0t0       TCP localhost:http-alt->localhost:63179 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   76u   IPv6 0xdd35324f7ba73383      0t0       TCP localhost:http-alt->localhost:63180 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   77u   IPv6 0x41b884a5bff10a57      0t0       TCP localhost:http-alt->localhost:63181 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   78u   IPv6 0x3bb9e5c7c1024fe7      0t0       TCP localhost:http-alt->localhost:63182 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   79u   IPv6 0x35f4b38e4453151c      0t0       TCP localhost:http-alt->localhost:63183 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   80u   IPv6 0x7fee632825c3da71      0t0       TCP localhost:http-alt->localhost:63184 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   81u   IPv6 0x2220e4a2533504fd      0t0       TCP localhost:http-alt->localhost:63185 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   82u   IPv6 0x1edd3c3fe37fc9b2      0t0       TCP localhost:http-alt->localhost:63186 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   83u   IPv6 0x2809f8f46c35e228      0t0       TCP localhost:http-alt->localhost:63187 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   84u   IPv6 0x93215138973a95dd      0t0       TCP localhost:http-alt->localhost:63188 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   85u   IPv6 0x9dc5dad6589817aa      0t0       TCP localhost:http-alt->localhost:63189 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   86u   IPv6 0x990d082cc5e4d562      0t0       TCP localhost:http-alt->localhost:63190 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   87u   IPv6 0xe91fb23c58dfcfea      0t0       TCP localhost:http-alt->localhost:63191 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   88u   IPv6 0x836ffe3ebfab2552      0t0       TCP localhost:http-alt->localhost:63192 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   89u   IPv6 0x1ac8dc884051b793      0t0       TCP localhost:http-alt->localhost:63193 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   90u   IPv6 0xecfaab719038ce13      0t0       TCP localhost:http-alt->localhost:63194 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   91u   IPv6 0x9bb6a88c7968a1ad      0t0       TCP localhost:http-alt->localhost:63195 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   92u   IPv6 0x31e5cf119007123d      0t0       TCP localhost:http-alt->localhost:63196 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   93u   IPv6  0x250fea05a5ccb00      0t0       TCP localhost:http-alt->localhost:63197 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   94u   IPv6 0x9ed35a9ea039545d      0t0       TCP localhost:http-alt->localhost:63198 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   95u   IPv6 0x892b5ca9dac93725      0t0       TCP localhost:http-alt->localhost:63199 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   96u   IPv6 0xf34c7da5d75d0383      0t0       TCP localhost:http-alt->localhost:63200 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   97u   IPv6 0x310e47ab3ab959d1      0t0       TCP localhost:http-alt->localhost:63201 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   98u   IPv6 0x8ef8e09ebf8ba2fe      0t0       TCP localhost:http-alt->localhost:63202 (CLOSE_WAIT)
server.ex 59846 vesakarvonen   99u   IPv6 0x7b65cb7875e2e3d2      0t0       TCP localhost:http-alt->localhost:63203 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  100u   IPv6 0x6ce508373b6aa471      0t0       TCP localhost:http-alt->localhost:63204 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  101u   IPv6 0x4bfea862f71e407c      0t0       TCP localhost:http-alt->localhost:63205 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  102u   IPv6 0xa7517af2024568c5      0t0       TCP localhost:http-alt->localhost:63206 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  103u   IPv6 0x7b19b2ca1f067ef0      0t0       TCP localhost:http-alt->localhost:63207 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  104u   IPv6 0x25533246b8b6d8c6      0t0       TCP localhost:http-alt->localhost:63208 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  105u   IPv6 0x4a5520f37d816303      0t0       TCP localhost:http-alt->localhost:63209 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  106u   IPv6 0xcc110499ffae9c55      0t0       TCP localhost:http-alt->localhost:63210 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  107u   IPv6 0x9a72f13ff992e617      0t0       TCP localhost:http-alt->localhost:63211 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  108u   IPv6 0x6b75c9910d0b364f      0t0       TCP localhost:http-alt->localhost:63212 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  109u   IPv6 0x993822a5c37ba34f      0t0       TCP localhost:http-alt->localhost:63213 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  110u   IPv6 0xb4c6aaaa3a7172e6      0t0       TCP localhost:http-alt->localhost:63214 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  111u   IPv6 0x798c1a1d5dd3349c      0t0       TCP localhost:http-alt->localhost:63215 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  112u   IPv6 0x6cc0df46db803d4a      0t0       TCP localhost:http-alt->localhost:63216 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  113u   IPv6 0x167c86bfbca86451      0t0       TCP localhost:http-alt->localhost:63217 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  114u   IPv6 0x9c5b65ab00db0e07      0t0       TCP localhost:http-alt->localhost:63218 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  115u   IPv6 0x193f958951e43edb      0t0       TCP localhost:http-alt->localhost:63219 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  116u   IPv6 0x123c978652bec633      0t0       TCP localhost:http-alt->localhost:63220 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  117u   IPv6 0x53ec3865b2a533aa      0t0       TCP localhost:http-alt->localhost:63221 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  118u   IPv6 0x691ec58a7f2b029c      0t0       TCP localhost:http-alt->localhost:63222 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  119u   IPv6 0x5eca2c1d09f1a4d3      0t0       TCP localhost:http-alt->localhost:63223 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  120u   IPv6 0x93afb2f715680cad      0t0       TCP localhost:http-alt->localhost:63224 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  121u   IPv6 0xc63f18480f8f4484      0t0       TCP localhost:http-alt->localhost:63225 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  122u   IPv6 0x1cac2026145e8fcf      0t0       TCP localhost:http-alt->localhost:63226 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  123u   IPv6 0x85a038785e7c9872      0t0       TCP localhost:http-alt->localhost:63227 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  124u   IPv6 0x7eb6a394633d5e96      0t0       TCP localhost:http-alt->localhost:63228 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  125u   IPv6 0xce393d6b4606a624      0t0       TCP localhost:http-alt->localhost:63229 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  126u   IPv6 0xbfce6dfa9463433b      0t0       TCP localhost:http-alt->localhost:63230 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  127u   IPv6 0x98fadc640571d08f      0t0       TCP localhost:http-alt->localhost:63231 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  128u   IPv6 0xd90772517f5d37e4      0t0       TCP localhost:http-alt->localhost:63232 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  129u   IPv6 0x202ddc5a7cdef485      0t0       TCP localhost:http-alt->localhost:63233 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  130u   IPv6 0x7f68f6965c9bc27d      0t0       TCP localhost:http-alt->localhost:63234 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  131u   IPv6  0xe2c65db6b2bbe74      0t0       TCP localhost:http-alt->localhost:63235 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  132u   IPv6 0xe383ce35055f5e6d      0t0       TCP localhost:http-alt->localhost:63236 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  133u   IPv6 0x27902f4d41f5a575      0t0       TCP localhost:http-alt->localhost:63237 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  134u   IPv6  0x28e3a1919491d6f      0t0       TCP localhost:http-alt->localhost:63238 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  135u   IPv6 0x8b829fc102fcc3ef      0t0       TCP localhost:http-alt->localhost:63239 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  136u   IPv6 0xdfd28761dd005fb7      0t0       TCP localhost:http-alt->localhost:63240 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  137u   IPv6 0x8b9919e10e21fae5      0t0       TCP localhost:http-alt->localhost:63241 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  138u   IPv6 0x893ab1cdbbd224e0      0t0       TCP localhost:http-alt->localhost:63242 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  139u   IPv6  0x5207e354a90dea0      0t0       TCP localhost:http-alt->localhost:63243 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  140u   IPv6 0xc0f7568337def11b      0t0       TCP localhost:http-alt->localhost:63244 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  141u   IPv6 0x2e70f9f63bbf8120      0t0       TCP localhost:http-alt->localhost:63245 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  142u   IPv6 0x531288d0b32802d0      0t0       TCP localhost:http-alt->localhost:63246 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  143u   IPv6 0xdd93c310a8f87f47      0t0       TCP localhost:http-alt->localhost:63247 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  144u   IPv6  0x607760660d32143      0t0       TCP localhost:http-alt->localhost:63248 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  145u   IPv6 0x9a6b416d61039a61      0t0       TCP localhost:http-alt->localhost:63249 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  146u   IPv6 0x1e315e846ffa372b      0t0       TCP localhost:http-alt->localhost:63250 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  147u   IPv6  0x5b0bf2390dc3493      0t0       TCP localhost:http-alt->localhost:63251 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  148u   IPv6  0x46ae8144826016e      0t0       TCP localhost:http-alt->localhost:63252 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  149u   IPv6 0xbdb74d81e03a3b66      0t0       TCP localhost:http-alt->localhost:63253 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  150u   IPv6 0x52af0f3f971a4259      0t0       TCP localhost:http-alt->localhost:63254 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  151u   IPv6 0x19d8bc0c75e9646a      0t0       TCP localhost:http-alt->localhost:63255 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  152u   IPv6 0xeb2f8b07f0ecf1c6      0t0       TCP localhost:http-alt->localhost:63256 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  153u   IPv6 0x457e72e209d5d9a0      0t0       TCP localhost:http-alt->localhost:63257 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  154u   IPv6 0xbe31c8317d1164a7      0t0       TCP localhost:http-alt->localhost:63258 (CLOSE_WAIT)
server.ex 59846 vesakarvonen  155u   IPv6 0xec4b1b6195b1870c      0t0       TCP localhost:http-alt->localhost:63259 (CLOSE_WAIT)
...

aantron added a commit that referenced this issue Oct 10, 2024
@aantron
Attempting to reproduce #347 locally
8dddedf
aantron added a commit that referenced this issue Oct 10, 2024
@aantron
Reproduce 347 in CI
efcfb09 
See #347.
aantron added a commit that referenced this issue Oct 10, 2024
@aantron
Reproduce 347 in CI
23f40f9 
See #347.
aantron added a commit that referenced this issue Oct 10, 2024
@aantron
Reproduce #347 in CI
87d735b
aantron added a commit that referenced this issue Oct 10, 2024
@aantron
Reproduce #347 in CI
a1ee71e
aantron added a commit that referenced this issue Oct 11, 2024
@aantron
Reproduce #347 in CI
b87fa14
aantron added a commit that referenced this issue Oct 11, 2024
@aantron
Reproduce #347 in CI
095f95e
aantron added a commit that referenced this issue Oct 11, 2024
@aantron
Reproduce #347 in CI
40f7861
@craff
Copy link

craff commented Oct 12, 2024

I think I have the same issue on a raspberry pi. I was using hyper, tried to close the message in Hyper.get / Hyper.post doest not solve the problem.

Are you letting the GC close the socket ? This is unsafe, no warranty that the GC (especially the major GC) will collect socket fast enough ?

I ask this, because socket are not freed immediatly after Hyper.get and Hyper.close exit. So my guess was socket closed by a finaliser in the GC.

@craff craff mentioned this issue Oct 13, 2024
Open
@craff
Copy link

craff commented Oct 13, 2024

I copy the message I dit on hyper:

I investigated more. Calling Message.close and/or using Hyper.request did not solve the problem.
I think it is deeper in Dream and may be related to issue 347 of dream.

The observed behavior is that in general 5-10 unused sockets are preserved for some unknown reasons, but after a few days, this increase to more that 1000 sockets and the program fails with EMFILES as expected.

Switching to http-lwt-client immediatly solved the issue, so it must be an issue related to Hyper/Dream. Now no unused socket is kept.

Now that I have switched my code in production to http-lwt-client, I will not be able to investigate more. Sorry.

@aantron
Copy link
Owner

aantron commented Oct 13, 2024

Thanks. I am still working on this, but greatly slowed down by that I haven't been able to observe it locally -- so far only in GHA.

@craff
Copy link

craff commented Oct 14, 2024 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
🐛 bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@craff @polytypic @aantron