SIGBUS: fatal error: unexpected signal during runtime execution

[sameh-farouk]

Description

after updating the VDC, the VDC becomes unreachable.
investigating the issue find that it starts with a BUS ERROR.

Version information

• Project commit/version: e.g. js-sdk 73616ba

Traceback/Logs/Alerts

VDC information

Creation time: 01/06/2021, 14:43:55                                                                                                                
Expiration time: 01/07/2021, 19:15:43                                                                                                              
Flavor: silver                                                                                                                                     
Grace Period:  No                                                                                                                                  
Master IP:    --> State: Down                                                                                                                      
Threebot IP:                                                                                                                                       
Threebot Domain:   --> State Down                                                                                                                  

+Workloads--------------+---------+-------------------+--------+-----------+----------------------------------------------+---------+---------+
| Wid   | Type          | State   | Farm              | PoolID | IPv4Units | NodeID                                       | NState  | Message |
+-------+---------------+---------+-------------------+--------+-----------+----------------------------------------------+---------+---------+
| 51642 | Zdb           | DEPLOY  | lochristi_dev_lab | 11529  | 1206035.0 | BpTAry1Na2s1J8RAHNyDsbvaBSM3FjR4gMXEKga3UPbs | Up      |         |
| 51643 | Zdb           | DEPLOY  | lochristi_dev_lab | 11529  | 1206034.0 | 2anfZwrzskXiUHPLTqH1veJAia8G6rW2eFLy5YFMa1MP | Up      |         |
| 51644 | Zdb           | DEPLOY  | lochristi_dev_lab | 11529  | 1206034.0 | BpTAry1Na2s1J8RAHNyDsbvaBSM3FjR4gMXEKga3UPbs | Up      |         |
| 51645 | Zdb           | DEPLOY  | lochristi_dev_lab | 11529  | 1206033.0 | BpTAry1Na2s1J8RAHNyDsbvaBSM3FjR4gMXEKga3UPbs | Up      |         |
| 51646 | Zdb           | DEPLOY  | lochristi_dev_lab | 11529  | 1206032.0 | 2anfZwrzskXiUHPLTqH1veJAia8G6rW2eFLy5YFMa1MP | Up      |         |
| 51647 | Zdb           | DEPLOY  | lochristi_dev_lab | 11529  | 1206032.0 | 48YmkyvfoXnXqEojpJieMfFPnCv2enEuqEJcmhvkcdAk | Up      |         |
| 51648 | Zdb           | DEPLOY  | lochristi_dev_lab | 11529  | 1206031.0 | 2anfZwrzskXiUHPLTqH1veJAia8G6rW2eFLy5YFMa1MP | Up      |         |
| 51649 | Zdb           | DEPLOY  | lochristi_dev_lab | 11529  | 1206031.0 | 48YmkyvfoXnXqEojpJieMfFPnCv2enEuqEJcmhvkcdAk | Up      |         |
| 51650 | Zdb           | DEPLOY  | lochristi_dev_lab | 11529  | 1206030.0 | 3NAkUYqm5iPRmNAnmLfjwdreqDssvsebj4uPUt9BxFPm | Up      |         |
| 51651 | Zdb           | DEPLOY  | lochristi_dev_lab | 11529  | 1206030.0 | 48YmkyvfoXnXqEojpJieMfFPnCv2enEuqEJcmhvkcdAk | Up      |         |
| 51672 | Kubernetes    | DEPLOY  | lochristi_dev_lab | 11529  | 1206029.0 | 2anfZwrzskXiUHPLTqH1veJAia8G6rW2eFLy5YFMa1MP | Up      |         |
| 51680 | Kubernetes    | DEPLOY  | lochristi_dev_lab | 11529  | 1206029.0 | BBmMHCw392RLiByVEUT2ac6y6qJpbW9xj8TRdgk5rMGN | Up      |         |
| 51685 | Subdomain     | DELETED | Gateway           | 11533  | 0.0       | 9PdutHsdDSxcKUUyDg8ovS1KWh47qLT5R9h5uoFgRUH2 | Gateway |         |
| 51689 | Reverse_proxy | DEPLOY  | Gateway           | 11533  | 0.0       | 9PdutHsdDSxcKUUyDg8ovS1KWh47qLT5R9h5uoFgRUH2 | Gateway |         |
| 51702 | Container     | DELETED | lochristi_dev_lab | 11529  | 1206028.0 | BpTAry1Na2s1J8RAHNyDsbvaBSM3FjR4gMXEKga3UPbs | Up      |         |
+-------+---------------+---------+-------------------+--------+-----------+----------------------------------------------+---------+---------+

threebot log from VDC Monitor (read from bottom to up)

stack trace unavailable\n
--
  | fatal error: unexpected signal during runtime execution\n
  | runtime stack:\n
  | \n
  | [signal SIGBUS: bus error code=0x2 addr=0x55887d87f799 pc=0x55887d1a5f6c]\n
  | panic during panic\n
  | fatal error: unexpected signal during runtime execution\n
  | runtime stack:\n
  | \n
  | [signal SIGBUS: bus error code=0x2 addr=0x55887d4f828e pc=0x55887d1fc744]\n
  | fatal error: unexpected signal during runtime execution\n
  | 3:17PM INF incoming stream, connect to local application remote add=174.138.14.103:18000\n
  | 3:17PM INF start forwarding\n
  | 3:17PM INF incoming stream, connect to local application remote add=174.138.14.103:18000\n
  | 3:17PM INF start forwarding\n
  | 3:17PM INF incoming stream, connect to local application remote add=174.138.14.103:18000\n
  | 3:17PM INF start forwarding\n
  | 3:17PM INF incoming stream, connect to local application remote add=174.138.14.103:18000\n
  | 3:17PM INF start forwarding\n
  | 3:17PM INF incoming stream, connect to local application remote add=174.138.14.103:18000\n
  | 3:17PM INF start forwarding\n
  | 3:17PM INF incoming stream, connect to local application remote add=174.138.14.103:18000\n
  | 3:17PM INF start forwarding\n
  | 3:17PM INF incoming stream, connect to local application remote add=174.138.14.103:18000\n
  | 3:17PM INF start forwarding\n
  | 3:17PM INF incoming stream, connect to local application remote add=174.138.14.103:18000\n
  | 3:17PM INF start forwarding\n
  | 3:17PM INF incoming stream, connect to local application remote add=174.138.14.103:18000\n
  | 3:16PM INF start forwarding\n
  | 3:16PM INF incoming stream, connect to local application remote add=174.138.14.103:18000\n
  | 3:16PM INF start forwarding\n
  | 3:16PM INF incoming stream, connect to local application remote add=174.138.14.103:18000\n
  | jumpscale/packages/vdc_dashboard/scripts/update.sh: line 81: /usr/bin/git: Transport endpoint is not connected\n
  | jumpscale/packages/vdc_dashboard/scripts/update.sh: line 80: /usr/bin/git: Transport endpoint is not connected\n
  | Error: Failed to install the defined dependencies!\n
  | jumpscale/packages/vdc_dashboard/scripts/update.sh: line 63: /usr/local/bin/poetry: Transport endpoint is not connected\n
  | jumpscale/packages/vdc_dashboard/scripts/update.sh: line 63: /usr/local/bin/poetry: Transport endpoint is not connected\n
  | jumpscale/packages/vdc_dashboard/scripts/update.sh: line 63: /usr/local/bin/poetry: Transport endpoint is not connected\n
  | jumpscale/packages/vdc_dashboard/scripts/update.sh: line 77: 687 Bus error poetry install --no-interaction\n
  | Switched to a new branch 'development_chatbot'\n

[DylanVerstraete]

In the node logs I can see the node received a decomission request for your container with id 51702-1

2021-06-01 17:50:05 | [+] provisiond: 2021-06-01T15:50:05Z info start decommissioning reservation duration=2562047h47m16.854775807s expired=false id=51702-1 tag= to-delete=true type=container
2021-06-01 17:50:05 | [+] provisiond: 2021-06-01T15:50:05Z info received ids for processing reservation-ids=["51702-1 [D: true]"]