Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Web server: Action extension_list_menuitems timed out for subscriber airdcpp-share-monitor #2

Closed
bluboyo opened this issue Dec 18, 2020 · 59 comments

Comments

@bluboyo
Copy link

bluboyo commented Dec 18, 2020

Web server: Action extension_list_menuitems timed out for subscriber airdcpp-share-monitor. And it times out so i have to restart it. airDc 4.0

@maksis
Copy link
Member

maksis commented Dec 18, 2020

Please test version 1.0.1 of the extension to see if it makes any difference

Related issue: airdcpp-web/airdcpp-release-validator#14

@bluboyo
Copy link
Author

bluboyo commented Dec 18, 2020

Finished one download now and no errors. Will push it and see what happens.

@bluboyo
Copy link
Author

bluboyo commented Dec 18, 2020

15:50:49] Extensions: Extension airdcpp-share-monitor timed out and was restarted

Same as releasevalidator i guess.

@bluboyo
Copy link
Author

bluboyo commented Dec 18, 2020

the other errors seem to be gone.

@maksis
Copy link
Member

maksis commented Dec 19, 2020

There are still a few similar issues left in an external dependency. See yuanchuan/node-watch#107

@bluboyo
Copy link
Author

bluboyo commented Dec 19, 2020

Alright. Let me know if i can help

@maksis
Copy link
Member

maksis commented Dec 20, 2020

Could you test this version: http://web-builds.airdcpp.net/tmp/airdcpp-share-monitor-1.0.2-beta.1.tgz (use Install extension from URL... under Options on the Extensions tab)

@bluboyo
Copy link
Author

bluboyo commented Dec 20, 2020

[12:11:50] Extensions: Extension airdcpp-share-monitor timed out and was restarted
[12:11:50] airdcpp-share-monitor: Adding 15 paths for monitoring...
[12:11:50] airdcpp-share-monitor: 15 paths were added for monitoring
[12:13:02] Extensions: Extension airdcpp-share-monitor timed out and was restarted
[12:13:03] airdcpp-share-monitor: Adding 15 paths for monitoring...
[12:13:03] airdcpp-share-monitor: 15 paths were added for monitoring
[12:13:32] Extensions: Extension airdcpp-share-monitor timed out and was restarted
[12:13:33] airdcpp-share-monitor: Adding 15 paths for monitoring...
[12:13:33] airdcpp-share-monitor: 15 paths were added for monitoring

Restarted dc after installation. Still the same.

@maksis
Copy link
Member

maksis commented Dec 20, 2020

You could also post the extension debug info according to https://github.com/airdcpp-web/airdcpp-webclient/blob/master/.github/CONTRIBUTING.md#extension-related-issues

Log files with the .old extension should be used in this case as the extension was restarted due to timeout.

@bluboyo
Copy link
Author

bluboyo commented Dec 20, 2020

This is from the error log.

-20 17:46:12:96] 137 409 Directory is inside an unfinished bundle
[2020-12-20 17:46:12:97] Could not refresh the path L:****: Directory is inside an unfinished bundle
[2020-12-20 17:46:12:142] 138 409 Directory is inside an unfinished bundle

@maksis
Copy link
Member

maksis commented Dec 20, 2020

That's normal, the output log should be more interesting

@bluboyo
Copy link
Author

bluboyo commented Dec 20, 2020

Check your mail

@maksis
Copy link
Member

maksis commented Dec 20, 2020

What was the time when the extension timed out on that run?

@bluboyo
Copy link
Author

bluboyo commented Dec 20, 2020

[17:40:33] Extensions: Extension airdcpp-share-monitor timed out and was restarted
[17:40:33] airdcpp-share-monitor: Adding 15 paths for monitoring...
[17:40:34] airdcpp-share-monitor: 15 paths were added for monitoring

[17:40:33] Extensions: Extension airdcpp-share-monitor timed out and was restarted
[17:40:33] airdcpp-share-monitor: Adding 15 paths for monitoring...
[17:40:34] airdcpp-share-monitor: 15 paths were added for monitoring

@bluboyo
Copy link
Author

bluboyo commented Dec 20, 2020

log was created 17:47

@maksis
Copy link
Member

maksis commented Dec 20, 2020

The log starts from 17:40:33. I'd need logs before the timeout to see that what was going on back then.

@bluboyo
Copy link
Author

bluboyo commented Dec 20, 2020

No older logs but i happens again at 17:48

@maksis
Copy link
Member

maksis commented Dec 20, 2020

What's the exact time with seconds?

@bluboyo
Copy link
Author

bluboyo commented Dec 20, 2020

[17:47:58] Web server: Disconnecting extension airdcpp-share-monitor because of ping timeout
[17:48:37] Extensions: Extension airdcpp-share-monitor timed out and was restarted

@maksis
Copy link
Member

maksis commented Dec 20, 2020

You could try increasing the socket ping timeout from the Web UI to see if the extension becomes more stable (Settings -> System -> Advanced)

@bluboyo
Copy link
Author

bluboyo commented Dec 20, 2020

Tested 30s now

@bluboyo
Copy link
Author

bluboyo commented Dec 20, 2020

Monster restart now.

[22:16:36] airdcpp-share-monitor: 15 paths were added for monitoring
[22:16:56] Extensions: Extension airdcpp-share-monitor timed out and was restarted
[22:16:56] airdcpp-share-monitor: Adding 15 paths for monitoring...
[22:17:15] airdcpp-share-monitor: 15 paths were added for monitoring
[22:17:55] Web server: Disconnecting extension airdcpp-share-monitor because of ping timeout
[22:18:01] Extensions: Extension airdcpp-share-monitor timed out and was restarted
[22:18:01] airdcpp-share-monitor: Adding 15 paths for monitoring...
[22:18:01] airdcpp-share-monitor: 15 paths were added for monitoring

[22:18:55] Web server: Disconnecting extension airdcpp-share-monitor because of ping timeout
[22:19:19] Extensions: Extension airdcpp-share-monitor timed out and was restarted
[22:19:19] airdcpp-share-monitor: Adding 15 paths for monitoring...
[22:19:19] airdcpp-share-monitor: 15 paths were added for monitoring
[22:20:19] Extensions: Extension airdcpp-share-monitor timed out and was restarted
[22:20:19] airdcpp-share-monitor: Adding 15 paths for monitoring...
[22:20:23] airdcpp-share-monitor: 15 paths were added for monitoring
[22:21:17] Extensions: Extension airdcpp-share-monitor timed out and was restarted
[22:21:17] airdcpp-share-monitor: Adding 15 paths for monitoring...
[22:21:17] airdcpp-share-monitor: 15 paths were added for monitoring

[22:23:17] Extensions: Extension airdcpp-share-monitor timed out and was restarted
[22:23:17] airdcpp-share-monitor: Adding 15 paths for monitoring...

[22:23:41] airdcpp-share-monitor: 15 paths were added for monitoring
[22:23:42] Extensions: Extension airdcpp-share-monitor timed out and was restarted
[22:23:42] airdcpp-share-monitor: Adding 15 paths for monitoring...
[22:23:45] airdcpp-share-monitor: 15 paths were added for monitoring
[22:24:05] Extensions: Extension airdcpp-share-monitor timed out and was restarted
[22:24:05] airdcpp-share-monitor: Adding 15 paths for monitoring...
[22:24:06] airdcpp-share-monitor: 15 paths were added for monitoring

@maksis
Copy link
Member

maksis commented Dec 20, 2020

Do you have any idea that why it might have started to happen at that point? What were you doing at that time? What's the load (CPU, disks)?

@bluboyo
Copy link
Author

bluboyo commented Dec 20, 2020

I can see that it was refreshing. The cpu load is high (airdc). Disk load is low.

@bluboyo
Copy link
Author

bluboyo commented Dec 20, 2020

Im using rclone with remote disks. Can that be it?

@maksis
Copy link
Member

maksis commented Dec 21, 2020

Im using rclone with remote disks. Can that be it?

I'm not sure yet. There weren't that many file changes before it timed out based on your log file so I don't know why it would be frozen for that long. Your system specs aren't that bad either.

The next thing that you could try would be to download the latest version of node from https://nodejs.org/dist/v15.4.0/node-v15.4.0-win-x64.zip (only replace Node.js/node.exe inside the application installation directory) to see if it runs the extension better. Currently application ships with node v12 which is quite old.

@bluboyo
Copy link
Author

bluboyo commented Dec 21, 2020

node.exe replaced. Lets see what happens.

@bluboyo
Copy link
Author

bluboyo commented Dec 21, 2020

[15:45:04] Extensions: 4 extensions were loaded (debug mode)
[15:45:04] airdcpp-share-monitor: Adding 15 paths for monitoring...
[15:45:04] airdcpp-share-monitor: 15 paths were added for monitoring
[15:45:04] Web server: Web server started
[15:45:05] Connectivity (IPv4): Port mapping: Successfully created port mappings (Transfers: 23264, Encrypted transfers: 27825, Search: 27942) on the "WANDevice" device with the MiniUPnP interface
[15:45:08] Share: File list refresh initiated
[15:46:29] Extensions: Extension airdcpp-share-monitor timed out and was restarted
[15:46:29] airdcpp-share-monitor: Adding 15 paths for monitoring...
[15:46:31] airdcpp-share-monitor: 15 paths were added for monitoring

[15:47:13] Web server: Disconnecting extension airdcpp-share-monitor because of ping timeout (I have 90s in settings)
[15:47:39] Extensions: Extension airdcpp-share-monitor timed out and was restarted
[15:47:44] airdcpp-share-monitor: Adding 15 paths for monitoring...
[15:47:44] airdcpp-share-monitor: 15 paths were added for monitoring
[15:47:44] Hashing: Hasher #1 created

@maksis
Copy link
Member

maksis commented Dec 22, 2020

Please test the latest version of the extension with the latest application beta build.

Actually I just noticed that the timeout values won't currently be saved in the settings file. If you want to test custom timeouts, you should only restart the web server and not the application. It will be fixed in a later build.

@bluboyo
Copy link
Author

bluboyo commented Dec 22, 2020

Will try

@bluboyo
Copy link
Author

bluboyo commented Dec 22, 2020

Trying some settings now. But if i try to stop the server in the win app it crashes. Have to do it form the browser.

@maksis
Copy link
Member

maksis commented Dec 22, 2020

It would be good to get the crash log

@bluboyo
Copy link
Author

bluboyo commented Dec 22, 2020

Cant find any crashlog and no crashlog in settings.

@bluboyo
Copy link
Author

bluboyo commented Dec 22, 2020

Cant see any errors in a few hours now. I have ping time out on 240s and new directory on 240s. Also had a crash when i restarted the webserver from browser. And a note on the updates when it should restart it only shuts down.

@maksis
Copy link
Member

maksis commented Dec 22, 2020

If the application freezes, you can follow the instructions at https://www.airdcpp.net/forum/viewtopic.php?f=7&t=447 for getting the necessary information

And a note on the updates when it should restart it only shuts down.

Do you mean extension updates?

@bluboyo
Copy link
Author

bluboyo commented Dec 22, 2020

No when i update the app. Is says restart to update but it just shuts down.

@bluboyo
Copy link
Author

bluboyo commented Dec 22, 2020

If the application freezes, you can follow the instructions at https://www.airdcpp.net/forum/viewtopic.php?f=7&t=447 for getting the necessary information

And a note on the updates when it should restart it only shuts down.

Do you mean extension updates?

Can really use that debug. It just dies. Boom dead. No frozen client.

@maksis
Copy link
Member

maksis commented Dec 22, 2020

Since I can't reproduce those issues and I'm not aware of anyone else experiencing them, you could try doing some troubleshooting, such as testing with a clean installation whether it behaves in the same way. If you have antivirus software (or something similar) running, you could also try disabling them. It would be better to post a new bug report for the application itself if the issue won't go away.

I released a new beta build that will also fix saving of the timeout values in the config file.

@bluboyo
Copy link
Author

bluboyo commented Dec 22, 2020

I can test some stuff. Is not a big issue but always nice to fix bugs. WIll download the beta and test.

@maksis
Copy link
Member

maksis commented Dec 28, 2020

Is everything still working fine with the extension?

@bluboyo
Copy link
Author

bluboyo commented Dec 28, 2020 via email

@maksis
Copy link
Member

maksis commented Dec 29, 2020

Timeouts? With socket ping timeout set to 240 seconds?

@bluboyo
Copy link
Author

bluboyo commented Dec 29, 2020

Timeouts and crasch when it cant access the log. Yes. Its 240s. The crash is almost always when i start dc. timeouts are random.

@maksis
Copy link
Member

maksis commented Jan 1, 2021

You could try the latest version and if it still doesn't work, I would need the same log data as before (.old logs and the respective system log messages). The logging has been improved in the recent versions.

Timeouts and crasch when it cant access the log

Does the log error happen even on the first time after restarting the computer? What's the error messages?

@bluboyo
Copy link
Author

bluboyo commented Jan 1, 2021

I have installed and restarted. No errors in the last hour. I dont really remember the times i restarted the computer. But i did this time. Will keep you updated.

@bluboyo
Copy link
Author

bluboyo commented Jan 2, 2021

No errors since yesterday. Quite high load. Lots of downloads and hashing.

@bluboyo
Copy link
Author

bluboyo commented Jan 2, 2021

Still no errors. The hashing is much faster to. It was much slower when i upgraded to 4.00. Nice work :)

@bluboyo
Copy link
Author

bluboyo commented Jan 3, 2021

Damn. Timed out and restarted. From log: [2021-1-3 7:14:45:94] Wake up detected (last alive 32351 ms ago), requesting restart...

@maksis
Copy link
Member

maksis commented Jan 3, 2021

Damn. Timed out and restarted. From log: [2021-1-3 7:14:45:94] Wake up detected (last alive 32351 ms ago), requesting restart...

The main thing for me isn't actually to know whether it times out, but to know what was happening when it timed out (and whether it's something than can be avoided). The timeout periods can't be increased indefinitely as the socket will get disconnected at some point in any case if the process is frozen for too long (see airdcpp-web/airdcpp-webclient#356).

@bluboyo
Copy link
Author

bluboyo commented Jan 3, 2021

Im going to set them in debug and see what happens. But now i had lots of crashes. The api for the rclone remote maxed out. Can that be it?

@bluboyo
Copy link
Author

bluboyo commented Jan 4, 2021

I have checked the logs and there is a 30s blindspot before the addon is restarted. I cant see any errors before so something is happening there.

@maksis
Copy link
Member

maksis commented Jan 4, 2021

You could monitor the CPU usage of the extension process according to https://social.technet.microsoft.com/Forums/lync/en-US/42afb7cd-6cb6-463c-b385-21a97162e3f5/monitoring-cpu-usage-of-individual-processes?forum=winservergen

You need to monitor the node process. It might also be a good idea to stop all other extensions as they all have the same process name so you won't start monitoring a wrong extension.

You should also edit the performance monitor graph settings from Properties as by default it will only show a duration of 120 seconds. Note that you may also add other processes for monitoring with different line colors, so you also add monitoring for AirDC, rsync and possible other processes that could cause CPU spikes. Additionally you could add separate graphs for the total IO operations for the same processes (Add counters -> Process -> IO * items -> Processes rsync, AirDC... ).

Maybe you will get some useful information from the time when the extension timed out.

@bluboyo
Copy link
Author

bluboyo commented Jan 4, 2021

Had a crasch now and rclone and airdc spiked in cpu at the same time. Node is very flat. No spikes at all.

@maksis
Copy link
Member

maksis commented Jan 4, 2021

So sounds like the process just got frozen, maybe because of total system load...

Please test the latest version of the extension. It will set the system sleep detection timeout based on the socket ping timeout that you have configured in the application settings. So you if still have ping timeout set to 240 seconds, the extension process can be frozen for the same amount of time before it commits suicide (requests a restart from the application).

@bluboyo
Copy link
Author

bluboyo commented Jan 4, 2021

thx. ye the load is high. Will try it.

@maksis
Copy link
Member

maksis commented Jan 8, 2021

Would you say that the issue has now been fixed?

@bluboyo
Copy link
Author

bluboyo commented Jan 8, 2021

Yes. Release validator still crasches. But yes share monitor feels stable now.

@maksis maksis closed this as completed Aug 26, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants