0000689: Shutdown never finishes

(0001305)
luc (reporter)
2016-09-24 14:20

I also experienced this randomly last weeks.
I didn't look deeply for the main cause, but I had the feeling it is occuring more frequently when networking bandwidth is limited or for example when WiFi connection is failing...

(0001308)
Collision (reporter)
2016-09-27 12:15

This issue seems to be related to the following.
ID: 0000566, Updated: 2015-07-03 Still not shooting down
ID: 0000567, Updated: 2015-04-04 YaCy can't shutdown properly. Hangs on HeapReader dumping. Have to kill it.
ID: 0000505, Updated: 2015-04-04 Can't shutdown

This problem remains unresolved on my peer.
Therefore I force shutdown YaCy everytime.

In Japan, the politics is becoming dictatorial. Information control is carried out. I think that YaCy IS THE MOST IMPORTANT THING for the Japanese citizen.

(0001309)
luc (reporter)
2016-09-28 09:55

I agree this is quite annoying. But looks like a difficult one to reproduce on demand and therefore to solve.

I will check during the coming days if I can find some fix to apply.

(0001310)
luc (reporter)
2016-09-28 15:31
edited on: 2016-09-28 15:44

Waw I already found a reproductible use case :
- on Debian Linux with a headless JRE, Firefox installed, and latest YaCy sources from GitHub
- start YaCy with no browser opened
- firefox opens on localhost:8090
- do something or not with YaCy and then shutdown
- shutdown does not work, until you close your browser

Version 1.90 is not affected byt this issue, it was introduced by this commit : https://github.com/yacy/yacy_search_server/commit/bf6709d196bff6445826fd42bb411847188d2a1a [^]
Please note that when Firefox is already opened, YaCy shutdowns correctly.

So I commited a fix targetting YaCy on Linux Platform, but also to make sure this issue can no more occur on other platforms : https://github.com/yacy/yacy_search_server/commit/b8f64581525377649eaf44bbf2e679fedab1f1ff [^]

(0001311)
luc (reporter)
2016-09-28 15:48
edited on: 2016-09-28 16:04

@Pythagoras , you are apparently running a YaCY version greater than 1.90. Does your peer shutdown correctly when you close your browser?

@Collision , which YaCy version do you currently run and on which OS?

What would really help to identify and eventually reproduce this kind of issue would be to provide a Thread Dump taken when your peer is blocked in the Shutdown step, either from YaCY with /Threaddump_p.html or with VisualVM using the 'Threads' tab and 'Thread dump' button.

(0001312)
Collision (reporter)
2016-09-28 20:06

@luc, thank you for your kindness.
I currently use YaCy 1.90/09000, Ubuntu 16.04.1 LTS, Linux kernel 4.4.0.
I attach three files dumped by VisualVM.

(0001313)
Pythagoras (reporter)
2016-09-28 21:13

Yes, you were right, the issue is indeed related to the browser. I confirm that when I close the browser the peer shuts down correctly. It also shuts down correctly if the browser were already running on startup. Tested on YaCy version 1.91/9188 (df51e4e).

(0001316)
Collision (reporter)
2016-09-29 03:29

In my case, whether Firefox browser was running or not, the result was the same.
I closed the browser after pushing the shutdown button, but YaCy never shut down.

(0001317)
luc (reporter)
2016-09-29 10:40

Thank you Collision for your dumps, they were very useful ant let me discover another reproductible scenario :
- shutdown blocks when config value crawler.MaxActiveThreads is greater than 200. This value can be configured in /PerformanceQueues_p.html admin page, in "Thread Pool Settings" table, row "Crawler Pool", column "maximum Active".

Do you have this value greater than 200?

I commited a fix (https://github.com/yacy/yacy_search_server/commit/dcdea2d02f815bcece1713cd2ef8c56c39f889db [^]) for this particular case, but it could also solve other potential concurrency issues in the shutdown process.

If you can run a test from latest sources, feedback is welcome!

(0001318)
Collision (reporter)
2016-09-29 12:30

Oh, yes! That's right! I have set "Crawler Pool - maximum Active" value to '600'.
By the way, it is difficult to run a test from latest sources for me, sorry.

(0001319)
luc (reporter)
2016-09-29 13:35

Ok Collision, no problem.
So you will just have to wait for the next release or for an update made available in the System updates.

I think we can clause this issue, but don't hesitate to report if you encounter other shutdown issues in 1.90 even with "Crawler Pool - maximum Active" set to 200.

(0001320)
Collision (reporter)
2016-10-03 11:07

I set "Crawler Pool - maximum Active" to 180, and I confirmed that YaCy shut down successfully. Thank you so much, luc.

(0001322)
BuBu (developer)
2016-10-04 19:03

see https://github.com/yacy/yacy_search_server/commit/dcdea2d02f815bcece1713cd2ef8c56c39f889db [^] by luccioman

Issue History
Date Modified	Username	Field	Change
2016-09-24 10:42	Pythagoras	New Issue
2016-09-24 14:20	luc	Note Added: 0001305
2016-09-27 12:15	Collision	Note Added: 0001308
2016-09-28 09:55	luc	Note Added: 0001309
2016-09-28 15:11	BuBu	Relationship added	related to 0000566
2016-09-28 15:12	BuBu	Relationship added	related to 0000241
2016-09-28 15:13	BuBu	Relationship added	related to 0000567
2016-09-28 15:19	BuBu	Relationship added	related to 0000505
2016-09-28 15:31	luc	Note Added: 0001310
2016-09-28 15:31	luc	Note Edited: 0001310	View Revisions
2016-09-28 15:43	luc	Note Edited: 0001310	View Revisions
2016-09-28 15:44	luc	Note Edited: 0001310	View Revisions
2016-09-28 15:48	luc	Note Added: 0001311
2016-09-28 15:48	luc	Note Edited: 0001311	View Revisions
2016-09-28 16:04	luc	Note Edited: 0001311	View Revisions
2016-09-28 20:04	Collision	File Added: VVMYaCyInfomation_1.txt
2016-09-28 20:05	Collision	File Added: VVMYaCyDumpImmediatelyAfterPressTheShutdownButton_1.txt
2016-09-28 20:06	Collision	File Added: VVMYaCyDumpLongAfterPressedTheShutdownButton_1.txt
2016-09-28 20:06	Collision	Note Added: 0001312
2016-09-28 21:13	Pythagoras	Note Added: 0001313
2016-09-29 03:29	Collision	Note Added: 0001316
2016-09-29 10:40	luc	Note Added: 0001317
2016-09-29 12:30	Collision	Note Added: 0001318
2016-09-29 13:35	luc	Note Added: 0001319
2016-10-03 11:07	Collision	Note Added: 0001320
2016-10-04 19:03	BuBu	Note Added: 0001322
2016-10-04 19:03	BuBu	Status	new => resolved
2016-10-04 19:03	BuBu	Resolution	open => fixed
2016-10-04 19:03	BuBu	Assigned To	=> administrator

Notes
(0001305) luc (reporter) 2016-09-24 14:20	I also experienced this randomly last weeks. I didn't look deeply for the main cause, but I had the feeling it is occuring more frequently when networking bandwidth is limited or for example when WiFi connection is failing...

(0001308) Collision (reporter) 2016-09-27 12:15	This issue seems to be related to the following. ID: 0000566, Updated: 2015-07-03 Still not shooting down ID: 0000567, Updated: 2015-04-04 YaCy can't shutdown properly. Hangs on HeapReader dumping. Have to kill it. ID: 0000505, Updated: 2015-04-04 Can't shutdown This problem remains unresolved on my peer. Therefore I force shutdown YaCy everytime. In Japan, the politics is becoming dictatorial. Information control is carried out. I think that YaCy IS THE MOST IMPORTANT THING for the Japanese citizen.

(0001309) luc (reporter) 2016-09-28 09:55	I agree this is quite annoying. But looks like a difficult one to reproduce on demand and therefore to solve. I will check during the coming days if I can find some fix to apply.

(0001310) luc (reporter) 2016-09-28 15:31 edited on: 2016-09-28 15:44	Waw I already found a reproductible use case : - on Debian Linux with a headless JRE, Firefox installed, and latest YaCy sources from GitHub - start YaCy with no browser opened - firefox opens on localhost:8090 - do something or not with YaCy and then shutdown - shutdown does not work, until you close your browser Version 1.90 is not affected byt this issue, it was introduced by this commit : https://github.com/yacy/yacy_search_server/commit/bf6709d196bff6445826fd42bb411847188d2a1a [^] Please note that when Firefox is already opened, YaCy shutdowns correctly. So I commited a fix targetting YaCy on Linux Platform, but also to make sure this issue can no more occur on other platforms : https://github.com/yacy/yacy_search_server/commit/b8f64581525377649eaf44bbf2e679fedab1f1ff [^]

(0001311) luc (reporter) 2016-09-28 15:48 edited on: 2016-09-28 16:04	@Pythagoras , you are apparently running a YaCY version greater than 1.90. Does your peer shutdown correctly when you close your browser? @Collision , which YaCy version do you currently run and on which OS? What would really help to identify and eventually reproduce this kind of issue would be to provide a Thread Dump taken when your peer is blocked in the Shutdown step, either from YaCY with /Threaddump_p.html or with VisualVM using the 'Threads' tab and 'Thread dump' button.

(0001312) Collision (reporter) 2016-09-28 20:06	@luc, thank you for your kindness. I currently use YaCy 1.90/09000, Ubuntu 16.04.1 LTS, Linux kernel 4.4.0. I attach three files dumped by VisualVM.

(0001313) Pythagoras (reporter) 2016-09-28 21:13	Yes, you were right, the issue is indeed related to the browser. I confirm that when I close the browser the peer shuts down correctly. It also shuts down correctly if the browser were already running on startup. Tested on YaCy version 1.91/9188 (df51e4e).

(0001316) Collision (reporter) 2016-09-29 03:29	In my case, whether Firefox browser was running or not, the result was the same. I closed the browser after pushing the shutdown button, but YaCy never shut down.

(0001317) luc (reporter) 2016-09-29 10:40	Thank you Collision for your dumps, they were very useful ant let me discover another reproductible scenario : - shutdown blocks when config value crawler.MaxActiveThreads is greater than 200. This value can be configured in /PerformanceQueues_p.html admin page, in "Thread Pool Settings" table, row "Crawler Pool", column "maximum Active". Do you have this value greater than 200? I commited a fix (https://github.com/yacy/yacy_search_server/commit/dcdea2d02f815bcece1713cd2ef8c56c39f889db [^]) for this particular case, but it could also solve other potential concurrency issues in the shutdown process. If you can run a test from latest sources, feedback is welcome!

(0001318) Collision (reporter) 2016-09-29 12:30	Oh, yes! That's right! I have set "Crawler Pool - maximum Active" value to '600'. By the way, it is difficult to run a test from latest sources for me, sorry.

(0001319) luc (reporter) 2016-09-29 13:35	Ok Collision, no problem. So you will just have to wait for the next release or for an update made available in the System updates. I think we can clause this issue, but don't hesitate to report if you encounter other shutdown issues in 1.90 even with "Crawler Pool - maximum Active" set to 200.

(0001320) Collision (reporter) 2016-10-03 11:07	I set "Crawler Pool - maximum Active" to 180, and I confirmed that YaCy shut down successfully. Thank you so much, luc.

(0001322) BuBu (developer) 2016-10-04 19:03	see https://github.com/yacy/yacy_search_server/commit/dcdea2d02f815bcece1713cd2ef8c56c39f889db [^] by luccioman