YaCy-Bugtracker - YaCy
View Issue Details
0000745YaCy[All Projects] Generalpublic2017-06-01 02:552021-03-26 09:35
YaCy 1.9 
0000745: Crawler speed reduces and Status log in GUI fails to refresh.
Noticed the remote crawer had slowed down after 2 hours.
Running 3 Yacys (Windows Version)in 1 playonlinux drive XP in use.
The Top VIRT memory is 45 % reduction in the figures when compaired to just running on Ubuntu 16.04.
I am downloading SP3 for playonlinux.

I wrote a program to extract errors and errors and warnings from the log files.
There is a 8 yacy log files.
No tags attached.
gz logfiles.tar.gz (1,309,594) 2017-06-01 02:55
txt Threads.txt (1,911) 2017-06-05 23:51
Issue History
2017-06-01 02:55smokingwheelsNew Issue
2017-06-01 02:55smokingwheelsFile Added: logfiles.tar.gz
2017-06-05 23:51smokingwheelsFile Added: Threads.txt
2017-06-06 06:56lucNote Added: 0001407
2017-06-06 14:53smokingwheelsNote Added: 0001408
2017-06-06 14:53smokingwheelsNote Edited: 0001408bug_revision_view_page.php?bugnote_id=1408#r403
2020-11-09 04:35bobpaulNote Added: 0001509
2020-11-09 04:35bobpaulNote Edited: 0001509bug_revision_view_page.php?bugnote_id=1509#r482
2020-11-09 04:36bobpaulNote Edited: 0001509bug_revision_view_page.php?bugnote_id=1509#r483
2021-03-26 09:35joyoungNote Added: 0001544
2021-03-26 09:35joyoungNote Edited: 0001544bug_revision_view_page.php?bugnote_id=1544#r504

2017-06-06 06:56   
Thank you smokingwheels for sharing a part of your threads dump. For this case this may be more helpful than the logs (for now, I did not found any clue about the issue by reading them). Can you share the full threads dump?

By the way, the part you shared may suggest there is a problem with file handlers. I already started cleaning up potential leaks in code dealing with FileInputStreams (https://github.com/yacy/yacy_search_server/commit/d98c04853dd39a452b0b98fb269a2013396cbb42 [^]). But there are probably other code paths to improve, notably the ones using RandomAccessFiles.
2017-06-06 14:53   
Ok Thanks.
I think there is a problem when doing a shutdown while crawling, it leaves the crawler Que's open on disk or something.

Clone and Run as root doing a crawl then shutdown Yacy.

I used chmod -R 755 /usr/yacy... then as a user moved Yacy's files to different place it still crawls when started but the 0003.stacks and folders have errors with cut and past they are all left behind.

Possible Fix is to create a crawler stacks index counter file separately thus retain compatibility with earlier versions, for times someone uses pkill java to terminate it.

2020-11-09 04:35   
(edited on: 2020-11-09 04:36)
that's a good idea
https://www.netbestgroup.com [^]

2021-03-26 09:35   
After the stealing of fire,Zeus became increasingly unkind to men.
https://www.thepornsitelists.com [^]