YaCy-Bugtracker - YaCy
View Issue Details
0000745YaCy[All Projects] Generalpublic2017-06-01 02:552019-07-28 08:38
smokingwheels 
 
normalminorrandom
newopen 
none 
YaCy 1.9 
 
0000745: Crawler speed reduces and Status log in GUI fails to refresh.
Noticed the remote crawer had slowed down after 2 hours.
Running 3 Yacys (Windows Version)in 1 playonlinux drive XP in use.
The Top VIRT memory is 45 % reduction in the figures when compaired to just running on Ubuntu 16.04.
I am downloading SP3 for playonlinux.


I wrote a program to extract errors and errors and warnings from the log files.
There is a 8 yacy log files.
No tags attached.
gz logfiles.tar.gz (1,309,594) 2017-06-01 02:55
http://mantis.tokeek.de/file_download.php?file_id=267&type=bug
txt Threads.txt (1,911) 2017-06-05 23:51
http://mantis.tokeek.de/file_download.php?file_id=268&type=bug
Issue History
2017-06-01 02:55smokingwheelsNew Issue
2017-06-01 02:55smokingwheelsFile Added: logfiles.tar.gz
2017-06-05 23:51smokingwheelsFile Added: Threads.txt
2017-06-06 06:56lucNote Added: 0001407
2017-06-06 14:53smokingwheelsNote Added: 0001408
2017-06-06 14:53smokingwheelsNote Edited: 0001408bug_revision_view_page.php?bugnote_id=1408#r403

Notes
(0001407)
luc   
2017-06-06 06:56   
Thank you smokingwheels for sharing a part of your threads dump. For this case this may be more helpful than the logs (for now, I did not found any clue about the issue by reading them). Can you share the full threads dump?

By the way, the part you shared may suggest there is a problem with file handlers. I already started cleaning up potential leaks in code dealing with FileInputStreams (https://github.com/yacy/yacy_search_server/commit/d98c04853dd39a452b0b98fb269a2013396cbb42 [^]). But there are probably other code paths to improve, notably the ones using RandomAccessFiles.
(0001408)
smokingwheels   
2017-06-06 14:53   
Ok Thanks.
I think there is a problem when doing a shutdown while crawling, it leaves the crawler Que's open on disk or something.

Clone and Run as root doing a crawl then shutdown Yacy.

I used chmod -R 755 /usr/yacy... then as a user moved Yacy's files to different place it still crawls when started but the 0003.stacks and folders have errors with cut and past they are all left behind.

Possible Fix is to create a crawler stacks index counter file separately thus retain compatibility with earlier versions, for times someone uses pkill java to terminate it.