View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0000769YaCy[All Projects] Generalpublic2017-09-30 03:142020-11-18 09:20
Assigned To 
PlatformOSOS Version
Product VersionYaCy 1.9 
Target VersionFixed in Version 
Summary0000769: http interface is not responding after crawling for a while.
DescriptionI set my peer to crawl at approx 10% of peak speed overnight (500 PPM).
Log files indicate it is still working.

I downloaded Visualvm and ran that and not sure of the command to get a heapdump?
Did a few screen dumps a threaddump and 2 yacy logfiles.
Also have a host file. [^] 6139421 bytes [^] 15157 bytes

This is a temporary site may run out or disappear in the future.
TagsNo tags attached.
Attached Filesgz file icon Screenshot from 2017-10-01 09-11-37.png.tar.gz [^] (122,278 bytes) 2017-10-01 03:39
log file icon hs_err_pid24830.log [^] (179,765 bytes) 2017-10-01 12:05
pdf file icon installing-an-easy-http-proxy-cache-polipo-805-lznv54.pdf [^] (36,239 bytes) 2017-10-02 01:55

- Relationships

-  Notes
smokingwheels (reporter)
2017-09-30 03:24
edited on: 2017-09-30 06:24

I have noticed on a forceful end task (pkill) when starting again it opens crawler stacks with 0 urls.
Also Pi config -XX:+UseParNewGC -XX:ParallelGCThreads=2
Old pc -XX:+UseParNewGC -XX:ParallelGCThreads=8 -d64

smokingwheels (reporter)
2017-10-01 03:31
edited on: 2017-10-01 03:41

Cloned and did a fresh peer after running e4defrag and started to crawl a few sites it was still crawling this morning. 500 ppm was the top crawl speed.
Using -XX:+UseParNewGC -XX:ParallelGCThreads=7 -d64
Why does the fresh install Version show up 1.921/9000?
All the others upgrades where 9388.
Did a heapdump and found the file. [^] 116304045 Bytes
I will try the other peer after upgrade today sometime.
The screen shot is of JVisualM Screenshot from 2017-10-01 09-11-37.png.tar.gz

smokingwheels (reporter)
2017-10-01 12:04
edited on: 2017-10-01 12:06

I enabled the jsresort and it crashed my peer.
I did an emergency restore of /DATA/index/*.* to another newfolder of a yacy server because of disk performance issues in the particular folder I was using.
I tested with dd if=/dev/zero of=/mnt/md0/test bs=512k count=20
and had over 600 MB/s an hot upgrade with yacy stopped I was getting lessthan 1 MB/s transfer.
I modified Grub. EDIT: to solve this, either remove enough RAM, or add “mem=8G” as kernel boot parameter (e.g. in /etc/default/grub on Ubuntu — don’t forget to run update-grub !) https://fhackts.wordpress.com/2014/03/10/very-slow-disk-write-performance-linux/ [^] I have no idea really but there is a hs_err_pid24830.log in the old yacy folder. #ShiftDel

smokingwheels (reporter)
2017-10-02 02:00

I am running a trial Polipo Proxy DNS cache server to see if that helps.

I ran DNS Benchmark for https://www.grc.com/dns/benchmark.htm [^] with Playonlinux and rebuilt the custom resolvers list this may help.
smokingwheels (reporter)
2017-10-03 15:38

Postprocessing Progress
busy:collecting 8087 documents from the collection for harvestkey null, partitioned by responsetime_i [^] [^] [^] [^] [^]
Something about long string.
I removed some html files later on.
luc (reporter)
2017-10-05 10:07

Hi smokingwheels, quite much data to analyze here :)

In the log files you provide in yacylogs30am.tar.gz, apart the traced OutOfMemory error, there is an interesting recurrent exception :

    at sun.nio.ch.SelectorImpl.keys(SelectorImpl.java:68)
    at org.eclipse.jetty.io.ManagedSelector.size(ManagedSelector.java:104)

There is maybe something to dig here. And this kind of error seems to be mentioned in some other projects using Jetty as a webserver...
smokingwheels (reporter)
2017-10-05 14:37
edited on: 2017-10-05 14:40

Hi Luc,
Dont worry to much I think my file system was set too high (775).
My Limit Crawler sits at 0 now.

Fixed now but thats the sort of thing that can happen if you leave the door open now days.

If you could run the jetty server thread in a controlled background process that may help. Sort of like a silent 503 but keep the caller online.

bestarticlelist (reporter)
2020-11-18 09:20

hooray! it's snowing! it's time to make a snowman.james runs out. he makes a large pile of snow. he places a huge snowball on top. he adds a headband and a hat. he adds an orange for the nostril. he adds coal for the eyes and buttons.within the night, james opens the door. what does he see? the snowman is transferring! james invitations him in. the snowman has by no means been interior a house. he says hey to the cat. he performs with paper towels.a moment later, the snowman takes james's hand and goes out.they move up, up, up into the air! they may be flying! what a terrific night time!the following morning, james jumps off the bed. he runs to the door.he desires to thank the snowman. however he is long past.
https://www.bestarticlelist.com [^]

- Issue History
Date Modified Username Field Change
2017-09-30 03:14 smokingwheels New Issue
2017-09-30 03:24 smokingwheels Note Added: 0001473
2017-09-30 06:24 smokingwheels Note Edited: 0001473 View Revisions
2017-10-01 03:31 smokingwheels Note Added: 0001474
2017-10-01 03:32 smokingwheels Note Edited: 0001474 View Revisions
2017-10-01 03:39 smokingwheels File Added: Screenshot from 2017-10-01 09-11-37.png.tar.gz
2017-10-01 03:41 smokingwheels Note Edited: 0001474 View Revisions
2017-10-01 12:04 smokingwheels Note Added: 0001475
2017-10-01 12:05 smokingwheels File Added: hs_err_pid24830.log
2017-10-01 12:06 smokingwheels Note Edited: 0001475 View Revisions
2017-10-02 01:55 smokingwheels File Added: installing-an-easy-http-proxy-cache-polipo-805-lznv54.pdf
2017-10-02 02:00 smokingwheels Note Added: 0001476
2017-10-03 15:38 smokingwheels Note Added: 0001479
2017-10-05 10:07 luc Note Added: 0001481
2017-10-05 14:37 smokingwheels Note Added: 0001482
2017-10-05 14:40 smokingwheels Note Edited: 0001482 View Revisions
2020-11-18 09:20 bestarticlelist Note Added: 0001514

Copyright © 2000 - 2020 MantisBT Team
Powered by Mantis Bugtracker