0000761: Remote crawler causes JVM GC overhead limit exceeded

Notes
(0001458) smokingwheels (reporter) 2017-07-03 00:05 edited on: 2017-07-03 01:59	Screenshot from 2017-07-03 06-00-08.png Remote crawler speed spikes. Unsure why. Related to http://mantis.tokeek.de/view.php?id=656 [^] Yacy will start but is swapping on my pc.

(0001459) luc (reporter) 2017-07-03 15:08	If you can launch again your peer with the JVM option "-XX:+HeapDumpOnOutOfMemoryError" enabled (this generates a heap memory dump once a OutOfMemory error is thrown), this could greatly help analysis. The generated hprof files can be quite large, so you may not want to share it directly, but instead open it with JVisualVM, and check some key points : - "Threads at the heap dump" - Find "N" biggest objects by retained size - check the "References" to the biggest objects in the "Instances" view

(0001460) smokingwheels (reporter) 2017-07-03 15:08 edited on: 2017-07-03 23:29	MEMORY performed necessary GC, freed 138355 KB (requested/available/average: 125000 / 261476 / 195498 KB) is showing up in the logs now. I have double/parallel GC threads now. #get javastart args JAVA_ARGS=" -server -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djsse.enableSNIExtension=false -XX:+UseParNewGC -XX:ParallelGCThreads=2 -d64 "; If you use a 32 bit system you must Remove -d64. I also added a space in the JAVA_ARGS=" -server. Must have Java JDK installed to use this option. My system has 4GB in total and I am trying these settings. javastart_Xmx=Xmx1920m javastart_Xms=Xms384m One little Tweet. W 2017/07/03 21:19:08 IODispatcher Could not add merge job to queue: Queue full -XX:ParallelGCThreads=4 Seems to be ok was 2 did not crash for 5.5 hours crawling.

(0001461) smokingwheels (reporter) 2017-07-04 10:07	Threads.txt.tar.gz Just a bunch of thread dumps from june to now. In peers with IPv6 only it causes a problem for my IPv4 peers not to sure really what to do.

(0001462) smokingwheels (reporter) 2017-07-04 14:11	Ok the Dumps are around 2GB. It gets stuck on this. I 2017/07/04 19:16:20 HeapReader * close HeapFile citation.index.20170703130353782.blob; trace: net.yacy.kelondro.blob.HeapModifier.close(HeapModifier.java:50) -> net.yacy.kelondro.blob.HeapModifier.close(HeapModifier.java:54) -> net.yacy.kelondro.blob.HeapModifier.finalize(HeapModifier.java:58) -> java.lang.System$2.invokeFinalize(System.java:1270) -> java.lang.ref.Finalizer.runFinalizer(Finalizer.java:98) -> java.lang.ref.Finalizer.access$100(Finalizer.java:34) -> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:210) I 2017/07/04 19:17:19 MEMORY * performed explicit GC, freed 1595366 KB (requested/available/average: 93851 / 1664166 / 350964 KB) I will have a look at the dumps with QB64 and see what I find going on what you said. Its too big to transfer anywhere.

(0001463) smokingwheels (reporter) 2017-07-04 21:34 edited on: 2017-07-04 22:04	I tried various option but an old PC. One program tells me the dump file are no good. Short Video of java 8 Visual VM running not sure which buttons to push, I tried a few. https://youtu.be/OQIx_XxU9v8 [^] I started Yacy then the Visual VM.

(0001464) smokingwheels (reporter) 2017-07-05 07:32	crashhostdump.txt.tar.gz It is a very small part of an XML dump from my peer. I increased the JVM and was able to start yacy long enough to get a 3 GB XML dump file. If you use a search function in a word processor look for the string zzzz this is the end of each XML record defined by each of the <doc><str name codes. The last number indicates the length of the line input of one line. Some are very long I have no idea what I am looking at so I will leave that for someone else. Values/text are over 500 000 in length in some cases.

(0001466) smokingwheels (reporter) 2017-07-05 19:28	https://youtu.be/AvRUntG9j30 [^] Is with 50 GC threads and lower memory settings but new install. I tried an upgrade on the crashing server and I could not reduce the memory settings. Currently running Xmx = 1024 Xms = 512. With 4 GC threads, I noticed the Survivor 0 and 1 had timing gaps on the trace. I know the 50 maybe overkill. I have tried 16 and upwards. The average Load of the system is higher when crawling but the actual Java load is normal my system still responds ok.

Issue History
Date Modified	Username	Field	Change
2017-07-02 23:51	smokingwheels	New Issue
2017-07-03 00:03	smokingwheels	File Added: Screenshot from 2017-07-03 06-00-08.png
2017-07-03 00:05	smokingwheels	Note Added: 0001458
2017-07-03 01:59	smokingwheels	Note Edited: 0001458	View Revisions
2017-07-03 15:08	luc	Note Added: 0001459
2017-07-03 15:08	smokingwheels	Note Added: 0001460
2017-07-03 15:10	smokingwheels	Note Edited: 0001460	View Revisions
2017-07-03 15:20	smokingwheels	Note Edited: 0001460	View Revisions
2017-07-03 23:29	smokingwheels	Note Edited: 0001460	View Revisions
2017-07-04 10:03	smokingwheels	File Added: Threads.txt.tar.gz
2017-07-04 10:07	smokingwheels	Note Added: 0001461
2017-07-04 14:11	smokingwheels	Note Added: 0001462
2017-07-04 21:34	smokingwheels	Note Added: 0001463
2017-07-04 22:04	smokingwheels	Note Edited: 0001463	View Revisions
2017-07-05 07:19	smokingwheels	File Added: crashhostdump.txt.tar.gz
2017-07-05 07:32	smokingwheels	Note Added: 0001464
2017-07-05 19:28	smokingwheels	Note Added: 0001466

Relationships