YaCy-Bugtracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0000269YaCy[All Projects] Generalpublic2013-07-25 02:032014-02-18 01:22
Reporterpr0vieh 
Assigned ToOrbiter 
PriorityhighSeveritycrashReproducibilitysometimes
StatusresolvedResolutionfixed 
ETAnone 
Platform64bitOSWindowsOS Version7
Product Version 
Target VersionFixed in Version 
Summary0000269: Yacy Freeze on DHT Only after 20 min with hight CPU Usage
DescriptionMoved out to logfile
TagsNo tags attached.
Attached Fileszip file icon java_pid6140.0001_Leak_Hunter.zip [^] (16,018 bytes) 2013-07-26 02:46
zip file icon java_pid6140.0001_Leak_Suspects.zip [^] (34,237 bytes) 2013-07-26 02:47
zip file icon java_pid6140.0001_System_Overview.zip [^] (25,428 bytes) 2013-07-26 02:47
log file icon yacy001.log [^] (41,053 bytes) 2013-07-27 13:49
zip file icon java_pid8248.0001_Leak_Suspects.zip [^] (34,358 bytes) 2013-08-01 01:40

- Relationships

-  Notes
(0000567)
pr0vieh (reporter)
2013-07-26 02:49

Hier mal ein Heap Dump aus Eclipse mit Memory Analysis ausgewertet sind html datein...
(0000591)
pr0vieh (reporter)
2013-08-01 01:41
edited on: 2013-08-01 01:43

Weiter Heap Dump @ 1.61/9021 ich hoffe es hilft euch ?!

(0000593)
ccompagnon (reporter)
2013-08-18 10:16
edited on: 2013-08-24 15:33

I have this issue too.
I had this issue on Ubuntu. I believed that I crawled too many pages.
Then I installed YaCy on a FreeBSD server and started it without crawling any pages. After 3 weeks --> 100% CPU.
Even with YaCy version 1.62/9025.
The GUI is not available and the log :
INDEX-TRANSFER-DISPATCHER STORE: Chunk ci6cUKzQoZ__ has failed to transmit index; marked peer as busy.
Load: 1.08935546875 (How the load can be greater than 1 ?)

(0000599)
ccompagnon (reporter)
2013-08-28 11:40

I'm using the "default profile" with 10%.
It seems to be better.
(0000600)
ccompagnon (reporter)
2013-08-28 18:04

Nope. Not better.
Even with remote crawling deactivated.
It seems to be overloaded when incoming connections increase (in my case more than 20 connections).
I will try to limit the number of incoming connections.
(0000601)
ccompagnon (reporter)
2013-08-28 19:57

Cannot limit the number of incoming connections.
YaCy still 100% loaded and :

java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at net.yacy.server.http.HTTPDFileHandler.invokeServlet(HTTPDFileHandler.java:1426)
    at net.yacy.server.http.HTTPDFileHandler.doResponse(HTTPDFileHandler.java:910)
    at net.yacy.server.http.HTTPDFileHandler.doGet(HTTPDFileHandler.java:250)
    at net.yacy.server.http.HTTPDemon.GET(HTTPDemon.java:403)
    at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at net.yacy.server.serverCore$Session.listen(serverCore.java:767)
    at net.yacy.server.serverCore$Session.run(serverCore.java:658)
Caused by: java.lang.OutOfMemoryError: Java heap space
W 2013/08/28 17:55:54 StackTrace Java heap space
java.lang.OutOfMemoryError: Java heap space
W 2013/08/28 17:55:54 StackTrace Java heap space
java.lang.OutOfMemoryError: Java heap space
W 2013/08/28 17:55:54 net.yacy.cora.federate.solr.connector.SolrServerConnector Java heap space
java.lang.OutOfMemoryError: Java heap space
(0000602)
ccompagnon (reporter)
2013-08-29 21:54

Set more memory to the JVM (--> 1024 MB)

But still 100%.
After some tests, the 100% comes when the number of incoming connections is up to 15-20.
Performances decrease from 12-13 incoming connections.
(0000603)
ccompagnon (reporter)
2013-08-31 17:04

Even with 10 incoming connections, YaCy crashes. (in my case after 1,5 days ater 100%)

Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
W 2013/08/31 14:56:01 StackTrace GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
W 2013/08/31 14:56:01 StackTrace GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
E 2013/08/31 14:56:01 BLOCKINGTHREAD Runtime Error in serverInstantThread.job, thread 'java.lang.reflect.Method.transferDocumentIndex.0': null
W 2013/08/31 14:57:59 net.yacy.cora.federate.solr.connector.SolrServerConnector Java heap space
java.lang.OutOfMemoryError: Java heap space
(0000608)
alexdobrianski (reporter)
2013-09-13 08:50

stuck 100% CPU with error "java.lang.OutOfMemoryError: Java heap space"
every time after 10 minutes of work when memory reached ~ 600k
first search: "to be or not to be"
second search: "to be or not to be" - 100% CPU time

tried everything - different memory size, different initial memory size, even changed HDD, different setting

every time the same - start - 10-15 min ==600K == 100% CPU time - -> kill javaw - start again - practically yacy is not working.
before reaching 600K search works fine and fast.
log:
W 2013/09/12 23:38:15 StackTrace Error executing query
java.io.IOException: Error executing query
    at net.yacy.cora.federate.solr.connector.AbstractSolrConnector.getDocumentById(AbstractSolrConnector.java:350)
    at net.yacy.cora.federate.solr.connector.MirrorSolrConnector.getDocumentById(MirrorSolrConnector.java:164)
    at net.yacy.cora.federate.solr.connector.ConcurrentUpdateSolrConnector.getDocumentById(ConcurrentUpdateSolrConnector.java:394)
    at net.yacy.search.index.Fulltext.getMetadata(Fulltext.java:359)
    at net.yacy.search.index.Fulltext.getMetadata(Fulltext.java:351)
    at net.yacy.peers.Protocol.transferIndex(Protocol.java:1384)
    at net.yacy.peers.Transmission$Chunk.transmit(Transmission.java:251)
    at net.yacy.peers.Dispatcher.transferDocumentIndex(Dispatcher.java:402)
    at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at net.yacy.kelondro.workflow.InstantBlockingThread.job(InstantBlockingThread.java:99)
    at net.yacy.kelondro.workflow.AbstractBlockingThread.run(AbstractBlockingThread.java:78)
    at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
    at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: Error executing query
    at net.yacy.cora.federate.solr.connector.EmbeddedSolrConnector.getResponseByParams(EmbeddedSolrConnector.java:145)
    at net.yacy.cora.federate.solr.connector.AbstractSolrConnector.getDocumentById(AbstractSolrConnector.java:345)
    ... 18 more
Caused by: java.lang.OutOfMemoryError: Java heap space
W 2013/09/12 23:38:19 StackTrace Error executing query
java.io.IOException: Error executing query
    at net.yacy.cora.federate.solr.connector.AbstractSolrConnector.getDocumentById(AbstractSolrConnector.java:350)
    at net.yacy.cora.federate.solr.connector.MirrorSolrConnector.getDocumentById(MirrorSolrConnector.java:164)
    at net.yacy.cora.federate.solr.connector.ConcurrentUpdateSolrConnector.getDocumentById(ConcurrentUpdateSolrConnector.java:394)
    at net.yacy.search.index.Fulltext.getMetadata(Fulltext.java:359)
    at net.yacy.search.index.Fulltext.getMetadata(Fulltext.java:351)
    at net.yacy.peers.Protocol.transferIndex(Protocol.java:1384)
    at net.yacy.peers.Transmission$Chunk.transmit(Transmission.java:251)
    at net.yacy.peers.Dispatcher.transferDocumentIndex(Dispatcher.java:402)
    at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at net.yacy.kelondro.workflow.InstantBlockingThread.job(InstantBlockingThread.java:99)
    at net.yacy.kelondro.workflow.AbstractBlockingThread.run(AbstractBlockingThread.java:78)
    at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
    at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: Error executing query
    at net.yacy.cora.federate.solr.connector.EmbeddedSolrConnector.getResponseByParams(EmbeddedSolrConnector.java:145)
    at net.yacy.cora.federate.solr.connector.AbstractSolrConnector.getDocumentById(AbstractSolrConnector.java:345)
    ... 18 more
Caused by: java.lang.OutOfMemoryError: Java heap space
W 2013/09/12 23:38:22 StackTrace Error executing query
java.io.IOException: Error executing query
    at net.yacy.cora.federate.solr.connector.AbstractSolrConnector.getDocumentById(AbstractSolrConnector.java:350)
    at net.yacy.cora.federate.solr.connector.MirrorSolrConnector.getDocumentById(MirrorSolrConnector.java:164)
    at net.yacy.cora.federate.solr.connector.ConcurrentUpdateSolrConnector.getDocumentById(ConcurrentUpdateSolrConnector.java:394)
    at net.yacy.search.index.Fulltext.getMetadata(Fulltext.java:359)
    at net.yacy.search.index.Fulltext.getMetadata(Fulltext.java:351)
    at net.yacy.peers.Protocol.transferIndex(Protocol.java:1384)
    at net.yacy.peers.Transmission$Chunk.transmit(Transmission.java:251)
    at net.yacy.peers.Dispatcher.transferDocumentIndex(Dispatcher.java:402)
    at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at net.yacy.kelondro.workflow.InstantBlockingThread.job(InstantBlockingThread.java:99)
    at net.yacy.kelondro.workflow.AbstractBlockingThread.run(AbstractBlockingThread.java:78)
    at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
    at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: Error executing query
    at net.yacy.cora.federate.solr.connector.EmbeddedSolrConnector.getResponseByParams(EmbeddedSolrConnector.java:145)
    at net.yacy.cora.federate.solr.connector.AbstractSolrConnector.getDocumentById(AbstractSolrConnector.java:345)
    ... 18 more
Caused by: java.lang.OutOfMemoryError: Java heap space


Does anybody have any idea what to do -

I tried year ago yacy - was the same problem (was winxp 32), - that time deleted yacy and wait for one year - now in is win7 - 64 bit and the same problem.

if needs to debug - just can somebody tell which environment needs to have.
(0000635)
Orbiter (manager)
2013-10-25 16:06

chances are very high that this is related to a very stupid memory leak which has been here around for years...
https://gitorious.org/yacy/rc1/commit/6aabc4e5c88193d0850cbbcc7c62e0eb7175ecb6 [^]
https://gitorious.org/yacy/rc1/commit/3c3cb7855555d5b2712bb1bc05cddb5c3175ec45 [^]
There are two fixes which address the memory leak only, but not the DHT process itself. But since the DHT process needs some temporary memory which this leak may have taken completely, the problem could be solved.

Please re-test.
(0000717)
Orbiter (manager)
2014-02-18 01:22

I believe I have fixed all these issues. There had been actually a lot of memory leaks, partly because some caching had been misconfigured, partly caused by bugs in the code.
There had been also a memory observer in the code which prevents running of memory-exhausting services, this observer also had bugs and had been fixed.
Additionally I implemented some load and memory check functions which prevent the running of DHT processes at all if the necessary ressources are not present.
With all these fixes a 20-million-document test-peer is now running smoothly.

- Issue History
Date Modified Username Field Change
2013-07-25 02:03 pr0vieh New Issue
2013-07-26 02:46 pr0vieh File Added: java_pid6140.0001_Leak_Hunter.zip
2013-07-26 02:47 pr0vieh File Added: java_pid6140.0001_Leak_Suspects.zip
2013-07-26 02:47 pr0vieh File Added: java_pid6140.0001_System_Overview.zip
2013-07-26 02:49 pr0vieh Note Added: 0000567
2013-07-27 13:49 Quix0r Description Updated View Revisions
2013-07-27 13:49 Quix0r File Added: yacy001.log
2013-08-01 01:40 pr0vieh File Added: java_pid8248.0001_Leak_Suspects.zip
2013-08-01 01:41 pr0vieh Note Added: 0000591
2013-08-01 01:43 pr0vieh Note Edited: 0000591 View Revisions
2013-08-18 10:16 ccompagnon Note Added: 0000593
2013-08-24 15:11 ccompagnon Note Edited: 0000593 View Revisions
2013-08-24 15:33 ccompagnon Note Edited: 0000593 View Revisions
2013-08-28 11:40 ccompagnon Note Added: 0000599
2013-08-28 18:04 ccompagnon Note Added: 0000600
2013-08-28 19:57 ccompagnon Note Added: 0000601
2013-08-29 21:54 ccompagnon Note Added: 0000602
2013-08-31 17:04 ccompagnon Note Added: 0000603
2013-09-13 08:50 alexdobrianski Note Added: 0000608
2013-10-25 16:06 Orbiter Note Added: 0000635
2013-10-31 14:37 smokingwheels Note Added: 0000638
2013-10-31 17:18 administrator Note Deleted: 0000638
2014-02-18 01:22 Orbiter Note Added: 0000717
2014-02-18 01:22 Orbiter Status new => resolved
2014-02-18 01:22 Orbiter Resolution open => fixed
2014-02-18 01:22 Orbiter Assigned To => Orbiter


Copyright © 2000 - 2019 MantisBT Team
Powered by Mantis Bugtracker