YaCy-Bugtracker - YaCy
View Issue Details
0000758YaCy[All Projects] Generalpublic2017-06-14 15:202017-06-15 23:03
Collision 
administrator 
immediatemajoralways
resolvedfixed 
none 
Intel Core-i7Ubuntu16.04 LTS
YaCy 1.9 
 
0000758: Cpu load is always 100%.
After I updated YaCy to version 1.921/9236, a problem occurred.
The CPU is always busy since YaCy started.
Start up YaCy 1.921/9236.
I upload a Visual VM thread dump file and a screenshot.
No tags attached.
png Screenshot from 2017-06-14 21-53-07.png (197,752) 2017-06-14 15:20
http://mantis.tokeek.de/file_download.php?file_id=287&type=bug
png

? threaddump-1497444704828.tdump (89,248) 2017-06-14 15:21
http://mantis.tokeek.de/file_download.php?file_id=288&type=bug
Issue History
2017-06-14 15:20CollisionNew Issue
2017-06-14 15:20CollisionFile Added: Screenshot from 2017-06-14 21-53-07.png
2017-06-14 15:21CollisionFile Added: threaddump-1497444704828.tdump
2017-06-14 15:32CollisionNote Added: 0001433
2017-06-15 07:24lucNote Added: 0001434
2017-06-15 08:48CollisionNote Added: 0001435
2017-06-15 08:54CollisionNote Added: 0001436
2017-06-15 12:30lucNote Added: 0001437
2017-06-15 13:52CollisionNote Added: 0001438
2017-06-15 14:03lucNote Added: 0001439
2017-06-15 15:15CollisionNote Added: 0001440
2017-06-15 15:16CollisionNote Edited: 0001440bug_revision_view_page.php?bugnote_id=1440#r427
2017-06-15 15:21CollisionNote Edited: 0001440bug_revision_view_page.php?bugnote_id=1440#r428
2017-06-15 17:56CollisionNote Added: 0001441
2017-06-15 23:03BuBuNote Added: 0001444
2017-06-15 23:03BuBuStatusnew => resolved
2017-06-15 23:03BuBuResolutionopen => fixed
2017-06-15 23:03BuBuAssigned To => administrator

Notes
(0001433)
Collision   
2017-06-14 15:32   
By the way, today the autocratic regime legislates the law which punishes collusion and the democracy of Japan is about to be destroyed.....
(0001434)
luc   
2017-06-15 07:24   
Your dump would suggest the load is mainly on Solr. Maybe the upgrade from Solr 5.5 to 6.6 is in cause...
Were you performing some search requests, crawls or any other operations on your peer while producing the thread dump? Or is your CPU always at 100% even when your peer is doing nothing else than waiting and eventually only answering to other peers?
(0001435)
Collision   
2017-06-15 08:48   
Just by starting up YaCy, I have not done any operation.
The overload condition will continue until YaCy is forcibly terminated.
In addition,the storage access is also busy for a long time after startup.
(0001436)
Collision   
2017-06-15 08:54   
It seems I can not access 'crawler monitor' and 'index browser'.
(0001437)
luc   
2017-06-15 12:30   
The crawler monitor page also takes more than one minute to display on a small test peer of mine.
Using the "Sampler" feature of JVisualVM reveals that much CPU is used in the "org.apache.solr.handler.component.SuggestComponent$SuggesterListener.buildSuggesterIndex()".

Looks like other people experienced this using Solr: see https://issues.apache.org/jira/browse/SOLR-6845 [^]

Can you check if modifying your YaCy defaults/solr/solrconfig.xml by adding "<str name="buildOnStartup">false</str>" to the "<searchComponent name="suggest" class="solr.SuggestComponent">" configuration solve your problem? It looks like it works finer for me after that modification.
(0001438)
Collision   
2017-06-15 13:52   
Yes! You are right, luc!
I tried as you said, the problem was solved.
Thank you very much for your kind reply.

By the way, does this problem affect other users?
(0001439)
luc   
2017-06-15 14:03   
Yes other users that applied the upgrade are potentially affected, but I guess the problem is more or less visible depending on the index size and the processor you have.
How many documents do you have in your index?

By the way, thanks to your quick feedback, I am more sure of the relevency of the proposed fix and I will commit on GitHub.
(0001440)
Collision   
2017-06-15 15:15   
(edited on: 2017-06-15 15:21)
I have 34,665,660 documents and 26,569,252 RWIs in my YaCy index.
YaCy is using 680.47 GB of HDD space.
'yacy/DATA/INDEX/freeworld/SEGMENTS/solr_6_6' contains 793 items, 554.3 GB used.

For reference, My PC has Intel Core-i7 3770T 4 core CPU, 32GB DDR3-SDRAM, 3TB HDD, 1Gbps FTTH network.

Thank you for your work!

(0001441)
Collision   
2017-06-15 17:56   
One more infomation: 1,367,747,785 webgraph Edges I have.
(0001444)
BuBu   
2017-06-15 23:03   
see commit by @luc
https://github.com/yacy/yacy_search_server/commit/f6e8d7171865087d518f17278ad35a082b6cbfff [^]