YaCy-Bugtracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0000758YaCy[All Projects] Generalpublic2017-06-14 15:202017-06-15 23:03
ReporterCollision 
Assigned Toadministrator 
PriorityimmediateSeveritymajorReproducibilityalways
StatusresolvedResolutionfixed 
ETAnone 
PlatformIntel Core-i7OSUbuntuOS Version16.04 LTS
Product VersionYaCy 1.9 
Target VersionFixed in Version 
Summary0000758: Cpu load is always 100%.
DescriptionAfter I updated YaCy to version 1.921/9236, a problem occurred.
The CPU is always busy since YaCy started.
Steps To ReproduceStart up YaCy 1.921/9236.
Additional InformationI upload a Visual VM thread dump file and a screenshot.
TagsNo tags attached.
Attached Filespng file icon Screenshot from 2017-06-14 21-53-07.png [^] (197,752 bytes) 2017-06-14 15:20


? file icon threaddump-1497444704828.tdump [^] (89,248 bytes) 2017-06-14 15:21

- Relationships

-  Notes
(0001433)
Collision (reporter)
2017-06-14 15:32

By the way, today the autocratic regime legislates the law which punishes collusion and the democracy of Japan is about to be destroyed.....
(0001434)
luc (reporter)
2017-06-15 07:24

Your dump would suggest the load is mainly on Solr. Maybe the upgrade from Solr 5.5 to 6.6 is in cause...
Were you performing some search requests, crawls or any other operations on your peer while producing the thread dump? Or is your CPU always at 100% even when your peer is doing nothing else than waiting and eventually only answering to other peers?
(0001435)
Collision (reporter)
2017-06-15 08:48

Just by starting up YaCy, I have not done any operation.
The overload condition will continue until YaCy is forcibly terminated.
In addition,the storage access is also busy for a long time after startup.
(0001436)
Collision (reporter)
2017-06-15 08:54

It seems I can not access 'crawler monitor' and 'index browser'.
(0001437)
luc (reporter)
2017-06-15 12:30

The crawler monitor page also takes more than one minute to display on a small test peer of mine.
Using the "Sampler" feature of JVisualVM reveals that much CPU is used in the "org.apache.solr.handler.component.SuggestComponent$SuggesterListener.buildSuggesterIndex()".

Looks like other people experienced this using Solr: see https://issues.apache.org/jira/browse/SOLR-6845 [^]

Can you check if modifying your YaCy defaults/solr/solrconfig.xml by adding "<str name="buildOnStartup">false</str>" to the "<searchComponent name="suggest" class="solr.SuggestComponent">" configuration solve your problem? It looks like it works finer for me after that modification.
(0001438)
Collision (reporter)
2017-06-15 13:52

Yes! You are right, luc!
I tried as you said, the problem was solved.
Thank you very much for your kind reply.

By the way, does this problem affect other users?
(0001439)
luc (reporter)
2017-06-15 14:03

Yes other users that applied the upgrade are potentially affected, but I guess the problem is more or less visible depending on the index size and the processor you have.
How many documents do you have in your index?

By the way, thanks to your quick feedback, I am more sure of the relevency of the proposed fix and I will commit on GitHub.
(0001440)
Collision (reporter)
2017-06-15 15:15
edited on: 2017-06-15 15:21

I have 34,665,660 documents and 26,569,252 RWIs in my YaCy index.
YaCy is using 680.47 GB of HDD space.
'yacy/DATA/INDEX/freeworld/SEGMENTS/solr_6_6' contains 793 items, 554.3 GB used.

For reference, My PC has Intel Core-i7 3770T 4 core CPU, 32GB DDR3-SDRAM, 3TB HDD, 1Gbps FTTH network.

Thank you for your work!

(0001441)
Collision (reporter)
2017-06-15 17:56

One more infomation: 1,367,747,785 webgraph Edges I have.
(0001444)
BuBu (developer)
2017-06-15 23:03

see commit by @luc
https://github.com/yacy/yacy_search_server/commit/f6e8d7171865087d518f17278ad35a082b6cbfff [^]

- Issue History
Date Modified Username Field Change
2017-06-14 15:20 Collision New Issue
2017-06-14 15:20 Collision File Added: Screenshot from 2017-06-14 21-53-07.png
2017-06-14 15:21 Collision File Added: threaddump-1497444704828.tdump
2017-06-14 15:32 Collision Note Added: 0001433
2017-06-15 07:24 luc Note Added: 0001434
2017-06-15 08:48 Collision Note Added: 0001435
2017-06-15 08:54 Collision Note Added: 0001436
2017-06-15 12:30 luc Note Added: 0001437
2017-06-15 13:52 Collision Note Added: 0001438
2017-06-15 14:03 luc Note Added: 0001439
2017-06-15 15:15 Collision Note Added: 0001440
2017-06-15 15:16 Collision Note Edited: 0001440 View Revisions
2017-06-15 15:21 Collision Note Edited: 0001440 View Revisions
2017-06-15 17:56 Collision Note Added: 0001441
2017-06-15 23:03 BuBu Note Added: 0001444
2017-06-15 23:03 BuBu Status new => resolved
2017-06-15 23:03 BuBu Resolution open => fixed
2017-06-15 23:03 BuBu Assigned To => administrator


Copyright © 2000 - 2017 MantisBT Team
Powered by Mantis Bugtracker