0000758: Cpu load is always 100%.

(0001433)
Collision (reporter)
2017-06-14 15:32

By the way, today the autocratic regime legislates the law which punishes collusion and the democracy of Japan is about to be destroyed.....

(0001434)
luc (reporter)
2017-06-15 07:24

Your dump would suggest the load is mainly on Solr. Maybe the upgrade from Solr 5.5 to 6.6 is in cause...
Were you performing some search requests, crawls or any other operations on your peer while producing the thread dump? Or is your CPU always at 100% even when your peer is doing nothing else than waiting and eventually only answering to other peers?

(0001435)
Collision (reporter)
2017-06-15 08:48

Just by starting up YaCy, I have not done any operation.
The overload condition will continue until YaCy is forcibly terminated.
In addition,the storage access is also busy for a long time after startup.

(0001436)
Collision (reporter)
2017-06-15 08:54

It seems I can not access 'crawler monitor' and 'index browser'.

(0001437)
luc (reporter)
2017-06-15 12:30

The crawler monitor page also takes more than one minute to display on a small test peer of mine.
Using the "Sampler" feature of JVisualVM reveals that much CPU is used in the "org.apache.solr.handler.component.SuggestComponent$SuggesterListener.buildSuggesterIndex()".

Looks like other people experienced this using Solr: see https://issues.apache.org/jira/browse/SOLR-6845 [^]

Can you check if modifying your YaCy defaults/solr/solrconfig.xml by adding "<str name="buildOnStartup">false</str>" to the "<searchComponent name="suggest" class="solr.SuggestComponent">" configuration solve your problem? It looks like it works finer for me after that modification.

(0001438)
Collision (reporter)
2017-06-15 13:52

Yes! You are right, luc!
I tried as you said, the problem was solved.
Thank you very much for your kind reply.

By the way, does this problem affect other users?

(0001439)
luc (reporter)
2017-06-15 14:03

Yes other users that applied the upgrade are potentially affected, but I guess the problem is more or less visible depending on the index size and the processor you have.
How many documents do you have in your index?

By the way, thanks to your quick feedback, I am more sure of the relevency of the proposed fix and I will commit on GitHub.

(0001440)
Collision (reporter)
2017-06-15 15:15
edited on: 2017-06-15 15:21

I have 34,665,660 documents and 26,569,252 RWIs in my YaCy index.
YaCy is using 680.47 GB of HDD space.
'yacy/DATA/INDEX/freeworld/SEGMENTS/solr_6_6' contains 793 items, 554.3 GB used.

For reference, My PC has Intel Core-i7 3770T 4 core CPU, 32GB DDR3-SDRAM, 3TB HDD, 1Gbps FTTH network.

Thank you for your work!

(0001441)
Collision (reporter)
2017-06-15 17:56

One more infomation: 1,367,747,785 webgraph Edges I have.

(0001444)
BuBu (developer)
2017-06-15 23:03

see commit by @luc
https://github.com/yacy/yacy_search_server/commit/f6e8d7171865087d518f17278ad35a082b6cbfff [^]

Notes
(0001433) Collision (reporter) 2017-06-14 15:32	By the way, today the autocratic regime legislates the law which punishes collusion and the democracy of Japan is about to be destroyed.....

(0001434) luc (reporter) 2017-06-15 07:24	Your dump would suggest the load is mainly on Solr. Maybe the upgrade from Solr 5.5 to 6.6 is in cause... Were you performing some search requests, crawls or any other operations on your peer while producing the thread dump? Or is your CPU always at 100% even when your peer is doing nothing else than waiting and eventually only answering to other peers?

(0001435) Collision (reporter) 2017-06-15 08:48	Just by starting up YaCy, I have not done any operation. The overload condition will continue until YaCy is forcibly terminated. In addition,the storage access is also busy for a long time after startup.

(0001436) Collision (reporter) 2017-06-15 08:54	It seems I can not access 'crawler monitor' and 'index browser'.

(0001437) luc (reporter) 2017-06-15 12:30	The crawler monitor page also takes more than one minute to display on a small test peer of mine. Using the "Sampler" feature of JVisualVM reveals that much CPU is used in the "org.apache.solr.handler.component.SuggestComponent$SuggesterListener.buildSuggesterIndex()". Looks like other people experienced this using Solr: see https://issues.apache.org/jira/browse/SOLR-6845 [^] Can you check if modifying your YaCy defaults/solr/solrconfig.xml by adding "<str name="buildOnStartup">false</str>" to the "<searchComponent name="suggest" class="solr.SuggestComponent">" configuration solve your problem? It looks like it works finer for me after that modification.

(0001438) Collision (reporter) 2017-06-15 13:52	Yes! You are right, luc! I tried as you said, the problem was solved. Thank you very much for your kind reply. By the way, does this problem affect other users?

(0001439) luc (reporter) 2017-06-15 14:03	Yes other users that applied the upgrade are potentially affected, but I guess the problem is more or less visible depending on the index size and the processor you have. How many documents do you have in your index? By the way, thanks to your quick feedback, I am more sure of the relevency of the proposed fix and I will commit on GitHub.

(0001440) Collision (reporter) 2017-06-15 15:15 edited on: 2017-06-15 15:21	I have 34,665,660 documents and 26,569,252 RWIs in my YaCy index. YaCy is using 680.47 GB of HDD space. 'yacy/DATA/INDEX/freeworld/SEGMENTS/solr_6_6' contains 793 items, 554.3 GB used. For reference, My PC has Intel Core-i7 3770T 4 core CPU, 32GB DDR3-SDRAM, 3TB HDD, 1Gbps FTTH network. Thank you for your work!

(0001441) Collision (reporter) 2017-06-15 17:56	One more infomation: 1,367,747,785 webgraph Edges I have.

(0001444) BuBu (developer) 2017-06-15 23:03	see commit by @luc https://github.com/yacy/yacy_search_server/commit/f6e8d7171865087d518f17278ad35a082b6cbfff [^]

Issue History
Date Modified	Username	Field	Change
2017-06-14 15:20	Collision	New Issue
2017-06-14 15:20	Collision	File Added: Screenshot from 2017-06-14 21-53-07.png
2017-06-14 15:21	Collision	File Added: threaddump-1497444704828.tdump
2017-06-14 15:32	Collision	Note Added: 0001433
2017-06-15 07:24	luc	Note Added: 0001434
2017-06-15 08:48	Collision	Note Added: 0001435
2017-06-15 08:54	Collision	Note Added: 0001436
2017-06-15 12:30	luc	Note Added: 0001437
2017-06-15 13:52	Collision	Note Added: 0001438
2017-06-15 14:03	luc	Note Added: 0001439
2017-06-15 15:15	Collision	Note Added: 0001440
2017-06-15 15:16	Collision	Note Edited: 0001440	View Revisions
2017-06-15 15:21	Collision	Note Edited: 0001440	View Revisions
2017-06-15 17:56	Collision	Note Added: 0001441
2017-06-15 23:03	BuBu	Note Added: 0001444
2017-06-15 23:03	BuBu	Status	new => resolved
2017-06-15 23:03	BuBu	Resolution	open => fixed
2017-06-15 23:03	BuBu	Assigned To	=> administrator

Relationships