YaCy-Bugtracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0000776YaCy[All Projects] Generalpublic2018-01-23 07:012018-11-23 04:37
Reportersmokingwheels 
Assigned To 
PrioritynormalSeverityminorReproducibilityrandom
StatusnewResolutionopen 
ETAnone 
PlatformOSOS Version
Product VersionYaCy 1.9 
Target VersionFixed in Version 
Summary0000776: THREADS WITH STATES: BLOCKED While crawling
DescriptionXmx1000m
Xms140m

YaCy Version: 1.921/9537
Assigned Memory = 932184064
Used Memory = 728007688
Available Memory = 204176376
 
 
this thread dump function can find threads that lock others, to enable this function start YaCy with 'startYACY.sh -l'
 
THREADS WITH STATES: BLOCKED
 
Thread= CrawlStacker_pool-1-thread-58 id=241 BLOCKED
at net.yacy.kelondro.index.RAMIndex.has(RAMIndex.java:309) [assert (key != null);]
at net.yacy.kelondro.index.RAMIndexCluster.has(RAMIndexCluster.java:343)
at net.yacy.kelondro.index.RowHandleMap.has(RowHandleMap.java:427)
at net.yacy.crawler.HostBalancer.has(HostBalancer.java:335)
at net.yacy.crawler.HostBalancer.push(HostBalancer.java:375)
at net.yacy.crawler.data.NoticedURL.push(NoticedURL.java:261)
at net.yacy.crawler.CrawlStacker.stackCrawl(CrawlStacker.java:485)
at net.yacy.crawler.CrawlStacker.process(CrawlStacker.java:267)
at net.yacy.crawler.CrawlStacker.process(CrawlStacker.java:209)
at net.yacy.kelondro.workflow.InstantBlockingThread.job(InstantBlockingThread.java:92)
at net.yacy.kelondro.workflow.AbstractBlockingThread.run(AbstractBlockingThread.java:187)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

 
Additional InformationI run a Pi-Hole with all these blocklists https://discourse.pi-hole.net/t/i-concatenated-every-blocklist-i-could-find/5184 [^]

https://pi-hole.net/ [^]
TagsNo tags attached.
Attached Files

- Relationships

-  Notes
(0001493)
smokingwheels (reporter)
2018-01-23 08:03

Crawler speed low

THREADS WITH STATES: BLOCKED
 
Thread= Embedded solr query: q={!cache=false raw f=id}ZP4vI7LO17aZ&rows=1&start=0&fl=id,load_date_dt&defType=edismax&qf=text_t^1.0 id=198 BLOCKED
at net.yacy.kelondro.index.BufferedObjectIndex.has(BufferedObjectIndex.java:215) [synchronized (this.backend) {]
at net.yacy.crawler.HostQueue.has(HostQueue.java:471)
at net.yacy.crawler.HostBalancer.has(HostBalancer.java:339)
at net.yacy.crawler.HostBalancer.push(HostBalancer.java:375)
at net.yacy.crawler.data.NoticedURL.push(NoticedURL.java:260)
at net.yacy.crawler.CrawlStacker.stackCrawl(CrawlStacker.java:489)
at net.yacy.crawler.CrawlStacker.process(CrawlStacker.java:267)
at net.yacy.crawler.CrawlStacker.process(CrawlStacker.java:209)
at net.yacy.kelondro.workflow.InstantBlockingThread.job(InstantBlockingThread.java:92)
at net.yacy.kelondro.workflow.AbstractBlockingThread.run(AbstractBlockingThread.java:187)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
(0001494)
luc (reporter)
2018-01-24 08:01

Hi and thanks for the report, smokingwheels.
This is true that the RowHandleMap used as a caching structure for the HostBalancer as indeed a synchronization lock even on read-only operations, which is far from optimal.

Switching to another structure with less synchronization locks, such as the standard ConcurrentHashMap should propably considered. I wonder how far it could improve (or not) performance on a heavy loaded peer.
Testing will be needed, especially for reproductible performance measurements and non regressions.
(0001495)
luc (reporter)
2018-01-31 07:05

In the end it looks like the RowHandleMap structure used as depth cache is not so bad compared to the ConcurrentHashMap (see commit https://github.com/yacy/yacy_search_server/commit/098ee639119031239157b1fb798f73544a060c07 [^]).

Smokingwheels, did you keep a complete copy of your Threads dump? It could be interesting to see which other threads were waiting instead of running.
(0001502)
smokingwheels (reporter)
2018-11-23 04:37

No sorry I had a System Crash but given time I could try again.
I had my ADSL connection removed with out my permission 3 weeks ago.
I only got the new service yesterday and am waiting for some cat 6 cable to connect up my pc again.

- Issue History
Date Modified Username Field Change
2018-01-23 07:01 smokingwheels New Issue
2018-01-23 08:03 smokingwheels Note Added: 0001493
2018-01-24 08:01 luc Note Added: 0001494
2018-01-31 07:05 luc Note Added: 0001495
2018-11-23 04:37 smokingwheels Note Added: 0001502


Copyright © 2000 - 2018 MantisBT Team
Powered by Mantis Bugtracker