0000776: THREADS WITH STATES: BLOCKED While crawling

Notes
(0001493) smokingwheels (reporter) 2018-01-23 08:03	Crawler speed low THREADS WITH STATES: BLOCKED Thread= Embedded solr query: q={!cache=false raw f=id}ZP4vI7LO17aZ&rows=1&start=0&fl=id,load_date_dt&defType=edismax&qf=text_t^1.0 id=198 BLOCKED at net.yacy.kelondro.index.BufferedObjectIndex.has(BufferedObjectIndex.java:215) [synchronized (this.backend) {] at net.yacy.crawler.HostQueue.has(HostQueue.java:471) at net.yacy.crawler.HostBalancer.has(HostBalancer.java:339) at net.yacy.crawler.HostBalancer.push(HostBalancer.java:375) at net.yacy.crawler.data.NoticedURL.push(NoticedURL.java:260) at net.yacy.crawler.CrawlStacker.stackCrawl(CrawlStacker.java:489) at net.yacy.crawler.CrawlStacker.process(CrawlStacker.java:267) at net.yacy.crawler.CrawlStacker.process(CrawlStacker.java:209) at net.yacy.kelondro.workflow.InstantBlockingThread.job(InstantBlockingThread.java:92) at net.yacy.kelondro.workflow.AbstractBlockingThread.run(AbstractBlockingThread.java:187) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

(0001494) luc (reporter) 2018-01-24 08:01	Hi and thanks for the report, smokingwheels. This is true that the RowHandleMap used as a caching structure for the HostBalancer as indeed a synchronization lock even on read-only operations, which is far from optimal. Switching to another structure with less synchronization locks, such as the standard ConcurrentHashMap should propably considered. I wonder how far it could improve (or not) performance on a heavy loaded peer. Testing will be needed, especially for reproductible performance measurements and non regressions.

(0001495) luc (reporter) 2018-01-31 07:05	In the end it looks like the RowHandleMap structure used as depth cache is not so bad compared to the ConcurrentHashMap (see commit https://github.com/yacy/yacy_search_server/commit/098ee639119031239157b1fb798f73544a060c07 [^]). Smokingwheels, did you keep a complete copy of your Threads dump? It could be interesting to see which other threads were waiting instead of running.

(0001502) smokingwheels (reporter) 2018-11-23 04:37	No sorry I had a System Crash but given time I could try again. I had my ADSL connection removed with out my permission 3 weeks ago. I only got the new service yesterday and am waiting for some cat 6 cable to connect up my pc again.

Issue History
Date Modified	Username	Field	Change
2018-01-23 07:01	smokingwheels	New Issue
2018-01-23 08:03	smokingwheels	Note Added: 0001493
2018-01-24 08:01	luc	Note Added: 0001494
2018-01-31 07:05	luc	Note Added: 0001495
2018-11-23 04:37	smokingwheels	Note Added: 0001502

Relationships