Anonymous | Login | Signup for a new account | 2021-01-27 23:04 CET | ![]() |
Main | My View | View Issues | Change Log | Roadmap |
View Issue Details [ Jump to Notes ] | [ Issue History ] [ Print ] | ||||||||||||
ID | Project | Category | View Status | Date Submitted | Last Update | ||||||||
0000776 | YaCy | [All Projects] General | public | 2018-01-23 07:01 | 2019-07-28 08:38 | ||||||||
Reporter | smokingwheels | ||||||||||||
Assigned To | |||||||||||||
Priority | normal | Severity | minor | Reproducibility | random | ||||||||
Status | new | Resolution | open | ||||||||||
ETA | none | ||||||||||||
Platform | OS | OS Version | |||||||||||
Product Version | YaCy 1.9 | ||||||||||||
Target Version | Fixed in Version | ||||||||||||
Summary | 0000776: THREADS WITH STATES: BLOCKED While crawling | ||||||||||||
Description | Xmx1000m Xms140m YaCy Version: 1.921/9537 Assigned Memory = 932184064 Used Memory = 728007688 Available Memory = 204176376 this thread dump function can find threads that lock others, to enable this function start YaCy with 'startYACY.sh -l' THREADS WITH STATES: BLOCKED Thread= CrawlStacker_pool-1-thread-58 id=241 BLOCKED at net.yacy.kelondro.index.RAMIndex.has(RAMIndex.java:309) [assert (key != null);] at net.yacy.kelondro.index.RAMIndexCluster.has(RAMIndexCluster.java:343) at net.yacy.kelondro.index.RowHandleMap.has(RowHandleMap.java:427) at net.yacy.crawler.HostBalancer.has(HostBalancer.java:335) at net.yacy.crawler.HostBalancer.push(HostBalancer.java:375) at net.yacy.crawler.data.NoticedURL.push(NoticedURL.java:261) at net.yacy.crawler.CrawlStacker.stackCrawl(CrawlStacker.java:485) at net.yacy.crawler.CrawlStacker.process(CrawlStacker.java:267) at net.yacy.crawler.CrawlStacker.process(CrawlStacker.java:209) at net.yacy.kelondro.workflow.InstantBlockingThread.job(InstantBlockingThread.java:92) at net.yacy.kelondro.workflow.AbstractBlockingThread.run(AbstractBlockingThread.java:187) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) | ||||||||||||
Additional Information | I run a Pi-Hole with all these blocklists https://discourse.pi-hole.net/t/i-concatenated-every-blocklist-i-could-find/5184 [^] https://pi-hole.net/ [^] | ||||||||||||
Tags | No tags attached. | ||||||||||||
Attached Files | |||||||||||||
![]() |
|
(0001493) smokingwheels (reporter) 2018-01-23 08:03 |
Crawler speed low THREADS WITH STATES: BLOCKED Thread= Embedded solr query: q={!cache=false raw f=id}ZP4vI7LO17aZ&rows=1&start=0&fl=id,load_date_dt&defType=edismax&qf=text_t^1.0 id=198 BLOCKED at net.yacy.kelondro.index.BufferedObjectIndex.has(BufferedObjectIndex.java:215) [synchronized (this.backend) {] at net.yacy.crawler.HostQueue.has(HostQueue.java:471) at net.yacy.crawler.HostBalancer.has(HostBalancer.java:339) at net.yacy.crawler.HostBalancer.push(HostBalancer.java:375) at net.yacy.crawler.data.NoticedURL.push(NoticedURL.java:260) at net.yacy.crawler.CrawlStacker.stackCrawl(CrawlStacker.java:489) at net.yacy.crawler.CrawlStacker.process(CrawlStacker.java:267) at net.yacy.crawler.CrawlStacker.process(CrawlStacker.java:209) at net.yacy.kelondro.workflow.InstantBlockingThread.job(InstantBlockingThread.java:92) at net.yacy.kelondro.workflow.AbstractBlockingThread.run(AbstractBlockingThread.java:187) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) |
(0001494) luc (reporter) 2018-01-24 08:01 |
Hi and thanks for the report, smokingwheels. This is true that the RowHandleMap used as a caching structure for the HostBalancer as indeed a synchronization lock even on read-only operations, which is far from optimal. Switching to another structure with less synchronization locks, such as the standard ConcurrentHashMap should propably considered. I wonder how far it could improve (or not) performance on a heavy loaded peer. Testing will be needed, especially for reproductible performance measurements and non regressions. |
(0001495) luc (reporter) 2018-01-31 07:05 |
In the end it looks like the RowHandleMap structure used as depth cache is not so bad compared to the ConcurrentHashMap (see commit https://github.com/yacy/yacy_search_server/commit/098ee639119031239157b1fb798f73544a060c07 [^]). Smokingwheels, did you keep a complete copy of your Threads dump? It could be interesting to see which other threads were waiting instead of running. |
(0001502) smokingwheels (reporter) 2018-11-23 04:37 |
No sorry I had a System Crash but given time I could try again. I had my ADSL connection removed with out my permission 3 weeks ago. I only got the new service yesterday and am waiting for some cat 6 cable to connect up my pc again. |
![]() |
|||
Date Modified | Username | Field | Change |
2018-01-23 07:01 | smokingwheels | New Issue | |
2018-01-23 08:03 | smokingwheels | Note Added: 0001493 | |
2018-01-24 08:01 | luc | Note Added: 0001494 | |
2018-01-31 07:05 | luc | Note Added: 0001495 | |
2018-11-23 04:37 | smokingwheels | Note Added: 0001502 |
Copyright © 2000 - 2021 MantisBT Team |