Greetings,

Thank you for trying YourKit Java Profiler.

Your evaluation license key: 

With My extra comments on *.java and *.js. Files  *.css slightly optimized


Potential deadlock: frozen threads found

It seems that the following threads have not changed their stack for more than 10 seconds.
These threads are possibly (but not necessarily!) in a deadlock or hung.

storeDocumentIndex_pool-1-thread-43 <--- Frozen for at least 14s
net.yacy.cora.federate.solr.connector.SolrServerConnector.add(SolrInputDocument) SolrServerConnector.java:237
net.yacy.cora.federate.solr.connector.MirrorSolrConnector.add(SolrInputDocument) MirrorSolrConnector.java:228
net.yacy.search.index.Fulltext.putDocument(SolrInputDocument) Fulltext.java:330
net.yacy.search.index.Segment.putDocument(SolrInputDocument) Segment.java:510
net.yacy.search.index.Segment.storeDocument(DigestURL, DigestURL, Map, CrawlProfile, ResponseHeader, Document, Condenser, SearchEvent, String, boolean, String, String) Segment.java:553
net.yacy.search.Switchboard.storeDocumentIndex(Response, Map, Document, Condenser, SearchEvent, String) Switchboard.java:2739
net.yacy.search.Switchboard.storeDocumentIndex(IndexingQueueEntry) Switchboard.java:2689
sun.reflect.GeneratedMethodAccessor8.invoke(Object, Object[])
sun.reflect.DelegatingMethodAccessorImpl.invoke(Object, Object[]) DelegatingMethodAccessorImpl.java:43
java.lang.reflect.Method.invoke(Object, Object[]) Method.java:498
net.yacy.kelondro.workflow.InstantBlockingThread.job(WorkflowJob) InstantBlockingThread.java:109
net.yacy.kelondro.workflow.AbstractBlockingThread.run() AbstractBlockingThread.java:124
java.util.concurrent.Executors$RunnableAdapter.call() Executors.java:511
java.util.concurrent.FutureTask.run() FutureTask.java:266
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) ThreadPoolExecutor.java:1149
java.util.concurrent.ThreadPoolExecutor$Worker.run() ThreadPoolExecutor.java:624
java.lang.Thread.run() Thread.java:748


Potential deadlock: frozen threads found

It seems that the following threads have not changed their stack for more than 10 seconds.
These threads are possibly (but not necessarily!) in a deadlock or hung.

commitScheduler-15-thread-1 <--- Frozen for at least 17s
sun.nio.ch.FileDispatcherImpl.force0(FileDescriptor, boolean) FileDispatcherImpl.java (native)
sun.nio.ch.FileDispatcherImpl.force(FileDescriptor, boolean) FileDispatcherImpl.java:76
sun.nio.ch.FileChannelImpl.force(boolean) FileChannelImpl.java:388
org.apache.lucene.util.IOUtils.fsync(Path, boolean) IOUtils.java:472
org.apache.lucene.store.FSDirectory.fsync(String) FSDirectory.java:327
org.apache.lucene.store.FSDirectory.sync(Collection) FSDirectory.java:285
org.apache.lucene.store.LockValidatingDirectoryWrapper.sync(Collection) LockValidatingDirectoryWrapper.java:68
org.apache.lucene.index.IndexWriter.startCommit(SegmentInfos) IndexWriter.java:4724
org.apache.lucene.index.IndexWriter.prepareCommitInternal(boolean[]) IndexWriter.java:3085
org.apache.lucene.index.IndexWriter.commitInternal(MergePolicy) IndexWriter.java:3244
org.apache.lucene.index.IndexWriter.commit() IndexWriter.java:3207
org.apache.solr.update.DirectUpdateHandler2.commit(CommitUpdateCommand) DirectUpdateHandler2.java:644
org.apache.solr.update.CommitTracker.run() CommitTracker.java:217
java.util.concurrent.Executors$RunnableAdapter.call() Executors.java:511
java.util.concurrent.FutureTask.run() FutureTask.java:266
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor$ScheduledFutureTask) ScheduledThreadPoolExecutor.java:180
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run() ScheduledThreadPoolExecutor.java:293
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) ThreadPoolExecutor.java:1149
java.util.concurrent.ThreadPoolExecutor$Worker.run() ThreadPoolExecutor.java:624
java.lang.Thread.run() Thread.java:748



Stacks at 2017-10-09 10:42:53 AM. Uptime is 15m 33s 800ms.

storeDocumentIndex_pool-1-thread-158 State: BLOCKED CPU usage on sample: 0ms
net.yacy.cora.federate.solr.connector.SolrServerConnector.add(SolrInputDocument) SolrServerConnector.java:237
net.yacy.cora.federate.solr.connector.MirrorSolrConnector.add(SolrInputDocument) MirrorSolrConnector.java:228
net.yacy.search.index.Fulltext.putDocument(SolrInputDocument) Fulltext.java:330
net.yacy.search.index.Segment.putDocument(SolrInputDocument) Segment.java:510
net.yacy.search.index.Segment.storeDocument(DigestURL, DigestURL, Map, CrawlProfile, ResponseHeader, Document, Condenser, SearchEvent, String, boolean, String, String) Segment.java:553
net.yacy.search.Switchboard.storeDocumentIndex(Response, Map, Document, Condenser, SearchEvent, String) Switchboard.java:2739
net.yacy.search.Switchboard.storeDocumentIndex(IndexingQueueEntry) Switchboard.java:2689
sun.reflect.GeneratedMethodAccessor13.invoke(Object, Object[])
sun.reflect.DelegatingMethodAccessorImpl.invoke(Object, Object[]) DelegatingMethodAccessorImpl.java:43
java.lang.reflect.Method.invoke(Object, Object[]) Method.java:498
net.yacy.kelondro.workflow.InstantBlockingThread.job(WorkflowJob) InstantBlockingThread.java:109
net.yacy.kelondro.workflow.AbstractBlockingThread.run() AbstractBlockingThread.java:124
java.util.concurrent.Executors$RunnableAdapter.call() Executors.java:511
java.util.concurrent.FutureTask.run() FutureTask.java:266
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) ThreadPoolExecutor.java:1149
java.util.concurrent.ThreadPoolExecutor$Worker.run() ThreadPoolExecutor.java:624
java.lang.Thread.run() Thread.java:748


Stacks at 2017-10-09 10:54:53 AM. Uptime is 27m 34s 800ms.

storeDocumentIndex_pool-1-thread-158 State: BLOCKED CPU usage on sample: 72ms
java.util.Collections$SynchronizedMap.get(Object) Collections.java:2584
net.yacy.cora.storage.SimpleARC.get(Object) SimpleARC.java:209
net.yacy.cora.storage.ConcurrentARC.get(Object) ConcurrentARC.java:173
net.yacy.cora.order.Digest.encodeMD5Raw(String) Digest.java:135
net.yacy.cora.document.id.DigestURL.urlHashComputation() DigestURL.java:287
net.yacy.cora.document.id.DigestURL.hash() DigestURL.java:230
net.yacy.search.index.Segment.storeDocument(DigestURL, DigestURL, Map, CrawlProfile, ResponseHeader, Document, Condenser, SearchEvent, String, boolean, String, String) Segment.java:582
net.yacy.search.Switchboard.storeDocumentIndex(Response, Map, Document, Condenser, SearchEvent, String) Switchboard.java:2739
net.yacy.search.Switchboard.storeDocumentIndex(IndexingQueueEntry) Switchboard.java:2689
sun.reflect.GeneratedMethodAccessor13.invoke(Object, Object[])
sun.reflect.DelegatingMethodAccessorImpl.invoke(Object, Object[]) DelegatingMethodAccessorImpl.java:43
java.lang.reflect.Method.invoke(Object, Object[]) Method.java:498
net.yacy.kelondro.workflow.InstantBlockingThread.job(WorkflowJob) InstantBlockingThread.java:109
net.yacy.kelondro.workflow.AbstractBlockingThread.run() AbstractBlockingThread.java:124
java.util.concurrent.Executors$RunnableAdapter.call() Executors.java:511
java.util.concurrent.FutureTask.run() FutureTask.java:266
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) ThreadPoolExecutor.java:1149
java.util.concurrent.ThreadPoolExecutor$Worker.run() ThreadPoolExecutor.java:624
java.lang.Thread.run() Thread.java:748


Stacks at 2017-10-09 10:55:07 AM. Uptime is 27m 48s 100ms.

IODispatcher State: WAITING CPU usage on sample: 0ms
sun.misc.Unsafe.park(boolean, long) Unsafe.java (native)
java.util.concurrent.locks.LockSupport.park(Object) LockSupport.java:175
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt() AbstractQueuedSynchronizer.java:836
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(int) AbstractQueuedSynchronizer.java:997
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(int) AbstractQueuedSynchronizer.java:1304
java.util.concurrent.Semaphore.acquire() Semaphore.java:312
net.yacy.kelondro.rwi.IODispatcher.run() IODispatcher.java:198

Stacks at 2017-10-09 10:58:36 AM. Uptime is 31m 17s 100ms.

BusyThread CrawlQueues.coreCrawlJob [DAEMON] State: BLOCKED CPU usage on sample: 0ms
net.yacy.crawler.robots.RobotsTxt.getEntry(String, ClientIdentification$Agent, boolean) RobotsTxt.java:169
net.yacy.crawler.robots.RobotsTxt.getEntry(MultiProtocolURL, ClientIdentification$Agent) RobotsTxt.java:138
net.yacy.crawler.data.Latency.waitingRobots(MultiProtocolURL, RobotsTxt, ClientIdentification$Agent) Latency.java:128
net.yacy.crawler.data.Latency.waitingRemaining(DigestURL, RobotsTxt, ClientIdentification$Agent) Latency.java:185
net.yacy.crawler.data.Latency.getDomainSleepTime(RobotsTxt, CrawlProfile, DigestURL) Latency.java:232
net.yacy.crawler.HostQueue.pop(boolean, CrawlSwitchboard, RobotsTxt) HostQueue.java:471
net.yacy.crawler.HostBalancer.pop(boolean, CrawlSwitchboard, RobotsTxt) HostBalancer.java:439
net.yacy.crawler.data.NoticedURL.pop(Balancer, boolean, CrawlSwitchboard, RobotsTxt) NoticedURL.java:300
net.yacy.crawler.data.NoticedURL.pop(NoticedURL$StackType, boolean, CrawlSwitchboard, RobotsTxt) NoticedURL.java:254
net.yacy.crawler.data.CrawlQueues.coreCrawlJob() CrawlQueues.java:330
net.yacy.search.Switchboard$9.jobImpl() Switchboard.java:984
net.yacy.kelondro.workflow.InstantBusyThread.job() InstantBusyThread.java:100
net.yacy.kelondro.workflow.AbstractBusyThread.run() AbstractBusyThread.java:172


Stacks at 2017-10-09 10:59:12 AM. Uptime is 31m 53s 300ms.

Embedded solr query: q={!cache=false raw f=id}eTk9zJ80aHL6&rows=1&start=0&fl=id,load_date_dt&defType=edismax&qf=text_t^1.0 State: WAITING CPU usage on sample: 33ms
sun.misc.Unsafe.park(boolean, long) Unsafe.java (native)
java.util.concurrent.locks.LockSupport.park(Object) LockSupport.java:175
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() AbstractQueuedSynchronizer.java:2039
java.util.concurrent.LinkedBlockingQueue.take() LinkedBlockingQueue.java:442
net.yacy.kelondro.workflow.WorkflowProcessor.take() WorkflowProcessor.java:123
net.yacy.kelondro.workflow.AbstractBlockingThread.run() AbstractBlockingThread.java:119
java.util.concurrent.Executors$RunnableAdapter.call() Executors.java:511
java.util.concurrent.FutureTask.run() FutureTask.java:266
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) ThreadPoolExecutor.java:1149
java.util.concurrent.ThreadPoolExecutor$Worker.run() ThreadPoolExecutor.java:624
java.lang.Thread.run() Thread.java:748

Stacks at 2017-10-09 11:00:17 AM. Uptime is 32m 58s 100ms.

Embedded solr query: q={!cache=false raw f=id}eTk9zJ80aHL6&rows=1&start=0&fl=id,load_date_dt&defType=edismax&qf=text_t^1.0 State: BLOCKED CPU usage on sample: 11ms
net.yacy.crawler.HostBalancer.push(Request, CrawlProfile, RobotsTxt) HostBalancer.java:291
net.yacy.crawler.data.NoticedURL.push(NoticedURL$StackType, Request, CrawlProfile, RobotsTxt) NoticedURL.java:163
net.yacy.crawler.CrawlStacker.stackCrawl(Request) CrawlStacker.java:356
net.yacy.crawler.CrawlStacker.job(Request) CrawlStacker.java:134
sun.reflect.GeneratedMethodAccessor3.invoke(Object, Object[])
sun.reflect.DelegatingMethodAccessorImpl.invoke(Object, Object[]) DelegatingMethodAccessorImpl.java:43
java.lang.reflect.Method.invoke(Object, Object[]) Method.java:498
net.yacy.kelondro.workflow.InstantBlockingThread.job(WorkflowJob) InstantBlockingThread.java:109
net.yacy.kelondro.workflow.AbstractBlockingThread.run() AbstractBlockingThread.java:124
java.util.concurrent.Executors$RunnableAdapter.call() Executors.java:511
java.util.concurrent.FutureTask.run() FutureTask.java:266
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) ThreadPoolExecutor.java:1149
java.util.concurrent.ThreadPoolExecutor$Worker.run() ThreadPoolExecutor.java:624
java.lang.Thread.run() Thread.java:748



Stacks at 2017-10-09 11:03:52 AM. Uptime is 36m 33s 400ms.

Embedded solr query: q={!cache=false raw f=id}YLsTkp6VLpe6&rows=1&start=0&fl=id,load_date_dt&defType=edismax&qf=text_t^1.0 State: BLOCKED CPU usage on sample: 12ms
net.yacy.crawler.HostBalancer.push(Request, CrawlProfile, RobotsTxt) HostBalancer.java:291
net.yacy.crawler.data.NoticedURL.push(NoticedURL$StackType, Request, CrawlProfile, RobotsTxt) NoticedURL.java:163
net.yacy.crawler.CrawlStacker.stackCrawl(Request) CrawlStacker.java:356
net.yacy.crawler.CrawlStacker.job(Request) CrawlStacker.java:134
sun.reflect.GeneratedMethodAccessor3.invoke(Object, Object[])
sun.reflect.DelegatingMethodAccessorImpl.invoke(Object, Object[]) DelegatingMethodAccessorImpl.java:43
java.lang.reflect.Method.invoke(Object, Object[]) Method.java:498
net.yacy.kelondro.workflow.InstantBlockingThread.job(WorkflowJob) InstantBlockingThread.java:109
net.yacy.kelondro.workflow.AbstractBlockingThread.run() AbstractBlockingThread.java:124
java.util.concurrent.Executors$RunnableAdapter.call() Executors.java:511
java.util.concurrent.FutureTask.run() FutureTask.java:266
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) ThreadPoolExecutor.java:1149
java.util.concurrent.ThreadPoolExecutor$Worker.run() ThreadPoolExecutor.java:624
java.lang.Thread.run() Thread.java:748









Restored code to github 1.921/9410 11:14 am


Stacks at 2017-10-09 11:17:37 AM. Uptime is 6m 34s 200ms.

job_pool-1-thread-88 State: BLOCKED CPU usage on sample: 87ms
net.yacy.kelondro.index.BufferedObjectIndex.has(byte[]) BufferedObjectIndex.java:178
net.yacy.crawler.HostQueue.has(byte[]) HostQueue.java:369
net.yacy.crawler.HostBalancer.has(byte[]) HostBalancer.java:205
net.yacy.crawler.HostBalancer.push(Request, CrawlProfile, RobotsTxt) HostBalancer.java:245
net.yacy.crawler.data.NoticedURL.push(NoticedURL$StackType, Request, CrawlProfile, RobotsTxt) NoticedURL.java:184
net.yacy.crawler.CrawlStacker.stackCrawl(Request) CrawlStacker.java:390
net.yacy.crawler.CrawlStacker.job(Request) CrawlStacker.java:135
sun.reflect.GeneratedMethodAccessor4.invoke(Object, Object[])
sun.reflect.DelegatingMethodAccessorImpl.invoke(Object, Object[]) DelegatingMethodAccessorImpl.java:43
java.lang.reflect.Method.invoke(Object, Object[]) Method.java:498
net.yacy.kelondro.workflow.InstantBlockingThread.job(WorkflowJob) InstantBlockingThread.java:101
net.yacy.kelondro.workflow.AbstractBlockingThread.run() AbstractBlockingThread.java:82
java.util.concurrent.Executors$RunnableAdapter.call() Executors.java:511
java.util.concurrent.FutureTask.run() FutureTask.java:266
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) ThreadPoolExecutor.java:1149
java.util.concurrent.ThreadPoolExecutor$Worker.run() ThreadPoolExecutor.java:624
java.lang.Thread.run() Thread.java:748


Stacks at 2017-10-09 11:18:07 AM. Uptime is 7m 4s 600ms.

job_pool-1-thread-87 State: BLOCKED CPU usage on sample: 27ms
net.yacy.crawler.HostBalancer.push(Request, CrawlProfile, RobotsTxt) HostBalancer.java:248
net.yacy.crawler.data.NoticedURL.push(NoticedURL$StackType, Request, CrawlProfile, RobotsTxt) NoticedURL.java:184
net.yacy.crawler.CrawlStacker.stackCrawl(Request) CrawlStacker.java:390
net.yacy.crawler.CrawlStacker.job(Request) CrawlStacker.java:135
sun.reflect.GeneratedMethodAccessor4.invoke(Object, Object[])
sun.reflect.DelegatingMethodAccessorImpl.invoke(Object, Object[]) DelegatingMethodAccessorImpl.java:43
java.lang.reflect.Method.invoke(Object, Object[]) Method.java:498
net.yacy.kelondro.workflow.InstantBlockingThread.job(WorkflowJob) InstantBlockingThread.java:101
net.yacy.kelondro.workflow.AbstractBlockingThread.run() AbstractBlockingThread.java:82
java.util.concurrent.Executors$RunnableAdapter.call() Executors.java:511
java.util.concurrent.FutureTask.run() FutureTask.java:266
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) ThreadPoolExecutor.java:1149
java.util.concurrent.ThreadPoolExecutor$Worker.run() ThreadPoolExecutor.java:624
java.lang.Thread.run() Thread.java:748


Stacks at 2017-10-09 11:18:44 AM. Uptime is 7m 41s 400ms.

job_pool-1-thread-87 State: BLOCKED CPU usage on sample: 32ms
net.yacy.crawler.HostBalancer.push(Request, CrawlProfile, RobotsTxt) HostBalancer.java:248
net.yacy.crawler.data.NoticedURL.push(NoticedURL$StackType, Request, CrawlProfile, RobotsTxt) NoticedURL.java:184
net.yacy.crawler.CrawlStacker.stackCrawl(Request) CrawlStacker.java:390
net.yacy.crawler.CrawlStacker.job(Request) CrawlStacker.java:135
sun.reflect.GeneratedMethodAccessor4.invoke(Object, Object[])
sun.reflect.DelegatingMethodAccessorImpl.invoke(Object, Object[]) DelegatingMethodAccessorImpl.java:43
java.lang.reflect.Method.invoke(Object, Object[]) Method.java:498
net.yacy.kelondro.workflow.InstantBlockingThread.job(WorkflowJob) InstantBlockingThread.java:101
net.yacy.kelondro.workflow.AbstractBlockingThread.run() AbstractBlockingThread.java:82
java.util.concurrent.Executors$RunnableAdapter.call() Executors.java:511
java.util.concurrent.FutureTask.run() FutureTask.java:266
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) ThreadPoolExecutor.java:1149
java.util.concurrent.ThreadPoolExecutor$Worker.run() ThreadPoolExecutor.java:624
java.lang.Thread.run() Thread.java:748



Stacks at 2017-10-09 11:18:46 AM. Uptime is 7m 43s 400ms.

qtp1281414889-113 State: BLOCKED CPU usage on sample: 4ms
net.yacy.kelondro.blob.ArrayStack.sizes() ArrayStack.java:508
net.yacy.kelondro.rwi.ReferenceContainerArray.sizes() ReferenceContainerArray.java:87
net.yacy.kelondro.rwi.IndexCell.sizes() IndexCell.java:622
net.yacy.kelondro.rwi.IndexCell.sizesMax() IndexCell.java:631
net.yacy.search.index.Segment.RWICount() Segment.java:343
net.yacy.search.Switchboard.updateMySeed() Switchboard.java:4172
status_p.respond(RequestHeader, serverObjects, serverSwitch) status_p.java:63
sun.reflect.GeneratedMethodAccessor10.invoke(Object, Object[])
sun.reflect.DelegatingMethodAccessorImpl.invoke(Object, Object[]) DelegatingMethodAccessorImpl.java:43
java.lang.reflect.Method.invoke(Object, Object[]) Method.java:498
net.yacy.http.servlets.YaCyDefaultServlet.invokeServlet(File, RequestHeader, serverObjects) YaCyDefaultServlet.java:673
net.yacy.http.servlets.YaCyDefaultServlet.handleTemplate(String, HttpServletRequest, HttpServletResponse) YaCyDefaultServlet.java:884
net.yacy.http.servlets.YaCyDefaultServlet.doGet(HttpServletRequest, HttpServletResponse) YaCyDefaultServlet.java:315
javax.servlet.http.HttpServlet.service(HttpServletRequest, HttpServletResponse) HttpServlet.java:687
javax.servlet.http.HttpServlet.service(ServletRequest, ServletResponse) HttpServlet.java:790
org.eclipse.jetty.servlet.ServletHolder.handle(Request, ServletRequest, ServletResponse) ServletHolder.java:841
org.eclipse.jetty.servlet.ServletHandler.doHandle(String, Request, HttpServletRequest, HttpServletResponse) ServletHandler.java:535
org.eclipse.jetty.server.handler.ScopedHandler.handle(String, Request, HttpServletRequest, HttpServletResponse) ScopedHandler.java:143
org.eclipse.jetty.security.SecurityHandler.handle(String, Request, HttpServletRequest, HttpServletResponse) SecurityHandler.java:513
org.eclipse.jetty.server.handler.HandlerWrapper.handle(String, Request, HttpServletRequest, HttpServletResponse) HandlerWrapper.java:132
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(String, Request, HttpServletRequest, HttpServletResponse) ScopedHandler.java:190
org.eclipse.jetty.server.session.SessionHandler.doHandle(String, Request, HttpServletRequest, HttpServletResponse) SessionHandler.java:1595
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(String, Request, HttpServletRequest, HttpServletResponse) ScopedHandler.java:188
org.eclipse.jetty.server.handler.ContextHandler.doHandle(String, Request, HttpServletRequest, HttpServletResponse) ContextHandler.java:1253
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(String, Request, HttpServletRequest, HttpServletResponse) ScopedHandler.java:168
org.eclipse.jetty.servlet.ServletHandler.doScope(String, Request, HttpServletRequest, HttpServletResponse) ServletHandler.java:473
org.eclipse.jetty.server.session.SessionHandler.doScope(String, Request, HttpServletRequest, HttpServletResponse) SessionHandler.java:1564
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(String, Request, HttpServletRequest, HttpServletResponse) ScopedHandler.java:166
org.eclipse.jetty.server.handler.ContextHandler.doScope(String, Request, HttpServletRequest, HttpServletResponse) ContextHandler.java:1155
org.eclipse.jetty.server.handler.ScopedHandler.handle(String, Request, HttpServletRequest, HttpServletResponse) ScopedHandler.java:141
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(String, Request, HttpServletRequest, HttpServletResponse) ContextHandlerCollection.java:219
org.eclipse.jetty.server.handler.HandlerWrapper.handle(String, Request, HttpServletRequest, HttpServletResponse) HandlerWrapper.java:132
net.yacy.http.CrashProtectionHandler.handle(String, Request, HttpServletRequest, HttpServletResponse) CrashProtectionHandler.java:33
org.eclipse.jetty.server.handler.HandlerWrapper.handle(String, Request, HttpServletRequest, HttpServletResponse) HandlerWrapper.java:132
org.eclipse.jetty.server.Server.handle(HttpChannel) Server.java:564
org.eclipse.jetty.server.HttpChannel.handle() HttpChannel.java:317
org.eclipse.jetty.server.HttpConnection.onFillable() HttpConnection.java:251
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded() AbstractConnection.java:279
org.eclipse.jetty.io.FillInterest.fillable() FillInterest.java:110
org.eclipse.jetty.io.ChannelEndPoint$2.run() ChannelEndPoint.java:124
org.eclipse.jetty.util.thread.Invocable.invokePreferred(Runnable, Invocable$InvocationType) Invocable.java:128
org.eclipse.jetty.util.thread.Invocable$InvocableExecutor.invoke(Runnable) Invocable.java:222
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce() EatWhatYouKill.java:294
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run() EatWhatYouKill.java:199
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Runnable) QueuedThreadPool.java:673
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run() QueuedThreadPool.java:591
java.lang.Thread.run() Thread.java:748

Stacks at 2017-10-09 11:19:46 AM. Uptime is 8m 43s 300ms.

qtp1281414889-205 State: BLOCKED CPU usage on sample: 3ms
net.yacy.kelondro.index.BufferedObjectIndex.size() BufferedObjectIndex.java:151
net.yacy.crawler.HostQueue.size() HostQueue.java:381
net.yacy.crawler.HostBalancer.size() HostBalancer.java:212
net.yacy.crawler.data.NoticedURL.size() NoticedURL.java:130
net.yacy.search.Switchboard.updateMySeed() Switchboard.java:4170
status_p.respond(RequestHeader, serverObjects, serverSwitch) status_p.java:63
sun.reflect.GeneratedMethodAccessor10.invoke(Object, Object[])
sun.reflect.DelegatingMethodAccessorImpl.invoke(Object, Object[]) DelegatingMethodAccessorImpl.java:43
java.lang.reflect.Method.invoke(Object, Object[]) Method.java:498
net.yacy.http.servlets.YaCyDefaultServlet.invokeServlet(File, RequestHeader, serverObjects) YaCyDefaultServlet.java:673
net.yacy.http.servlets.YaCyDefaultServlet.handleTemplate(String, HttpServletRequest, HttpServletResponse) YaCyDefaultServlet.java:884
net.yacy.http.servlets.YaCyDefaultServlet.doGet(HttpServletRequest, HttpServletResponse) YaCyDefaultServlet.java:315
javax.servlet.http.HttpServlet.service(HttpServletRequest, HttpServletResponse) HttpServlet.java:687
javax.servlet.http.HttpServlet.service(ServletRequest, ServletResponse) HttpServlet.java:790
org.eclipse.jetty.servlet.ServletHolder.handle(Request, ServletRequest, ServletResponse) ServletHolder.java:841
org.eclipse.jetty.servlet.ServletHandler.doHandle(String, Request, HttpServletRequest, HttpServletResponse) ServletHandler.java:535
org.eclipse.jetty.server.handler.ScopedHandler.handle(String, Request, HttpServletRequest, HttpServletResponse) ScopedHandler.java:143
org.eclipse.jetty.security.SecurityHandler.handle(String, Request, HttpServletRequest, HttpServletResponse) SecurityHandler.java:513
org.eclipse.jetty.server.handler.HandlerWrapper.handle(String, Request, HttpServletRequest, HttpServletResponse) HandlerWrapper.java:132
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(String, Request, HttpServletRequest, HttpServletResponse) ScopedHandler.java:190
org.eclipse.jetty.server.session.SessionHandler.doHandle(String, Request, HttpServletRequest, HttpServletResponse) SessionHandler.java:1595
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(String, Request, HttpServletRequest, HttpServletResponse) ScopedHandler.java:188
org.eclipse.jetty.server.handler.ContextHandler.doHandle(String, Request, HttpServletRequest, HttpServletResponse) ContextHandler.java:1253
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(String, Request, HttpServletRequest, HttpServletResponse) ScopedHandler.java:168
org.eclipse.jetty.servlet.ServletHandler.doScope(String, Request, HttpServletRequest, HttpServletResponse) ServletHandler.java:473
org.eclipse.jetty.server.session.SessionHandler.doScope(String, Request, HttpServletRequest, HttpServletResponse) SessionHandler.java:1564
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(String, Request, HttpServletRequest, HttpServletResponse) ScopedHandler.java:166
org.eclipse.jetty.server.handler.ContextHandler.doScope(String, Request, HttpServletRequest, HttpServletResponse) ContextHandler.java:1155
org.eclipse.jetty.server.handler.ScopedHandler.handle(String, Request, HttpServletRequest, HttpServletResponse) ScopedHandler.java:141
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(String, Request, HttpServletRequest, HttpServletResponse) ContextHandlerCollection.java:219
org.eclipse.jetty.server.handler.HandlerWrapper.handle(String, Request, HttpServletRequest, HttpServletResponse) HandlerWrapper.java:132
net.yacy.http.CrashProtectionHandler.handle(String, Request, HttpServletRequest, HttpServletResponse) CrashProtectionHandler.java:33
org.eclipse.jetty.server.handler.HandlerWrapper.handle(String, Request, HttpServletRequest, HttpServletResponse) HandlerWrapper.java:132
org.eclipse.jetty.server.Server.handle(HttpChannel) Server.java:564
org.eclipse.jetty.server.HttpChannel.handle() HttpChannel.java:317
org.eclipse.jetty.server.HttpConnection.onFillable() HttpConnection.java:251
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded() AbstractConnection.java:279
org.eclipse.jetty.io.FillInterest.fillable() FillInterest.java:110
org.eclipse.jetty.io.ChannelEndPoint$2.run() ChannelEndPoint.java:124
org.eclipse.jetty.util.thread.Invocable.invokePreferred(Runnable, Invocable$InvocationType) Invocable.java:128
org.eclipse.jetty.util.thread.Invocable$InvocableExecutor.invoke(Runnable) Invocable.java:222
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce() EatWhatYouKill.java:294
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run() EatWhatYouKill.java:199
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Runnable) QueuedThreadPool.java:673
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run() QueuedThreadPool.java:591
java.lang.Thread.run() Thread.java:748


Potential deadlock: frozen threads found

It seems that the following threads have not changed their stack for more than 10 seconds.
These threads are possibly (but not necessarily!) in a deadlock or hung.

InetAddress.getByName_pool-1-thread-462 <--- Frozen for at least 13s
java.net.Inet6AddressImpl.lookupAllHostAddr(String) Inet6AddressImpl.java (native)
java.net.InetAddress$2.lookupAllHostAddr(String) InetAddress.java:928
java.net.InetAddress.getAddressesFromNameService(String, InetAddress) InetAddress.java:1323
java.net.InetAddress.getAllByName0(String, InetAddress, boolean) InetAddress.java:1276
java.net.InetAddress.getAllByName(String, InetAddress) InetAddress.java:1192
java.net.InetAddress.getAllByName(String) InetAddress.java:1126
java.net.InetAddress.getByName(String) InetAddress.java:1076
net.yacy.cora.protocol.Domains$2.call() Domains.java:960
net.yacy.cora.protocol.Domains$2.call() Domains.java:957
java.util.concurrent.FutureTask.run() FutureTask.java:266
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) ThreadPoolExecutor.java:1149
java.util.concurrent.ThreadPoolExecutor$Worker.run() ThreadPoolExecutor.java:624
java.lang.Thread.run() Thread.java:748


Stacks at 2017-10-09 11:21:23 AM. Uptime is 10m 20s 500ms.

qtp1281414889-217 State: BLOCKED CPU usage on sample: 0ms
java.io.ExpiringCache.get(String) ExpiringCache.java:75
java.io.UnixFileSystem.canonicalize(String) UnixFileSystem.java:152
java.io.File.getCanonicalPath() File.java:618
net.yacy.http.servlets.YaCyDefaultServlet.rewriteClassFile(File) YaCyDefaultServlet.java:763
net.yacy.http.servlets.YaCyDefaultServlet.handleTemplate(String, HttpServletRequest, HttpServletResponse) YaCyDefaultServlet.java:841
net.yacy.http.servlets.YaCyDefaultServlet.doGet(HttpServletRequest, HttpServletResponse) YaCyDefaultServlet.java:315
javax.servlet.http.HttpServlet.service(HttpServletRequest, HttpServletResponse) HttpServlet.java:687
javax.servlet.http.HttpServlet.service(ServletRequest, ServletResponse) HttpServlet.java:790
org.eclipse.jetty.servlet.ServletHolder.handle(Request, ServletRequest, ServletResponse) ServletHolder.java:841
org.eclipse.jetty.servlet.ServletHandler.doHandle(String, Request, HttpServletRequest, HttpServletResponse) ServletHandler.java:535
org.eclipse.jetty.server.handler.ScopedHandler.handle(String, Request, HttpServletRequest, HttpServletResponse) ScopedHandler.java:143
org.eclipse.jetty.security.SecurityHandler.handle(String, Request, HttpServletRequest, HttpServletResponse) SecurityHandler.java:524
org.eclipse.jetty.server.handler.HandlerWrapper.handle(String, Request, HttpServletRequest, HttpServletResponse) HandlerWrapper.java:132
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(String, Request, HttpServletRequest, HttpServletResponse) ScopedHandler.java:190
org.eclipse.jetty.server.session.SessionHandler.doHandle(String, Request, HttpServletRequest, HttpServletResponse) SessionHandler.java:1595
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(String, Request, HttpServletRequest, HttpServletResponse) ScopedHandler.java:188
org.eclipse.jetty.server.handler.ContextHandler.doHandle(String, Request, HttpServletRequest, HttpServletResponse) ContextHandler.java:1253
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(String, Request, HttpServletRequest, HttpServletResponse) ScopedHandler.java:168
org.eclipse.jetty.servlet.ServletHandler.doScope(String, Request, HttpServletRequest, HttpServletResponse) ServletHandler.java:473
org.eclipse.jetty.server.session.SessionHandler.doScope(String, Request, HttpServletRequest, HttpServletResponse) SessionHandler.java:1564
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(String, Request, HttpServletRequest, HttpServletResponse) ScopedHandler.java:166
org.eclipse.jetty.server.handler.ContextHandler.doScope(String, Request, HttpServletRequest, HttpServletResponse) ContextHandler.java:1155
org.eclipse.jetty.server.handler.ScopedHandler.handle(String, Request, HttpServletRequest, HttpServletResponse) ScopedHandler.java:141
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(String, Request, HttpServletRequest, HttpServletResponse) ContextHandlerCollection.java:219
org.eclipse.jetty.server.handler.HandlerWrapper.handle(String, Request, HttpServletRequest, HttpServletResponse) HandlerWrapper.java:132
net.yacy.http.CrashProtectionHandler.handle(String, Request, HttpServletRequest, HttpServletResponse) CrashProtectionHandler.java:33
org.eclipse.jetty.server.handler.HandlerWrapper.handle(String, Request, HttpServletRequest, HttpServletResponse) HandlerWrapper.java:132
org.eclipse.jetty.server.Server.handle(HttpChannel) Server.java:564
org.eclipse.jetty.server.HttpChannel.handle() HttpChannel.java:317
org.eclipse.jetty.server.HttpConnection.onFillable() HttpConnection.java:251
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded() AbstractConnection.java:279
org.eclipse.jetty.io.FillInterest.fillable() FillInterest.java:110
org.eclipse.jetty.io.ChannelEndPoint$2.run() ChannelEndPoint.java:124
org.eclipse.jetty.util.thread.Invocable.invokePreferred(Runnable, Invocable$InvocationType) Invocable.java:128
org.eclipse.jetty.util.thread.Invocable$InvocableExecutor.invoke(Runnable) Invocable.java:222
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce() EatWhatYouKill.java:294
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run() EatWhatYouKill.java:199
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Runnable) QueuedThreadPool.java:673
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run() QueuedThreadPool.java:591
java.lang.Thread.run() Thread.java:748


Stacks at 2017-10-09 11:22:02 AM. Uptime is 10m 59s 200ms.

BusyThread Switchboard.dhtTransferJob [DAEMON] State: BLOCKED CPU usage on sample: 177ms
net.yacy.kelondro.blob.ArrayStack.keys(boolean, byte[]) ArrayStack.java:542
net.yacy.kelondro.rwi.ReferenceContainerArray$ReferenceContainerIterator.<init>(ReferenceContainerArray, byte[], boolean, boolean) ReferenceContainerArray.java:134
net.yacy.kelondro.rwi.ReferenceContainerArray.referenceContainerIterator(byte[], boolean, boolean) ReferenceContainerArray.java:114
net.yacy.kelondro.rwi.IndexCell.referenceContainerIterator(byte[], boolean, boolean, boolean) IndexCell.java:562
net.yacy.peers.Dispatcher.selectContainers(byte[], byte[], int, int, int, boolean) Dispatcher.java:175
net.yacy.peers.Dispatcher.selectContainers(byte[], byte[], int, int, int) Dispatcher.java:158
net.yacy.peers.Dispatcher.selectContainersEnqueueToBuffer(byte[], byte[], int, int, int) Dispatcher.java:306
net.yacy.search.Switchboard.dhtTransferJob() Switchboard.java:3935
net.yacy.search.Switchboard$12.jobImpl() Switchboard.java:1192
net.yacy.kelondro.workflow.InstantBusyThread.job() InstantBusyThread.java:64
net.yacy.kelondro.workflow.AbstractBusyThread.run() AbstractBusyThread.java:215

Stacks at 2017-10-09 11:22:34 AM. Uptime is 11m 30s 800ms.

qtp1281414889-121 State: BLOCKED CPU usage on sample: 4ms
org.apache.lucene.index.IndexWriter.isClosed() IndexWriter.java:4891
org.apache.lucene.index.StandardDirectoryReader.isCurrent() StandardDirectoryReader.java:350
org.apache.lucene.index.FilterDirectoryReader.isCurrent() FilterDirectoryReader.java:124  <2 recursive calls>
org.apache.solr.handler.admin.LukeRequestHandler$$Lambda$182.get()
org.apache.solr.handler.admin.LukeRequestHandler.closeSafe(LukeRequestHandler$IOSupplier) LukeRequestHandler.java:605
org.apache.solr.handler.admin.LukeRequestHandler.getIndexInfo(DirectoryReader) LukeRequestHandler.java:582
org.apache.solr.handler.admin.LukeRequestHandler.handleRequestBody(SolrQueryRequest, SolrQueryResponse) LukeRequestHandler.java:138
org.apache.solr.handler.RequestHandlerBase.handleRequest(SolrQueryRequest, SolrQueryResponse) RequestHandlerBase.java:173
org.apache.solr.core.SolrCore.execute(SolrRequestHandler, SolrQueryRequest, SolrQueryResponse) SolrCore.java:2477
org.apache.solr.client.solrj.embedded.EmbeddedSolrServer.request(SolrRequest, String) EmbeddedSolrServer.java:179
org.apache.solr.client.solrj.SolrRequest.process(SolrClient, String) SolrRequest.java:160
org.apache.solr.client.solrj.SolrRequest.process(SolrClient) SolrRequest.java:177
net.yacy.cora.federate.solr.connector.SolrServerConnector.getIndexBrowser(boolean) SolrServerConnector.java:421
net.yacy.cora.federate.solr.connector.SolrServerConnector.getSegmentCount() SolrServerConnector.java:345
net.yacy.cora.federate.solr.connector.MirrorSolrConnector.getSegmentCount() MirrorSolrConnector.java:133
status_p.respond(RequestHeader, serverObjects, serverSwitch) status_p.java:87
sun.reflect.GeneratedMethodAccessor10.invoke(Object, Object[])
sun.reflect.DelegatingMethodAccessorImpl.invoke(Object, Object[]) DelegatingMethodAccessorImpl.java:43
java.lang.reflect.Method.invoke(Object, Object[]) Method.java:498
net.yacy.http.servlets.YaCyDefaultServlet.invokeServlet(File, RequestHeader, serverObjects) YaCyDefaultServlet.java:673
net.yacy.http.servlets.YaCyDefaultServlet.handleTemplate(String, HttpServletRequest, HttpServletResponse) YaCyDefaultServlet.java:884
net.yacy.http.servlets.YaCyDefaultServlet.doGet(HttpServletRequest, HttpServletResponse) YaCyDefaultServlet.java:315
javax.servlet.http.HttpServlet.service(HttpServletRequest, HttpServletResponse) HttpServlet.java:687
javax.servlet.http.HttpServlet.service(ServletRequest, ServletResponse) HttpServlet.java:790
org.eclipse.jetty.servlet.ServletHolder.handle(Request, ServletRequest, ServletResponse) ServletHolder.java:841
org.eclipse.jetty.servlet.ServletHandler.doHandle(String, Request, HttpServletRequest, HttpServletResponse) ServletHandler.java:535
org.eclipse.jetty.server.handler.ScopedHandler.handle(String, Request, HttpServletRequest, HttpServletResponse) ScopedHandler.java:143
org.eclipse.jetty.security.SecurityHandler.handle(String, Request, HttpServletRequest, HttpServletResponse) SecurityHandler.java:513
org.eclipse.jetty.server.handler.HandlerWrapper.handle(String, Request, HttpServletRequest, HttpServletResponse) HandlerWrapper.java:132
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(String, Request, HttpServletRequest, HttpServletResponse) ScopedHandler.java:190
org.eclipse.jetty.server.session.SessionHandler.doHandle(String, Request, HttpServletRequest, HttpServletResponse) SessionHandler.java:1595
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(String, Request, HttpServletRequest, HttpServletResponse) ScopedHandler.java:188
org.eclipse.jetty.server.handler.ContextHandler.doHandle(String, Request, HttpServletRequest, HttpServletResponse) ContextHandler.java:1253
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(String, Request, HttpServletRequest, HttpServletResponse) ScopedHandler.java:168
org.eclipse.jetty.servlet.ServletHandler.doScope(String, Request, HttpServletRequest, HttpServletResponse) ServletHandler.java:473
org.eclipse.jetty.server.session.SessionHandler.doScope(String, Request, HttpServletRequest, HttpServletResponse) SessionHandler.java:1564
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(String, Request, HttpServletRequest, HttpServletResponse) ScopedHandler.java:166
org.eclipse.jetty.server.handler.ContextHandler.doScope(String, Request, HttpServletRequest, HttpServletResponse) ContextHandler.java:1155
org.eclipse.jetty.server.handler.ScopedHandler.handle(String, Request, HttpServletRequest, HttpServletResponse) ScopedHandler.java:141
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(String, Request, HttpServletRequest, HttpServletResponse) ContextHandlerCollection.java:219
org.eclipse.jetty.server.handler.HandlerWrapper.handle(String, Request, HttpServletRequest, HttpServletResponse) HandlerWrapper.java:132
net.yacy.http.CrashProtectionHandler.handle(String, Request, HttpServletRequest, HttpServletResponse) CrashProtectionHandler.java:33
org.eclipse.jetty.server.handler.HandlerWrapper.handle(String, Request, HttpServletRequest, HttpServletResponse) HandlerWrapper.java:132
org.eclipse.jetty.server.Server.handle(HttpChannel) Server.java:564
org.eclipse.jetty.server.HttpChannel.handle() HttpChannel.java:317
org.eclipse.jetty.server.HttpConnection.onFillable() HttpConnection.java:251
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded() AbstractConnection.java:279
org.eclipse.jetty.io.FillInterest.fillable() FillInterest.java:110
org.eclipse.jetty.io.ChannelEndPoint$2.run() ChannelEndPoint.java:124
org.eclipse.jetty.util.thread.Invocable.invokePreferred(Runnable, Invocable$InvocationType) Invocable.java:128
org.eclipse.jetty.util.thread.Invocable$InvocableExecutor.invoke(Runnable) Invocable.java:222
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce() EatWhatYouKill.java:294
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run() EatWhatYouKill.java:199
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Runnable) QueuedThreadPool.java:673
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run() QueuedThreadPool.java:591
java.lang.Thread.run() Thread.java:748


Stacks at 2017-10-09 11:23:15 AM. Uptime is 12m 12s 200ms.

qtp1281414889-112 State: BLOCKED CPU usage on sample: 3ms
java.lang.Object.wait(long) Object.java (native)
java.lang.Object.wait() Object.java:502
org.eclipse.jetty.server.HttpInput.blockForContent() HttpInput.java:541
org.eclipse.jetty.server.HttpInputOverHTTP.blockForContent() HttpInputOverHTTP.java:42
org.eclipse.jetty.server.HttpInput$1.blockForContent(HttpInput) HttpInput.java:1062
org.eclipse.jetty.server.HttpInput.read(byte[], int, int) HttpInput.java:304
org.apache.commons.fileupload.MultipartStream$ItemInputStream.makeAvailable() MultipartStream.java:999
org.apache.commons.fileupload.MultipartStream$ItemInputStream.read(byte[], int, int) MultipartStream.java:903
java.io.FilterInputStream.read(byte[], int, int) FilterInputStream.java:133
org.apache.commons.fileupload.util.LimitedInputStream.read(byte[], int, int) LimitedInputStream.java:134
java.io.FilterInputStream.read(byte[]) FilterInputStream.java:107
org.apache.commons.fileupload.util.Streams.copy(InputStream, OutputStream, boolean, byte[]) Streams.java:100
org.apache.commons.fileupload.util.Streams.copy(InputStream, OutputStream, boolean) Streams.java:70
org.apache.commons.fileupload.FileUploadBase.parseRequest(RequestContext) FileUploadBase.java:347
org.apache.commons.fileupload.servlet.ServletFileUpload.parseRequest(HttpServletRequest) ServletFileUpload.java:115
net.yacy.http.servlets.YaCyDefaultServlet.parseMultipart(HttpServletRequest, serverObjects) YaCyDefaultServlet.java:1261
net.yacy.http.servlets.YaCyDefaultServlet.handleTemplate(String, HttpServletRequest, HttpServletResponse) YaCyDefaultServlet.java:874
net.yacy.http.servlets.YaCyDefaultServlet.doGet(HttpServletRequest, HttpServletResponse) YaCyDefaultServlet.java:315
net.yacy.http.servlets.YaCyDefaultServlet.doPost(HttpServletRequest, HttpServletResponse) YaCyDefaultServlet.java:377
javax.servlet.http.HttpServlet.service(HttpServletRequest, HttpServletResponse) HttpServlet.java:707
javax.servlet.http.HttpServlet.service(ServletRequest, ServletResponse) HttpServlet.java:790
org.eclipse.jetty.servlet.ServletHolder.handle(Request, ServletRequest, ServletResponse) ServletHolder.java:841
org.eclipse.jetty.servlet.ServletHandler.doHandle(String, Request, HttpServletRequest, HttpServletResponse) ServletHandler.java:535
org.eclipse.jetty.server.handler.ScopedHandler.handle(String, Request, HttpServletRequest, HttpServletResponse) ScopedHandler.java:143
org.eclipse.jetty.security.SecurityHandler.handle(String, Request, HttpServletRequest, HttpServletResponse) SecurityHandler.java:524
org.eclipse.jetty.server.handler.HandlerWrapper.handle(String, Request, HttpServletRequest, HttpServletResponse) HandlerWrapper.java:132
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(String, Request, HttpServletRequest, HttpServletResponse) ScopedHandler.java:190
org.eclipse.jetty.server.session.SessionHandler.doHandle(String, Request, HttpServletRequest, HttpServletResponse) SessionHandler.java:1595
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(String, Request, HttpServletRequest, HttpServletResponse) ScopedHandler.java:188
org.eclipse.jetty.server.handler.ContextHandler.doHandle(String, Request, HttpServletRequest, HttpServletResponse) ContextHandler.java:1253
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(String, Request, HttpServletRequest, HttpServletResponse) ScopedHandler.java:168
org.eclipse.jetty.servlet.ServletHandler.doScope(String, Request, HttpServletRequest, HttpServletResponse) ServletHandler.java:473
org.eclipse.jetty.server.session.SessionHandler.doScope(String, Request, HttpServletRequest, HttpServletResponse) SessionHandler.java:1564
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(String, Request, HttpServletRequest, HttpServletResponse) ScopedHandler.java:166
org.eclipse.jetty.server.handler.ContextHandler.doScope(String, Request, HttpServletRequest, HttpServletResponse) ContextHandler.java:1155
org.eclipse.jetty.server.handler.ScopedHandler.handle(String, Request, HttpServletRequest, HttpServletResponse) ScopedHandler.java:141
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(String, Request, HttpServletRequest, HttpServletResponse) ContextHandlerCollection.java:219
org.eclipse.jetty.server.handler.HandlerWrapper.handle(String, Request, HttpServletRequest, HttpServletResponse) HandlerWrapper.java:132
net.yacy.http.CrashProtectionHandler.handle(String, Request, HttpServletRequest, HttpServletResponse) CrashProtectionHandler.java:33
org.eclipse.jetty.server.handler.HandlerWrapper.handle(String, Request, HttpServletRequest, HttpServletResponse) HandlerWrapper.java:132
org.eclipse.jetty.server.Server.handle(HttpChannel) Server.java:564
org.eclipse.jetty.server.HttpChannel.handle() HttpChannel.java:317
org.eclipse.jetty.server.HttpConnection.onFillable() HttpConnection.java:251
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded() AbstractConnection.java:279
org.eclipse.jetty.io.FillInterest.fillable() FillInterest.java:110
org.eclipse.jetty.io.ChannelEndPoint$2.run() ChannelEndPoint.java:124
org.eclipse.jetty.util.thread.Invocable.invokePreferred(Runnable, Invocable$InvocationType) Invocable.java:128
org.eclipse.jetty.util.thread.Invocable$InvocableExecutor.invoke(Runnable) Invocable.java:222
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce() EatWhatYouKill.java:294
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run() EatWhatYouKill.java:199
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Runnable) QueuedThreadPool.java:673
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run() QueuedThreadPool.java:591
java.lang.Thread.run() Thread.java:748

Lots of these

Stacks at 2017-10-09 11:23:24 AM. Uptime is 12m 21s 200ms.

BusyThread CrawlQueues.coreCrawlJob [DAEMON] State: BLOCKED CPU usage on sample: 0ms
net.yacy.crawler.robots.RobotsTxt.getEntry(String, ClientIdentification$Agent, boolean) RobotsTxt.java:163
net.yacy.crawler.robots.RobotsTxt.getEntry(MultiProtocolURL, ClientIdentification$Agent) RobotsTxt.java:125
net.yacy.crawler.data.Latency.waitingRobots(MultiProtocolURL, RobotsTxt, ClientIdentification$Agent) Latency.java:125
net.yacy.crawler.data.Latency.waitingRemaining(DigestURL, RobotsTxt, ClientIdentification$Agent) Latency.java:216
net.yacy.crawler.data.Latency.getDomainSleepTime(RobotsTxt, CrawlProfile, DigestURL) Latency.java:285
net.yacy.crawler.HostQueue.pop(boolean, CrawlSwitchboard, RobotsTxt) HostQueue.java:455
net.yacy.crawler.HostBalancer.pop(boolean, CrawlSwitchboard, RobotsTxt) HostBalancer.java:420
net.yacy.crawler.data.NoticedURL.pop(Balancer, boolean, CrawlSwitchboard, RobotsTxt) NoticedURL.java:332
net.yacy.crawler.data.NoticedURL.pop(NoticedURL$StackType, boolean, CrawlSwitchboard, RobotsTxt) NoticedURL.java:285
net.yacy.crawler.data.CrawlQueues.coreCrawlJob() CrawlQueues.java:307
net.yacy.search.Switchboard$9.jobImpl() Switchboard.java:1141
net.yacy.kelondro.workflow.InstantBusyThread.job() InstantBusyThread.java:64
net.yacy.kelondro.workflow.AbstractBusyThread.run() AbstractBusyThread.java:215


Stacks at 2017-10-09 11:23:30 AM. Uptime is 12m 27s 200ms.

BusyThread CrawlQueues.coreCrawlJob [DAEMON] State: BLOCKED CPU usage on sample: 0ms
net.yacy.crawler.robots.RobotsTxt.getEntry(String, ClientIdentification$Agent, boolean) RobotsTxt.java:163
net.yacy.crawler.robots.RobotsTxt.getEntry(MultiProtocolURL, ClientIdentification$Agent) RobotsTxt.java:125
net.yacy.crawler.data.Latency.waitingRobots(MultiProtocolURL, RobotsTxt, ClientIdentification$Agent) Latency.java:125
net.yacy.crawler.data.Latency.waitingRemaining(DigestURL, RobotsTxt, ClientIdentification$Agent) Latency.java:216
net.yacy.crawler.data.Latency.getDomainSleepTime(RobotsTxt, CrawlProfile, DigestURL) Latency.java:285
net.yacy.crawler.HostQueue.pop(boolean, CrawlSwitchboard, RobotsTxt) HostQueue.java:455
net.yacy.crawler.HostBalancer.pop(boolean, CrawlSwitchboard, RobotsTxt) HostBalancer.java:420
net.yacy.crawler.data.NoticedURL.pop(Balancer, boolean, CrawlSwitchboard, RobotsTxt) NoticedURL.java:332
net.yacy.crawler.data.NoticedURL.pop(NoticedURL$StackType, boolean, CrawlSwitchboard, RobotsTxt) NoticedURL.java:285
net.yacy.crawler.data.CrawlQueues.coreCrawlJob() CrawlQueues.java:307
net.yacy.search.Switchboard$9.jobImpl() Switchboard.java:1141
net.yacy.kelondro.workflow.InstantBusyThread.job() InstantBusyThread.java:64
net.yacy.kelondro.workflow.AbstractBusyThread.run() AbstractBusyThread.java:215

Stacks at 2017-10-09 11:24:38 AM. Uptime is 13m 34s 900ms.

BusyThread CrawlQueues.coreCrawlJob [DAEMON] State: BLOCKED CPU usage on sample: 5ms
net.yacy.crawler.robots.RobotsTxt.getEntry(String, ClientIdentification$Agent, boolean) RobotsTxt.java:163
net.yacy.crawler.robots.RobotsTxt.getEntry(MultiProtocolURL, ClientIdentification$Agent) RobotsTxt.java:125
net.yacy.crawler.data.Latency.waitingRobots(MultiProtocolURL, RobotsTxt, ClientIdentification$Agent) Latency.java:125
net.yacy.crawler.data.Latency.waitingRemaining(DigestURL, RobotsTxt, ClientIdentification$Agent) Latency.java:216
net.yacy.crawler.data.Latency.getDomainSleepTime(RobotsTxt, CrawlProfile, DigestURL) Latency.java:285
net.yacy.crawler.HostQueue.pop(boolean, CrawlSwitchboard, RobotsTxt) HostQueue.java:455
net.yacy.crawler.HostBalancer.pop(boolean, CrawlSwitchboard, RobotsTxt) HostBalancer.java:420
net.yacy.crawler.data.NoticedURL.pop(Balancer, boolean, CrawlSwitchboard, RobotsTxt) NoticedURL.java:332
net.yacy.crawler.data.NoticedURL.pop(NoticedURL$StackType, boolean, CrawlSwitchboard, RobotsTxt) NoticedURL.java:285
net.yacy.crawler.data.CrawlQueues.coreCrawlJob() CrawlQueues.java:307
net.yacy.search.Switchboard$9.jobImpl() Switchboard.java:1141
net.yacy.kelondro.workflow.InstantBusyThread.job() InstantBusyThread.java:64
net.yacy.kelondro.workflow.AbstractBusyThread.run() AbstractBusyThread.java:215



Stacks at 2017-10-09 11:24:42 AM. Uptime is 13m 39s 200ms.

pTHuKZW-DefI_pool-1-thread-21 State: BLOCKED CPU usage on sample: 3ms
net.yacy.kelondro.blob.HeapReader.containsKey(byte[]) HeapReader.java:391
net.yacy.kelondro.blob.ArrayStack$1.call() ArrayStack.java:598
net.yacy.kelondro.blob.ArrayStack$1.call() ArrayStack.java:595
java.util.concurrent.FutureTask.run() FutureTask.java:266
java.util.concurrent.Executors$RunnableAdapter.call() Executors.java:511
java.util.concurrent.FutureTask.run() FutureTask.java:266
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) ThreadPoolExecutor.java:1149
java.util.concurrent.ThreadPoolExecutor$Worker.run() ThreadPoolExecutor.java:624
java.lang.Thread.run() Thread.java:748


Stacks at 2017-10-09 11:25:00 AM. Uptime is 13m 57s 400ms.

job_pool-1-thread-86 State: BLOCKED CPU usage on sample: 53ms
net.yacy.crawler.HostBalancer.push(Request, CrawlProfile, RobotsTxt) HostBalancer.java:248
net.yacy.crawler.data.NoticedURL.push(NoticedURL$StackType, Request, CrawlProfile, RobotsTxt) NoticedURL.java:184
net.yacy.crawler.CrawlStacker.stackCrawl(Request) CrawlStacker.java:390
net.yacy.crawler.CrawlStacker.job(Request) CrawlStacker.java:135
sun.reflect.GeneratedMethodAccessor4.invoke(Object, Object[])
sun.reflect.DelegatingMethodAccessorImpl.invoke(Object, Object[]) DelegatingMethodAccessorImpl.java:43
java.lang.reflect.Method.invoke(Object, Object[]) Method.java:498
net.yacy.kelondro.workflow.InstantBlockingThread.job(WorkflowJob) InstantBlockingThread.java:101
net.yacy.kelondro.workflow.AbstractBlockingThread.run() AbstractBlockingThread.java:82
java.util.concurrent.Executors$RunnableAdapter.call() Executors.java:511
java.util.concurrent.FutureTask.run() FutureTask.java:266
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) ThreadPoolExecutor.java:1149
java.util.concurrent.ThreadPoolExecutor$Worker.run() ThreadPoolExecutor.java:624
java.lang.Thread.run() Thread.java:748

Stacks at 2017-10-09 11:25:21 AM. Uptime is 14m 17s 900ms.

job_pool-1-thread-86 State: BLOCKED CPU usage on sample: 12ms
net.yacy.crawler.HostBalancer.push(Request, CrawlProfile, RobotsTxt) HostBalancer.java:248
net.yacy.crawler.data.NoticedURL.push(NoticedURL$StackType, Request, CrawlProfile, RobotsTxt) NoticedURL.java:184
net.yacy.crawler.CrawlStacker.stackCrawl(Request) CrawlStacker.java:390
net.yacy.crawler.CrawlStacker.job(Request) CrawlStacker.java:135
sun.reflect.GeneratedMethodAccessor4.invoke(Object, Object[])
sun.reflect.DelegatingMethodAccessorImpl.invoke(Object, Object[]) DelegatingMethodAccessorImpl.java:43
java.lang.reflect.Method.invoke(Object, Object[]) Method.java:498
net.yacy.kelondro.workflow.InstantBlockingThread.job(WorkflowJob) InstantBlockingThread.java:101
net.yacy.kelondro.workflow.AbstractBlockingThread.run() AbstractBlockingThread.java:82
java.util.concurrent.Executors$RunnableAdapter.call() Executors.java:511
java.util.concurrent.FutureTask.run() FutureTask.java:266
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) ThreadPoolExecutor.java:1149
java.util.concurrent.ThreadPoolExecutor$Worker.run() ThreadPoolExecutor.java:624
java.lang.Thread.run() Thread.java:748

Stacks at 2017-10-09 11:25:00 AM. Uptime is 13m 57s 400ms.

job_pool-1-thread-88 State: BLOCKED CPU usage on sample: 23ms
net.yacy.kelondro.index.RowHandleSet.has(byte[]) RowHandleSet.java:204
net.yacy.repository.Blacklist.isListed(Blacklist$BlacklistType, DigestURL) Blacklist.java:561
net.yacy.crawler.CrawlStacker.checkAcceptanceChangeable(DigestURL, CrawlProfile, int) CrawlStacker.java:486
net.yacy.crawler.CrawlStacker.stackCrawl(Request) CrawlStacker.java:349
net.yacy.crawler.CrawlStacker.job(Request) CrawlStacker.java:135
sun.reflect.GeneratedMethodAccessor4.invoke(Object, Object[])
sun.reflect.DelegatingMethodAccessorImpl.invoke(Object, Object[]) DelegatingMethodAccessorImpl.java:43
java.lang.reflect.Method.invoke(Object, Object[]) Method.java:498
net.yacy.kelondro.workflow.InstantBlockingThread.job(WorkflowJob) InstantBlockingThread.java:101
net.yacy.kelondro.workflow.AbstractBlockingThread.run() AbstractBlockingThread.java:82
java.util.concurrent.Executors$RunnableAdapter.call() Executors.java:511
java.util.concurrent.FutureTask.run() FutureTask.java:266
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) ThreadPoolExecutor.java:1149
java.util.concurrent.ThreadPoolExecutor$Worker.run() ThreadPoolExecutor.java:624
java.lang.Thread.run() Thread.java:748

Stacks at 2017-10-09 11:25:21 AM. Uptime is 14m 17s 900ms.

job_pool-1-thread-88 State: BLOCKED CPU usage on sample: 8ms
net.yacy.crawler.HostBalancer.push(Request, CrawlProfile, RobotsTxt) HostBalancer.java:248
net.yacy.crawler.data.NoticedURL.push(NoticedURL$StackType, Request, CrawlProfile, RobotsTxt) NoticedURL.java:184
net.yacy.crawler.CrawlStacker.stackCrawl(Request) CrawlStacker.java:390
net.yacy.crawler.CrawlStacker.job(Request) CrawlStacker.java:135
sun.reflect.GeneratedMethodAccessor4.invoke(Object, Object[])
sun.reflect.DelegatingMethodAccessorImpl.invoke(Object, Object[]) DelegatingMethodAccessorImpl.java:43
java.lang.reflect.Method.invoke(Object, Object[]) Method.java:498
net.yacy.kelondro.workflow.InstantBlockingThread.job(WorkflowJob) InstantBlockingThread.java:101
net.yacy.kelondro.workflow.AbstractBlockingThread.run() AbstractBlockingThread.java:82
java.util.concurrent.Executors$RunnableAdapter.call() Executors.java:511
java.util.concurrent.FutureTask.run() FutureTask.java:266
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) ThreadPoolExecutor.java:1149
java.util.concurrent.ThreadPoolExecutor$Worker.run() ThreadPoolExecutor.java:624
java.lang.Thread.run() Thread.java:748


Stacks at 2017-10-09 11:25:00 AM. Uptime is 13m 57s 400ms.

Thread-934 [DAEMON] State: BLOCKED CPU usage on sample: 0ms
net.yacy.cora.federate.solr.connector.SolrServerConnector.add(SolrInputDocument) SolrServerConnector.java:211
net.yacy.cora.federate.solr.connector.MirrorSolrConnector.add(SolrInputDocument) MirrorSolrConnector.java:204
net.yacy.search.index.ErrorCache.push(DigestURL, int, CrawlProfile, FailCategory, String, int) ErrorCache.java:122
net.yacy.crawler.retrieval.HTTPLoader.load(Request, CrawlProfile, int, int, Blacklist$BlacklistType, ClientIdentification$Agent) HTTPLoader.java:418
net.yacy.crawler.retrieval.HTTPLoader.load(Request, CrawlProfile, int, Blacklist$BlacklistType, ClientIdentification$Agent) HTTPLoader.java:83
net.yacy.repository.LoaderDispatcher.loadInternal(Request, CacheStrategy, int, Blacklist$BlacklistType, ClientIdentification$Agent) LoaderDispatcher.java:242
net.yacy.repository.LoaderDispatcher.load(Request, CacheStrategy, int, Blacklist$BlacklistType, ClientIdentification$Agent) LoaderDispatcher.java:180
net.yacy.repository.LoaderDispatcher.load(Request, CacheStrategy, Blacklist$BlacklistType, ClientIdentification$Agent) LoaderDispatcher.java:151
net.yacy.crawler.data.CrawlQueues$Loader.run() CrawlQueues.java:749


Lots of these and crawler slowed down

Stacks at 2017-10-09 11:54:00 AM. Uptime is 42m 57s 0ms.

Thread-1159 State: BLOCKED CPU usage on sample: 0ms
net.yacy.cora.federate.solr.connector.SolrServerConnector.add(SolrInputDocument) SolrServerConnector.java:211
net.yacy.cora.federate.solr.connector.MirrorSolrConnector.add(SolrInputDocument) MirrorSolrConnector.java:204
net.yacy.search.index.ErrorCache.push(DigestURL, int, CrawlProfile, FailCategory, String, int) ErrorCache.java:122
net.yacy.crawler.data.CrawlQueues$Loader.run() CrawlQueues.java:776

Stacks at 2017-10-09 12:00:33 PM. Uptime is 49m 30s 100ms.

qtp1281414889-3308 State: BLOCKED CPU usage on sample: 0ms
java.lang.ClassLoader.defineClass1(String, byte[], int, int, ProtectionDomain, String) ClassLoader.java (native)
java.lang.ClassLoader.defineClass(String, byte[], int, int, ProtectionDomain) ClassLoader.java:763
java.lang.ClassLoader.defineClass(String, byte[], int, int) ClassLoader.java:642
net.yacy.server.serverClassLoader.loadClass(File) serverClassLoader.java:93
net.yacy.http.servlets.YaCyDefaultServlet.rewriteMethod(File) YaCyDefaultServlet.java:793
net.yacy.http.servlets.YaCyDefaultServlet.invokeServlet(File, RequestHeader, serverObjects) YaCyDefaultServlet.java:673
net.yacy.http.servlets.YaCyDefaultServlet.handleTemplate(String, HttpServletRequest, HttpServletResponse) YaCyDefaultServlet.java:884
net.yacy.http.servlets.YaCyDefaultServlet.doGet(HttpServletRequest, HttpServletResponse) YaCyDefaultServlet.java:315
net.yacy.http.servlets.YaCyDefaultServlet.doPost(HttpServletRequest, HttpServletResponse) YaCyDefaultServlet.java:377
javax.servlet.http.HttpServlet.service(HttpServletRequest, HttpServletResponse) HttpServlet.java:707
javax.servlet.http.HttpServlet.service(ServletRequest, ServletResponse) HttpServlet.java:790
org.eclipse.jetty.servlet.ServletHolder.handle(Request, ServletRequest, ServletResponse) ServletHolder.java:841
org.eclipse.jetty.servlet.ServletHandler.doHandle(String, Request, HttpServletRequest, HttpServletResponse) ServletHandler.java:535
org.eclipse.jetty.server.handler.ScopedHandler.handle(String, Request, HttpServletRequest, HttpServletResponse) ScopedHandler.java:143
org.eclipse.jetty.security.SecurityHandler.handle(String, Request, HttpServletRequest, HttpServletResponse) SecurityHandler.java:524
org.eclipse.jetty.server.handler.HandlerWrapper.handle(String, Request, HttpServletRequest, HttpServletResponse) HandlerWrapper.java:132
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(String, Request, HttpServletRequest, HttpServletResponse) ScopedHandler.java:190
org.eclipse.jetty.server.session.SessionHandler.doHandle(String, Request, HttpServletRequest, HttpServletResponse) SessionHandler.java:1595
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(String, Request, HttpServletRequest, HttpServletResponse) ScopedHandler.java:188
org.eclipse.jetty.server.handler.ContextHandler.doHandle(String, Request, HttpServletRequest, HttpServletResponse) ContextHandler.java:1253
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(String, Request, HttpServletRequest, HttpServletResponse) ScopedHandler.java:168
org.eclipse.jetty.servlet.ServletHandler.doScope(String, Request, HttpServletRequest, HttpServletResponse) ServletHandler.java:473
org.eclipse.jetty.server.session.SessionHandler.doScope(String, Request, HttpServletRequest, HttpServletResponse) SessionHandler.java:1564
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(String, Request, HttpServletRequest, HttpServletResponse) ScopedHandler.java:166
org.eclipse.jetty.server.handler.ContextHandler.doScope(String, Request, HttpServletRequest, HttpServletResponse) ContextHandler.java:1155
org.eclipse.jetty.server.handler.ScopedHandler.handle(String, Request, HttpServletRequest, HttpServletResponse) ScopedHandler.java:141
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(String, Request, HttpServletRequest, HttpServletResponse) ContextHandlerCollection.java:219
org.eclipse.jetty.server.handler.HandlerWrapper.handle(String, Request, HttpServletRequest, HttpServletResponse) HandlerWrapper.java:132
net.yacy.http.CrashProtectionHandler.handle(String, Request, HttpServletRequest, HttpServletResponse) CrashProtectionHandler.java:33
org.eclipse.jetty.server.handler.HandlerWrapper.handle(String, Request, HttpServletRequest, HttpServletResponse) HandlerWrapper.java:132
org.eclipse.jetty.server.Server.handle(HttpChannel) Server.java:564
org.eclipse.jetty.server.HttpChannel.handle() HttpChannel.java:317
org.eclipse.jetty.server.HttpConnection.onFillable() HttpConnection.java:251
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded() AbstractConnection.java:279
org.eclipse.jetty.io.FillInterest.fillable() FillInterest.java:110
org.eclipse.jetty.io.ChannelEndPoint$2.run() ChannelEndPoint.java:124
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Runnable) QueuedThreadPool.java:673
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run() QueuedThreadPool.java:591
java.lang.Thread.run() Thread.java:748

Stacks at 2017-10-09 12:02:52 PM. Uptime is 51m 49s 200ms.

storeDocumentIndex_pool-1-thread-93 State: BLOCKED CPU usage on sample: 29ms
java.util.logging.FileHandler.publish(LogRecord) FileHandler.java:679
java.util.logging.Logger.log(LogRecord) Logger.java:738
org.slf4j.impl.JDK14LoggerAdapter.log(String, Level, String, Throwable) JDK14LoggerAdapter.java:582
org.slf4j.impl.JDK14LoggerAdapter.info(String) JDK14LoggerAdapter.java:277
org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish() LogUpdateProcessorFactory.java:187
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest, SolrQueryResponse) ContentStreamHandlerBase.java:78
org.apache.solr.handler.RequestHandlerBase.handleRequest(SolrQueryRequest, SolrQueryResponse) RequestHandlerBase.java:173
org.apache.solr.core.SolrCore.execute(SolrRequestHandler, SolrQueryRequest, SolrQueryResponse) SolrCore.java:2477
org.apache.solr.client.solrj.embedded.EmbeddedSolrServer.request(SolrRequest, String) EmbeddedSolrServer.java:179
org.apache.solr.client.solrj.SolrRequest.process(SolrClient, String) SolrRequest.java:160
org.apache.solr.client.solrj.SolrClient.add(String, SolrInputDocument, int) SolrClient.java:173
org.apache.solr.client.solrj.SolrClient.add(SolrInputDocument, int) SolrClient.java:190
net.yacy.cora.federate.solr.connector.SolrServerConnector.add(SolrInputDocument) SolrServerConnector.java:213
net.yacy.cora.federate.solr.connector.MirrorSolrConnector.add(SolrInputDocument) MirrorSolrConnector.java:204
net.yacy.search.index.Fulltext.putDocument(SolrInputDocument) Fulltext.java:349
net.yacy.search.index.Segment.putDocument(SolrInputDocument) Segment.java:555
net.yacy.search.index.Segment.storeDocument(DigestURL, DigestURL, Map, CrawlProfile, ResponseHeader, Document, Condenser, SearchEvent, String, boolean, String, String) Segment.java:616
net.yacy.search.Switchboard.storeDocumentIndex(Response, Map, Document, Condenser, SearchEvent, String) Switchboard.java:3252
net.yacy.search.Switchboard.storeDocumentIndex(IndexingQueueEntry) Switchboard.java:3186
sun.reflect.GeneratedMethodAccessor13.invoke(Object, Object[])
sun.reflect.DelegatingMethodAccessorImpl.invoke(Object, Object[]) DelegatingMethodAccessorImpl.java:43
java.lang.reflect.Method.invoke(Object, Object[]) Method.java:498
net.yacy.kelondro.workflow.InstantBlockingThread.job(WorkflowJob) InstantBlockingThread.java:101
net.yacy.kelondro.workflow.AbstractBlockingThread.run() AbstractBlockingThread.java:82
java.util.concurrent.Executors$RunnableAdapter.call() Executors.java:511
java.util.concurrent.FutureTask.run() FutureTask.java:266
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) ThreadPoolExecutor.java:1149
java.util.concurrent.ThreadPoolExecutor$Worker.run() ThreadPoolExecutor.java:624
java.lang.Thread.run() Thread.java:748



Stacks at 2017-10-09 12:04:32 PM. Uptime is 53m 29s 0ms.

condenseDocument_pool-1-thread-80 State: BLOCKED CPU usage on sample: 3ms
net.yacy.cora.storage.SimpleARC.get(Object) SimpleARC.java:143
net.yacy.cora.storage.ConcurrentARC.get(Object) ConcurrentARC.java:140
net.yacy.cora.order.Digest.encodeMD5Raw(String) Digest.java:122
net.yacy.cora.document.id.DigestURL.urlHashComputation() DigestURL.java:280
net.yacy.cora.document.id.DigestURL.hash() DigestURL.java:206
net.yacy.cora.document.id.DigestURL.hashCode() DigestURL.java:181
java.util.HashMap.hash(Object) HashMap.java:338
java.util.HashMap.put(Object, Object) HashMap.java:611
net.yacy.document.Document.resortLinks() Document.java:593
net.yacy.document.Document.getImages() Document.java:531
net.yacy.document.Condenser.<init>(Document, VocabularyScraper, boolean, boolean, WordCache, boolean, boolean, int) Condenser.java:83
net.yacy.search.Switchboard.condenseDocument(IndexingQueueEntry) Switchboard.java:3145
sun.reflect.GeneratedMethodAccessor6.invoke(Object, Object[])
sun.reflect.DelegatingMethodAccessorImpl.invoke(Object, Object[]) DelegatingMethodAccessorImpl.java:43
java.lang.reflect.Method.invoke(Object, Object[]) Method.java:498
net.yacy.kelondro.workflow.InstantBlockingThread.job(WorkflowJob) InstantBlockingThread.java:101
net.yacy.kelondro.workflow.AbstractBlockingThread.run() AbstractBlockingThread.java:82
java.util.concurrent.Executors$RunnableAdapter.call() Executors.java:511
java.util.concurrent.FutureTask.run() FutureTask.java:266
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) ThreadPoolExecutor.java:1149
java.util.concurrent.ThreadPoolExecutor$Worker.run() ThreadPoolExecutor.java:624
java.lang.Thread.run() Thread.java:748


Potential deadlock: frozen threads found

It seems that the following threads have not changed their stack for more than 10 seconds.
These threads are possibly (but not necessarily!) in a deadlock or hung.

commitScheduler-15-thread-1 <--- Frozen for at least 1m 1s
sun.nio.ch.FileDispatcherImpl.force0(FileDescriptor, boolean) FileDispatcherImpl.java (native)
sun.nio.ch.FileDispatcherImpl.force(FileDescriptor, boolean) FileDispatcherImpl.java:76
sun.nio.ch.FileChannelImpl.force(boolean) FileChannelImpl.java:388
org.apache.lucene.util.IOUtils.fsync(Path, boolean) IOUtils.java:472
org.apache.lucene.store.FSDirectory.fsync(String) FSDirectory.java:327
org.apache.lucene.store.FSDirectory.sync(Collection) FSDirectory.java:285
org.apache.lucene.store.LockValidatingDirectoryWrapper.sync(Collection) LockValidatingDirectoryWrapper.java:68
org.apache.lucene.index.IndexWriter.startCommit(SegmentInfos) IndexWriter.java:4724
org.apache.lucene.index.IndexWriter.prepareCommitInternal(boolean[]) IndexWriter.java:3085
org.apache.lucene.index.IndexWriter.commitInternal(MergePolicy) IndexWriter.java:3244
org.apache.lucene.index.IndexWriter.commit() IndexWriter.java:3207
org.apache.solr.update.DirectUpdateHandler2.commit(CommitUpdateCommand) DirectUpdateHandler2.java:644
org.apache.solr.update.CommitTracker.run() CommitTracker.java:217
java.util.concurrent.Executors$RunnableAdapter.call() Executors.java:511
java.util.concurrent.FutureTask.run() FutureTask.java:266
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor$ScheduledFutureTask) ScheduledThreadPoolExecutor.java:180
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run() ScheduledThreadPoolExecutor.java:293
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) ThreadPoolExecutor.java:1149
java.util.concurrent.ThreadPoolExecutor$Worker.run() ThreadPoolExecutor.java:624
java.lang.Thread.run() Thread.java:748



Stacks at 2017-10-09 12:06:19 PM. Uptime is 55m 16s 500ms.

BusyThread CrawlQueues.coreCrawlJob [DAEMON] State: BLOCKED CPU usage on sample: 52ms
net.yacy.kelondro.blob.MapHeap.containsKey(Object) MapHeap.java:242
net.yacy.crawler.data.Cache.has(byte[]) Cache.java:307
net.yacy.crawler.data.Latency.getDomainSleepTime(RobotsTxt, CrawlProfile, DigestURL) Latency.java:284
net.yacy.crawler.HostQueue.pop(boolean, CrawlSwitchboard, RobotsTxt) HostQueue.java:455
net.yacy.crawler.HostBalancer.pop(boolean, CrawlSwitchboard, RobotsTxt) HostBalancer.java:420
net.yacy.crawler.data.NoticedURL.pop(Balancer, boolean, CrawlSwitchboard, RobotsTxt) NoticedURL.java:332
net.yacy.crawler.data.NoticedURL.pop(NoticedURL$StackType, boolean, CrawlSwitchboard, RobotsTxt) NoticedURL.java:282
net.yacy.crawler.data.CrawlQueues.coreCrawlJob() CrawlQueues.java:326
net.yacy.search.Switchboard$9.jobImpl() Switchboard.java:1141
net.yacy.kelondro.workflow.InstantBusyThread.job() InstantBusyThread.java:64
net.yacy.kelondro.workflow.AbstractBusyThread.run() AbstractBusyThread.java:215



Stacks at 2017-10-09 12:06:40 PM. Uptime is 55m 37s 500ms.

BusyThread CrawlQueues.coreCrawlJob [DAEMON] State: BLOCKED CPU usage on sample: 43ms
net.yacy.kelondro.blob.MapHeap.containsKey(Object) MapHeap.java:242
net.yacy.crawler.data.Cache.has(byte[]) Cache.java:307
net.yacy.crawler.data.Latency.getDomainSleepTime(RobotsTxt, CrawlProfile, DigestURL) Latency.java:284
net.yacy.crawler.HostQueue.pop(boolean, CrawlSwitchboard, RobotsTxt) HostQueue.java:455
net.yacy.crawler.HostBalancer.pop(boolean, CrawlSwitchboard, RobotsTxt) HostBalancer.java:420
net.yacy.crawler.data.NoticedURL.pop(Balancer, boolean, CrawlSwitchboard, RobotsTxt) NoticedURL.java:332
net.yacy.crawler.data.NoticedURL.pop(NoticedURL$StackType, boolean, CrawlSwitchboard, RobotsTxt) NoticedURL.java:282
net.yacy.crawler.data.CrawlQueues.coreCrawlJob() CrawlQueues.java:326
net.yacy.search.Switchboard$9.jobImpl() Switchboard.java:1141
net.yacy.kelondro.workflow.InstantBusyThread.job() InstantBusyThread.java:64
net.yacy.kelondro.workflow.AbstractBusyThread.run() AbstractBusyThread.java:215



Stacks at 2017-10-09 12:06:46 PM. Uptime is 55m 42s 700ms.

BusyThread CrawlQueues.coreCrawlJob [DAEMON] State: BLOCKED CPU usage on sample: 14ms
net.yacy.kelondro.blob.MapHeap.containsKey(Object) MapHeap.java:242
net.yacy.crawler.data.Cache.has(byte[]) Cache.java:307
net.yacy.crawler.data.Latency.getDomainSleepTime(RobotsTxt, CrawlProfile, DigestURL) Latency.java:284
net.yacy.crawler.HostQueue.pop(boolean, CrawlSwitchboard, RobotsTxt) HostQueue.java:455
net.yacy.crawler.HostBalancer.pop(boolean, CrawlSwitchboard, RobotsTxt) HostBalancer.java:420
net.yacy.crawler.data.NoticedURL.pop(Balancer, boolean, CrawlSwitchboard, RobotsTxt) NoticedURL.java:332
net.yacy.crawler.data.NoticedURL.pop(NoticedURL$StackType, boolean, CrawlSwitchboard, RobotsTxt) NoticedURL.java:282
net.yacy.crawler.data.CrawlQueues.coreCrawlJob() CrawlQueues.java:326
net.yacy.search.Switchboard$9.jobImpl() Switchboard.java:1141
net.yacy.kelondro.workflow.InstantBusyThread.job() InstantBusyThread.java:64
net.yacy.kelondro.workflow.AbstractBusyThread.run() AbstractBusyThread.java:215

Crawler slow...


Stacks at 2017-10-09 12:06:54 PM. Uptime is 55m 51s 400ms.

BusyThread CrawlQueues.coreCrawlJob [DAEMON] State: BLOCKED CPU usage on sample: 24ms
net.yacy.crawler.HostBalancer.pop(boolean, CrawlSwitchboard, RobotsTxt) HostBalancer.java:439
net.yacy.crawler.data.NoticedURL.pop(Balancer, boolean, CrawlSwitchboard, RobotsTxt) NoticedURL.java:332
net.yacy.crawler.data.NoticedURL.pop(NoticedURL$StackType, boolean, CrawlSwitchboard, RobotsTxt) NoticedURL.java:282
net.yacy.crawler.data.CrawlQueues.coreCrawlJob() CrawlQueues.java:326
net.yacy.search.Switchboard$9.jobImpl() Switchboard.java:1141
net.yacy.kelondro.workflow.InstantBusyThread.job() InstantBusyThread.java:64
net.yacy.kelondro.workflow.AbstractBusyThread.run() AbstractBusyThread.java:215