YaCy-Bugtracker - YaCy
View Issue Details
0000640YaCy[All Projects] Generalpublic2016-02-28 06:502016-07-07 01:18
BuBu 
BuBu 
normalmajorrandom
resolvedfixed 
none 
win
 
 
0000640: crawler error loop crawling local hard drive in Intranet mode
Crawl queue at one point starts showing error and looping with same same error.

in YaCy Intranet Mode

reason in kelendro.table.removeOne()
lr.getPrimaryKeyBytes() return null and this.table is null

with assertion exception is thrown, without it's even worse
no certain way to reproduce but happens frequently
on different instances (crawling different drives)
log snippet

E 2016/02/28 06:28:01 CRAWLER LOCALCRAWL[0, 1422, 0, 0]: CANNOT FETCH ENTRY: file.size() = 0
    at net.yacy.crawler.data.NoticedURL.pop(NoticedURL.java:332)
java.io.IOException: file.size() = 0
    at net.yacy.crawler.data.NoticedURL.pop(NoticedURL.java:282)
    at net.yacy.crawler.HostBalancer.pop(HostBalancer.java:450)
    at net.yacy.crawler.data.NoticedURL.pop(NoticedURL.java:332)
    at net.yacy.crawler.data.CrawlQueues.coreCrawlJob(CrawlQueues.java:315)
    at net.yacy.crawler.data.NoticedURL.pop(NoticedURL.java:282)
    at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
    at net.yacy.crawler.data.CrawlQueues.coreCrawlJob(CrawlQueues.java:315)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
    at net.yacy.kelondro.workflow.InstantBusyThread.job(InstantBusyThread.java:105)
    at net.yacy.kelondro.workflow.AbstractBusyThread.run(AbstractBusyThread.java:215)
W 2016/02/28 06:28:01 ConcurrentLog java.lang.AssertionError: file.size() = 0
java.lang.AssertionError: file.size() = 0
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at net.yacy.kelondro.table.Table.removeOne(Table.java:832)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at net.yacy.kelondro.workflow.InstantBusyThread.job(InstantBusyThread.java:105)
    at net.yacy.kelondro.index.BufferedObjectIndex.removeOne(BufferedObjectIndex.java:268)
    at net.yacy.kelondro.workflow.AbstractBusyThread.run(AbstractBusyThread.java:215)
    at net.yacy.crawler.HostQueue.pop(HostQueue.java:424)
W 2016/02/28 06:28:01 ConcurrentLog java.lang.AssertionError: file.size() = 0
    at net.yacy.crawler.HostBalancer.pop(HostBalancer.java:418)
java.lang.AssertionError: file.size() = 0
    at net.yacy.crawler.data.NoticedURL.pop(NoticedURL.java:332)
    at net.yacy.kelondro.table.Table.removeOne(Table.java:832)
    at net.yacy.crawler.data.NoticedURL.pop(NoticedURL.java:282)
    at net.yacy.kelondro.index.BufferedObjectIndex.removeOne(BufferedObjectIndex.java:268)
    at net.yacy.crawler.data.CrawlQueues.coreCrawlJob( at net.yacy.crawler.HostQueue.pop(HostQueue.java:424)
CrawlQueues.java:315)
    at net.yacy.crawler.HostBalancer.pop(HostBalancer.java:418)
    at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
    at net.yacy.crawler.data.NoticedURL.pop(NoticedURL.java:332)
    at net.yacy.crawler.data.NoticedURL.pop(NoticedURL.java:282)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at net.yacy.crawler.data.CrawlQueues.coreCrawlJob(CrawlQueues.java:315)
    at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at net.yacy.kelondro.workflow.InstantBusyThread.job(InstantBusyThread.java:105)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at net.yacy.kelondro.workflow.AbstractBusyThread.run(AbstractBusyThread.java:215)
    at net.yacy.kelondro.workflow.InstantBusyThread.job(InstantBusyThread.java:105)
E 2016/02/28 06:28:01 CRAWLER LOCALCRAWL[0, 1422, 0, 0]: CANNOT FETCH ENTRY: file.size() = 0
    at net.yacy.kelondro.workflow.AbstractBusyThread.run(AbstractBusyThread.java:215)
java.io.IOException: file.size() = 0
    at net.yacy.crawler.HostBalancer.pop(HostBalancer.java:450)
    at net.yacy.crawler.data.NoticedURL.pop(NoticedURL.java:332)
    at net.yacy.crawler.data.NoticedURL.pop(NoticedURL.java:282)
    at net.yacy.crawler.data.CrawlQueues.coreCrawlJob(CrawlQueues.java:315)
    at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at net.yacy.kelondro.workflow.InstantBusyThread.job(InstantBusyThread.java:105)
    at net.yacy.kelondro.workflow.AbstractBusyThread.run(AbstractBusyThread.java:215)
W 2016/02/28 06:28:01 ConcurrentLog java.lang.AssertionError: file.size() = 0
java.lang.AssertionError: file.size() = 0
    at net.yacy.kelondro.table.Table.removeOne(Table.java:832)
E 2016/02/28 06:28:01 CRAWLER LOCALCRAWL[0, 1422, 0, 0]: CANNOT FETCH ENTRY: file.size() = 0
No tags attached.
has duplicate 0000668resolved BuBu Crawl Queue NPE in Intranet Mode, crawling file:// 
Issue History
2016-02-28 06:50BuBuNew Issue
2016-02-28 06:51BuBuDescription Updatedbug_revision_view_page.php?rev_id=352#r352
2016-02-28 06:53BuBuNote Added: 0001222
2016-02-28 06:58BuBuSummarycrawler error loop crawling local hard drive => crawler error loop crawling local hard drive in Intranet mode
2016-02-28 07:01BuBuSteps to Reproduce Updatedbug_revision_view_page.php?rev_id=354#r354
2016-07-02 00:58BuBuRelationship addedhas duplicate 0000668
2016-07-07 01:18BuBuNote Added: 0001258
2016-07-07 01:18BuBuStatusnew => resolved
2016-07-07 01:18BuBuResolutionopen => fixed
2016-07-07 01:18BuBuAssigned To => BuBu

Notes
(0001222)
BuBu   
2016-02-28 06:53   
Log snippet w/o assertion

E 2016/02/27 00:58:08 CRAWLER LOCALCRAWL[0, 837, 0, 0]: CANNOT FETCH ENTRY: null
java.io.IOException
    at net.yacy.crawler.HostBalancer.pop(HostBalancer.java:450)
    at net.yacy.crawler.data.NoticedURL.pop(NoticedURL.java:332)
    at net.yacy.crawler.data.NoticedURL.pop(NoticedURL.java:282)
    at net.yacy.crawler.data.CrawlQueues.coreCrawlJob(CrawlQueues.java:316)
    at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at net.yacy.kelondro.workflow.InstantBusyThread.job(InstantBusyThread.java:105)
    at net.yacy.kelondro.workflow.AbstractBusyThread.run(AbstractBusyThread.java:215)
I 2016/02/27 00:58:08 HostBalancer (re-)initialized the round-robin queue; 2 hosts.
W 2016/02/27 00:58:08 ConcurrentLog java.lang.NullPointerException
java.lang.NullPointerException
E 2016/02/27 00:58:08 CRAWLER LOCALCRAWL[0, 837, 0, 0]: CANNOT FETCH ENTRY: null
java.io.IOException
    at net.yacy.crawler.HostBalancer.pop(HostBalancer.java:450)
    at net.yacy.crawler.data.NoticedURL.pop(NoticedURL.java:332)
    at net.yacy.crawler.data.NoticedURL.pop(NoticedURL.java:282)
    at net.yacy.crawler.data.CrawlQueues.coreCrawlJob(CrawlQueues.java:316)
    at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at net.yacy.kelondro.workflow.InstantBusyThread.job(InstantBusyThread.java:105)
    at net.yacy.kelondro.workflow.AbstractBusyThread.run(AbstractBusyThread.java:215)
I 2016/02/27 00:58:08 HostBalancer (re-)initialized the round-robin queue; 2 hosts.
(0001258)
BuBu   
2016-07-07 01:18   
v1.91/9006
https://github.com/yacy/yacy_search_server/commit/22db449f2ab8c6b0d596b6eee57fd9bc13136a2a [^]