YaCy-Bugtracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0000640YaCy[All Projects] Generalpublic2016-02-28 06:502016-07-07 01:18
ReporterBuBu 
Assigned ToBuBu 
PrioritynormalSeveritymajorReproducibilityrandom
StatusresolvedResolutionfixed 
ETAnone 
PlatformOSwinOS Version
Product Version 
Target VersionFixed in Version 
Summary0000640: crawler error loop crawling local hard drive in Intranet mode
DescriptionCrawl queue at one point starts showing error and looping with same same error.

in YaCy Intranet Mode

reason in kelendro.table.removeOne()
lr.getPrimaryKeyBytes() return null and this.table is null

with assertion exception is thrown, without it's even worse
Steps To Reproduceno certain way to reproduce but happens frequently
on different instances (crawling different drives)
Additional Informationlog snippet

E 2016/02/28 06:28:01 CRAWLER LOCALCRAWL[0, 1422, 0, 0]: CANNOT FETCH ENTRY: file.size() = 0
    at net.yacy.crawler.data.NoticedURL.pop(NoticedURL.java:332)
java.io.IOException: file.size() = 0
    at net.yacy.crawler.data.NoticedURL.pop(NoticedURL.java:282)
    at net.yacy.crawler.HostBalancer.pop(HostBalancer.java:450)
    at net.yacy.crawler.data.NoticedURL.pop(NoticedURL.java:332)
    at net.yacy.crawler.data.CrawlQueues.coreCrawlJob(CrawlQueues.java:315)
    at net.yacy.crawler.data.NoticedURL.pop(NoticedURL.java:282)
    at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
    at net.yacy.crawler.data.CrawlQueues.coreCrawlJob(CrawlQueues.java:315)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
    at net.yacy.kelondro.workflow.InstantBusyThread.job(InstantBusyThread.java:105)
    at net.yacy.kelondro.workflow.AbstractBusyThread.run(AbstractBusyThread.java:215)
W 2016/02/28 06:28:01 ConcurrentLog java.lang.AssertionError: file.size() = 0
java.lang.AssertionError: file.size() = 0
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at net.yacy.kelondro.table.Table.removeOne(Table.java:832)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at net.yacy.kelondro.workflow.InstantBusyThread.job(InstantBusyThread.java:105)
    at net.yacy.kelondro.index.BufferedObjectIndex.removeOne(BufferedObjectIndex.java:268)
    at net.yacy.kelondro.workflow.AbstractBusyThread.run(AbstractBusyThread.java:215)
    at net.yacy.crawler.HostQueue.pop(HostQueue.java:424)
W 2016/02/28 06:28:01 ConcurrentLog java.lang.AssertionError: file.size() = 0
    at net.yacy.crawler.HostBalancer.pop(HostBalancer.java:418)
java.lang.AssertionError: file.size() = 0
    at net.yacy.crawler.data.NoticedURL.pop(NoticedURL.java:332)
    at net.yacy.kelondro.table.Table.removeOne(Table.java:832)
    at net.yacy.crawler.data.NoticedURL.pop(NoticedURL.java:282)
    at net.yacy.kelondro.index.BufferedObjectIndex.removeOne(BufferedObjectIndex.java:268)
    at net.yacy.crawler.data.CrawlQueues.coreCrawlJob( at net.yacy.crawler.HostQueue.pop(HostQueue.java:424)
CrawlQueues.java:315)
    at net.yacy.crawler.HostBalancer.pop(HostBalancer.java:418)
    at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
    at net.yacy.crawler.data.NoticedURL.pop(NoticedURL.java:332)
    at net.yacy.crawler.data.NoticedURL.pop(NoticedURL.java:282)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at net.yacy.crawler.data.CrawlQueues.coreCrawlJob(CrawlQueues.java:315)
    at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at net.yacy.kelondro.workflow.InstantBusyThread.job(InstantBusyThread.java:105)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at net.yacy.kelondro.workflow.AbstractBusyThread.run(AbstractBusyThread.java:215)
    at net.yacy.kelondro.workflow.InstantBusyThread.job(InstantBusyThread.java:105)
E 2016/02/28 06:28:01 CRAWLER LOCALCRAWL[0, 1422, 0, 0]: CANNOT FETCH ENTRY: file.size() = 0
    at net.yacy.kelondro.workflow.AbstractBusyThread.run(AbstractBusyThread.java:215)
java.io.IOException: file.size() = 0
    at net.yacy.crawler.HostBalancer.pop(HostBalancer.java:450)
    at net.yacy.crawler.data.NoticedURL.pop(NoticedURL.java:332)
    at net.yacy.crawler.data.NoticedURL.pop(NoticedURL.java:282)
    at net.yacy.crawler.data.CrawlQueues.coreCrawlJob(CrawlQueues.java:315)
    at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at net.yacy.kelondro.workflow.InstantBusyThread.job(InstantBusyThread.java:105)
    at net.yacy.kelondro.workflow.AbstractBusyThread.run(AbstractBusyThread.java:215)
W 2016/02/28 06:28:01 ConcurrentLog java.lang.AssertionError: file.size() = 0
java.lang.AssertionError: file.size() = 0
    at net.yacy.kelondro.table.Table.removeOne(Table.java:832)
E 2016/02/28 06:28:01 CRAWLER LOCALCRAWL[0, 1422, 0, 0]: CANNOT FETCH ENTRY: file.size() = 0
TagsNo tags attached.
Attached Files

- Relationships
has duplicate 0000668resolvedBuBu Crawl Queue NPE in Intranet Mode, crawling file:// 

-  Notes
(0001222)
BuBu (developer)
2016-02-28 06:53

Log snippet w/o assertion

E 2016/02/27 00:58:08 CRAWLER LOCALCRAWL[0, 837, 0, 0]: CANNOT FETCH ENTRY: null
java.io.IOException
    at net.yacy.crawler.HostBalancer.pop(HostBalancer.java:450)
    at net.yacy.crawler.data.NoticedURL.pop(NoticedURL.java:332)
    at net.yacy.crawler.data.NoticedURL.pop(NoticedURL.java:282)
    at net.yacy.crawler.data.CrawlQueues.coreCrawlJob(CrawlQueues.java:316)
    at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at net.yacy.kelondro.workflow.InstantBusyThread.job(InstantBusyThread.java:105)
    at net.yacy.kelondro.workflow.AbstractBusyThread.run(AbstractBusyThread.java:215)
I 2016/02/27 00:58:08 HostBalancer (re-)initialized the round-robin queue; 2 hosts.
W 2016/02/27 00:58:08 ConcurrentLog java.lang.NullPointerException
java.lang.NullPointerException
E 2016/02/27 00:58:08 CRAWLER LOCALCRAWL[0, 837, 0, 0]: CANNOT FETCH ENTRY: null
java.io.IOException
    at net.yacy.crawler.HostBalancer.pop(HostBalancer.java:450)
    at net.yacy.crawler.data.NoticedURL.pop(NoticedURL.java:332)
    at net.yacy.crawler.data.NoticedURL.pop(NoticedURL.java:282)
    at net.yacy.crawler.data.CrawlQueues.coreCrawlJob(CrawlQueues.java:316)
    at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at net.yacy.kelondro.workflow.InstantBusyThread.job(InstantBusyThread.java:105)
    at net.yacy.kelondro.workflow.AbstractBusyThread.run(AbstractBusyThread.java:215)
I 2016/02/27 00:58:08 HostBalancer (re-)initialized the round-robin queue; 2 hosts.
(0001258)
BuBu (developer)
2016-07-07 01:18

v1.91/9006
https://github.com/yacy/yacy_search_server/commit/22db449f2ab8c6b0d596b6eee57fd9bc13136a2a [^]

- Issue History
Date Modified Username Field Change
2016-02-28 06:50 BuBu New Issue
2016-02-28 06:51 BuBu Description Updated View Revisions
2016-02-28 06:53 BuBu Note Added: 0001222
2016-02-28 06:58 BuBu Summary crawler error loop crawling local hard drive => crawler error loop crawling local hard drive in Intranet mode
2016-02-28 07:01 BuBu Steps to Reproduce Updated View Revisions
2016-07-02 00:58 BuBu Relationship added has duplicate 0000668
2016-07-07 01:18 BuBu Note Added: 0001258
2016-07-07 01:18 BuBu Status new => resolved
2016-07-07 01:18 BuBu Resolution open => fixed
2016-07-07 01:18 BuBu Assigned To => BuBu


Copyright © 2000 - 2019 MantisBT Team
Powered by Mantis Bugtracker