YaCy-Bugtracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0000479YaCy[All Projects] Generalpublic2014-09-27 19:182015-06-07 16:24
ReporterLA_FORGE 
Assigned ToOrbiter 
PrioritynormalSeveritycrashReproducibilityalways
StatusassignedResolutionreopened 
ETAnone 
PlatformOSOS Version
Product VersionYaCy 1.6 
Target VersionFixed in Version 
Summary0000479: YaCy Webinterface stops working during Highspeed-Crawl
DescriptionAfter some hours of highspeed-crawling the Webinterface isn't reachable anymore.
Steps To ReproduceStart a crawl process with a crawl-depth of 6. Let it run a few hours. Then try to reach e.g.
 http://localhost:8090/Crawler_p.html?pause=localcrawler [^]
Additional InformationI'm able to exclude a memory problem here (Not enough RAM) because the Logfile doesn't show 'Out of memory error'
TagsNo tags attached.
Attached Files

- Relationships

-  Notes
(0000932)
LA_FORGE (reporter)
2014-09-28 11:31

These are the log entries. The Web interface shows no response.

I 2014/09/28 10:35:37 SWITCHBOARD *Indexed 1199 words in URL http://landresources.montana.edu/nm/Modules/NM2.pdf [^] [D6RO2dPTL1IQ]
        Description: NM2layoutB.p65
        MimeType: application/pdf | Charset: UTF-8 | Size: 32520 bytes |
        LinkStorageTime: 1 ms | indexStorageTime: 5 ms
I 2014/09/28 10:35:37 HeapReader close HeapFile citation.index.20140928083437811.blob; trace: net.yacy.kelondro.blob.HeapModifier.close(HeapModifier.java:82) -> net.yacy.kelondro.blob.ArrayStack.unmountBLOB(ArrayStack.java:276) -> net.yacy.kelondro.blob.ArrayStack.unmountBestMatch(ArrayStack.java:323) -> net.yacy.kelondro.rwi.ReferenceContainerArray.shrinkBestSmallFiles(ReferenceContainerArray.java:403) -> net.yacy.kelondro.rwi.IndexCell.shrink(IndexCell.java:191) -> net.yacy.kelondro.rwi.IndexCell.access$900(IndexCell.java:65) -> net.yacy.kelondro.rwi.IndexCell$FlushThread.flushBuffer(IndexCell.java:172) -> net.yacy.kelondro.rwi.IndexCell$FlushThread.run(IndexCell.java:122)
I 2014/09/28 10:35:37 HeapReader close HeapFile citation.index.20140928083319659.blob; trace: net.yacy.kelondro.blob.HeapModifier.close(HeapModifier.java:82) -> net.yacy.kelondro.blob.ArrayStack.unmountBLOB(ArrayStack.java:276) -> net.yacy.kelondro.blob.ArrayStack.unmountBestMatch(ArrayStack.java:324) -> net.yacy.kelondro.rwi.ReferenceContainerArray.shrinkBestSmallFiles(ReferenceContainerArray.java:403) -> net.yacy.kelondro.rwi.IndexCell.shrink(IndexCell.java:191) -> net.yacy.kelondro.rwi.IndexCell.access$900(IndexCell.java:65) -> net.yacy.kelondro.rwi.IndexCell$FlushThread.flushBuffer(IndexCell.java:172) -> net.yacy.kelondro.rwi.IndexCell$FlushThread.run(IndexCell.java:122)
I 2014/09/28 10:35:37 RICELL-shrink1 unmountBestMatch(2.0, 67108864)
I 2014/09/28 10:35:37 IODispatcher appended merge job of files citation.index.20140928083319659.blob, citation.index.20140928083437811.blob to citation.index.20140928083537816.blob
I 2014/09/28 10:35:37 BLOBArray merging citation.index.20140928083319659.blob with citation.index.20140928083437811.blob
I 2014/09/28 10:35:37 HeapReader close HeapFile text.index.20140928083516826.blob; trace: net.yacy.kelondro.blob.HeapModifier.close(HeapModifier.java:82) -> net.yacy.kelondro.blob.ArrayStack.unmountBLOB(ArrayStack.java:276) -> net.yacy.kelondro.blob.ArrayStack.unmountBestMatch(ArrayStack.java:323) -> net.yacy.kelondro.rwi.ReferenceContainerArray.shrinkBestSmallFiles(ReferenceContainerArray.java:403) -> net.yacy.kelondro.rwi.IndexCell.shrink(IndexCell.java:191) -> net.yacy.kelondro.rwi.IndexCell.access$900(IndexCell.java:65) -> net.yacy.kelondro.rwi.IndexCell$FlushThread.flushBuffer(IndexCell.java:172) -> net.yacy.kelondro.rwi.IndexCell$FlushThread.run(IndexCell.java:122)
I 2014/09/28 10:35:37 HeapReader close HeapFile text.index.20140928083443819.blob; trace: net.yacy.kelondro.blob.HeapModifier.close(HeapModifier.java:82) -> net.yacy.kelondro.blob.ArrayStack.unmountBLOB(ArrayStack.java:276) -> net.yacy.kelondro.blob.ArrayStack.unmountBestMatch(ArrayStack.java:324) -> net.yacy.kelondro.rwi.ReferenceContainerArray.shrinkBestSmallFiles(ReferenceContainerArray.java:403) -> net.yacy.kelondro.rwi.IndexCell.shrink(IndexCell.java:191) -> net.yacy.kelondro.rwi.IndexCell.access$900(IndexCell.java:65) -> net.yacy.kelondro.rwi.IndexCell$FlushThread.flushBuffer(IndexCell.java:172) -> net.yacy.kelondro.rwi.IndexCell$FlushThread.run(IndexCell.java:122)
I 2014/09/28 10:35:37 RICELL-shrink1 unmountBestMatch(2.0, 67108864)
I 2014/09/28 10:35:37 IODispatcher appended merge job of files text.index.20140928083443819.blob, text.index.20140928083516826.blob to text.index.20140928083537901.blob
I 2014/09/28 10:35:37 BLOBArray merging text.index.20140928083443819.blob with text.index.20140928083516826.blob
I 2014/09/28 10:35:37 HeapReader close HeapFile text.index.20140928083437814.blob; trace: net.yacy.kelondro.blob.HeapModifier.close(HeapModifier.java:82) -> net.yacy.kelondro.blob.ArrayStack.unmountBLOB(ArrayStack.java:276) -> net.yacy.kelondro.blob.ArrayStack.unmountBestMatch(ArrayStack.java:323) -> net.yacy.kelondro.rwi.ReferenceContainerArray.shrinkBestSmallFiles(ReferenceContainerArray.java:403) -> net.yacy.kelondro.rwi.IndexCell.shrink(IndexCell.java:191) -> net.yacy.kelondro.rwi.IndexCell.access$900(IndexCell.java:65) -> net.yacy.kelondro.rwi.IndexCell$FlushThread.flushBuffer(IndexCell.java:172) -> net.yacy.kelondro.rwi.IndexCell$FlushThread.run(IndexCell.java:122)
I 2014/09/28 10:35:37 HeapReader close HeapFile text.index.20140928083337809.blob; trace: net.yacy.kelondro.blob.HeapModifier.close(HeapModifier.java:82) -> net.yacy.kelondro.blob.ArrayStack.unmountBLOB(ArrayStack.java:276) -> net.yacy.kelondro.blob.ArrayStack.unmountBestMatch(ArrayStack.java:324) -> net.yacy.kelondro.rwi.ReferenceContainerArray.shrinkBestSmallFiles(ReferenceContainerArray.java:403) -> net.yacy.kelondro.rwi.IndexCell.shrink(IndexCell.java:191) -> net.yacy.kelondro.rwi.IndexCell.access$900(IndexCell.java:65) -> net.yacy.kelondro.rwi.IndexCell$FlushThread.flushBuffer(IndexCell.java:172) -> net.yacy.kelondro.rwi.IndexCell$FlushThread.run(IndexCell.java:122)
I 2014/09/28 10:35:37 RICELL-shrink1 unmountBestMatch(2.0, 67108864)
I 2014/09/28 10:35:37 IODispatcher appended merge job of files text.index.20140928083337809.blob, text.index.20140928083437814.blob to text.index.20140928083537918.blob
I 2014/09/28 10:35:38 HeapWriter wrote a dump for the 92210 index entries of citation.index.20140928083537816.blob in 55 milliseconds.
I 2014/09/28 10:35:38 HeapReader saturation of citation.index.20140928083537816.blob.igy_iBBdluEC.idx: keylength = 5, vallength = 3, size = 92210, maximum saving for index-compression = 0 MB, exact saving for value-compression = 0 MB
I 2014/09/28 10:35:38 HeapReader using a dump of the index of /home/dlw343/yacy/DATA/INDEX/freeworld/SEGMENTS/default/citation.index.20140928083537816.blob.
I 2014/09/28 10:35:38 BLOBArray merged citation.index.20140928083319659.blob with citation.index.20140928083437811.blob into /home/dlw343/yacy/DATA/INDEX/freeworld/SEGMENTS/default/citation.index.20140928083537816.blob
I 2014/09/28 10:35:38 HeapWriter wrote a dump for the 86977 index entries of text.index.20140928083537901.blob in 39 milliseconds.
I 2014/09/28 10:35:39 HeapReader saturation of text.index.20140928083537901.blob.TWR84pLcFB17.idx: keylength = 5, vallength = 3, size = 86977, maximum saving for index-compression = 0 MB, exact saving for value-compression = 0 MB
I 2014/09/28 10:35:39 HeapReader using a dump of the index of /home/dlw343/yacy/DATA/INDEX/freeworld/SEGMENTS/default/text.index.20140928083537901.blob.
I 2014/09/28 10:35:39 BLOBArray merged text.index.20140928083443819.blob with text.index.20140928083516826.blob into /home/dlw343/yacy/DATA/INDEX/freeworld/SEGMENTS/default/text.index.20140928083537901.blob
I 2014/09/28 10:35:39 BLOBArray merging text.index.20140928083337809.blob with text.index.20140928083437814.blob
I 2014/09/28 10:35:40 HeapWriter wrote a dump for the 131723 index entries of text.index.20140928083537918.blob in 59 milliseconds.
I 2014/09/28 10:35:40 HeapReader saturation of text.index.20140928083537918.blob.PWtvcyE264QV.idx: keylength = 5, vallength = 4, size = 131723, maximum saving for index-compression = 0 MB, exact saving for value-compression = 0 MB
I 2014/09/28 10:35:40 HeapReader using a dump of the index of /home/dlw343/yacy/DATA/INDEX/freeworld/SEGMENTS/default/text.index.20140928083537918.blob.
I 2014/09/28 10:35:40 BLOBArray merged text.index.20140928083337809.blob with text.index.20140928083437814.blob into /home/dlw343/yacy/DATA/INDEX/freeworld/SEGMENTS/default/text.index.20140928083537918.blob
I 2014/09/28 10:35:49 REJECTED http://www.dumontverlag.de/ [^] - cannot load: load error - java.io.IOException: Client can't execute: No route to host duration=15097
I 2014/09/28 10:36:38 HeapReader close HeapFile text.index.20140928083537901.blob; trace: net.yacy.kelondro.blob.HeapModifier.close(HeapModifier.java:82) -> net.yacy.kelondro.blob.ArrayStack.unmountBLOB(ArrayStack.java:276) -> net.yacy.kelondro.blob.ArrayStack.unmountSmallest(ArrayStack.java:343) -> net.yacy.kelondro.rwi.ReferenceContainerArray.shrinkAnySmallFiles(ReferenceContainerArray.java:411) -> net.yacy.kelondro.rwi.IndexCell.shrink(IndexCell.java:198) -> net.yacy.kelondro.rwi.IndexCell.access$900(IndexCell.java:65) -> net.yacy.kelondro.rwi.IndexCell$FlushThread.flushBuffer(IndexCell.java:172) -> net.yacy.kelondro.rwi.IndexCell$FlushThread.run(IndexCell.java:122)
I 2014/09/28 10:36:38 HeapReader close HeapFile text.index.20140928083537918.blob; trace: net.yacy.kelondro.blob.HeapModifier.close(HeapModifier.java:82) -> net.yacy.kelondro.blob.ArrayStack.unmountBLOB(ArrayStack.java:276) -> net.yacy.kelondro.blob.ArrayStack.unmountSmallest(ArrayStack.java:344) -> net.yacy.kelondro.rwi.ReferenceContainerArray.shrinkAnySmallFiles(ReferenceContainerArray.java:411) -> net.yacy.kelondro.rwi.IndexCell.shrink(IndexCell.java:198) -> net.yacy.kelondro.rwi.IndexCell.access$900(IndexCell.java:65) -> net.yacy.kelondro.rwi.IndexCell$FlushThread.flushBuffer(IndexCell.java:172) -> net.yacy.kelondro.rwi.IndexCell$FlushThread.run(IndexCell.java:122)
I 2014/09/28 10:36:38 RICELL-shrink2 unmountSmallest(67108864)
I 2014/09/28 10:36:38 IODispatcher appended merge job of files text.index.20140928083537901.blob, text.index.20140928083537918.blob to text.index.20140928083638022.blob
I 2014/09/28 10:36:38 BLOBArray merging text.index.20140928083537901.blob with text.index.20140928083537918.blob
I 2014/09/28 10:36:39 HeapWriter wrote a dump for the 175931 index entries of text.index.20140928083638022.blob in 37 milliseconds.
I 2014/09/28 10:36:39 HeapReader saturation of text.index.20140928083638022.blob.BZWMpngGw_EB.idx: keylength = 5, vallength = 4, size = 175931, maximum saving for index-compression = 0 MB, exact saving for value-compression = 0 MB
I 2014/09/28 10:36:39 HeapReader using a dump of the index of /home/dlw343/yacy/DATA/INDEX/freeworld/SEGMENTS/default/text.index.20140928083638022.blob.
I 2014/09/28 10:36:39 BLOBArray merged text.index.20140928083537901.blob with text.index.20140928083537918.blob into /home/dlw343/yacy/DATA/INDEX/freeworld/SEGMENTS/default/text.index.20140928083638022.blob
I 2014/09/28 10:39:41 YACY rulebasedUpdateInfo: not an automatic update selected
I 2014/09/28 10:39:41 RESOURCE OBSERVER resources ok
I 2014/09/28 10:44:38 IODispatcher appended dump job for file citation.index.20140928084438507.blob
I 2014/09/28 10:44:38 ReferenceContainerCache creating rwi heap dump 'citation.index.20140928084438507.blob', 33202 rwi's
I 2014/09/28 10:44:38 HeapWriter wrote a dump for the 33202 index entries of citation.index.20140928084438507.blob in 15 milliseconds.
I 2014/09/28 10:44:38 ReferenceContainerCache finished rwi heap dump: 33202 terms, 0 term/data relations in 207 milliseconds
I 2014/09/28 10:44:38 HeapReader saturation of citation.index.20140928084438507.blob.KgtPG5Hewdjo.idx: keylength = 4, vallength = 3, size = 33202, maximum saving for index-compression = 0 MB, exact saving for value-compression = 0 MB
I 2014/09/28 10:44:38 HeapReader using a dump of the index of /home/dlw343/yacy/DATA/INDEX/freeworld/SEGMENTS/default/citation.index.20140928084438507.blob.
I 2014/09/28 10:45:17 IODispatcher appended dump job for file text.index.20140928084517663.blob
I 2014/09/28 10:45:17 ReferenceContainerCache creating rwi heap dump 'text.index.20140928084517663.blob', 24221 rwi's
I 2014/09/28 10:45:17 HeapWriter wrote a dump for the 24221 index entries of text.index.20140928084517663.blob in 11 milliseconds.
I 2014/09/28 10:45:17 ReferenceContainerCache finished rwi heap dump: 24221 terms, 0 term/data relations in 185 milliseconds
I 2014/09/28 10:45:17 HeapReader saturation of text.index.20140928084517663.blob.o-mGWcMlZyJc.idx: keylength = 4, vallength = 3, size = 24221, maximum saving for index-compression = 0 MB, exact saving for value-compression = 0 MB
I 2014/09/28 10:45:17 HeapReader using a dump of the index of /home/dlw343/yacy/DATA/INDEX/freeworld/SEGMENTS/default/text.index.20140928084517663.blob.
I 2014/09/28 10:45:38 HeapReader close HeapFile citation.index.20140928084438507.blob; trace: net.yacy.kelondro.blob.HeapModifier.close(HeapModifier.java:82) -> net.yacy.kelondro.blob.ArrayStack.unmountBLOB(ArrayStack.java:276) -> net.yacy.kelondro.blob.ArrayStack.unmountSmallest(ArrayStack.java:343) -> net.yacy.kelondro.rwi.ReferenceContainerArray.shrinkAnySmallFiles(ReferenceContainerArray.java:411) -> net.yacy.kelondro.rwi.IndexCell.shrink(IndexCell.java:198) -> net.yacy.kelondro.rwi.IndexCell.access$900(IndexCell.java:65) -> net.yacy.kelondro.rwi.IndexCell$FlushThread.flushBuffer(IndexCell.java:172) -> net.yacy.kelondro.rwi.IndexCell$FlushThread.run(IndexCell.java:122)
I 2014/09/28 10:45:38 HeapReader close HeapFile citation.index.20140928083537816.blob; trace: net.yacy.kelondro.blob.HeapModifier.close(HeapModifier.java:82) -> net.yacy.kelondro.blob.ArrayStack.unmountBLOB(ArrayStack.java:276) -> net.yacy.kelondro.blob.ArrayStack.unmountSmallest(ArrayStack.java:344) -> net.yacy.kelondro.rwi.ReferenceContainerArray.shrinkAnySmallFiles(ReferenceContainerArray.java:411) -> net.yacy.kelondro.rwi.IndexCell.shrink(IndexCell.java:198) -> net.yacy.kelondro.rwi.IndexCell.access$900(IndexCell.java:65) -> net.yacy.kelondro.rwi.IndexCell$FlushThread.flushBuffer(IndexCell.java:172) -> net.yacy.kelondro.rwi.IndexCell$FlushThread.run(IndexCell.java:122)
I 2014/09/28 10:45:38 RICELL-shrink2 unmountSmallest(67108864)
I 2014/09/28 10:45:38 IODispatcher appended merge job of files citation.index.20140928084438507.blob, citation.index.20140928083537816.blob to citation.index.20140928084538611.blob
I 2014/09/28 10:45:38 BLOBArray merging citation.index.20140928084438507.blob with citation.index.20140928083537816.blob
I 2014/09/28 10:45:38 HeapReader close HeapFile text.index.20140928084517663.blob; trace: net.yacy.kelondro.blob.HeapModifier.close(HeapModifier.java:82) -> net.yacy.kelondro.blob.ArrayStack.unmountBLOB(ArrayStack.java:276) -> net.yacy.kelondro.blob.ArrayStack.unmountSmallest(ArrayStack.java:343) -> net.yacy.kelondro.rwi.ReferenceContainerArray.shrinkAnySmallFiles(ReferenceContainerArray.java:411) -> net.yacy.kelondro.rwi.IndexCell.shrink(IndexCell.java:198) -> net.yacy.kelondro.rwi.IndexCell.access$900(IndexCell.java:65) -> net.yacy.kelondro.rwi.IndexCell$FlushThread.flushBuffer(IndexCell.java:172) -> net.yacy.kelondro.rwi.IndexCell$FlushThread.run(IndexCell.java:122)
I 2014/09/28 10:45:38 HeapReader close HeapFile text.index.20140928083638022.blob; trace: net.yacy.kelondro.blob.HeapModifier.close(HeapModifier.java:82) -> net.yacy.kelondro.blob.ArrayStack.unmountBLOB(ArrayStack.java:276) -> net.yacy.kelondro.blob.ArrayStack.unmountSmallest(ArrayStack.java:344) -> net.yacy.kelondro.rwi.ReferenceContainerArray.shrinkAnySmallFiles(ReferenceContainerArray.java:411) -> net.yacy.kelondro.rwi.IndexCell.shrink(IndexCell.java:198) -> net.yacy.kelondro.rwi.IndexCell.access$900(IndexCell.java:65) -> net.yacy.kelondro.rwi.IndexCell$FlushThread.flushBuffer(IndexCell.java:172) -> net.yacy.kelondro.rwi.IndexCell$FlushThread.run(IndexCell.java:122)
I 2014/09/28 10:45:38 RICELL-shrink2 unmountSmallest(67108864)
I 2014/09/28 10:45:38 IODispatcher appended merge job of files text.index.20140928084517663.blob, text.index.20140928083638022.blob to text.index.20140928084538719.blob
I 2014/09/28 10:45:38 BLOBArray merging text.index.20140928084517663.blob with text.index.20140928083638022.blob
I 2014/09/28 10:45:39 HeapWriter wrote a dump for the 117767 index entries of citation.index.20140928084538611.blob in 55 milliseconds.
I 2014/09/28 10:45:39 HeapReader saturation of citation.index.20140928084538611.blob.Jj-QZnptIdGg.idx: keylength = 5, vallength = 3, size = 117767, maximum saving for index-compression = 0 MB, exact saving for value-compression = 0 MB
I 2014/09/28 10:45:39 HeapReader using a dump of the index of /home/dlw343/yacy/DATA/INDEX/freeworld/SEGMENTS/default/citation.index.20140928084538611.blob.
I 2014/09/28 10:45:39 BLOBArray merged citation.index.20140928084438507.blob with citation.index.20140928083537816.blob into /home/dlw343/yacy/DATA/INDEX/freeworld/SEGMENTS/default/citation.index.20140928084538611.blob
I 2014/09/28 10:45:40 HeapWriter wrote a dump for the 183078 index entries of text.index.20140928084538719.blob in 110 milliseconds.
I 2014/09/28 10:45:41 HeapReader saturation of text.index.20140928084538719.blob.MXQ0g_W1e2TS.idx: keylength = 5, vallength = 4, size = 183078, maximum saving for index-compression = 0 MB, exact saving for value-compression = 0 MB
I 2014/09/28 10:45:41 HeapReader using a dump of the index of /home/dlw343/yacy/DATA/INDEX/freeworld/SEGMENTS/default/text.index.20140928084538719.blob.
I 2014/09/28 10:45:41 BLOBArray merged text.index.20140928084517663.blob with text.index.20140928083638022.blob into /home/dlw343/yacy/DATA/INDEX/freeworld/SEGMENTS/default/text.index.20140928084538719.blob
(0000933)
LA_FORGE (reporter)
2014-09-28 16:33

After a reboot I gave it another try but it still freezes:

I 2014/09/28 15:55:55 SWITCHBOARD *Indexed 636 words in URL http://www.wiki-aventurica.de/wiki/Siebenstreich_(Waffe) [^] [mN3DcIW6RiQC]
        Description: Siebenstreich (Waffe) รข Wiki Aventurica, das DSA-Fanprojekt
        MimeType: text/html | Charset: UTF-8 | Size: 8471 bytes |
        LinkStorageTime: 36 ms | indexStorageTime: 0 ms
I 2014/09/28 15:55:56 SWITCHBOARD CRAWL: ADDED 96 LINKS FROM http://www.didacta-bildungsklick.tv/sonderthemen/, [^] STACKING TIME = 2, PARSING TIME = 5
I 2014/09/28 15:55:56 SWITCHBOARD CRAWL: ADDED 160 LINKS FROM http://hofrat.rsp-blogs.de/2006/06/, [^] STACKING TIME = 3, PARSING TIME = 7
I 2014/09/28 15:56:15 YACY rulebasedUpdateInfo: not an automatic update selected
I 2014/09/28 15:56:15 RESOURCE OBSERVER resources ok
I 2014/09/28 15:56:25 HeapReader close HeapFile citation.index.20140928135552447.blob; trace: net.yacy.kelondro.blob.HeapModifier.close(HeapModifier.java:82) -> net.yacy.kelondro.blob.ArrayStack.unmountBLOB(ArrayStack.java:276) -> net.yacy.kelondro.blob.ArrayStack.unmountSmallest(ArrayStack.java:343) -> net.yacy.kelondro.rwi.ReferenceContainerArray.shrinkAnySmallFiles(ReferenceContainerArray.java:411) -> net.yacy.kelondro.rwi.IndexCell.shrink(IndexCell.java:198) -> net.yacy.kelondro.rwi.IndexCell.access$900(IndexCell.java:65) -> net.yacy.kelondro.rwi.IndexCell$FlushThread.flushBuffer(IndexCell.java:172) -> net.yacy.kelondro.rwi.IndexCell$FlushThread.run(IndexCell.java:122)
I 2014/09/28 15:56:25 HeapReader close HeapFile citation.index.20140928135325343.blob; trace: net.yacy.kelondro.blob.HeapModifier.close(HeapModifier.java:82) -> net.yacy.kelondro.blob.ArrayStack.unmountBLOB(ArrayStack.java:276) -> net.yacy.kelondro.blob.ArrayStack.unmountSmallest(ArrayStack.java:344) -> net.yacy.kelondro.rwi.ReferenceContainerArray.shrinkAnySmallFiles(ReferenceContainerArray.java:411) -> net.yacy.kelondro.rwi.IndexCell.shrink(IndexCell.java:198) -> net.yacy.kelondro.rwi.IndexCell.access$900(IndexCell.java:65) -> net.yacy.kelondro.rwi.IndexCell$FlushThread.flushBuffer(IndexCell.java:172) -> net.yacy.kelondro.rwi.IndexCell$FlushThread.run(IndexCell.java:122)
I 2014/09/28 15:56:25 RICELL-shrink2 unmountSmallest(67108864)
I 2014/09/28 15:56:25 IODispatcher appended merge job of files citation.index.20140928135552447.blob, citation.index.20140928135325343.blob to citation.index.20140928135625452.blob
I 2014/09/28 15:56:25 BLOBArray merging citation.index.20140928135552447.blob with citation.index.20140928135325343.blob
I 2014/09/28 15:56:27 HeapWriter wrote a dump for the 803964 index entries of citation.index.20140928135625452.blob in 174 milliseconds.
I 2014/09/28 15:56:28 HeapReader saturation of citation.index.20140928135625452.blob.oX_dP40CSECP.idx: keylength = 6, vallength = 4, size = 803964, maximum saving for index-compression = 4 MB, exact saving for value-compression = 3 MB
I 2014/09/28 15:56:28 HeapReader using a dump of the index of /home/dlw343/yacy/DATA/INDEX/webportal/SEGMENTS/default/citation.index.20140928135625452.blob.
I 2014/09/28 15:56:28 BLOBArray merged citation.index.20140928135552447.blob with citation.index.20140928135325343.blob into /home/dlw343/yacy/DATA/INDEX/webportal/SEGMENTS/default/citation.index.20140928135625452.blob
I 2014/09/28 16:05:52 IODispatcher appended dump job for file citation.index.20140928140552466.blob
I 2014/09/28 16:05:52 ReferenceContainerCache creating rwi heap dump 'citation.index.20140928140552466.blob', 1036 rwi's
I 2014/09/28 16:05:52 HeapWriter wrote a dump for the 1036 index entries of citation.index.20140928140552466.blob in 2 milliseconds.
I 2014/09/28 16:05:52 ReferenceContainerCache finished rwi heap dump: 1036 terms, 0 term/data relations in 15 milliseconds
I 2014/09/28 16:05:52 HeapReader saturation of citation.index.20140928140552466.blob.jNaFzANwVGDG.idx: keylength = 3, vallength = 2, size = 1036, maximum saving for index-compression = 0 MB, exact saving for value-compression = 0 MB
I 2014/09/28 16:05:52 HeapReader using a dump of the index of /home/dlw343/yacy/DATA/INDEX/webportal/SEGMENTS/default/citation.index.20140928140552466.blob.
I 2014/09/28 16:06:25 HeapReader close HeapFile citation.index.20140928140552466.blob; trace: net.yacy.kelondro.blob.HeapModifier.close(HeapModifier.java:82) -> net.yacy.kelondro.blob.ArrayStack.unmountBLOB(ArrayStack.java:276) -> net.yacy.kelondro.blob.ArrayStack.unmountSmallest(ArrayStack.java:343) -> net.yacy.kelondro.rwi.ReferenceContainerArray.shrinkAnySmallFiles(ReferenceContainerArray.java:411) -> net.yacy.kelondro.rwi.IndexCell.shrink(IndexCell.java:198) -> net.yacy.kelondro.rwi.IndexCell.access$900(IndexCell.java:65) -> net.yacy.kelondro.rwi.IndexCell$FlushThread.flushBuffer(IndexCell.java:172) -> net.yacy.kelondro.rwi.IndexCell$FlushThread.run(IndexCell.java:122)
I 2014/09/28 16:06:25 HeapReader close HeapFile citation.index.20140928135625452.blob; trace: net.yacy.kelondro.blob.HeapModifier.close(HeapModifier.java:82) -> net.yacy.kelondro.blob.ArrayStack.unmountBLOB(ArrayStack.java:276) -> net.yacy.kelondro.blob.ArrayStack.unmountSmallest(ArrayStack.java:344) -> net.yacy.kelondro.rwi.ReferenceContainerArray.shrinkAnySmallFiles(ReferenceContainerArray.java:411) -> net.yacy.kelondro.rwi.IndexCell.shrink(IndexCell.java:198) -> net.yacy.kelondro.rwi.IndexCell.access$900(IndexCell.java:65) -> net.yacy.kelondro.rwi.IndexCell$FlushThread.flushBuffer(IndexCell.java:172) -> net.yacy.kelondro.rwi.IndexCell$FlushThread.run(IndexCell.java:122)
I 2014/09/28 16:06:25 RICELL-shrink2 unmountSmallest(67108864)
I 2014/09/28 16:06:25 IODispatcher appended merge job of files citation.index.20140928140552466.blob, citation.index.20140928135625452.blob to citation.index.20140928140625472.blob
I 2014/09/28 16:06:25 BLOBArray merging citation.index.20140928140552466.blob with citation.index.20140928135625452.blob
I 2014/09/28 16:06:27 HeapWriter wrote a dump for the 804509 index entries of citation.index.20140928140625472.blob in 175 milliseconds.
I 2014/09/28 16:06:28 HeapReader saturation of citation.index.20140928140625472.blob.mR0eetndmMJx.idx: keylength = 6, vallength = 4, size = 804509, maximum saving for index-compression = 4 MB, exact saving for value-compression = 3 MB
I 2014/09/28 16:06:28 HeapReader using a dump of the index of /home/dlw343/yacy/DATA/INDEX/webportal/SEGMENTS/default/citation.index.20140928140625472.blob.
I 2014/09/28 16:06:28 BLOBArray merged citation.index.20140928140552466.blob with citation.index.20140928135625452.blob into /home/dlw343/yacy/DATA/INDEX/webportal/SEGMENTS/default/citation.index.20140928140625472.blob
(0000934)
LA_FORGE (reporter)
2014-09-28 17:13

Doesn't occur with v1.7/9000
(0000966)
Orbiter (manager)
2014-12-08 10:43

this is difficult to reconstruct since every crawl may be computed differently event with the same start point because concurrency may change order and the number of threads may be different. In recent YaCy changes there had been fixes to the maximum number of threads that are taken from the OS and these had been reduced to 1/4 (older versions may have reserved about 400 threads, current YaCy takes at most about 100 and runs down to lower than 50 if the crawl tasks are finished.
Please check the performance again and if the problem occurs again, reopen the issue.
(0001014)
LA_FORGE (reporter)
2015-03-06 18:37

Happening again in v1.82/9000

Can I do something to help the developers? Run YaCy in Eclipse to fetch some additional debug info for you.
(0001037)
Scarfmonster (reporter)
2015-04-17 02:01

What I noticed is that the httpd queue gets full because YaCy doesn't process the connections from other peers fast enough, and then there is not enough threads to serve the admin pages. It seems that after a certain point yacy can't keep up with itself and then gets DDOSed by it's own network since the connections keep hanging indefinitely and more and more keep coming as other peers try to ping it or transfer RWIs.
(0001038)
BuBu (developer)
2015-04-17 03:15

From last comment, I'll going to experimentally activate the Jetty QoSFilter (prioritizing requests from localhost)
(0001040)
LA_FORGE (reporter)
2015-04-23 19:37

Great! Thank you very much.
(0001041)
BuBu (developer)
2015-04-23 21:48

unfortunately..... I'm tested the QoSFilter locally for some days and had one Situation of unresponsive frontend after some larger crawls
and the QoS didn't solve/prevent the Situation (as others reported, too with the DHT Connections still doing stuff... but not the frontend)
Expected better... and 'm baffled.
(0001042)
LA_FORGE (reporter)
2015-04-25 18:41

Oops! I'm sorry to hear that. I never had this problem wit the 1.72 version. With 1.82 I'm running in some kind of deadlocks and the frontend doesn't work for almost 20 minutes. After this time, it's working again very well.
(0001047)
LA_FORGE (reporter)
2015-05-15 17:39

After 2 days of a hung in deadlock, the following exception was thrown by jetty:

javax.servlet.ServletException: /mnt/1800gb/yacy2/htroot/CrawlResults.html
    at net.yacy.http.servlets.YaCyDefaultServlet.handleTemplate(YaCyDefaultServlet.java:831)
    at net.yacy.http.servlets.YaCyDefaultServlet.doGet(YaCyDefaultServlet.java:317)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:800)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:553)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
    at net.yacy.http.CrashProtectionHandler.handle(CrashProtectionHandler.java:33)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
    at org.eclipse.jetty.server.Server.handle(Server.java:497)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:313)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248)
    at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:626)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:546)
    at java.lang.Thread.run(Unknown Source)
(0001048)
LA_FORGE (reporter)
2015-05-16 13:24

Related to the unresponsive webinterface I found this in the Logs (1.82/9000):

W 2015/05/16 13:22:01 ConcurrentLog net.yacy.kelondro.util.kelondroException: C:
\Users\Stefan\YaCy\DATA\INDEX\freeworld\QUEUES\CrawlerCoreStacks\home."+k+".com.
443\0005.stack
net.yacy.kelondro.util.kelondroException: C:\Users\Stefan\YaCy\DATA\INDEX\freewo
rld\QUEUES\CrawlerCoreStacks\home."+k+".com.443\0005.stack
        at net.yacy.kelondro.table.Table.<init>(Table.java:228)
        at net.yacy.kelondro.index.OnDemandOpenFileIndex.getIndex(OnDemandOpenFi
leIndex.java:61)
        at net.yacy.kelondro.index.OnDemandOpenFileIndex.size(OnDemandOpenFileIn
dex.java:153)
        at net.yacy.kelondro.index.BufferedObjectIndex.size(BufferedObjectIndex.
java:152)
        at net.yacy.crawler.HostQueue.size(HostQueue.java:360)
        at net.yacy.crawler.HostBalancer.size(HostBalancer.java:193)
        at net.yacy.crawler.data.NoticedURL.stackSize(NoticedURL.java:142)
        at net.yacy.crawler.data.CrawlQueues.coreCrawlJobSize(CrawlQueues.java:2
32)
        at Crawler_p.respond(Crawler_p.java:102)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at net.yacy.http.servlets.YaCyDefaultServlet.invokeServlet(YaCyDefaultSe
rvlet.java:661)
        at net.yacy.http.servlets.YaCyDefaultServlet.handleTemplate(YaCyDefaultS
ervlet.java:825)
        at net.yacy.http.servlets.YaCyDefaultServlet.doGet(YaCyDefaultServlet.ja
va:317)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:800
)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java
:587)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.j
ava:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.jav
a:542)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandl
er.java:223)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandl
er.java:1125)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:
515)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandle
r.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandle
r.java:1059)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.j
ava:141)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(Cont
extHandlerCollection.java:215)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper
.java:97)
        at net.yacy.http.CrashProtectionHandler.handle(CrashProtectionHandler.ja
va:33)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper
.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:497)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:313)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.jav
a:248)
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java
:540)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPoo
l.java:626)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool
.java:546)
        at java.lang.Thread.run(Unknown Source)
(0001066)
LA_FORGE (reporter)
2015-05-31 15:04

I recently discovered it's not a bug! I monitored the traffic on the port yacy listens on and found a huge botnet that's neary permanetly penetrating my HostBrowser with automated & excessive queries!
(0001067)
LA_FORGE (reporter)
2015-06-07 16:24

We need a better protection against botnet attacks on the IndexBrowser; my suggestion:

http://mantis.tokeek.de/view.php?id=583 [^]

- Issue History
Date Modified Username Field Change
2014-09-27 19:18 LA_FORGE New Issue
2014-09-28 10:31 LA_FORGE Note Added: 0000931
2014-09-28 11:26 LA_FORGE Note Deleted: 0000931
2014-09-28 11:31 LA_FORGE Note Added: 0000932
2014-09-28 16:33 LA_FORGE Note Added: 0000933
2014-09-28 17:13 LA_FORGE Note Added: 0000934
2014-12-08 10:43 Orbiter Note Added: 0000966
2014-12-08 10:43 Orbiter Status new => resolved
2014-12-08 10:43 Orbiter Resolution open => fixed
2014-12-08 10:43 Orbiter Assigned To => Orbiter
2015-03-06 18:37 LA_FORGE Note Added: 0001014
2015-03-06 18:37 LA_FORGE Status resolved => feedback
2015-03-06 18:37 LA_FORGE Resolution fixed => reopened
2015-04-17 02:01 Scarfmonster Note Added: 0001037
2015-04-17 03:15 BuBu Note Added: 0001038
2015-04-23 19:37 LA_FORGE Note Added: 0001040
2015-04-23 19:37 LA_FORGE Status feedback => assigned
2015-04-23 21:48 BuBu Note Added: 0001041
2015-04-25 18:41 LA_FORGE Note Added: 0001042
2015-05-15 17:39 LA_FORGE Note Added: 0001047
2015-05-16 13:24 LA_FORGE Note Added: 0001048
2015-05-31 15:04 LA_FORGE Note Added: 0001066
2015-06-07 16:24 LA_FORGE Note Added: 0001067


Copyright © 2000 - 2019 MantisBT Team
Powered by Mantis Bugtracker