YaCy-Bugtracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0000566YaCy[All Projects] Generalpublic2015-03-27 22:292016-09-28 15:11
ReporterPhrixos-IT 
Assigned To 
PriorityhighSeveritymajorReproducibilityalways
StatusnewResolutionopen 
ETAnone 
PlatformXeonOSWindowsOS Version2003
Product VersionYaCy 1.8 
Target VersionFixed in Version 
Summary0000566: Still not shooting down
DescriptionWaiting now about 3 houres to shutdown using stopYacy.bat
Additional InformationTestet publisehed woraround worked 1 times.
TagsNo tags attached.
Attached Fileslog file icon yacy00.log [^] (954,911 bytes) 2015-05-03 14:56
log file icon yacy00_(2015_ 06_27).log [^] (793,492 bytes) 2015-06-27 10:29
png file icon Screenshot from 2015-06-27 17:09:08.png [^] (3,081,699 bytes) 2015-06-27 10:30
png file icon YaCy_Starting.png [^] (672,473 bytes) 2015-06-28 06:36
png file icon YaCy_Shutdown.png [^] (165,676 bytes) 2015-06-28 20:27


log file icon yacy00_2015_07_03_Starting.log [^] (160,796 bytes) 2015-07-03 07:44

- Relationships
related to 0000505resolvedBuBu [YaCy-git] Can't shutdown 
related to 0000689resolvedadministrator Shutdown never finishes 
related to 0000567new YaCy can't shutdown properly. Hangs on HeapReader dumping. Have to kill it. 

-  Notes
(0001025)
Phrixos-IT (reporter)
2015-03-27 22:30

I tooks about 1 day to start up! 15 GB Ran 6 cores 2,74 Mio Documents
(0001026)
Phrixos-IT (reporter)
2015-03-28 07:26

now I'm at about 12 houres shutdown it still uses 17% CPU an 7.337.596 GB RAM
(0001028)
Collision (reporter)
2015-04-03 09:24
edited on: 2015-05-03 15:00

I can confirm this issue.
YaCy version 1.83/9118.
Ubuntu 14.04 LTS, 24GB RAM (Allocating 18,000 MiB for YaCy), 4 core 8 thread CPU, 3TB HDD, Java 7u75, 1Gbps FTTH, language: Japanese.
YaCy is never shut down...

RAM used: 14.68 GB
RAM max: 17.2 GB
DISK used: (approx.) 395.61 GB
DISK free: 1,095.11 GB
The current size of the cache: 8126 MB for 305312 files, 27 KB / file in average
Maximum number of words in cache: 8000000

I attached a log file; yacy00.log.

(0001080)
Davide (reporter)
2015-06-27 17:31

When YaCy starts I notice a read burst on the disks lasting several minutes, with sequential accesses. This delays the boot until reads complete.

If the accesses are sequential, the disks sustain high transfer speed and YaCy starts quickly. You can check with `atop` if your accesses are sequential or random. The latter may be due to too many processes accessing the disk(s) concurrently, or filesystem fragmentation.
(0001081)
Collision (reporter)
2015-06-28 06:43

Thanks for the advice, Davide.
I tried 'atop'. But I do not understand the meaning of the results.
So, I upload the screenshot. (YaCy_Starting.png)
(0001082)
Davide (reporter)
2015-06-28 19:24

From your atop screenshots, is seems this time YaCy took a fair 30 minutes to start, after which your sda is reported as beginning to idle, indicating end of initial disk burst and start of cruise, regular service, which is a normal delay to me and apparently indicates a fairly large index, perhaps 500GB, given your disk performance.

You have 650% of wait time on the cpu, out of 800% (8 cores) so the start up strain is solely on the disk. The accesses are also sequential so nothing else appears to compete for disk IO, at least this time, during this set of atop samplings.

Based on the large amount of iops reported (about 800, given by 8000 divided by 10 seconds) you either have a hardware RAID (not visible from atop) or your read requests are pulled from the large 4GB RAM cache, speeding up accesses toward the disk. If the latter is the case and you don't have a hardware RAID, your RAM cache appears to be already populated with index data, supposedly read previously from sda; this would indicate that you have started YaCy more than once since the last computer reboot, and it is possible that the first start of YaCy was much slower than this, due to the cache being empty.

But from this set of samplings all seems OK.
(0001083)
Davide (reporter)
2015-06-28 19:29

To be clear, large iops doesn't indicate that reads are served by the RAM cache, but still the 4GB cache may be playing a role here.
(0001084)
Collision (reporter)
2015-06-28 20:28

I thank for your description, Davide!
As you said, I had started YaCy once before.(I have a single HDD)

I attached a new screenshot image file which is taken after issuing a command to shut down YaCy by pressing the GUI Shutdown button.
(0001085)
Collision (reporter)
2015-07-03 07:54

I attached a new log file that contains the following lines.

E 2015/07/03 14:23:29 HeapReader reclen == 0 at seek pos 11412961 in file /home/xxxx/yacy/DATA/HTCACHE/file.array/wB0BCU2HDv0Y.20150701175815495.blob

E 2015/07/03 14:23:35 HeapReader reclen == 0 at seek pos 26795858 in file /home/xxxx/yacy/DATA/HTCACHE/file.array/wB0BCU2HDv0Y.20150630170808531.blob

E 2015/07/03 14:24:32 HeapReader reclen == 0 at seek pos 20400906 in file /home/xxxx/yacy/DATA/HTCACHE/file.array/wB0BCU2HDv0Y.20150701190217890.blob

W 2015/07/03 14:28:20 HeapReader BLOB wB0BCU2HDv0Y.20150701180820484.blob: skiped not wellformed key \00\00\00\00\00\00\00\00\00\00\00\00 at seek pos 24778774

E 2015/07/03 14:29:48 HeapReader reclen == 0 at seek pos 40333151 in file /home/xxxx/yacy/DATA/HTCACHE/file.array/wB0BCU2HDv0Y.20150702031206918.blob

W 2015/07/03 14:29:58 Cache file and metadata size is not equal, starting a cleanup thread...

This file is recorded at the beginning of the YaCy.

- Issue History
Date Modified Username Field Change
2015-03-27 22:29 Phrixos-IT New Issue
2015-03-27 22:30 Phrixos-IT Note Added: 0001025
2015-03-28 07:26 Phrixos-IT Note Added: 0001026
2015-04-03 09:24 Collision Note Added: 0001028
2015-04-03 09:41 Collision Note Edited: 0001028 View Revisions
2015-04-04 04:45 Collision Note Edited: 0001028 View Revisions
2015-04-04 16:15 BuBu Relationship added related to 0000505
2015-04-04 23:41 BuBu Relationship added related to 0000567
2015-05-03 14:56 Collision File Added: yacy00.log
2015-05-03 15:00 Collision Note Edited: 0001028 View Revisions
2015-06-27 10:29 Collision File Added: yacy00_(2015_ 06_27).log
2015-06-27 10:30 Collision File Added: Screenshot from 2015-06-27 17:09:08.png
2015-06-27 17:31 Davide Note Added: 0001080
2015-06-28 06:36 Collision File Added: YaCy_Starting.png
2015-06-28 06:43 Collision Note Added: 0001081
2015-06-28 19:24 Davide Note Added: 0001082
2015-06-28 19:29 Davide Note Added: 0001083
2015-06-28 20:27 Collision File Added: YaCy_Shutdown.png
2015-06-28 20:28 Collision Note Added: 0001084
2015-07-03 07:44 Collision File Added: yacy00_2015_07_03_Starting.log
2015-07-03 07:54 Collision Note Added: 0001085
2016-09-28 15:11 BuBu Relationship added related to 0000689


Copyright © 2000 - 2019 MantisBT Team
Powered by Mantis Bugtracker