YaCy-Bugtracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0000751YaCyWishlist - Wunschlistepublic2017-06-07 20:082017-06-15 22:59
Reportersmokingwheels 
Assigned Toadministrator 
PrioritynoneSeveritytextReproducibilityN/A
StatusresolvedResolutionfixed 
ETAnone 
PlatformWindows Linux MACOSOS Version
Product Version 
Target VersionFixed in Version 
Summary0000751: Thread Dumps for anyone's PDF's screen dumps and what was happening at the time.
DescriptionA Sort of Torture and Test thread record for anyone to use and add things.

Eg the Mouse movement becomes laggy or slow to respond

  
TagsNo tags attached.
Attached Filesgz file icon yacy-Debug.tar.gz [^] (1,132,522 bytes) 2017-06-07 20:16
pdf file icon YaCy 'anonufe-11011755-15'_ Crawler 600 mb cloud.pdf [^] (217,731 bytes) 2017-06-07 20:40
pdf file icon YaCy 'webportal'_ Console Status result of hosts file.pdf [^] (141,988 bytes) 2017-06-07 22:41
gz file icon Debug 06-08-2017 8-35.tar.gz [^] (1,979,731 bytes) 2017-06-08 03:49
gz file icon Debug Threads 9-07--53 slow crawling.tar.gz [^] (2,591,553 bytes) 2017-06-08 03:52
pdf file icon YaCy 'anonufe-11011755-15'_ Console Status crawler very slow9-07-53.pdf [^] (115,044 bytes) 2017-06-08 03:58
gz file icon Debug 06-08-17 10-06-42.tar.gz [^] (3,461,256 bytes) 2017-06-08 16:28
gz file icon Threads.txt.tar.gz [^] (10,345 bytes) 2017-06-12 04:15
? file icon Yacy_java_memory.ods [^] (17,664 bytes) 2017-06-12 23:40
png file icon Yacy_java_calculator.png [^] (176,507 bytes) 2017-06-12 23:45


gz file icon Quad-cloud-and-home-using-trial-memory-settings.tar.gz [^] (403,556 bytes) 2017-06-13 11:45

- Relationships

-  Notes
(0001412)
smokingwheels (reporter)
2017-06-07 20:15
edited on: 2017-06-07 20:18

Local and Cloud Debug threads.
File names like -18 is equal to renice -18 -p 1234 eg java
Note Yacy Code is modified though.

yacy-Debug.tar.gz [^] (1,132,522 bytes) 2017-06-07 20:16

Upload file first then reference in post.

(0001413)
smokingwheels (reporter)
2017-06-07 20:41

YaCy 'anonufe-11011755-15'_ Crawler 600 mb cloud.pdf [^] (217,731 bytes) 2017-06-07 20:33
1 CPU 1 GB

Crawler has paused not enough memory. pending: collection=191059


YaCy version: 1.92/9231
Uptime: 2 days 03:04
Java version: 1.8.0_131
Processors: 1
Load: 1.19
Threads: 251/20, peak:526, total:55345

Was crawling at approx 300 PPM
(0001414)
smokingwheels (reporter)
2017-06-07 22:44

YaCy 'webportal'_ Console Status result of hosts file.pdf [^] (141,988 bytes) 2017-06-07 22:41

http://winhelp2002.mvps.org/HOSTS.HTM [^] There are many other sites to choose from.

Blocked Threads not so often.
(0001415)
smokingwheels (reporter)
2017-06-08 03:51

Debug 06-08-2017 8-35.tar.gz [^] (1,979,731 bytes) 2017-06-08 03:49

Crawling webportal running ok
20 dumps
(0001416)
smokingwheels (reporter)
2017-06-08 03:57
edited on: 2017-06-08 04:51

Debug Threads 9-07--53 slow crawling.tar.gz [^] (2,591,553 bytes) 2017-06-08 03:52

Noticed crawler speed dropped down 0-500 ppm
Started Logger recording
20 Dumps
Shutdown yacy.
190 k in collection.

YaCy 'anonufe-11011755-15'_ Console Status crawler very slow9-07-53.pdf [^] (115,044 bytes) 2017-06-08 03:58
Status page during the event.

(0001417)
luc (reporter)
2017-06-08 09:35

Wow great job!

Not easy to analyze but very valuable.

The "Debug Threads 9-07--53 slow crawling.tar.gz" traces seems to clearly show that there are some threads inter-blocked by operations on the crawler Cache, likely all waiting for this one :
"Thread= ArrayStack.DELETE_EXECUTOR_pool-1-thread-3821 id=31967 TIMED_WAITING
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:436)".

I guess this won't be easy to solve. Not sure for now why that read operation on a heap file could be blocking so long...
(0001418)
smokingwheels (reporter)
2017-06-08 16:20
edited on: 2017-06-08 18:27

Does this link work its a file access log while crawling?


https://www.dropbox.com/s/5vi8lzqoee1vwty/disk.txt.tar.gz?dl=0 [^]

(0001419)
smokingwheels (reporter)
2017-06-08 16:31
edited on: 2017-06-08 16:42

Debug 06-08-17 10-06-42.tar.gz

While crawling slowly before I terminated it.

Webportal is running ok in the cloud but that is on a different class of hardware.

The particular DNS servers you use may affect it also because my router was hanging after 12 hours or so of crawling.

(0001421)
luc (reporter)
2017-06-08 23:41

I guess the dropbox link should work, but personally I don't want to install dropbox on my computer... by the way you provided already much to analyze, which should inspire some improvements to try.
(0001423)
smokingwheels (reporter)
2017-06-09 23:42

Thanks for the feedback @Luc that is good news to hear.
It took 4-5 hours to work out a TAB key on my Linux PC needs 100ms per code and a double release code to make it work the send 10 at a time...

I have just Upgraded my home setup to 1.921/9236
I guess I will keep my cloud at the version I did created this post in, while testing from home the new version.


Just a minor issue crawler briefly slowed down.
I was using the mouse to refresh the output.

THREADS WITH STATES: BLOCKED
 
Thread= Finalizer daemon id=3 BLOCKED
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

 
Thread= Java2D Disposer daemon id=124 BLOCKED
at java.lang.Thread.run(Thread.java:748)
(0001424)
smokingwheels (reporter)
2017-06-09 23:54

I have room in my Dropbox to store the modified version, I am thinking along the lines of trying 8 $5.00 modules in the next experiment.

I think I can use or share or generate a link that does not show up in the public folder. I will look into that first.
(0001425)
smokingwheels (reporter)
2017-06-10 01:17
edited on: 2017-06-10 12:49

I ran out of JAVA heap space HTTP 500 doing a crawl for awhile.
1.921/9236 600 MB

I think the start up config of javastart_Xms=Xms is normally about 130 MB from my memory.
I have found around 200 mb is more stable but it depends on the memory available of the particular system in question. eg Raspberry PI, $5.00 cloud.
For Linux if JAVA will Shell to a Bash prompt and return to the adjust it semi automatic for 1 time configuration of a peer. https://www.cyberciti.biz/faq/ram-size-linux/ [^]
You maybe already may have the figure in the system somewhere (I think I did look up the command java shell back in version 1.69X).
$ less /proc/meminfo > memory.txt
OR
$ cat /proc/meminfo > memory.txt

The startup Linux nice setting I think was 0 and is now 19 I will look into that

In my New cloud install the JAVA Nice setting is set to 0 https://twitter.com/smokingwheels/status/873488505099370497 [^]

After restart Nice is 10.

After Increasing JVM Nice is 19.

(0001426)
smokingwheels (reporter)
2017-06-10 08:34

Just installed a spare PC will try.
ssh address-of-B 'wget -O - http://server-C/whatever' [^] >> whatever
I will also used it for backups.
I guess in the rush or the chances I take, I did not start a new copy after a usual few mods.

Its great my internet has very little bandwidth left with the default 6000 crawl I was setting up over 8000 before.

My little keyboard is now doing a few speed tests. If you would like to see 1 or 2 I can post them just let me know?
(0001427)
smokingwheels (reporter)
2017-06-10 11:38

Uploading on my ADSL2 connection takes 2 Hours, I Could install Ubuntu with the desktop version in my cloud because you have room for 2 custom ISO's and run this code modifier remotely.
https://youtu.be/3hOkcVbE0yQ [^]

Just at the end of this month I will be hit by 10% GST from my Cloud provider.
(0001429)
smokingwheels (reporter)
2017-06-12 04:14

1.921/236
HTTP Error 408 Request timeout
I got this when checking new site to crawl in browser.

Is there anyway to drop the site from the crawler Que if it happens too much like 3 times?
(0001430)
smokingwheels (reporter)
2017-06-12 04:19
edited on: 2017-06-12 21:03

1.921/236
Threads.txt.tar.gz [^] (10,345 bytes) 2017-06-12 04:15

at net.yacy.kelondro.index.RAMIndex.has(RAMIndex.java:137) [assert (key != null);]
Could be %3Bamp% in the url not to sure.

I did list it in the blacklist and the error seems to have gone.

(0001431)
smokingwheels (reporter)
2017-06-12 23:44
edited on: 2017-06-12 23:49

1.921/236
Yacy_java_memory.ods

I made a Java memory calculator based on the default settings. I am testing at the moment on a 1 GB P4 doing a crawl on adsl2.

Yacy_java_calculator.png
Just a Quick reference guide to try for Non Linux users. The java settings are all in eg 1024 256 128 increments etc.

(0001432)
smokingwheels (reporter)
2017-06-13 11:56

1.921/236
Quad-cloud-and-home-using-trial-memory-settings.tar.gz
Contains 2 pics Memory usage the cloud has 8 GB.
The GC seems to be working differently. Really too early to tell though.
(0001442)
BuBu (developer)
2017-06-15 22:59

improved by @luc
https://github.com/yacy/yacy_search_server/commit/a7394b479b4a2ecb9bd0c9696355e5d1008b8742 [^]

- Issue History
Date Modified Username Field Change
2017-06-07 20:08 smokingwheels New Issue
2017-06-07 20:15 smokingwheels Note Added: 0001412
2017-06-07 20:16 smokingwheels File Added: yacy-Debug.tar.gz
2017-06-07 20:17 smokingwheels Note Edited: 0001412 View Revisions
2017-06-07 20:18 smokingwheels Note Edited: 0001412 View Revisions
2017-06-07 20:40 smokingwheels File Added: YaCy 'anonufe-11011755-15'_ Crawler 600 mb cloud.pdf
2017-06-07 20:41 smokingwheels Note Added: 0001413
2017-06-07 22:41 smokingwheels File Added: YaCy 'webportal'_ Console Status result of hosts file.pdf
2017-06-07 22:44 smokingwheels Note Added: 0001414
2017-06-08 03:49 smokingwheels File Added: Debug 06-08-2017 8-35.tar.gz
2017-06-08 03:51 smokingwheels Note Added: 0001415
2017-06-08 03:52 smokingwheels File Added: Debug Threads 9-07--53 slow crawling.tar.gz
2017-06-08 03:57 smokingwheels Note Added: 0001416
2017-06-08 03:58 smokingwheels File Added: YaCy 'anonufe-11011755-15'_ Console Status crawler very slow9-07-53.pdf
2017-06-08 03:59 smokingwheels Note Edited: 0001416 View Revisions
2017-06-08 04:51 smokingwheels Note Edited: 0001416 View Revisions
2017-06-08 09:35 luc Note Added: 0001417
2017-06-08 16:20 smokingwheels Note Added: 0001418
2017-06-08 16:28 smokingwheels File Added: Debug 06-08-17 10-06-42.tar.gz
2017-06-08 16:31 smokingwheels Note Added: 0001419
2017-06-08 16:42 smokingwheels Note Edited: 0001419 View Revisions
2017-06-08 18:27 smokingwheels Note Edited: 0001418 View Revisions
2017-06-08 23:41 luc Note Added: 0001421
2017-06-09 23:42 smokingwheels Note Added: 0001423
2017-06-09 23:54 smokingwheels Note Added: 0001424
2017-06-10 01:17 smokingwheels Note Added: 0001425
2017-06-10 08:34 smokingwheels Note Added: 0001426
2017-06-10 11:38 smokingwheels Note Added: 0001427
2017-06-10 12:44 smokingwheels Note Edited: 0001425 View Revisions
2017-06-10 12:49 smokingwheels Note Edited: 0001425 View Revisions
2017-06-12 04:14 smokingwheels Note Added: 0001429
2017-06-12 04:15 smokingwheels File Added: Threads.txt.tar.gz
2017-06-12 04:19 smokingwheels Note Added: 0001430
2017-06-12 21:03 smokingwheels Note Edited: 0001430 View Revisions
2017-06-12 23:40 smokingwheels File Added: Yacy_java_memory.ods
2017-06-12 23:44 smokingwheels Note Added: 0001431
2017-06-12 23:45 smokingwheels File Added: Yacy_java_calculator.png
2017-06-12 23:49 smokingwheels Note Edited: 0001431 View Revisions
2017-06-13 11:45 smokingwheels File Added: Quad-cloud-and-home-using-trial-memory-settings.tar.gz
2017-06-13 11:56 smokingwheels Note Added: 0001432
2017-06-15 22:59 BuBu Note Added: 0001442
2017-06-15 22:59 BuBu Status new => resolved
2017-06-15 22:59 BuBu Resolution open => fixed
2017-06-15 22:59 BuBu Assigned To => administrator


Copyright © 2000 - 2017 MantisBT Team
Powered by Mantis Bugtracker