Anonymous | Login | Signup for a new account | 2022-05-19 14:03 CEST | ![]() |
Main | My View | View Issues | Change Log | Roadmap |
View Issue Details [ Jump to Notes ] | [ Issue History ] [ Print ] | ||||||||
ID | Project | Category | View Status | Date Submitted | Last Update | ||||
0000751 | YaCy | Wishlist - Wunschliste | public | 2017-06-07 20:08 | 2017-06-15 22:59 | ||||
Reporter | smokingwheels | ||||||||
Assigned To | administrator | ||||||||
Priority | none | Severity | text | Reproducibility | N/A | ||||
Status | resolved | Resolution | fixed | ||||||
ETA | none | ||||||||
Platform | Windows Linux MAC | OS | OS Version | ||||||
Product Version | |||||||||
Target Version | Fixed in Version | ||||||||
Summary | 0000751: Thread Dumps for anyone's PDF's screen dumps and what was happening at the time. | ||||||||
Description | A Sort of Torture and Test thread record for anyone to use and add things. Eg the Mouse movement becomes laggy or slow to respond | ||||||||
Tags | No tags attached. | ||||||||
Attached Files | ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() | ||||||||
![]() |
|
(0001412) smokingwheels (reporter) 2017-06-07 20:15 edited on: 2017-06-07 20:18 |
Local and Cloud Debug threads. File names like -18 is equal to renice -18 -p 1234 eg java Note Yacy Code is modified though. yacy-Debug.tar.gz [^] (1,132,522 bytes) 2017-06-07 20:16 Upload file first then reference in post. |
(0001413) smokingwheels (reporter) 2017-06-07 20:41 |
YaCy 'anonufe-11011755-15'_ Crawler 600 mb cloud.pdf [^] (217,731 bytes) 2017-06-07 20:33 1 CPU 1 GB Crawler has paused not enough memory. pending: collection=191059 YaCy version: 1.92/9231 Uptime: 2 days 03:04 Java version: 1.8.0_131 Processors: 1 Load: 1.19 Threads: 251/20, peak:526, total:55345 Was crawling at approx 300 PPM |
(0001414) smokingwheels (reporter) 2017-06-07 22:44 |
YaCy 'webportal'_ Console Status result of hosts file.pdf [^] (141,988 bytes) 2017-06-07 22:41 http://winhelp2002.mvps.org/HOSTS.HTM [^] There are many other sites to choose from. Blocked Threads not so often. |
(0001415) smokingwheels (reporter) 2017-06-08 03:51 |
Debug 06-08-2017 8-35.tar.gz [^] (1,979,731 bytes) 2017-06-08 03:49 Crawling webportal running ok 20 dumps |
(0001416) smokingwheels (reporter) 2017-06-08 03:57 edited on: 2017-06-08 04:51 |
Debug Threads 9-07--53 slow crawling.tar.gz [^] (2,591,553 bytes) 2017-06-08 03:52 Noticed crawler speed dropped down 0-500 ppm Started Logger recording 20 Dumps Shutdown yacy. 190 k in collection. YaCy 'anonufe-11011755-15'_ Console Status crawler very slow9-07-53.pdf [^] (115,044 bytes) 2017-06-08 03:58 Status page during the event. |
(0001417) luc (reporter) 2017-06-08 09:35 |
Wow great job! Not easy to analyze but very valuable. The "Debug Threads 9-07--53 slow crawling.tar.gz" traces seems to clearly show that there are some threads inter-blocked by operations on the crawler Cache, likely all waiting for this one : "Thread= ArrayStack.DELETE_EXECUTOR_pool-1-thread-3821 id=31967 TIMED_WAITING at java.io.RandomAccessFile.readFully(RandomAccessFile.java:436)". I guess this won't be easy to solve. Not sure for now why that read operation on a heap file could be blocking so long... |
(0001418) smokingwheels (reporter) 2017-06-08 16:20 edited on: 2017-06-08 18:27 |
Does this link work its a file access log while crawling? https://www.dropbox.com/s/5vi8lzqoee1vwty/disk.txt.tar.gz?dl=0 [^] |
(0001419) smokingwheels (reporter) 2017-06-08 16:31 edited on: 2017-06-08 16:42 |
Debug 06-08-17 10-06-42.tar.gz While crawling slowly before I terminated it. Webportal is running ok in the cloud but that is on a different class of hardware. The particular DNS servers you use may affect it also because my router was hanging after 12 hours or so of crawling. |
(0001421) luc (reporter) 2017-06-08 23:41 |
I guess the dropbox link should work, but personally I don't want to install dropbox on my computer... by the way you provided already much to analyze, which should inspire some improvements to try. |
(0001423) smokingwheels (reporter) 2017-06-09 23:42 |
Thanks for the feedback @Luc that is good news to hear. It took 4-5 hours to work out a TAB key on my Linux PC needs 100ms per code and a double release code to make it work the send 10 at a time... I have just Upgraded my home setup to 1.921/9236 I guess I will keep my cloud at the version I did created this post in, while testing from home the new version. Just a minor issue crawler briefly slowed down. I was using the mouse to refresh the output. THREADS WITH STATES: BLOCKED Thread= Finalizer daemon id=3 BLOCKED at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) Thread= Java2D Disposer daemon id=124 BLOCKED at java.lang.Thread.run(Thread.java:748) |
(0001424) smokingwheels (reporter) 2017-06-09 23:54 |
I have room in my Dropbox to store the modified version, I am thinking along the lines of trying 8 $5.00 modules in the next experiment. I think I can use or share or generate a link that does not show up in the public folder. I will look into that first. |
(0001425) smokingwheels (reporter) 2017-06-10 01:17 edited on: 2017-06-10 12:49 |
I ran out of JAVA heap space HTTP 500 doing a crawl for awhile. 1.921/9236 600 MB I think the start up config of javastart_Xms=Xms is normally about 130 MB from my memory. I have found around 200 mb is more stable but it depends on the memory available of the particular system in question. eg Raspberry PI, $5.00 cloud. For Linux if JAVA will Shell to a Bash prompt and return to the adjust it semi automatic for 1 time configuration of a peer. https://www.cyberciti.biz/faq/ram-size-linux/ [^] You maybe already may have the figure in the system somewhere (I think I did look up the command java shell back in version 1.69X). $ less /proc/meminfo > memory.txt OR $ cat /proc/meminfo > memory.txt The startup Linux nice setting I think was 0 and is now 19 I will look into that In my New cloud install the JAVA Nice setting is set to 0 https://twitter.com/smokingwheels/status/873488505099370497 [^] After restart Nice is 10. After Increasing JVM Nice is 19. |
(0001426) smokingwheels (reporter) 2017-06-10 08:34 |
Just installed a spare PC will try. ssh address-of-B 'wget -O - http://server-C/whatever' [^] >> whatever I will also used it for backups. I guess in the rush or the chances I take, I did not start a new copy after a usual few mods. Its great my internet has very little bandwidth left with the default 6000 crawl I was setting up over 8000 before. My little keyboard is now doing a few speed tests. If you would like to see 1 or 2 I can post them just let me know? |
(0001427) smokingwheels (reporter) 2017-06-10 11:38 |
Uploading on my ADSL2 connection takes 2 Hours, I Could install Ubuntu with the desktop version in my cloud because you have room for 2 custom ISO's and run this code modifier remotely. https://youtu.be/3hOkcVbE0yQ [^] Just at the end of this month I will be hit by 10% GST from my Cloud provider. |
(0001429) smokingwheels (reporter) 2017-06-12 04:14 |
1.921/236 HTTP Error 408 Request timeout I got this when checking new site to crawl in browser. Is there anyway to drop the site from the crawler Que if it happens too much like 3 times? |
(0001430) smokingwheels (reporter) 2017-06-12 04:19 edited on: 2017-06-12 21:03 |
1.921/236 Threads.txt.tar.gz [^] (10,345 bytes) 2017-06-12 04:15 at net.yacy.kelondro.index.RAMIndex.has(RAMIndex.java:137) [assert (key != null);] Could be %3Bamp% in the url not to sure. I did list it in the blacklist and the error seems to have gone. |
(0001431) smokingwheels (reporter) 2017-06-12 23:44 edited on: 2017-06-12 23:49 |
1.921/236 Yacy_java_memory.ods I made a Java memory calculator based on the default settings. I am testing at the moment on a 1 GB P4 doing a crawl on adsl2. Yacy_java_calculator.png Just a Quick reference guide to try for Non Linux users. The java settings are all in eg 1024 256 128 increments etc. |
(0001432) smokingwheels (reporter) 2017-06-13 11:56 |
1.921/236 Quad-cloud-and-home-using-trial-memory-settings.tar.gz Contains 2 pics Memory usage the cloud has 8 GB. The GC seems to be working differently. Really too early to tell though. |
(0001442) BuBu (developer) 2017-06-15 22:59 |
improved by @luc https://github.com/yacy/yacy_search_server/commit/a7394b479b4a2ecb9bd0c9696355e5d1008b8742 [^] |
![]() |
|||
Date Modified | Username | Field | Change |
2017-06-07 20:08 | smokingwheels | New Issue | |
2017-06-07 20:15 | smokingwheels | Note Added: 0001412 | |
2017-06-07 20:16 | smokingwheels | File Added: yacy-Debug.tar.gz | |
2017-06-07 20:17 | smokingwheels | Note Edited: 0001412 | View Revisions |
2017-06-07 20:18 | smokingwheels | Note Edited: 0001412 | View Revisions |
2017-06-07 20:40 | smokingwheels | File Added: YaCy 'anonufe-11011755-15'_ Crawler 600 mb cloud.pdf | |
2017-06-07 20:41 | smokingwheels | Note Added: 0001413 | |
2017-06-07 22:41 | smokingwheels | File Added: YaCy 'webportal'_ Console Status result of hosts file.pdf | |
2017-06-07 22:44 | smokingwheels | Note Added: 0001414 | |
2017-06-08 03:49 | smokingwheels | File Added: Debug 06-08-2017 8-35.tar.gz | |
2017-06-08 03:51 | smokingwheels | Note Added: 0001415 | |
2017-06-08 03:52 | smokingwheels | File Added: Debug Threads 9-07--53 slow crawling.tar.gz | |
2017-06-08 03:57 | smokingwheels | Note Added: 0001416 | |
2017-06-08 03:58 | smokingwheels | File Added: YaCy 'anonufe-11011755-15'_ Console Status crawler very slow9-07-53.pdf | |
2017-06-08 03:59 | smokingwheels | Note Edited: 0001416 | View Revisions |
2017-06-08 04:51 | smokingwheels | Note Edited: 0001416 | View Revisions |
2017-06-08 09:35 | luc | Note Added: 0001417 | |
2017-06-08 16:20 | smokingwheels | Note Added: 0001418 | |
2017-06-08 16:28 | smokingwheels | File Added: Debug 06-08-17 10-06-42.tar.gz | |
2017-06-08 16:31 | smokingwheels | Note Added: 0001419 | |
2017-06-08 16:42 | smokingwheels | Note Edited: 0001419 | View Revisions |
2017-06-08 18:27 | smokingwheels | Note Edited: 0001418 | View Revisions |
2017-06-08 23:41 | luc | Note Added: 0001421 | |
2017-06-09 23:42 | smokingwheels | Note Added: 0001423 | |
2017-06-09 23:54 | smokingwheels | Note Added: 0001424 | |
2017-06-10 01:17 | smokingwheels | Note Added: 0001425 | |
2017-06-10 08:34 | smokingwheels | Note Added: 0001426 | |
2017-06-10 11:38 | smokingwheels | Note Added: 0001427 | |
2017-06-10 12:44 | smokingwheels | Note Edited: 0001425 | View Revisions |
2017-06-10 12:49 | smokingwheels | Note Edited: 0001425 | View Revisions |
2017-06-12 04:14 | smokingwheels | Note Added: 0001429 | |
2017-06-12 04:15 | smokingwheels | File Added: Threads.txt.tar.gz | |
2017-06-12 04:19 | smokingwheels | Note Added: 0001430 | |
2017-06-12 21:03 | smokingwheels | Note Edited: 0001430 | View Revisions |
2017-06-12 23:40 | smokingwheels | File Added: Yacy_java_memory.ods | |
2017-06-12 23:44 | smokingwheels | Note Added: 0001431 | |
2017-06-12 23:45 | smokingwheels | File Added: Yacy_java_calculator.png | |
2017-06-12 23:49 | smokingwheels | Note Edited: 0001431 | View Revisions |
2017-06-13 11:45 | smokingwheels | File Added: Quad-cloud-and-home-using-trial-memory-settings.tar.gz | |
2017-06-13 11:56 | smokingwheels | Note Added: 0001432 | |
2017-06-15 22:59 | BuBu | Note Added: 0001442 | |
2017-06-15 22:59 | BuBu | Status | new => resolved |
2017-06-15 22:59 | BuBu | Resolution | open => fixed |
2017-06-15 22:59 | BuBu | Assigned To | => administrator |
Copyright © 2000 - 2022 MantisBT Team |