View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0000626YaCy[All Projects] Generalpublic2015-12-09 13:522016-01-06 20:14
Assigned ToBuBu 
PlatformJDK 1.7.0_51OSDebian GNU/LinuxOS Version8.2 (Jessie)
Product VersionYaCy 1.8 
Target VersionFixed in Version 
Summary0000626: Low memory setup use ends up in OutOfMemoryError
DescriptionUsing YaCy in P2P mode intensively with a low memory setup almost always ends up in the hour with YaCy beeing stuck on a search query and logging multiple OutOfMemoryErrors.
Steps To ReproduceUse a fresh YaCy installation or an existing one
Set RAM max less or equal than 1024 MB
No crawl is/has been run
Run various P2P successive search queries (1 to 3 search terms) producing from zero to thousands results
In the hour, YaCy almost always ends up by freezing on a search query and becomes unusable.
Console or log show multiple OutOfMemoryError traces.

Note : when activating GeoNames, OutOfMemoryError error uccurs even more quickly.
Additional InformationFirst analysis :
 - I have only 4GB physical RAM and increasing memory use to 2GB makes the whole system far too slow when having other open applications because of swapping
 - I created a heap dump with jvisualvm when OutOfMemoryError occurs :
 - Inspecting dump to find biggest objects reveals the largest are :
  - LRUCache (250MB) corresponding to local Solr documentCache
  - ByteArrayOutputStream (116MB) used by a a SolrQueryResponse
  - document cache is full : size=64 as declared in solrconfig.xml
 - local Solr collection1 only contains 6692 documents
 - looking for large documents in local Solr reveals there are some documents with very big (up to 6MB) text_t field : example sku=https://tools.wmflabs.org/quick-intersection/index.php?lang=fr&project=wikipedia&cats=Portail:Informatique&ns=*&depth=-1&max=30000&start=0&format=html [^]

 - when activating Geonames, biggest object in dump is net.yacy.cora.language.synonyms.AutotaggingLibrary (298MB)

Idea to solve this issue :
- clear local Solr document cache when memory is getting low?
TagsNo tags attached.
Attached Filespng file icon dump_analysis.png [^] (173,805 bytes) 2015-12-09 13:52

png file icon dump_analysis11122015.png [^] (187,948 bytes) 2015-12-11 20:30

png file icon dump_analysis11122015_jdk_1.7.0_80.png [^] (186,527 bytes) 2015-12-11 20:30

txt file icon console12122015.txt [^] (82,224 bytes) 2015-12-12 01:13 [Show Content]

- Relationships

-  Notes
sixcooler (developer)
2015-12-09 20:57

Hello luc,

clearing the Solr-caches is what YaCy does - but it depends on getting the point when to do that.
Try setting 'Minimum of Required Memory' to suspend tasks when low on Memory at /PerformanceQueues_p.html and use the 'Generation Memory Strategy' at /PerformanceMemory_p.html

Cu, sixcooler.
luc (reporter)
2015-12-09 23:16

Thank you very much for this advice. I was not really aware such fine performance tuning was available from administration interface.
By the way, not really easy to handle for the beginner, who may imagine default parameters to be sufficent for basic use.
I will try playing with all these parameters and see if I find my way!
luc (reporter)
2015-12-11 00:04

Some notes :
 - with my JVM (Oracle HotSpot), GenerationMemoryStrategy might be more reliable than StandardMemoryStrategy : when running MemoryControl.java main test, the two strategies run in similar time, but when standard strategy is used, test always ends in OutOfMemoryError, whereas when generation strategy is used, low memory is successfully detected and program ends normally.
 - however, switching to Generation Memory Strategy alone doesn't solve my problem : OutOfMemory is still finally thrown, on requests as simple as 'spring framework'
 - it is easier and more accurate to generate heap dumps automatically on memory error by runnning HotSpot JVM with these options : -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/path_to_dump
 - Solr EmbeddedSolrServer.request function may not be optimized and is marked in source code as TODO
luc (reporter)
2015-12-11 20:20

A few other notes :
 - upgraded to Oracle JDK 1.7.0_80 as recommended by Solr : https://lucene.apache.org/solr/5_3_1/SYSTEM_REQUIREMENTS.html [^]
 - tried playing with 'Minimum of Required Memory' field at /PerformanceQueues_p.html, up to 200MB on each row

But these trials didn't solve the problem...

Then I searched SolrCache.clear() call hierarchy in YaCy source :
 - most calls are made only after OutOfMemoryError has been thrown : typically in SolrServerConnector :
            try {
                this.server.add(solrdocs, -1);
            } catch (final Throwable e) {
                clearCaches(); // prevent further OOM if this was caused by OOM
 -> a good idea, but I think it is a little bit late, as other threads may have already suffered from the OutOfMemory error, and overall statibility might not be recovered
 - the only other calls I found is when deleting index (example : in ServerSolrConnector.deleteById), and the most usefull to my mind is in yacysearch because it makes a preventive clean :
            // check available memory and clean up if necessary
            if ( !MemoryControl.request(8000000L, false) ) {

But to my mind the checked size should be configurable, and it is too small : on the heap analysis I attached, solr is using around 116MB in a ByteArrayOutputStream to build its response...

 - Last but not least : in my local solr I have found documents with even larger text_t field than in initial description. One example :
  - sku : http://www.dshs.state.tx.us/vs/marriagedivorce/marridx/marr88.zip [^]
  - text_t : 23MB

-> I don't understand why the whole document is stored in the text_t field?? I don't kown Solr very well, but it appears they recommand to keep documents as small as possible... And isn't the data put in text_t field already in Solr internal index? In a worst case scenario, we could imagine a cache with 64 documents of 100MB (the default max crawl size) : which gives : 64x100MB=6400MB! Am I missing something?
sixcooler (developer)
2015-12-11 20:36

The CachedSolrConnector has a method called addToCache(...) which is used quite often. In this method we clear all the caches when we have a 'MemoryControl.shortStatus(). This status become true when something called more memory than being available.
That's why I asked you also to tune the minimum memory for the tasks:
Once YaCy fails calling for that amount of memory, it will clear its caches.
luc (reporter)
2015-12-11 20:59

Ok, but... it looks like "CachedSolrConnector was replaced by ConcurrentUpdateSolrConnector" : and it is even you who wrote this comment ;)
See PerformanceMemory_p.html line 228 and a commit by sixcooler of 2013-05-02 15:47:21. Damn git history!!
More seriously I don't find any references to CachedSolrConnector in the current code.
sixcooler (developer)
2015-12-11 21:30

Wow - seams to be time to remove that thingy :-)

OK - than it should be FullText.putDocument(...) FullText.clearCache() and Segment.clearCache() that are doing this job pretty much the same way :-)

At the end the short status is needed - that came up to get a very quick status whenever somthing failed because of low memory.
You can simulate this status on /PerformanceMemory_p.html
luc (reporter)
2015-12-11 21:51
edited on: 2015-12-11 21:52

You are true I had missed this in the hierarchy call : SolrCache.clear() -> EmbeddedSolrConnector.clearCaches -> InstanceMirror.clearCaches -> FullText.clearCaches -> FullText.putDocument
But all heap dumps I made show the memory error is not thrown when putting documents but when requesting using SolrConnector.getResponseByParams... I suppose at that time there would be enough space to put another document (so no call to clearCaches), but building response with getResponseByParams takes much memory. And once a OutOfMemory error is thrown it seems the whole app become unstable and can not really recover. A few more queries, and another is thrown, etc... From the browser view, it frozen.
Thats why I wonder if the root of all that would not be too large text_t fields in some (many?) solr documents.

sixcooler (developer)
2015-12-12 00:06

Do you have an stacktrace of such an OOM?
The text_t contains all the visible stuff of an document - in order to be searchable an to generate the snippets for the results.

I agree with you that this is a problem - I worked around for me by deleting documents that are to big.
(delete by query -> process scheduler)
luc (reporter)
2015-12-12 01:22

I attached console12122015.txt : a copy of my console when YaCy is not responding.
YaCy was ran from zero (default configuration and no DATA), in P2P mode. Launched from ant "run" task with the folowing parameters :
      <jvmarg line="-Xms256m"/>
      <jvmarg line="-Xmx1204m"/>
First OutOfMemory error occured about 15 minutes after startup. Then YaCy began to become slow, to finally stop answering search queries : that is when console was copied.
luc (reporter)
2015-12-12 22:02
edited on: 2015-12-12 22:03

I understand text_t is indexed to performs search queries, but when it becomes large is it really useful to store it in cache? To generate snippets, only the beginning is needed...
By the way, this is not the only field increasing memory footprint : in my last heap dump, I found in cache some solr documents around 20MB including more than 6000 "synonyms_sxt" field entries (example : sku="http://www.ecoccs.com/resources_links.html" [^]).
Coming Solr 5.4 release might enhance some points : for example, worked has been done on JavaBinCodec, which is used in SolrConnector.getResponseByParams (https://issues.apache.org/jira/browse/SOLR-7971 [^]).

luc (reporter)
2015-12-13 15:56
edited on: 2015-12-13 15:57

I finally found and began testing a first serious optimization : as synonyms can be searched but do not need to be retrieved (they are rebuild when loading from surrogates), set "stored" in collection schema to false works fine, and synonyms are no more returned in search results or in documents cache :
In schema.xml :
<field name="synonyms_sxt" type="string" indexed="true" stored="false" multiValued="true"/>

But I noticed another array type field in my dumps wich can be very large : inboundlinks_urlstub_sxt. It can hold thousands of items and retain some MB of memory for only one document in cache. The solution won't be as simple as for synonyms : it store parameter can not be set to false, as it is needed by HyperLinkGraph...

Concerning text_t field, retrieval is also needed for example for highlighting. But maybe using a copy field limited to some first characters to highlight could be considered sufficient...

luc (reporter)
2015-12-14 09:51

I have to say my last idea is not so good : setting some fields to stored="false" in schema surely reduce memory footprint, but only worked because it was set only on my node, and remote peers still continued to retrieve and transmit theses values.
So I tested a last preventive solution : in Protocol.solrQuery, I check text_t field size, and add it to local index only if below 1MB. For now it is what works the best for me, and it do not need large refactorings. So I propose to add this as a new configurable option, initialized at first startup with a default value calculated from total available memory.
Further tests are needed of course.
sixcooler (developer)
2015-12-14 22:11

Your idear sounds cool - but please keep in mind to make it optional.
Many user won't like such a feature :-)

Your OOM is spezial: 'GC overhead limit exceeded' - I've never seen this on YaCy.
Are you using a GC param on the command?
luc (reporter)
2015-12-14 23:39

Yes I understand it must be optional.

I run YaCy with defaults parameters, in normal or debug mode. My only custom setting is memory heap set to 1024M.
I noticed OutOfMemory errors occured from time to time since I've been using YaCy (August 2015). When I tried to reproduce the error, I realized it occurs quite quickly when searching intensively : no concurrent queries, only successive various unrelated searches, always waiting one finishes to launch another...
BuBu (developer)
2015-12-15 02:35

may be considere also existing options e.g.
- to limit parsers filesize in /Settings_p.html
  "Maximum allowed file size in bytes that should be downloaded. Larger files will be skipped. -1 means unlimited.",
- disable zip, tar, ....archive parser (adding decompressed content to index) if one would like to get rid of big index docs.
luc (reporter)
2015-12-15 02:56

Ok, these options are useful, but apply only to crawling process, isn't it?
I have reproduced the scenario described here many times, starting from an empty and fresh YaCY install...
All options I have found could not prevent receiving large Solr documents from remote peers in P2P mode, once "Index remote results" option is activated, which is the default.
Don't you think it would be interesting to have a choice between refusing to index remote results and between allowing everything?
luc (reporter)
2015-12-16 03:44

I have finalized my modifications : now when I limit remote documents to 1MB, I have no more exceptions when running my (gentle) stress test case, even with maximum heap memory set to 600MB!
With this setting there are not so much documents rejected.
I also corrected a vicious ClosedByInterruptException case.
See pull request : https://github.com/yacy/yacy_search_server/pull/34 [^]
BuBu (developer)
2016-01-06 20:14

config option proposal by @luc has been committed
pull request https://github.com/yacy/yacy_search_server/pull/34 [^]

- Issue History
Date Modified Username Field Change
2015-12-09 13:52 luc New Issue
2015-12-09 13:52 luc File Added: dump_analysis.png
2015-12-09 20:57 sixcooler Note Added: 0001157
2015-12-09 23:16 luc Note Added: 0001158
2015-12-11 00:04 luc Note Added: 0001159
2015-12-11 20:20 luc Note Added: 0001160
2015-12-11 20:30 luc File Added: dump_analysis11122015.png
2015-12-11 20:30 luc File Added: dump_analysis11122015_jdk_1.7.0_80.png
2015-12-11 20:36 sixcooler Note Added: 0001161
2015-12-11 20:59 luc Note Added: 0001162
2015-12-11 21:30 sixcooler Note Added: 0001163
2015-12-11 21:51 luc Note Added: 0001164
2015-12-11 21:52 luc Note Edited: 0001164 View Revisions
2015-12-12 00:06 sixcooler Note Added: 0001165
2015-12-12 01:13 luc File Added: console12122015.txt
2015-12-12 01:22 luc Note Added: 0001166
2015-12-12 22:02 luc Note Added: 0001167
2015-12-12 22:03 luc Note Edited: 0001167 View Revisions
2015-12-13 15:56 luc Note Added: 0001169
2015-12-13 15:57 luc Note Edited: 0001169 View Revisions
2015-12-14 09:51 luc Note Added: 0001170
2015-12-14 22:11 sixcooler Note Added: 0001171
2015-12-14 23:39 luc Note Added: 0001172
2015-12-15 02:35 BuBu Note Added: 0001173
2015-12-15 02:56 luc Note Added: 0001174
2015-12-16 03:44 luc Note Added: 0001176
2016-01-06 20:14 BuBu Note Added: 0001193
2016-01-06 20:14 BuBu Status new => resolved
2016-01-06 20:14 BuBu Resolution open => fixed
2016-01-06 20:14 BuBu Assigned To => BuBu

Copyright © 2000 - 2021 MantisBT Team
Powered by Mantis Bugtracker