YaCy-Bugtracker - YaCy
|View Issue Details|
|ID||Project||Category||View Status||Date Submitted||Last Update|
|0000774||YaCy||Wishlist - Wunschliste||public||2017-12-27 16:32||2018-06-22 09:06|
|Target Version||Fixed in Version|
|Summary||0000774: [Feature request] Add crawler filter based on document language|
|Description||Currently the crawler has a raw localization filter based on the IP country code of the host server; an improvement would be to also offer a filter based on the actual language detected from the document content.|
_ Main pros:
it would be easier to setup a large crawler job aimed at one specific language spanning a wide array of domain names and subdomains without having to specify individual regex filter rules for each of those domains, since domain names occasionally contain a country code; also consider that the crawler may be allowed to extent to domain names or hosts not initially listed in the crawler job specification, therefore making it harder to specify such rules.
The current filter based on country code may also be inadequate as a single host may offer documents in multiple languages.
_ Main cons:
a simple DNS lookup wouldn't be enough to execute the filter rules, as the document needs to be downloaded and parsed in order to pass it thru the filter.
|Steps To Reproduce|
|Tags||No tags attached.|
|2017-12-27 16:32||Davide||New Issue|
|2018-06-22 09:06||luc||Note Added: 0001498|
|2018-06-22 09:06||luc||Note Edited: 0001498||bug_revision_view_page.php?bugnote_id=1498#r466|