YaCy-Bugtracker - YaCy
View Issue Details
0000527YaCyWishlist - Wunschlistepublic2015-01-25 16:142019-07-28 08:45
oyvinds 
 
normalminorN/A
newopen 
none 
YaCy 1.8 
 
0000527: Wishlish: Different Proxy Settings for Crawling and YaCy<->Yacy
I would like to be able to use no proxy for YaCy<->Yacy communications but do use a http proxy for http and https connections to websites being crawled.

A huge amount of websites have, for some reason, hired Cloudflare to deny their users service. It is unclear exactly why a large number of website owners decide to deny a large portion of their users service but they clearly do.

I would like to be able to crawl with a proxy that goes through Tor and still do YaCy<->YaCy communications over non-Tor. The result of this is that all the sites blocked by Cloudflare would not be crawled and therefore not show up in the search results.

I do not mind that they can see that I am running a YaCy node and them tracking the search entries is not that bad. YaCy really should start encrypting all yacy<->yacy communications, seriously, why the flying fuck isn't it doing this - it would be so each for each yacy node to generate a keypair and use this for yacy<->yacy but that is a another bug.

I really do like to be able to visit the search results I get from YaCy over Tor. A bad result of doing crawls over the clearnet and visiting search results over Tor is that some sites do not get blocked by Cloudflare over clearnet and are included in the search results but they don't work over Tor. Thus it would be better to do the crawls over Tor but this aparently also makes yacy<->yacy go over Tor and that is not required. Thus one proxy setting for yacy<->yacy and another for crawling would be great.
Lol, yacysearch
Issue History
2015-01-25 16:14oyvindsNew Issue
2019-07-28 08:43NopolopTag Attached: Lol
2019-07-28 08:45NopolopTag Attached: yacysearch

There are no notes attached to this issue.