View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0000579YaCyWishlist - Wunschlistepublic2015-05-23 01:222015-05-24 04:55
Assigned To 
PlatformOSGnu/LinuxOS VersionDebian 7
Product VersionYaCy 1.8 
Target VersionFixed in Version 
Summary0000579: Allow to freely customize UA string
DescriptionSome websites alter the content they provide in function of the supplied UA string. Others discriminate and ban UAs.

This is the case of Amazon, which delivers no content if it detects "yacybot" in the UA string.

curl -A yacybot http://www.amazon.com/gp/product/B00M55C0NS/ref=s9_qpp_gw_p147_i3 [^]

Some might deem Amazon's as an appropriate and correct choice. I deem this as an opportunistic discrimination against freedom of information. Does Amazon ban googlebot? Of course not. Google brings customers into Amazon, despite draining a lot way more bandwidth than YaCy. Maybe YaCy doesn't bring many customers /right now/, but it will eventually grow with a larger userbase in future.

Being YaCy a decentralized system, everybody should be free to decide their UA string for themselves, if their situation makes the UA string something to be concerned about.
Steps To Reproduce# Works:
curl -A CustomerWallet http://www.amazon.com/gp/product/B00M55C0NS/ref=s9_qpp_gw_p147_i3 [^]

# Banned:
curl -A yacybot http://www.amazon.com/gp/product/B00M55C0NS/ref=s9_qpp_gw_p147_i3 [^]
TagsNo tags attached.
Attached Files

- Relationships

-  Notes
BuBu (developer)
2015-05-24 01:20

Sorry, but that is just not true
(your Statement about Amazon blocking on yacybot).
BuBu (developer)
2015-05-24 02:14

your Amazon issue fixed in v1.83/9215

commit https://github.com/yacy/yacy_search_server/commit/8a9622c31cbd8e05b9da0bc448b729594fffd299 [^]
Davide (reporter)
2015-05-24 02:31

Yes it is true, run the example.

What is fixed in v1.83/9215? My example runs on curl and is reproducible.
Davide (reporter)
2015-05-24 04:55

Thanks I now understand that the commit you referenced fixes a parsing error, which I guess affected Amazon.
But, still, I find Amazon blocking yacybot on UA-string level.

- Issue History
Date Modified Username Field Change
2015-05-23 01:22 Davide New Issue
2015-05-24 01:20 BuBu Note Added: 0001055
2015-05-24 02:14 BuBu Note Added: 0001056
2015-05-24 02:31 Davide Note Added: 0001057
2015-05-24 04:55 Davide Note Added: 0001059

Copyright © 2000 - 2021 MantisBT Team
Powered by Mantis Bugtracker