I agree that Places and other similar parts of google's business are highly ethi...

paganel · on Nov 24, 2011

> I don't understand how it's related to this particular issue though.

Because when it first started I think there was a mutual, non-spoken agreement between web content-owners and Google, sort of "I give you our content for free to use it on your search engine and you bring me back visitors by trying to be impartial in your SERPs; but I don't expect you to compete against me, not now, not ever".

That didn't last long, to say the least, but said content owners are now prisoners of Google (yes, it's a big word, but that's the fact). If my personal experience counts, I let Googlebot crawl my website like crazy, at speed rates for which I would certainly ban any other potential search engine bots, and that's because Google brings in 98% of my search engine traffic. And I'm sure I'm not the only one banning strange-looking bots, which could belong to potential new, wonderful search engines, but which at this rate will never see the light of day.

And, unfortunately, things like DuckDuckGo are sub-optimal outside of the English sphere of influence, at least that's what I could see in my case.

1010101111001 · on Nov 24, 2011

If you want to build a search engine, there's no need to crawl the web. Just crawl Google. That's what Microsoft did.

The inequities of crawling are in my opinion a huge issue with the web that receives relatively little discussion.

If a majority of websites were to ban Googlebot, Google would be brought to its knees. Slowly.

Their index would go stale.

I like Google. But I do think crawling, who gets to do it and who does not is an underappreciated issue. Google is very lucky.

1010101111001 · on Nov 25, 2011

I'm guessing the reason this was downvoted was the first paragraph. It was meant as sarcasm. Bing fans on HN I did not expect.

billpatrianakos · on Nov 24, 2011

How are we blocking Googlebot? If we're using robots.txt then they can simply ignore it. Googlebot can begin to identify itself differently. There are a million ways to get around a Googlebot ban and I wouldn't be the guy who thinks he's smarter than Google. You'll lose that one. They'll find a way.

But anyway, this isnt really relevant. Can you tie it in for us?

1010101111001 · on Nov 24, 2011

Blocking is not done via robots.txt. It would more likely be IP-based.

Impossible to block Google? Probably true. But only because they have been allowed to grow so large as to be indefeasible. And the reason they've been able to do that is because of what the commenter said: websites allowed them to crawl, fast and hard, year after year. This is not true for all bots.

I'm not sure how this is relevant to the article and the specific issue. I don't disgree with what Google is doing in this case. And I understand why and where Google is headed.

The issue of who is allowed to crawl and who is not is something the commenter raised. It's a huge issue that people take for granted, in my opinion.