@nipos @resist1984 But that's why they are respecting the same flags as search engines. If you don't allow searchengines to index a part of your page, wayback machine won't do so. And without a robots.txt or something comparable, it won't happen...

@frommMoritz @nipos @resist1984 no, they aren't respecting these flags anymore

https://www.heise.de/newsticker/meldung/Archivierung-des-Internets-Internet-Archive-ignoriert-kuenftig-robots-txt-3693558.html

@dadosch
I think it was later clarified, that it meant retrospect deletion.
Before the change the wayback machine had the problem that when someone new bought the domain and restricted the access via robots.txt, the whole archive of the site of the previous owner was deleted.
This was not only bad because of accidental deletion, but people could even intentionally destroy parts of the archive by buying old domains.
For just denying access to the archive bot have a look at the "noarchive" flag.

@nipos @resist1984 they only save websites that allow crawlers. So disabling crawlers for website means it won't be saved.

@nedelne_rano @resist1984 @frommMoritz There's a big difference in making content searchable or cloning it completely forever.If it's in the search and the author decides to delete it,search links will return Error 404 after clicking it.Yes,there may be some other cache but I'm talking about pure search results.This isn't problematic.If you delete it and there's an exact copy of the page which isnt removed,this is a problem in some cases.

@nipos

wayback machine respects robots.

https://blog.reputationx.com/block-wayback-machine

And claiming that it is "one more reason to use cloudflare" is kinda wierd.

And yes, you can get your site removed from wayback machine.

@nedelne_rano @resist1984 @frommMoritz

@fortune
@nipos @nedelne_rano @resist1984 @frommMoritz
You can also request do not be added anymore

@fortune @nipos @nedelne_rano @resist1984 @frommMoritz

https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/

@nipos
@nedelne_rano @resist1984 @frommMoritz
Google cache

@nipos @nedelne_rano @resist1984 @frommMoritz
Also one of my hobbies is archiving

@nipos @nedelne_rano @resist1984 @frommMoritz and yet history is important. There is a balance to be found here somewhere.

If you don't want your information public, don't make it public. Facebook already disallows crawling, so does Twitter, by the way. So your point is mostly moot anyway.

CloudFlare unilaterally deciding to screw over one of the main projects keeping Internet history is not that balance.

@nipos
Whether your dislike of the Wayback machine is justified or not -- putting them behind access restrictions like this only limits access by people who care about privacy, and disabled people -- hardly "just dessert"
@resist1984

@Mr_Teatime @resist1984 Nope because I explicitly whitelisted Tor in my Cloudflare settings resulting in Tor users being able to access the site without seeing a shitty Google captcha 😉

@nipos @Mr_Teatime #CloudFlare w/ #Tor whitelisted is even worse, b/c then Tor users don't know they are interacting w/a CF MitM. Tor users then unwittingly support a Tor adversary.

@Mr_Teatime @nipos archive.org does #Tor users a service b/c it helps bypass the #CAPTCHA (if needed) & ensures the target site is not rewarded w/traffic or interaction.

@nipos @Mr_Teatime there is a very useful browser plugin that detects #CloudFlare & automatically redirects to the archive of the page.

@resist1984
Do you have the name (or a link) to hand?
@nipos

@Mr_Teatime @nipos the Firefox plugin that redirects CF sites is called "Block Cloudflare MITM Attack" and is posted here: https://addons.mozilla.org/en-US/firefox/user/14218621/. Description is in cyrillic but don't let that scare you off. This plugin will outright block CF sites: https://gitlab.com/gkrishnaks/cloud-firewall

@resist1984 I was recently talking about Cloudflare with non-techie colleagues, and was trying to summarise what's wrong with Cloudflare. How would you put it?

The main issue is that it sees so much of all of the traffic, and that it serves as an obstacle to anonymous/TOR browsing. Anything else that should really be added and would likely make sense to them?

The starting point was that they saw it as something that helps people fend off DDoS attacks

@g at a high level, #CloudFlare is very similar to #SpamHaus. In both cases you have a vigilante extremist org so fixated on attacking their enemy that they have no regard for collateral damage to harmless users. Ppl cannot protect their own #privacy by running their own mail server b/c of SpamHaus, & ppl cannot protect their own identity b/c CF DoS's *all* #Tor users.

@g #CloudFlare also harms non-Tor users by MitMing the connection. CF sees every username & unhashed password even when a TLS padlock is present.

@g a side-effect of CF DoSing #Tor users is to push ppl off Tor (to escape the #CAPTCHAs). That's disabling ppl of their #privacy protection.

@g w.r.t finding off DDoS attacks, note 1st that any CDN will offer that.. no reason to use #CloudFlare. Also, once you have a DDoS attack, CF is no longer gratis. CF will force you to upgrade to premium b/c the attack counts toward your bandwidth allowance.

@g also realize that a DDoS attack is impractical from #Tor, b/c the exit nodes are too few & bandwidth is insufficent for that.

@g so it's crude & reckless to DoS #Tor in an anti-DoS effort. They want ppl to believe Tor is a DoS threat, but they're actually on an anti-bot agenda. Yet not all bots are malicious.

@g I could write a book on this. I'll also mention that #CloudFlare uses #Google's #CAPTCHA, & that's a #privacy abuse in itself. Google links your logged in cookie w/the CF site the CAPTCHA is on.

@resist1984 oh, right, makes sense, didn't think of it

@resist1984 yes, good point, so it's completely unnecessary... which makes clear this is intentional

@resist1984 Right. But other CDN will likely have similarly problematic behaviour. Let's say I'm a news organisation. Now google has a programme whereby it gives DDoS protection gratis to news organisations. If you do have DDoS problems, then this seems like a deal. But of course, google being google, it comes with all sorts of its own issues.

Full disclosure: my employer currently benefits of DDoS protection for free from https://equalit.ie/ (at first sight, it looks good, but I never really looked into that, will see). Targted DDoS had been a major issue in a few occasions

@g some problems like having visibility on all traffic are shared across all CDNs, so it's best to avoid CDNs entirely if possible. But if you must use a CDN, #CloudFlare is the worst of the worst.. it shelters criminals and harms #humanrights

@resist1984 well understood, that's a separate Cloudlfare controversy, but I'm aware of it

@resist1984 that's a great point, as indeed it impacts so much the browsing experience

@resist1984 I didn't think of it. And to be honest, I don't really understand how... admittedly, because I have a limited understanding of how CDN work. Don't they just pass things over using the host certificate? I'm obviously misunderstanding how this works

@g the TLS tunnel terminates at #CloudFlare, so CF sees all traffic. It must work that way. If CF were to simply proxy all encrypted traffic to the origin, then it would fail to relieve the originating server of workload.

@g this article covers it in detail: http://cryto.net/~joepie91/blog/2016/07/14/cloudflare-we-have-a-problem/

@resist1984 thanks a lot, will read!

@g np. And note there may or may not be a 2nd tunnel between the originating server & CF, but in either case the end user sees a padlock

@resist1984 right, in principle, you have no way to know what happens on the other side of the tunnel, good point

@resist1984 thanks a lot for making these points much clearer to me, much appreciated!

@resist1984

So if I have a SSL certificate from Let's Encrypt setup, but still sit behind Cloudflare, is my data still compromised?

@resist1984 oh, right, so TLS tunnel terminates there, and if it looks good, it's then passed over in a new tunnel... got it

@gerowen indeed, #Google is using #CAPTCHA to exploit labor, which in some situations in the US amounts to a 13th amendment issue (involuntary servitude). Google's CAPTCHA also gives preferential treatment to logged in Google users, which adds to the tracking of those users by associating them to the site where the captcha is presented.