I doubt it would help. My employer uses Akamai as a CDN & security provider for our websites. Their bot analysis tools regularly flag distributed bot activity that can come from a handful or a few thousand IPs. They do a range of browser fingerprinting, TLS fingerprinting, etc. to uniquely identify traffic across ranges of IP’s. I’m sure Google/Youtube has the ability to do this as well.

Any given client would need to regularly randomize the order of headers in requests, randomly include/exclude optional headers, and also randomize TLS negotiation to try to circumvent all the fingerprinting these big corporations perform.

DigiCert recently was forced to invalidate something like 50,000 of their DNS-challenge based certs because of a bug in their system, and they gave companies like mine only 24 hours to renew them before invalidating the old ones…

My employer had an EV cert for years on our primary domain. The C-suites, etc. thought it was important. Then one of our engineers who focuses on SEO demonstrated how the EV cert slowed down page loads enough that search engines like Google might take notice. Apparently EV certs trigger an additional lookup by the browser to confirm the extended validity.

Once the powers-that-be understood that the EV cert wasn’t offering any additional usefulness, and might be impacting our SEO performance (however small) they had us get rid of it and use a good old OV cert instead.

If you have ssh open to the world then it’s better to disable root logins entirely and also disable passwords, relying on ssh keys instead.

Port 22 is the default SSH port and it receives a TON of malicious traffic any time it’s open to the whole internet. 20 years ago I saw a newly installed server with a weak root password get infected by an IP address in China less than an hour after being connected to the open internet.

With all the bots out there these days it would probably take a lot less time if we ran the same experiment again.

Well OPSEC is the stated cause. Who knows how the person was initially identified and tracked. For all we know he was quickly identified through some sort of Tor backdoor that the feds have figured out, but they used that to watch for an unrelated OPSEC mistake they could take advantage of. That way the Tor backdoor remains protected.

Exactly. Tor was originally created so that people in repressive countries could access otherwise blocked content in a way it couldn’t be easily traced back to them.

It wasn’t designed to protect the illegal activities of people in first world countries that have teams of computer forensics experts at dozens of law enforcement agencies that have demonstrated experience in tracking down users of services like Tor, bitcoin, etc.

Oh there are definitely ways to circumvent many bot protections if you really want to work at it. Like a lot of web protection tools/systems, it’s largely about frustrating the attacker to the point that they give up and move on.

Having said that, I know Akamai can detect at least some instances where browsers are controlled as you suggested. My employer (which is an Akamai customer and why I know a bit about all this) uses tools from a company called Saucelabs for some automated testing. My understanding is that our QA teams can create tests that launch Chrome (or other browsers) and script their behavior to log into our website, navigate around, test different functionality, etc. I know that Akamai can recognize this traffic as potentially malicious because we have to configure the Akamai WAF to explicitly allow this traffic to our sites. I believe Akamai classifies this traffic as a “headless” Chrome impersonator bot.

When any browser, app, etc. makes an HTTP request, the request consists of a series of lines (headers) that define the details of the request, and what is expected in the response. For example:

GET /home.html HTTP/1.1
Host: developer.mozilla.org
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:50.0) Gecko/20100101 Firefox/50.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Referer: https://developer.mozilla.org/testpage.html
Connection: keep-alive
Upgrade-Insecure-Requests: 1
Cache-Control: max-age=0

The thing is, many of these headers are optional, and there’s no requirement regarding their order. As a result, virtually every web browser, every programming framework, etc. sends different headers and/or orders them differently. So by looking at what headers are included in a request, the order of the headers, and in some cases the values of some headers, it’s possible to tell if a person is using Firefox or Chrome, even if you use a plug-in to spoof your User-Agent to look like you’re using Safari.

Then there’s what is known as TLS fingerprinting, which can also be used to help identify a browser/app/programming language. Since so many sites use/require HTTPS these days it provides another way to collect details of an end user. Before the HTTP request is sent, the client & server have to negotiate the encryption to use. Similar to the HTTP headers, there are a number of optional encryption protocols & ciphers that can be used. Once again, different browsers, etc. will offer different ciphers & in different orders. The TLS fingerprint for Googlebot is likely very different than the one for Firefox, or for the Java HTTP library or the Python requests package, etc.

On top of all this Akamai uses other knowledge & tricks to determine bots vs. humans, not all of which is public knowledge. One thing they know, for example, is the set of IP addresses that Google’s bots operate out of. (Google likely publishes it somewhere) So if they see a User-Agent identifying itself as Googlebot they know it’s fake if it didn’t come from one of Google’s IP’s. Akamai also occasionally injects JavaScript, cookies, etc. into a request to see how the client responds. Lots of bots don’t process JavaScript, or only support a subset of it. Some bots also ignore cookies, and others even modify cookies to try to trick servers.

It’s through a combination of all the above plus other sorts of analysis that Akamai doesn’t publicize that they can identify bot vs human traffic pretty reliably.

Exactly. The only truly effectively way I’ve ever found to block bots is to use a service like Akamai. They have an add-on called Bot Manager that identifies requests as bots in real time. They have a library of over 1000 known bots and can also identify unknown bots built on different frameworks, bots that impersonate well known bots like Googlebot, etc. This service is expensive, but effective…

Not easily. The scammer likely has your current address & contact info, but knows nothing about your history.

To confirm your identity when you contact these reporting agencies they will use details from your credit history by asking detailed questions the scammer likely won’t know. For example it might be questions like these:

• What kind of car did you purchase in 2005?

1. Honda
2. Ford
3. Saab
4. Jeep
5. None of the above

• Which one of these companies did you work for previously?

1. IBM
2. Pizza Hut
3. Macy’s
4. Jiffy Lube
5. None of the above

They’ll throw 3 or 4 questions like these at you that you’ll have to answer correctly. They might involve places you used to live, banks you have had accounts with, etc. The chances of a scammer with your SSN knowing all these details about you is pretty tiny.

The credit monitoring companies have your up-to-date contact information (and verified) when you put the freeze in place. Now, should a third party try to open an account, etc. in your name it should be blocked from happening and the credit monitoring company should contact you.

If a scammer tries to unfreeze or otherwise modify your account with them they should also contact you.

If/when they contact you or you request your account be unfrozen then they’ll use old credit history to confirm your identity. These are a series of three or four random questions that a scammer is unlikely to know. For example they might ask you what kind of car you purchased in 2005, then give you 4 options, like Ford, Honda, Jaguar, or BMW, and then also a “nine of the above” option. Then they might ask you which of the following street addresses you used to live at, and list 4 seemingly random addresses, one of which you might have lived at.

Years ago I worked at a company where they based server root/admin passwords on song lyrics. The person who came up with it clearly liked classic rock. I still remember at least one of them:

4ThoseAboutToRockWeSaluteYou!

I don’t understand why Cloudflare gets bashed so much over this… EVERY CDN out there does exactly the same thing. It’s how CDN’s work. Whether it’s Akamai, AWS, Google Cloud CDN, Fastly, Microsoft Azure CDN, or some other provider, they all do the same thing. In order to operate properly they need access to unencrypted content so that they can determine how to cache it properly and serve it from those caches instead of always going back to your origin server.

My employer uses both Akamai and AWS, and we’re well aware of this fact and what it means.

robots.txt is 100% honor based. Well known bots like Googlebot, Bingbot, etc. definitely honor them. But there are also plenty of bots that completely ignore them.

I would hope the bots used to collect LLM training data honors them, but there’s no way to know for certain. And all it really takes is one bot ignoring it for the content of your website to end up in a random set of training data…

Try using “curl -A” to specify a User-Agent string that matches Chrome or Firefox.

deleted by creator

I hope you realize that virtually every CDN provider does the exact same thing in similar ways. Sites that use Akamai, AWS, Google cloud, Fastly, etc. all give those companies access to unencrypted content. It’s just how CDNs work…

‘21 Model Y long range. Overall it drives well, and the supercharger network is really nice. We took it on a trip up & down a good portion of the east coast last year and never had any issues charging it. We have a couple 30 lb dogs that love going for rides, so things like dog mode are really nice as well.

Things I really do not like:

• The reliance on cameras for all sorts of features like auto high beams and auto wipers on top of traffic aware cruise control (aka autopilot) (and full self driving, if you have it). I regularly have the wipers go off on clear, sunny days. The auto high beams are so unreliable I don’t use them, and that means no autopilot at night. I have no faith in even trying out FSD because of how glitchy everything else is.
• The minimal use of physical controls. I have to take my eyes off the road just to switch wiper speed/mode.
• Software updates have, more than once, changed my settings for things like autopilot without warning, and I’ve only discovered it when driving and turning autopilot on.
• The maps have lots of routing issues. It shows roads in my neighborhood that don’t yet exist (new development under construction), regularly routes me wrong ways (there’s a left turn near my home that it thinks it can’t take so it tries to route me two sides if a triangle as a result), and on our road trip we found a stretch of highway that it thought it couldn’t drive on and kept trying to route us along side streets. And there’s no way I know to report these issues so they can be fixed. Apps like Waze make that trivial.

Pretty much all of these are reasons why I refuse to even try FSD and discourage others from using it. About the only way I’ll give it another chance is if a truly independent third party tests it and says all these issues have been resolved.

I admit I own a Tesla. Given all the recent erratic behavior:

• Not only will I not recommend Teslas to anybody who might ask about it, I will warn them to look at company & CEO behavior over the years, and actively discourage others from buying one.
• When the time comes, I will not be replacing my current car with another Tesla. I will still likely go with an EV, but by then there should be significantly more good (better) options available.

About the only way I’ll change either of these will be for Elon to step down and completely remove himself from any control over Tesla. But I don’t see that happening and I certainly won’t be holding my breath.