A site for solving at least some of your technical problems...
Weird HTTP requests from Googlebot
Submitted by Alexis Wilke on Wed, 04/06/2011 - 21:49
Somehow I receive many weird requests from one of Googlebot's. These use some very strange HTTP header, if I can even call that a header! modsecurity blocks all of them at once, but it just seems very strange since it has nothing to do with HTTP.
The two requests I have seen go like this:
0x16 0x03 0x01
0x80 0x2B 0x03 0x01
As you can see I show you the hexadecimal codes because these are controls!!! (Ctrl-V, Ctrl-C, Ctrl-A in the first request, and PAD, +, Ctrl-C, Ctrl-A)
I just have no idea what that is. I'm glad that modsecurity kicks those request out, but I'm wondering why am I receiving them in the first place?!
If you know what those are, I'd appreciate a comment!
Re: Weird HTTP requests from Googlebot
Yeah, I never really had a good answer for why a client would send binary codes like so.
However, with newer technologies, we now have HTTP/2 and HTTP/3 which both make use of binary codes. HTTP/2 is still over TCP, but has some jamming issues so Google decided to come up with a better solution and that's HTTP/3 which uses UDP packets.
So it is not unlikely that you are receiving HTTP/2 data.
The HTTP/2 protocol was introduced in 2015. Now that protocol works with all modern browsers. But if a hacker is trying to penetrate your server, they may be trying to send your server frames that would look like valid data in order to hijack an existing connection.
As for 10 years ago, Google already had a very first protocol version they called SPDY. That would send compressed data. Again, I would imagine some hackers were trying to get their way through by sending what could look like valid requests even if totally invalid to our servers (it could be valid for some server).
Re: Weird HTTP requests from Googlebot
I've been getting the first one (^V^C^A) in my log quite a bit, nearly half the time with just those three, but many times followed by other control characters and occasionally with a single plain ASCII character mixed in. Unfortunately, this is the only result I found on Google addressing the issue, and it's over 10 years old. :-(
None of the instances in my log (going back 8 weeks) show anything other than "-" for the bot/user agent/etc., so it doesn't appear (at least in my case) that they are coming from any actual Googlebots. The great majority of them, within the same burst of HTTP requests do something I don't allow (using an HTTP verb I don't support, trying to log in to a website that has no such facility, trying to execute an external command, etc.), but other than that, I'm mystified.