Prevent X-Forwarded-For Spoofing or Manipulation
When using an inline proxy like our ADC-as-a-Service or Web Application and API Protection service, you’ll often want to know the original client IP address for security, to track in your logs for stats or for other reasons.
In our KB article entitled Getting the original client IP with X-Forwarded-For in your code, we’ve already discussed how to enable and capture this information using an HTTP header called X-Forwarded-For (XFF). But in this article, we’re going to discuss an important reason why you may wish to rename this header to something less common.
In our other articles, we stick with the industry standard and frequently suggest naming this header X-Forwarded-For because many appliances, systems and servers have functionality built-in to capture this data by out-of-the-box with ease. And while this might be the easy thing to do, it may not be the most secure. Many other cloud providers do not allow you to change the name of this header, but at Total Uptime, we do, and in here we’ll explain why.
The most common X-Forwarded-For header problem
Have you ever seen an X-Forwarded-For HTTP header look like this:
“X-Forwarded-For: 192.168.1.100, 203.0.113.14”
In the above sample, there are two IP addresses in the header. If at first glance you think this is invalid, it’s actually not. According to IETF RFC 2616, Section 4.2, multiple proxies between the client and your server are permitted to simply append the IP to the header. So here, the first IP might be an on-premise proxy, and the 2nd IP is probably the one that the Total Uptime network inserted (assuming you’ve enabled this feature, of course).
If your web application is coded to simply grab the first or leftmost IP address in the list, the above is one example where you’d be grabbing the wrong one. A non-routable private IP address won’t give you any valuable information about the customer. And even if it is a routable IP, do you know that it is the right one? If you think simply modifying your code to grab the rightmost one from the string is the way to go, you could also run into trouble. Do you have another load balancer or proxy between Total Uptime and your servers? If you do, you’ll be grabbing the IP of that device!
A number of expert posts online like to recommend writing a script to grab the leftmost IP that is not a private, non-routable IP address, and there is merit to that thinking, but what about a double proxy? Perhaps the same user with an on-premise proxy is also using an Internet proxy to obfuscate his/her whereabouts? Now the first two are not really the ones you want, but rather it might be the 3rd one.
X-Forwarded-Garbage?
Because this is an optional header (and one with X- at the beginning too, which is supposed to mean it has yet to become a standard, but that should be a separate rant and actually something that this IEFT draft recommended be deprecated), it doesn’t even have to be an IP address! A user could put garbage in there when making the request. If you’ve ever played with your browser’s developer tools, you know that you can specify headers when making requests, so why not specify the X-Forwarded-For header and put something useless in there like “localhost” or to be nefarious, an IP address that is legitimate, but not yours. The real spoofer likes to use proxy chaining too, and this combination makes it quite difficult to find the real IP.
What if we go the extra mile and put in a null byte? Now we’re getting into sneaky territory! Perhaps the user really wants the server to throw a 400 error so they can fingerprint the underlying operating system, fill up the log files or some such. How about setting it to be a string of valid IP addresses, none of which are yours. If your code is configured to grab the 2nd one from the left, do you know what you’re grabbing is legitimate?
A Real Solution to Prevent XFF Spoofing
As you can now clearly see, X-Forwarded-For has a number of limitations and vulnerabilities. Well, maybe vulnerabilities is the wrong word, but we can definitely agree that it could be quite unreliable. So what’s the solution? How about a different HTTP header name!
Something that Total Uptime supports where other load balancers or ADCs do not is allowing you to set any name for that header you want. If you go to the device edit dialog and specify something different, like TUT-IP (not a good idea, that’s just an easy example), you can now configure your server (or intermediate proxy) to look for that instead. Why is that important? Because it’s unique.
Now, when connections are sent from the Total Uptime platform to your devices, any existing X-Forwarded-For header is passed along unchanged. The TCP Source IP is then entered into your new HTTP Header (e.g. TUT-IP) so you can more reliably determine the actual connecting IP address to our platform.
But wait, there’s more!
There are security experts out there who will say this is far from secure, and there is truth to that, but it is clearly more difficult to circumvent than the common implementation, and that’s the first place to start. Then these experts will say that they can easily determine what the new HTTP header name is by making an HTTP request with the TRACE method.
And that’s true, such a strategy might reveal with that new HTTP header name is because the TRACE method simply echoes back to the client whatever string has been sent to the server. So when the nefarious individual sends his request, the load balancer adds the new TUT-IP header and then sends it to the server and the server may oblige and send it back to the nefarious individual.
Once the attacker knows what the header is, they can spoof it again. But we’ve thought of that too, and if your modern web server doesn’t have TRACE disabled by default, you can go into the public facing port options in our UI and turn off the HTTP trace method. While you’re there, you may want to turn off any of the ones you don’t need, which should really only be GET, unless you accept form POSTs too.
Conclusion
In conclusion, it’s quite possible to shore this up and make it far more difficult to spoof the connecting IP address.
Looking for code samples for obtaining the X-Forwarded-For header? Check out this article.