How to troubleshoot application timeout issues
Occasionally certain applications that proxy through the ADC-as-a-Service platform will encounter timeout issues. This happens primarily with long-lived connections, such as file downloads over http (TCP port 80) or https (TCP port 443) as well as SSH or SFTP connections, typically over TCP port 22.
Solving these timeout issues is generally pretty simple, but diagnosing them can be a challenge. So our remedy for solving should be used as your diagnosis tool. We highly recommend that if you are encountering application timeout issues, you increase the timeout values shown here in this KB article to a very high value to see if it goes away. If it does, you can then try reducing the values until you are comfortable with it. The lowest value possible is always the best setting.
Two locations for timeout settings
There are two primary locations where these timeout values are set. The first is in the public-facing port options dialog, and the second is in the back-end server/device port options.
Public Facing Port Options
To find the settings, each protocol/port has it’s own settings menu as shown below. This is where you will find it:
Once you click that little edit button, a dialog such as the one shown below will be visible.
The value you are looking for is the Client Idle Timeout. It’s true that the SSL dialog has another option below the Client Idle Timeout for SSL Session reuse, but that’s something completely different and has no impact on what we’re discussing here.
This Client idle Timeout value is set to 180 seconds by default. This means that if the client appears to be idle – that is, not interacting with the application for 3 minutes – the system will assume the client has left and will terminate the connection.
Of course, if the client does anything during that 3 minute window, the timer is reset. So if you consider a typical website, users generally click around. If they stop clicking around, the page will remain in their browser even past the 3 minute window, but then if they click through to a page after that time elapses, they will start a new session. Of course, for a website this may not be a big deal, and your server-side timeout might be a lot longer, so 180 seconds works in most situations.
Where it doesn’t always work is with those long-lived connections as mentioned earlier. In theory a long running HTTP download or SSH session should keep the connection active, but it doesn’t always. Sometimes there is just enough idle to trigger the timeout. Consider an FTP client, for example. If you upload a file and then wait for 3 minutes, the connection remains open waiting for further action. But if you wait longer than the 3 minutes and then transfer another file, it usually has to renegotiate a new connection since it’s been connected.
So try bumping up this value first when trying to diagnose connection timeout issues. The highest value is 7200 seconds and the lowest is 1. Bump it to 7200 (2 hours) just to see if that helps. That’s easier than adding a minute and testing over and over again. If it does, great! Start brining it down until you find a happy medium.
Device Timeout Options
There are additional timeout settings in the device itself. To find these, you need to edit your device by double clicking it from the left-side “All Devices” section within the configuration builder as shown below. If you hold your mouse over one of the devices, the menu provides instructions for getting into the edit menu.
Once the device is open, go to the “Ports and Protocols” tab to view the settings there on a per-port basis as shown below:
Here you will find two timeout settings for every port. The Client Idle Timeout is the client-side connection, which generally mirrors the public facing port options we just discussed above. The maximum value for this back-end connection is 1800 seconds (30 minutes). Again, we recommend bumping this up to the maximum value as your first test.
The Device Idle Timeout is for the back-end connection between the Total Uptime proxy and the device. Set this to the maximum of 1800 seconds as well.
In Conclusion
Hopefully the above is helpful in troubleshooting application timeout issues. If you still encounter issues after increasing these values, then it is possible you have something else going on. It’s rare, but we’re happy to help troubleshoot further. Create a support case and we’ll see if we can run a live traffic packet capture to dig into this further.