Steps Involved in HTTPS Transmission: Apache Server & Browser Handshake

[davee123]

Does anyone know the steps involved with an HTTPS transmission? Particularly with an Apache server? Is there some sort of a handshake for every call? Or is the key transmitted only once to a client browser per session? If so, what governs such a timeout, and how does the webserver identify each client? Does a browser typically validate the site's certificate on every call?

We've got a webserver that's "slow" sometimes for clients (30+ seconds), but we have no idea why. Each of our CGIs respond within 1-2 seconds of receiving a request, but we have reason to believe that "something" is delaying users initiating calls with us. We know the CGIs aren't filling up the machine they're running on, and we're told we aren't hitting the number of MaxClients for Apache. Could we be filling up the number of handshakes for Apache? Could their browser be taking too long to validate us with the Certificate Authority?

I'm honestly not even sure where to look for the slowdown, so I'm curious what the atomic steps are in the process of a typical HTTPS call.

DaveE

 

[Grep]

HTTPS is just HTTP over an encrypted TLS connection, so what you want to look at, really, is TLS:

http://en.wikipedia.org/wiki/Transport_Layer_Security

 

[Coin]

Is the CGI talking to a database server, like mysql or something?

Can you replicate the bad behavior on demand, say by simulating heavy load? If so, does the problem go away if you temporarily switch to HTTP?

What you are describing sounds like a pretty normal LAMP problem. Probably you are running out of some resource such as RAM or database client connections. I don't think knowing more about the internals of https/TLS will help you here (though I will say, I have seen some webservers consume more RAM when serving https than http, because of the additional work https requires done). I would try to see if you can replicate the problem under controlled conditions, and in either case see if anything weird is happening in 'top' when the problem appears (high load avg? low memory? a startling number of apache processes? Database server going nuts?). If the problem is you're running out of RAM I suggest investigating mpm-worker.

 

[Coin]

Also "validating with the certificate authority", in https/ssl/tls, does not require hitting the authority server and cannot time out. "Validating" means the web browser compares a certificate you sent it with a cached copy of the CA's public key or some such.

 

[davee123]

Is the CGI talking to a database server, like mysql or something?

The CGIs are doing very little. They make a connection (through an internal firewall) to a Perl server on a separate machine. We don't believe the internal firewall is overloaded, and we know the Perl server(s) similarly aren't overloaded. We also know the Apache "MaxClients" isn't being hit (or whatever it's called in Apache-- I'm forgetting the exact name).

Can you replicate the bad behavior on demand, say by simulating heavy load? If so, does the problem go away if you temporarily switch to HTTP?

We can't replicate the problem. We've tried various browsers (clients with problems have been using MSIE 6,7,8, which we've tested), both inside and outside our network. Some clients complain about it, but most don't.

We can't really experiment, which is unfortunate-- although we're trying to coordinate with clients to figure it out.

What you are describing sounds like a pretty normal LAMP problem. Probably you are running out of some resource such as RAM or database client connections.

If it were a resource issue, I would expect that our internal staff (and other clients) would have similar problems during heavy loads. But it seems that particular clients are repeatedly having issues. It's very unclear at the moment, since details from customers are sketchy. A few of them HAVE suddenly "gotten better", but some are constantly experiencing issues.

We've been running the site for roughly 12 years, we've had resource issues before which we could track, but this one seems very different. Recently (on the 11th), we changed:

1) The domain name. Shouldn't be a big deal.

2) The certificate. We now use TrustWave rather than Verisign. It's also a 256-byte key rather than a 128-byte key.

3) Some of the page references. Used to be that ALL the content was loaded from a single domain. Now, some of it is loaded from a SEPARATE domain (albeit it's all on the same actual webserver)

There's other things behind the scenes that we've changed (like which MySQL database we use, etc), but we can verify that those AREN'T causing the problem. The problem is visible before the MySQL connections are even established or used, and before any actual processing is done-- by the time the connection is received, the problem's already happening.

Honestly, I don't think any of the things that have changed ought to be causing a slowdown-- or, if they do, it shouldn't be taking 30+ seconds. Maybe a fraction of a second for the larger encryption, or some sort of strange browser config that raises security flags that we've got 2 different domains in the same page. So, I'm grasping at straws.

I will say that we've had various unexpected problems with TrustWave certificates. Some clients don't recognize them by default, and some automated software (Java mostly) similarly didn't recognize them and outrightly failed on page requests to us. Hence, my distrust of the certificate, and my suspicion that it could be HTTPS related. And given that I don't really understand the details of HTTPS, I'm curious what steps are involved so we can identify them.

For instance, from what I gather, Apache recognizes a particular IP/port combination for an HTTPS key. If our client has 2 computers on the same network, and they're going after our website, they'll come across as the same IP, but (I think?) different dynamic ports. However, they'll recycle them at some point-- IIRC, ports only go up to 2^15 or so? So, if one computer logs on and gets one HTTPS key at 13:44 using port 12345, but then their buddy logs on at 13:48, and ALSO gets the SAME recycled port of 12345, then our Apache server can't tell the difference between them, and sends the incorrect HTTPS key. Now, I expect that their network guys do something to make sure this doesn't happen-- but I have no idea.

Similarly, how does Apache store these IP/port keys? Is there a limit to the number that it stores, or a timeout associated with them? Does it delete them on a client signal? (If so, what signal?) Could it be that since we now are hosting on an increased number of domains on the same webserver (same amount of TRAFFIC, mind you, just now diversified to multiple domains) that we're hitting some boundary on IP/port keys? Does the IP/port key stored differently depending on the domain that the client entered? (It darn well better, I guess!)

Anyway, I'm not really sure where to go at the moment-- the certificate seems like the most LIKELY candidate, but really, nothing I can think of OUGHT to be causing the problem. Each possibility I can come up with either doesn't fit, or doesn't seem likely in terms of causing a 30+ second delay. 30+ second delays are typically resource issues (waiting for an available slot or outright timing out), but there's nothing I can find on our end that would seem to indicate a resource issue. And of course, we've never observed any problems, and neither have most clients, so it makes me inclined to think it's a resource issue on THEIR end, but I still don't see how.

DaveE

 

[Coin]

"The CGIs are doing very little. They make a connection (through an internal firewall) to a Perl server on a separate machine. We don't believe the internal firewall is overloaded, and we know the Perl server(s) similarly aren't overloaded. We also know the Apache "MaxClients" isn't being hit (or whatever it's called in Apache-- I'm forgetting the exact name)."

I would focus hard on this. There are lots of things that could go wrong here. A connection could fail (this would not *necessarily* require heavy load). If you are referring to the remote server by name your DNS server could be hanging on one out of every 30,000 requests.

"Either 1 second, or 30" is suspicious-- if its EXACTLY 30 and that isn't just a way for your users to express "a long time" then that sounds like some kind of timeout.

Incidentally note, running out of "a resource" in general does not *have* to imply being overloaded, especially since your cgi apparently takes a long time to complete-- I have seen untuned Apache do pathological things when cgi takes a long time to return or rather long enough that new requests are frequently coming in before the old request is done being serviced.

I think you just need more information about the problem. Here is one thing I would suggest. It is possible to configure your apache logs to include, for each request, the time taken to service the request (it's one of the % codes). This will include your cgi runtime. Get this added to your logs and then just wait a couple days and see what the distribution of request-complete times is. This will get you lots of potentially valuable information like, how often does the problem occur, when it occurs how long does the request take to complete, do long-return events come in "bunches" or are they randomly distributed, and once you start fiddling with stuff it can tell you "has the problem gone away", all without having to depend on unreliable users...

 

[davee123]

I would focus hard on this. There are lots of things that could go wrong here. A connection could fail (this would not *necessarily* require heavy load). If you are referring to the remote server by name your DNS server could be hanging on one out of every 30,000 requests.

Not really-- we put in a lot of error handling on that end years ago for precisely that reason. If there's failures, they get logged to the Apache error log, and that's not happening. Also, THAT part of the architecture hasn't been changed in ... about 4 years now (and has existed for about 9 years). Oh, and if that were timing out, we'd also see a 30 second delay in terms of when the CGIs themselves actually complete, which we're not seeing. You can tell when that happens because they get out of step in the Apache log, or by the times they post to the Apache error log (which is where we print our debug).

"Either 1 second, or 30" is suspicious-- if its EXACTLY 30 and that isn't just a way for your users to express "a long time" then that sounds like some kind of timeout.

What I wouldn't give for an answer to THAT question! Most of our clients say "your website is slow!" It's only after repeated prodding that we get any sort of description, which is still extremely vague.

especially since your cgi apparently takes a long time to complete

Well, that's expressly what's NOT happening :) The CGIs complete lickety-split. The delay appears to be between the time the client clicks on a link and the time the CGI actually begins to execute.

I think you just need more information about the problem. Here is one thing I would suggest. It is possible to configure your apache logs to include, for each request, the time taken to service the request (it's one of the % codes). This will include your cgi runtime.

I didn't know you could get the overall "time to service request" in the logs-- that would be very helpful in other instances, although I'm not sure it will help in this instance. Again, since the delay seems to take place before the call gets to the CGIs, I'm curious what steps are taken PRIOR to that point. If everyone's standing in line for a handshake with our Apache server, when they finally DO get around to making a call with us, it'll be really quick. But that probably wouldn't show up in the overall request service time (since, I'm guessing, the handshake is a separate interaction).

DaveE

 

[davee123]

So, for the curious, we tracked it down. We added %T to the Apache log files (total request duration) to verify that what we were seeing about the CGIs was correct. And it was. The CGIs indeed were running quickly.

BUT! It shed light on another problem: ExtJS. We had used ExtJS in the past, but not in a lot of places. Now, we're using it on ALL our pages. And here we were thinking that of COURSE the web requests for static content (JS, CSS, etc) were going to be speedy and not worth looking into. But no. Some of the Ext libraries were taking as much as 376 seconds to send to the client!

In particular, as with many JS libraries, there's a "debug" version and a "trim" version. The "trim" version removes all whitespace formatting, and shortens variable names to 1-2 letters, in an effort to make the file as small as possible. But it's an unreadable mess if you want to debug it. Hence, the "full" version is called "debug"-- it's 2.3 megabytes. And we were accidentally using the debug version.

Couple that with the fact that the disk on that machine is old and slow. So many calls to a huge file were apparently getting backlogged.

So, that explained why we were seeing a significant lag in time in-between CGI calls-- the JS couldn't run in order to make the next request because the JS library was still loading.

DaveE

 

[Coin]

So, for the curious, we tracked it down. We added %T to the Apache log files (total request duration) to verify that what we were seeing about the CGIs was correct. And it was. The CGIs indeed were running quickly.

BUT! It shed light on another problem: ExtJS. We had used ExtJS in the past, but not in a lot of places. Now, we're using it on ALL our pages. And here we were thinking that of COURSE the web requests for static content (JS, CSS, etc) were going to be speedy and not worth looking into. But no. Some of the Ext libraries were taking as much as 376 seconds to send to the client!

In particular, as with many JS libraries, there's a "debug" version and a "trim" version. The "trim" version removes all whitespace formatting, and shortens variable names to 1-2 letters, in an effort to make the file as small as possible. But it's an unreadable mess if you want to debug it. Hence, the "full" version is called "debug"-- it's 2.3 megabytes. And we were accidentally using the debug version.

Couple that with the fact that the disk on that machine is old and slow. So many calls to a huge file were apparently getting backlogged.

So, that explained why we were seeing a significant lag in time in-between CGI calls-- the JS couldn't run in order to make the next request because the JS library was still loading.

DaveE

DaveE, that's interesting, one thing you might want to try in future if you're not doing this already is installing Firebug (or installing Chrome and using the developer tools). Both these tools will give you elaborate graphs showing resource dependencies and load times.

 

[davee123]

DaveE, that's interesting, one thing you might want to try in future if you're not doing this already is installing Firebug (or installing Chrome and using the developer tools). Both these tools will give you elaborate graphs showing resource dependencies and load times.

Yep, got 'em! Also, a good one for Firefox is the Live HTTP Headers plugin-- let's you see the request headers the browser's making as it makes them, which is handy for some of the hidden headers that you might be sending.

Sadly, for this one, they didn't really help us. Nobody on our side was having the issue-- it apparently was a problem for clients that had a lot of traffic and who also had small browser caches. We asked our clients if they could install Firefox for debug purposes, but of course they don't have administrative abilities on their PC's, so it made it all the more annoying to debug.

DaveE