Since the release of iOS 5 we’ve received several bug reports about images randomly displaying the wrong image(s) on our front page for smartphones. It seems to be completely random and could affect any of the images, anywhere and anyhow. During one week we would receive anywhere between two and ten reports about this error. In the same time period we would do about 4.5 million page impressions(2.9 million of them from iPhone) on the front alone.
Safari has replaced the LIVE-icon with an image of Mancini
Our front page is manually edited with a tool called Dr. Front, so we started off with eliminating possible sources like bad HTML markup, duplicate filenames and human error. No errors could be found in the markup and there were no colliding filenames. Any of these sources would also mean that this problem would occur in a more consistent manner across different browsers and OS.
The common denominator for all reports were iOS 5, and most of them with slow and lossy internet access.
HTTP Pipelining
We recently discovered that iOS 5 enabled HTTP Pipelining by default. In short terms HTTP Pipelining enables the browser to send several HTTP requests on the same connection before receiving the responses. It can even send new requests while still receiving an earlier response. Pipelining is still pretty new; Opera was the only browser that enabled it by default, Firefox have support that’s disabled and Chrome will support it in Chrome 17.
Firefox’ implementation seems to never send a new request while receiving a response while Opera and Safari do. Even with extensive testing with both Firefox and Opera we have never seen the images get replaced.
PS! We also found a small gotcha in Firefox, you MUST send the Server-header. Firefox refuses to use pipelining against a server it does not know.
Our infrastructure
When sending a request to http://touch.vg.no you will first hit our F5 Big-IP load balancer. The Big-IP terminates the connections and use layer 7 load balancing to determine where the request should be forwared. In this case that is a Varnish cluster consisting of 6 servers. These talk to different Apache servers, depending on the hostname. Static elements are served from static.vg.no will go through the same load balancer, the same Varnish cluster but end with a Lighttpd server.
What happens?
So far we’ve only been able to reproduce the bug two times while sniffing the traffic with Wireshark. When Safari switches the image(s) it’s always images in the same TCP stream and in both recorded cases when it sends a request before the previous response is complete.
What Safari does is that all the image(s) that are requested in the middle of a response end up containing the image currently being transferred. It also uses the cache headers of the same image thats being transferred, so if the cache headers (Expires/Etag) allows it, Safari will keep this image (with the wrong content), but with the right filename. Therefore a “soft” refresh without emptying the cache will still display the wrong image.
Tcpdumps from Wireshark:
In this dump the problem occurs in stream 8 when requesting “/drfront/images/2011-12/12/86-a17ea747-830df041.jpeg“. This image is not in the cache and gets downloaded. During the transfer Safari requests “/gfx/snippetMinmote590.png?20091130“. Even if this is a 304 Not Modified response the result is that the local version of “snippetMinmote590.png” contains and will be displayed with the image content of “86-a17ea747-830df041.jpeg”.
In this dump it is stream 9 and the exact same behaviour, just with different elements. “/drfront/images/2011-12/12/86-e951cc7b-4ad1fd2a.jpeg” results in new download and in the middle “/gfx/ikon_vgtv.png” is requested. This will also result in a 304 Not Modified but contain “86-e951cc7b-4ad1fd2a.jpeg”.
The solution?
We don’t know yet. We can’t simply turn off pipelining as the only requirement is the “Connection: Keep-alive” header in the response. If the browser discovers keep-alive and supports pipelining it is going to use pipelining. We can increase the amount of hosts we deliver our images from, in an attempt to reduce the usefulness of pipelines, but that is really not a good solution.