Rather like my series of Windows 10 presentations for the user group community, here’s hoping that this is actually (really!) going to be the final word on IE10+ Cookies and History roaming…
You may remember the series of articles I did previously on the challenges of persisting Cookies and History between sessions, particularly on XenApp or non-persistent XenDesktop, but equally, in any environment where users move from device to device. Part #1 and part #2 concentrated on testing the actual capabilities of various methods for achieving roaming, whereas part #3 ran through some tricks you could use to restore the previous functionality. Today, we are going to run through the actual mechanics of what is happening in Internet Explorer browsing sessions, how these hang together in different situations, and offer you ways of managing this with IE11 on Windows 10 Anniversary Update, which (not surprisingly, and very much in keeping with Windows 10 in general) brings a whole host of new challenges to the table.
But first things first.
Introduction
Cookies, as we should all well know, are small files that allow browser sessions to record stateful information. They’re probably much more of a pain for users in the EU then they are for users beyond them, as every website that utilizes any form of tracking cookie is required by EU law to get the user to provide their consent to it. So if Cookies don’t persist reliably, every time a user in the EU visits the BBC website (for instance), they will see the intrusive cookie warning reappearing (see below). However, even those beyond the borders of the EU will be familiar with Cookies when used as authentication cookies, providing the saved logon information for browsing sessions when used with services such as Google.
History is another thing entirely, and should not be confused with “address bar history” (which is what you see, suggestions-wise, if you’re like me and prefer to type directly into the browser’s address bar). For purposes of this article, and to align myself with Microsoft’s notation, I will refer to “address bar history” as TypedURL History. The “History” that we are discussing from hereon in is the browser’s own History feature, accessed in IE by clicking on the “star” icon and then choosing the History tab, or simply pressing the Ctrl-H keyboard shortcut from the browser. There is an image of it below.
So, now we are clear on what we mean by Cookies and History, what did we learn from the previous set of articles that were published?
Here are a rundown of the key takeaways from the previous set of articles:-
- Everything works fine as long as you use a local profile, never use a different device and never have your profile removed or corrupted 🙂
- IE10 or higher makes persisting Cookies and History data very problematic compared to earlier IE versions
- In order to roam Cookies, you need to use a roaming profile or spoof a roaming profile before starting browsing
- In order to roam History individually, or Cookies and History together, you need to use a tool like AppSense Environment Manager or another UEM tool that can manage to deal with the WebCache file
- Folder Redirection, for these types of folders (lots of small files and lots of reads and writes), is BAD and gives inconsistent results
The key to everything is the “WebCache” file, a Jet Blue database with a name usually along the lines of webcachev01.dat, which performs various Internet Explorer functions that were previously the domain of old-fashioned index.dat files.
The WebCache file
This file sits in %LOCALAPPDATA%\Microsoft\Windows\WebCache, and by default is marked as both Hidden and System, making it invisible unless you’ve set up the required options in Explorer. The default size of this file on Windows 10 weighs in at around 25MB.
When the user is using the session, the webcache file is locked in use by a Scheduled Task called CacheTask which is so secretive, its Action is referred to by the cryptic name of “Custom Handler”. The task starts when any user logs on to the system.
Now, let’s get into the inner workings of the webcache file.
The webcache file doesn’t just hold Cookies and History. It contains a whole load of other stuff too. For some things, the webcache file provides complementary pointers to files held on disk in other parts of the user profile. Here’s a simplified rundown of its contents:-
- Internet Explorer History
- Pointers to Cookies which are held in %LOCALAPPDATA%\Microsoft\Windows\INetCookies
- Pointers to Temporary Internet Files which are held in %LOCALAPPDATA%\Microsoft\Windows\INetCache
- Pointers to DOMStore (Document Object Model storage) objects which are held in %USERPROFILE%\AppData\LocalLow\Microsoft\Internet Explorer\DOMStore
(which are kind of like http cookies, except that the accessibility for different windows is somewhat easier) - Details of Enterprise Mode (EMIE) settings
- Details of Compatibility Mode sites
- On Windows 10, each Modern App also accesses the webcache file, each using separate containers inside it to store cookies, History and Temporary Internet Files
The webcache file is relied on, along with files from other parts of the user profile, to provide access to all of this data which makes up a sizeable part of the Internet Explorer “experience”. This combination of both filesystem entries and the webcache file itself goes some way to explaining some of the aberrant behaviour which we saw in the earlier articles. For instance, if you redirect Cookies and History but do not persist the webcache file itself, you have some of the actual files present but not all of the required pointers to these files. This results in odd and sometimes contradictory behaviour.
However, some of you may be aware that even though History is ostensibly fully stored within the webcache file, there is still a folder in the user profile (%LOCALAPPDATA%\Microsoft\Windows\History) that appears to contain History entries. See the image below for an example of this.
If you dig a little deeper though – such as browsing to the same folder from a command prompt – you will see something completely different
The folder simply contains an empty container.dat file. It is clear that the actual “History” folder in the user profile seems to use some sort of shell extension, possibly with a COM surrogate process, to provide Explorer a “view” into the webcache file itself. This is possibly for backwards compatibility reasons, to allow applications to still write to the webcache through the filesystem maybe.
As for Cookies, they are still physically stored on the filesystem, in %LOCALAPPDATA%\Microsoft\Windows\INetCookies, but without the accompanying pointers in the webcache file, they are useless. If you remove either the Cookies themselves or the webcache pointers to them from the user profile, all of the Cookie warnings will reappear next time you visit a website.
But how does a roaming profile, or a spoofed roaming profile, still manage to provide Cookie persistence, even though they are supposed to be stored in %LOCALAPPDATA%? For those of you who don’t deal a lot with standard Microsoft roaming profiles, you can only exclude directories from a roaming profile, not include them. So anything stored in %LOCALAPPDATA% usually can’t be persisted…
Microsoft got around this with a bit of a fudge, which we covered to some degree in part #3 of the original series of articles. Rather than reading from %LOCALAPPDATA%\Microsoft\Windows\INetCookies, when a profile type is roaming, this is switched to %APPDATA%\Microsoft\Windows\INetCookies. The Cookies are then stored with a combination of .dat and .cookie file extensions rather than the normal format. When Internet Explorer is launched on a new device, the .cookie files are copied from %APPDATA%\Microsoft\Windows\INetCookies into %LOCALAPPDATA%\Microsoft\Windows\INetCookies and imported into the webcache file. When Internet Explorer closes, the changed cookies are copied back to the %APPDATA% location, allowing persistence of Cookies (but not, sadly, of History or anything else).
Issues
So, we can currently attack the persistence issue in two ways:-
- Use AppSense Environment Manager (either Personalization or Policy), or another compatible UEM tool, to roam the webcache file and associated folders
- Use a Microsoft roaming profile (or spoof one) to roam Cookies, and perform a scripted persistence of the webcache file to maintain the rest of the associated browser settings
This works well enough for most instances, but, over time, some issues arise.
Primarily, in XenApp/RDSH environments or other non-persistent settings, the webcache file gets bigger and bigger. Even light browsing seems to fill it up quickly. Within a few days, my test machines have webcache files of 50MB+. AppSense Personalization Server compresses the file, which works well enough when it is new and filled with white space, but over time you will see even this start to increase. Windows 10, because of the overhead of storing Modern App settings as well (assuming you use them), only makes it even worse. After a week of light browsing, Windows 10 machines, particularly those using Edge, will see a lot of increase.
Now, first of all there is the extrapolation of this across thousands of users and the increase in storage capacity required. Do you want to be centrally storing a file that could be about 500MB for thousands of users? Secondly, if you are copying this back in at logon time, there is the drag on the logon as the file is brought across the network. The problems soon mount up.
I suppose you could just delete the file when it gets too big, and simply force your users to accept the fact that the Cookie warnings will reappear and their History will be cleared (although TypedURL History should still be fine). But aside from the negative user experience and end-user frustration you’re going to cause, what about settings you may have configured that are also stored in the webcache file? Enterprise Mode settings, Compatibility Mode lists – they’re all in here. So not only do your users get annoyed, they might also have to phone the helpdesk to get things working again – it’s clear that deletion isn’t the best option around.
Could you maybe edit the file or clean it up? Sure – if you’re feeling brave. There is certainly content inside the webcache file that is completely extraneous – for instance, the pointers to Temporary Internet Files, which are utterly useless, because who in their right mind roams Temporary Internet Files? (Even worse, if you want to see some hideous side-effects, try redirecting them!) However, if you break anything within the file’s structure when you’re trying to edit it or clean it up, Windows 10 (and probably earlier versions too) – like it does with many things – simply recreates it from scratch, getting you firmly back in the same boat of deletion that you’re trying to avoid. Isn’t there possibly a better answer?
WebCache Manager
Well, as of right now, there is (just one!) other answer. Ex-AppSense bod Pete Jones has created a tool called WebCache Manager, available from Avanite. Currently it runs as a simple command line, but will soon be available in a version that runs as a service. It takes over management of the webcache file for you, so can clear up History and Cookies down to a specific number of days, manage the Temporary Internet Files side of the database, and even perform a full defrag of the file to save space (not just a compression, a full defragmentation).
Also, how much History is enough? How many users would notice if you removed all but their last month’s worth of History? Using WebCache Manager, you can run a quick logoff command or script which will clean out as much or as little of the file as you require.
Installing WebCache Manager is a piece of cake – just run the setup.exe file or deploy it through your tool of choice, and it’s on the endpoint in about ten seconds flat.
Here’s the command we ran on our test Windows 10 clients as a logoff script (you could also call this through AppSense Environment Manager or in fact any way you feel like)
The /h 21 parameter tells it to maintain the last 21 days of History in the database
The /c 21 parameter tells it to maintain the last 21 days of Cookies in the database
The /t switch tells it to purge Temporary Internet Files from the database
The /cf and /tf switches tell it to purge the Cookies and Temporary Internet Files from the filesystem that are no longer referenced in the database
The /cp switch performs a defrag of the webcache file once finished
So not only does it reduce the size of the webcache file, it also cleans up the filesystem as well, ensuring that the user’s profile has no extraneous data in it whatsoever, so even if you maintained every single item for Cookies and History the user had ever accessed., you could still use it to remove the useless Temporary Internet Files data.
It’s also very quick to run, when used in our testing – we didn’t observe any tangible difference in the logoff times of the users involved.
Just to illustrate, here is the webcache file after we’ve run the tool for the user who had the 200MB+ cache on Windows 10…
…and as you can see, quite a difference is observed – down to 28MB in size!
Summary
So, is this the last word on this webcache file, and Internet Explorer roaming in general? Let’s hope so, and hope that Microsoft don’t see fit to make any further changes to the way the browsing functions in these respects!
In summary, you can take a few approaches to managing this:-
- Persist the webcache file (using one of the two methods mentioned earlier), and potentially encounter issues with session performance and storage at a later date (particularly on Windows 10)
- Remove the webcache file from persistence, and have a negative impact on user experience as Cookies, History and other required data doesn’t roam
- Persist the webcache file and use WebCache Manager to keep it down to a size that balances acceptable user experience with acceptable session performance and storage requirements
Over at HTG we’re all big fans of technologies that offer simple, easy-to-use, sustainable solutions to problems that we have in the here and now. If you’re suffering because of this particular issue, then WebCache Manager offers you a proven solution.
Credits
I have to give thanks to Pete Jones and Jonathan Murray over at Avanite for their invaluable help in divining the inner mysteries of the webcache file!
Bootnote
Blog migration is almost fully complete, and The AppSense Bigot will soon morph fully into The Enterprise Eightfold Path. Stay tuned for an official rebrand hopefully some time next week!
Share
