I use Fiddler to trace and analyze my HTTP traffic. It’s a simple and visual way to debug a web scraping application.
Fiddler v188.8.131.52 can raise an out-of-memory exception when you try to save a lot of sessions in one SAZ file.
In my case, I tried to save 6,704 sessions in one SAZ file after recording a nine-hour site scraping session.
Fiddler got more than halfway through the list of sessions before halting with this exception:
As a workaround, I selected the first 3352 sessions and saved them in a “Part 1” SAZ file. For more than a minute the status bar looked like this, the number incrementing quickly from 0 to 3,352:
3,352 / 6,681 Collecting data... #1408 (ID: 1408)
The operation completed successfully. With no errors I saved the next 3352-ish sessions to a “Part 2” SAZ file. Slightly less than 6,704 sessions were saved in total. I don’t know why.
The possibility of an OutOfMemoryException is a known issue on the Fiddler website:
Fiddler works by storing the entire request and response in memory.
If you are performing a huge download (hundreds of megabytes) it’s possible that
Fiddler cannot find a free memory block large enough to hold the entire contiguous response,
and hence you’ll run into this out of memory problem. It’s also possible that if you have
thousands of sessions in the Fiddler session list, even a relatively small memory block will not
be available to store a response a few megabytes in size. You can reduce the incidence of this
problem by clearing the session list (CTRL+X) or configuring it to automatically trim to the most
recent two hundred sessions (Click the Filters tab, and click the “Keep only the most recent
sessions” option at the bottom).
This doesn’t completely explain to me why saving a an SAZ file of a large trace should do the same. It’s as if Fiddler is trying to allocate all the space for the SAZ file in memory as one contiguous block.
Seems like the quick fix would be to run Fiddler on a 64-bit version of Windows:
Fiddler2 now supports running on 64bit computers.
If you’re on a 64-bit machine, you’ll never hit a problem.
64-bit Windows has a much larger address space than 32-bit Windows.
But wouldn’t it be great if Fiddler could stream the session to disk in small sequential chunks instead of saving one monolithic chunk? Because of the workaround I used here (two monolithic chunks), my trace is now forever cut into two files. Not a big deal in practice, but it would be neater to have all the related data in one file.