Home / cfmx file upload memory leaks and limitations
So for the past few months I've been having some stability issues with one of our environments. We would periodically have the service gobble up all available JVM heap and die with the well known java.lang.OutOfMemory error. What was confusing me was that it was not necessarily load-related, it could hum along during peak usage (for this server that's about 5-6 requests per second) and not have any issues at all. Then, on other days, I'd watch the heap spike out of control, or rather jump 100-200mb of allocated objects and then level off... then maybe an hour later do the same. All the usual suspects (gc tuning, jrun.xml tuning, etc) were addressed, with no significant change in behavior. A graph of allocated heap would look something like:
Well last week I finally was able to correlate these spikes with a specific activity: FILE UPLOADS! We have a site that has always had steady usage of file uploads, but lately the content has contained a lot of podcasts at around 40-50mb a pop. It had never dawned on me that file uploads would be so devastating to the cf server. I even went as far as getting a heapdump off the server (different than a thread dump) - something you can do with a jvm newer than 1.4.2_12 by adding the -XX:+HeapDumpOnOutOfMemoryError flag to your config. With a heap dump in hand you can analyze it using the jhat utility provided with JDK6 updater 1. jhat will set up a little webserver for you on port 7000 that you can use to browse the heap dump (basically a snapshot of the heap at the time your server choked) to see what the heck was allocated. In my case it was about 900mb of byte arrays (byte[]) - all allocated by underlying cf request processing classes. I at least now knew my code was in the clear! You can read more about heap dumps here: http://blogs.sun.com/alanb/entry/heap_dumps_are_back_with
So - lets lay down some facts:
1) This is a fully patched 6.1 environment.
2) Yes, I'm aware there is a patch for file upload memory leaks recently released for 7.0.2 (http://www.adobe.com/cfusion/knowledgebase/index.cfm?id=kb401239)
3) I wish it was as simple as upgrading the server, however, in my testing, only half the issue is really addressed.
Here's where it gets tricky - yes, a fully patched 7.0.2 cf box won't leak like a sieve during large fileuploads. However, it's still vulnerable from the perspective of file size and scalability - during a file upload, cf allocates almost 3x the size of the file in memory. Just because it now gives that memory back afterwards is not enough - if you have a few people uploading large files at the same time, and the server is running with a decent amount of heap already allocated to other applications, you could still get a OutOfMemory error. This is easy to test by simply capping the jvm heap at 250-300mb (-Xmx300m) and then uploading a file bigger than 200mb - you'll get a 500 null error and corresponding OutOfMemoryError in the logs. Most modern web browers (IE and Firefox) and webservers (IIS and Apache) support http multipart requests up to 2gb. Another interesting piece that you may not know is that this will happen with or without a <cffile action="upload".../> tag being executed in the request - the memory consumption happens way before any of your cfml is processed, as file uploads are dealt with at the same time the "form" scope is populated. This to me also seems like a gaping Denial-Of-Service attack hole - fire a few large multipart requests at ANY cfm page and you can bring the server to its knees, whether the application attempts to "upload" the file or not. This is a little different than the security bulletin regarding a similar issue posted a while ago, in which file uploads that are stopped mid transfer were getting left on disk - theoretically could be used to exhaust the disk space on the server.
So of course I had to come up with a workaround (whether this issue gets fixed in Scorpio we'll have to see). These are the steps I took:
1) I wrote a servlet filter in java to intercept the request before it gets to your "cfmservlet" (basically the entry point for all CF requests). This needs to be installed and configured in your web.xml. To be safe, I only mapped this filter to the url that I knew would be dealing with fileuploads.
2) If the request is multipart encoded, I use the Apache commons-fileupload 1.2 (http://jakarta.apache.org/commons/fileupload/) to parse the request. You can tell the fileupload library that files over a certain size should be streamed to disk - I set this at 5mb. The common libaries are usually pretty easy to program with.
3) I then "wrap" the request using an HttpServletRequestWrapper and override some of the usual suspects that cf is probably using to determine if a file upload exists. I had to look at the source of the Oreilly multipart parser that cf uses to figure out what made it was looking for - it checks both the "content-type" request header and the alternate getContentType() ServletRequest method to see if either say "multipart/form-encoded" - instead my "wrapped" request always responds with "application/x-www-form-urlencoded" - which makes cf think that it's a regular form POST.
4) Unfortunately, this also prevents cf from properly populating the "form" scope because multipart requests are broken up differently than non-multipart ones and cf simply doesn't seem any POSTed parameters - there may be a way around this but I simply added some code in application.cfm to pull any values out of my "wrapped" request and place into the form scope.
5) I wrote a simple <cf_big_upload/> tag that returns the same stuff you would get in the "cffile." scope, however all the tag does is inspect the "wrapped" request object and if there is a file present under the field name you requested, it moves the file from the server's temp location to the destination of your choice - using DiskFileItem.write("/path/to/final/resting/place/"))
6) At the end of the request any uploaded files still in the temp location are deleted. This functionality is part of commons-fileupload.
Long story short - on the train tonight I successfully uploaded a 800mb file on a 6.1 server with -Xmx280m ... memory usage increased negligably and all of the other processing that needed to take place in addition to the file upload happened without issue.
If anyone is interested in the code or more detail in how this works let me know - I didn't want to post it without some proper load testing scenarios, but I'm still excited that I can work around this issue without a major upgrade (or even waiting for a "next" release).
