Member Avatar for PeteComcar

I've got a problem with our live server. Occasionally a page will not load, and the following error is displayed in plain black text on a white background:

Server Error
The server encountered an internal error and was unable to complete your request
Application server is busy. Either there are too many concurrent requests

If I look in the error log for the website, at the time of the error using the following command:

grep "12:17:33" error_log

lines such as this are returned:

[Tue May 03 12:17:33 2011] [notice] jrApache[14352: 21512] returning error page for JRun too busy or out of memory

Then if I search in the Coldfusion cfserver.log I find these errors, a few seconds before hand:

05/03 12:17:29 error Corrupt form data: premature ending
java.io.IOException: Corrupt form data: premature ending
at com.oreilly.servlet.multipart.MultipartParser.<init>(MultipartParser.java:177)
at com.oreilly.servlet.multipart.MultipartParser.<init>(MultipartParser.java:99)
at coldfusion.filter.FormScope.fillMultipart(FormScope.java:190)
at coldfusion.filter.FusionContext.SymTab_initForRequest(FusionContext.java:369)
at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:33)
at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:53)
at coldfusion.filter.RequestThrottleFilter.invoke(RequestThrottleFilter.java:126)
at coldfusion.CfmServlet.service(CfmServlet.java:200)
at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89)
at jrun.servlet.FilterChain.doFilter(FilterChain.java:86)
at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42)
at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)
at jrun.servlet.FilterChain.doFilter(FilterChain.java:94)
at jrun.servlet.FilterChain.service(FilterChain.java:101)
at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:106)
at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42)
at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:286)
at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543)
at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320)
at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)

05/03 12:17:29 error (JRun Service: ProxyService [jrun.servlet.jrpp.JRunProxyService@500b675]) JRunPRoxyServer.invokeRunnable:
java.lang.IllegalStateException
at jrun.servlet.JRunResponse.getWriter(JRunResponse.java:205)
at jrun.servlet.JRunResponse.sendError(JRunResponse.java:597)
at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:328)
at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543)
at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320)
at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)

java.lang.IllegalStateException
at jrun.servlet.JRunResponse.getWriter(JRunResponse.java:205)
at jrun.servlet.JRunResponse.sendError(JRunResponse.java:597)
at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:328)
at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543)
at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320)
at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)

I don't really know what to do from here.

If the page is refreshed then it loads without error. I don't know how to replicate the error, it doesn't seem to happen on any particular page or at any articular time.

I've tried setting:

Maximum number of simultaneous Template requests: 100

in the Coldfusion administrator, I've done that based on Steven Erat's post:

But I'm still getting errors. Would increasing this to a higher value help?

And then there's the "premature form ending" part of the longer Java error, I'm not sure if that's related and don't understand what it means anyway.

I've also looked through my access logs, and ruling out any images or js files that get called at the time of any errors, I've got a variety of CFM pages that return a 503 http status, and some of them are CFM pages that get returned with a content type of Javascript, and are never requested after a form post.

So in summary, I'm getting some pretty blunt error messages, and I can reproduce them at present, and the page loads fine if I just hit F5, and I'd like to know if anyone can help me to understand the source of the problem.

Oh and one last thing, we're running a RHEL 5 server with CF9, and we've upgraded recently from CF7. Before the upgrade this error was not happening.

  • 2 Contributors
  • 3 Replies
  • 2K Views
  • 1 Week Discussion Span
  • Latest Post Latest Post by cfwebdeveloper

Dani AI

Generated

What you are seeing is two sides of the same event: a multipart/form-data POST is getting truncated, then the web-server connector reports the CF/JRun back end as “busy.” The stacktrace shows ColdFusion’s multipart parser calling the O’Reilly COS library; that specific IOException (“Corrupt form data: premature ending”) occurs when the request body ends before the boundary is found, most often due to the web server or connector interrupting the upload/POST. (servlets.com)

A few practical checks have helped me stabilize this on CF9/JRun:

  • Tune the Apache-to-JRun connector timeouts (ConnectTimeout/RecvTimeout/SendTimeout) so slow POSTs are not dropped early; these are set via JRunConfig in your web server config. (3gpp2.org)
  • Keep concurrency settings aligned. Apache’s worker limits should not outpace what JRun/CF can actually service, and ColdFusion’s Request Tuning should reflect realistic throughput so excess requests queue instead of failing. The Admin docs explain the “Maximum number of simultaneous template requests” and the queue timeout behavior. ()

Quick diagnostic you can drop into the target action to see whether requests arrive incomplete and which headers clients send:

<cfscript>
req = getHttpRequestData();
h = req.headers;
cflog(file="upload_debug",
  text="uri=#cgi.request_uri# len=#h['content-length']# te=#(structKeyExists(h,'transfer-encoding')?h['transfer-encoding']:'')# ct=#h['content-type']#");
</cfscript>

If you see missing/odd Content-Length or repeated failures under load, focus on the connector and thread limits above. Also, ’s JVM update likely helped by resetting JRun threads and moving to a supported Java level for CF9; double-check the JVM you are on matches Adobe’s CF9 support matrix. (coldfusion.adobe.com)

If it resurfaces, capture a short window with Apache error_log (connector verbose on) plus CF request metrics, and compare timestamps with any “busy or out of memory” notices to confirm whether it is a timeout or true saturation, as hinted. (3gpp2.org)

Recommended Answers

All 3 Replies

Member Avatar for cfwebdeveloper

Make sure the CF Admin settings from CF7 carried over to CF9.

Member Avatar for PeteComcar

All my settings were carried over, that wasn't the problem.

The issue seems to have been resolved, although I can't put my finger on why.

I installed FusionReactor to try and better monitor the server while the error was happening, and in doing so had to update my JVM and restart CF9. Since then, I've not had the error.

My guess is the newer JVM, although I suppose it might have been the CF9 reboot, perhaps some settings I'd made hadn't take effect yet and that kicked them in. I very much doubt installing FusionReactor fixed it, but I really can't say that it didn't.

Member Avatar for cfwebdeveloper

Yeah, keep an eye on it. If it comes back let us know. I've seen this happen but without a real solution.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.