Member Avatar for MacTechie

Hi I have a 2003 Server It is a Dell PE 2800 The server is running no services other than those necessary to boot. It is running a custom application that runs on all other servers with no problems, the other servers are the same or similar dell servers. When the server freezes what happens is the users notice they have no connectivity to the Custom application When investigating on the server there is mouse control but no keyboard input is available. The server must be hard booted on reboot and investigation of the event viewer the error codes 333 the source is Application Popup the description reads "An I/O operation initiated byt the Registry failed unrecoverably. The Registry could not read in or write out or flush one of the files that contain the system's image of the Registry. The other error code is Event ID 2019 Source: Srv Description is "The server was unable to allocate from the system non-paged pool because the pool was empty. These errors occur multiple times until the server is unable to respond and is ultimately hard booted. There are no other errors on reboot. In researching other forums I am not seeing any real consensus on how when where or why this occurs. This server is not running any anti-virus anymore as we noted that some anti-virus applications caused problems. There are NO other applications installed on this server other than the Custom Application which does appear to be causing the problems and also works fine on all other servers. Does anyone have any ideas??

  • 7 Contributors
  • 10 Replies
  • 419 Views
  • 1 Year Discussion Span
  • Latest Post Latest Post by xrugger

Dani AI

Generated

The symptoms described by and the resource-growth observations others added (for example ’s note about rising kernel memory and ’s filter-driver suspicion) point toward a kernel‑mode resource leak or a failing kernel component rather than a simple user‑mode app fault. Microsoft’s writeup on this class of failures and the diagnostic hotfix/registry option is a good place to start. ()

Collect reproducible evidence before changing the server in production. Configure Windows to produce kernel (or full) crash dumps and capture one during failure — KB970054 documents a hotfix and the RegistryFlushErrorSubside option that can be used to force a Stop for diagnostics; then analyze the dump with the Windows debuggers (WinDbg, using commands like !analyze -v) to get a stack and likely culprit. Keep symbol configuration and symbols ready so analysis points to real modules. ()

For a leaking nonpaged/paged pool, use PoolMon to find the growing pool tag (capture repeated snapshots and look for tags whose Bytes/Diff climb over time). Once you have a tag, map it to a driver (KB298102 shows techniques such as scanning drivers with the tag), and then run Driver Verifier on the suspected driver(s) in a test window to force and capture a bugcheck for definitive identification — Driver Verifier must be used only on test or maintenance windows because it can crash the host. (learn.microsoft.com)

Don’t ignore firmware/storage/controller differences between identical servers: compare driver/firmware builds against the machines that are stable, run vendor diagnostics and update firmware/drivers (Dell’s support pages for the PowerEdge line are the right starting point), and escalate to the hardware or ISV vendor with the collected crash dumps, PoolMon output and driver lists if a third‑party filter driver is implicated. These artifacts are what will get a definitive fix. (dell.com)

(Use the hotfix/registry change only for diagnostics, back up the system first, and perform all intrusive tests—Driver Verifier, firmware updates, forced stops—in a maintenance window or lab copy.)

Recommended Answers

All 10 Replies

Member Avatar for daveman

Hi I have a 2003 Server It is a Dell PE 2800 The server is running no services other than those necessary to boot. It is running a custom application that runs on all other servers with no problems, the other servers are the same or similar dell servers. When the server freezes what happens is the users notice they have no connectivity to the Custom application When investigating on the server there is mouse control but no keyboard input is available. The server must be hard booted on reboot and investigation of the event viewer the error codes 333 the source is Application Popup the description reads "An I/O operation initiated byt the Registry failed unrecoverably. The Registry could not read in or write out or flush one of the files that contain the system's image of the Registry. The other error code is Event ID 2019 Source: Srv Description is "The server was unable to allocate from the system non-paged pool because the pool was empty. These errors occur multiple times until the server is unable to respond and is ultimately hard booted. There are no other errors on reboot. In researching other forums I am not seeing any real consensus on how when where or why this occurs. This server is not running any anti-virus anymore as we noted that some anti-virus applications caused problems. There are NO other applications installed on this server other than the Custom Application which does appear to be causing the problems and also works fine on all other servers. Does anyone have any ideas??

Hi,
We had a similar problem yesterday - just the ID333 part anyway.

Try running task manager looking at processes with 'handles' selected to view (click on the view tab & tick the 'handles' box.

We found two processes in there that were building up to many 10s of thousands of handles - when I looked the names up on google one of them came out as a trojan (not found by AV). We stopped the process & it came right back, so I did a search for the file & changed its .exe to .xxx then killed the process again & it stayed dead.

Also if you look in Performance I bet you'll find that the counter which shows Kernel memory is slowly counting up - when that gets high enough the error 333 kicks off because there isnt enough memory left for the system to read & write.

Once we were happy that every thing was still running OK we did a search through the registry for the file names - they all appeared together in one key - and we deleted those keys (made a backup first).

Touch wood the server has now been running for about 30 hours!

Hope this helps

Member Avatar for jhicken

I'm having the same issue. Do you remember what the processes or the filenames were?

Member Avatar for daveman

I'm having the same issue. Do you remember what the processes or the filenames were?

Its a while ago now but if I remember right the names were different every time we deleted the process.

Just look for a process which is grabbing thousands of handles.

Member Avatar for jbennet

I believe there is a hotfix for this issue

Member Avatar for DimaYasny

I believe there is a hotfix for this issue

yeah, it's called antivirus ;)

Member Avatar for jbennet

No, i had a similar issue on windows 2000 server. It was a problem with svchost where it malfunctioned and used more and more resources until the server crashed.

Member Avatar for DimaYasny

No, i had a similar issue on windows 2000 server. It was a problem with svchost where it malfunctioned and used more and more resources until the server crashed.

I know, I was just joking :)
sysinternals' process explorer usually shows exactly what the problem is in such cases

Member Avatar for ITCCTV

Hi I have a 2003 Server It is a Dell PE 2800 The server is running no services other than those necessary to boot. It is running a custom application that runs on all other servers with no problems, the other servers are the same or similar dell servers. When the server freezes what happens is the users notice they have no connectivity to the Custom application When investigating on the server there is mouse control but no keyboard input is available. The server must be hard booted on reboot and investigation of the event viewer the error codes 333 the source is Application Popup the description reads "An I/O operation initiated byt the Registry failed unrecoverably. The Registry could not read in or write out or flush one of the files that contain the system's image of the Registry. The other error code is Event ID 2019 Source: Srv Description is "The server was unable to allocate from the system non-paged pool because the pool was empty. These errors occur multiple times until the server is unable to respond and is ultimately hard booted. There are no other errors on reboot. In researching other forums I am not seeing any real consensus on how when where or why this occurs. This server is not running any anti-virus anymore as we noted that some anti-virus applications caused problems. There are NO other applications installed on this server other than the Custom Application which does appear to be causing the problems and also works fine on all other servers. Does anyone have any ideas??

I have the same problem I have changed the memories , format disk install windows again and still nothing I have the same problem with the server PE2850

help please

Member Avatar for DimaYasny

do you have dell support?

Member Avatar for xrugger

We have had a number of servers catching this disease and have a ticket open with Microsoft. Microsoft has said it has to do with a third party application. Antivirus, and our backup utility, it seems that some filter drivers that get loaded into the kernal begin going haywire that makes me agree with the 'handles overload' that has been suggested here. Microsoft has not been of much help.

It seems that the issue has manifested itself differently each time so what we did to fix one box does not work on the next, and back to digging through the registry we go. Though now we are seeing some other disturbing symptoms:

The server will also exhibit the following behavior, at the logon screen the screen is black, at console or RDP, and cannot be logged into, and reboot and profile removal has not helped.

we will be looking at the services and handles to see which ones they are this time. This error will crash a server, and render it inoperable with a lot of tedious work to get it back up and sorted out, and this is truly time we do not have as the sytems it has plagued are mission critical.

any new info or advice anyone has will help a ton.

Hi,
We had a similar problem yesterday - just the ID333 part anyway.

Try running task manager looking at processes with 'handles' selected to view (click on the view tab & tick the 'handles' box.

We found two processes in there that were building up to many 10s of thousands of handles - when I looked the names up on google one of them came out as a trojan (not found by AV). We stopped the process & it came right back, so I did a search for the file & changed its .exe to .xxx then killed the process again & it stayed dead.

Also if you look in Performance I bet you'll find that the counter which shows Kernel memory is slowly counting up - when that gets high enough the error 333 kicks off because there isnt enough memory left for the system to read & write.

Once we were happy that every thing was still running OK we did a search through the registry for the file names - they all appeared together in one key - and we deleted those keys (made a backup first).

Touch wood the server has now been running for about 30 hours!

Hope this helps

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.