ora-09817 ora-09945 error 28 no space left on device

[fo2sh]

Hello,

I rebooted my server and it won't startup due to the below error, now server is not getting up, its throwing the below error and keep trying to connect to Oracle, so I am unable to access the server to free some space if its the issue as per some googling, any advice how to overcome this issue? how can i startup my server in order to diagnose these errors?

ERROR:
ORA-09817: Write to audit file failed.
Linux-x86_64 Error: 28: No space left on device
Additional information: 12
ORA-09945: Unable to initialize the audit trail file
Linux-x86_64 Error: 28: No space left on device

Thanks in advnace for helping me in this!

[rproffitt]

Sorry fo2sh, but the error seems to tell you something interesting. Are you unable to check free space?

[fo2sh]

actually I am unable to startup server :)
so when starting the server it starts booting process then hang on a page where it gives this error and trying to connect to Oracle with no luck due to the error, so any solution maybe to be done in boot screen for example to disable starting up oracle or something during boot? :^)

[rproffitt]

@fo2sh. The message told you a reason why. But are you reporting it was too unclear?

Tell the forum how you got this far without basic knowledge of disk space, etc.

[fo2sh]

This is a SUSE server with Oracle 11g installed, I wanted to disable hyperthreading option, so in order to do it i have to do it via boot screen, so i performed reboot so i can access boot screen, after reboot i encountered this error keeping in mind that first reboot i faced this error was before even disabling hyperthreading so it has nothing to do with disabling hyperthreading.

To be honest i am not sure about space there before reboot but i can say 90% i didn't have any FS utilization because i have proper cleaning jobs and rman scripts and server is running since 4 years without any running out of space due to the proper cleaning jobs i have.

Hope this clarifies how i encountered this error.

[rproffitt]

Let's say you changed something. Now put it back the way it was.

As to space, you have yet to share much about space on all devices in your system tree (fstab info.)

Parting advice. Your server is working? Don't fix it.

[fo2sh]

OK, this server I am facing the problem with is redundant and replica with Oracle Streams configure with a master DB server, I can give the partitioning that i have on master server which is the same as redundant, please find it below.

/dev/cciss/c0d0p1  9.4G  4.2G  5.2G  45% /
devtmpfs           4.0G  204K  4.0G   1% /dev
tmpfs              4.0G     0  4.0G   0% /dev/shm
/dev/cciss/c0d0p5   19G  6.1G   13G  33% /usr/fdamra/db
/dev/cciss/c0d0p6  481G  286G  195G  60% /usr/fdamra/db/oracle/oradata
/dev/cciss/c0d0p2   47G  1.1G   46G   3% /var

and fstab below

/dev/cciss/c0d0p3    swap                 swap       defaults              0 0
/dev/cciss/c0d0p1    /                    xfs        defaults              1 1
/dev/cciss/c0d0p5    /usr/fdamra/db       xfs        defaults              1 2
/dev/cciss/c0d0p6    /usr/fdamra/db/oracle/oradata xfs        defaults              1 2
/dev/cciss/c0d0p2    /var                 xfs        defaults              1 2
proc                 /proc                proc       defaults              0 0
sysfs                /sys                 sysfs      noauto                0 0
debugfs              /sys/kernel/debug    debugfs    noauto                0 0
usbfs                /proc/bus/usb        usbfs      noauto                0 0
devpts               /dev/pts             devpts     mode=0620,gid=5       0 0

Now I can't get the status on redundant server because as I said its not booting up, when you start the server it hanges with screen telling the above ORA errors.

I only changed hyperthreading and i changed it back the way it was, keeping in mind that i encountered this error after server reboot and before even perform any change.

[fo2sh]

This is the screen i am hanged with during boot.

[rproffitt]

The file systems with 0, 1 and 3 per cent are suspect. Why not work on that?

PS. Added with edit. Compare free space on these mount points on your working server to this failed server.

[JamesCherrill]

This may be a dumb question, but does your server have the equivalent to a Windows or Mac boot in safe mode that would get you to a working minimal OS without starting any non-essential services such as SQL or connecting to any external servers?

[fo2sh]

Hi, I couldn't find safe mode option unfortunately :(

[JamesCherrill]

I Googled "SUSE server safe mode" and found a load of info that looked relevant to me.