I am running a Lotus Domino 8.5 server on Centos 5.2 x64 Linux. The installation is on a Intel server with Adaptec RAID controller 1.7Tb and the Notesdata directory is on an external Dell PowerVault MD1000 5.5Tb with RAID 5 configuration. I also have LVM2 installed.
On Wednesday last week we had a power surge. I have 2 x 3KVA UPS installed, but for some reason the external Powervault RAID lost the connection to the hard drives. After i rebooted, i check the RAID BIOS and found that 1 drive failed and the RAID is rebuilding.
After the process completed, i started up the server. FSCK gave me error that the volume is inconsistent and i should run fsck manually. So i ran fsck -y /dev/VolGroup02-LogVol02
This ran for almost 10hours, and supposedly fixed hundreds of errors. Once the server booted up, the file system was mounted as read only. I tried unmounting and remounting the volume, also checked fstab for valid mount points.
fdisk -l stated that the volume does not contain a valid partition table. So i bought 3 x 1tb drives and started transferring the data. I cleared and re-initialized the RAID and created a new RAID5 config.
This time i created a GPT partition table and formatted the file system to ext3, transferred the data back using Krusader and preserved ownership and permissions.
I ran an offline fixup, compact and updall.
I then fired up the Domino server to find that my names.nsf is gone. So copied one from a backup server. On startup i found many errors about files that do not exist. I copied the missing files from the backup server and all went well.
Only problem i am stuck with is user mail files. Some files when opening will give error "File is not a database". The mail files have sizes ranging from 260Mb to 4Gb.
I tried running a fixup, but got the same error that the file is not a database. I have replicas of the user mail files on my backup server but there are a couple of mails missing and i am unable to replicate databases hence the error.
What i have done for the time being is rename the problematic mail files and pushed new replicas from the backup server. The result is a loss of mail and folders for dates 12 - 22 March.
What can i do to recover/repair these mail files?