This is the page that was displayed as www.cs.Helsinki.FI home page during the break December 28-30, 1998. The progress report grew during the break.

We are upgrading our hardware at cs.Helsinki.FI

Most of the services at cs.Helsinki.FI are currently unavailable.
These services include at least

NFS services at fs.cs.Helsinki.FI
FTP services at ftp.cs.Helsinki.FI
WWW services at www.cs.Helsinki.FI
Mail services at mail.cs.Helsinki.FI
The reason for this unavailability of services is a hardware upgrade of the file server fs.cs.Helsinki.FI.
We estimate that all systems should be operational again at the latest on Wednesday, December 30, 1998 at 13:00 GMT (15:00 local time EET).

Progress report:

Backup of old systems: completed Dec 28, 09:30 EET
New hardware installation: completed Dec 28, 13:05 EET
Data transfer: estimated time of completion Dec 29, 22:00 EET
- First attempt aborted due to I/O errors, retry Dec 28, 14:57 EET
- Also second attempt aborted due to SCSI I/O errors, probable cause is a faulty SCSI cable that connects our DLT tape library on the same SCSI bus - DLT tape library disconnected
- Third attempt to transfer home directories started Dec 28, 16:42 EET, estimated time of completion Dec 29, 01:30 EET
- Home directories have been transfered, transfer of other directories started Dec 29, 08:48 EET
- Some bad news Dec 29, 09:20 EET: while transferring data from the oldest RAID unit (which is going to be retired) the system entered a SCSI reset loop where two SCSI buses were repeatedly reset, no useful activity could be observed
- We moved (logically) the new home directory RAID unit to the old file server for file system check which started Dec 29, 10:40 EET
- File system check for the oldest RAID unit started on the new file server Dec 29, 10:43 EET
- More bad news Dec 29, 12:10 EET: the oldest RAID unit seems to be failing totally
- We decided to restore the files will from backup tapes (which are very fresh), however, this seemed to be problematic because the faulty cable and/or terminator and/or SCSI controller
- Dec 29, 13:20 some new cables, terminators and SCSI controllers were ordered, they will arrive tomorrow morning
- After some more or less heavy reconfiguration (and reusing parts from the dead RAID unit) we have now rebuilt the file server and restoring the backup has started, time Dec 29, 14:42 EET, estimated time of completion Dec 29, 16:15 EET
- Dec 29, 16:11 EET: everything has been restored we start some integrity tests
- It looks like we have found a nasty operating system bug which can corrupt a file system even if the file system is mounted read-only
- Dec 29, 18:31 EET: started the final pass in the process of finding what really has been corrupted, so far we have found 6 corrupted inodes out of 2066087 in use - that's only 0.00029% but too much of course
- Finally the last check was complete and it turned out that only those 6 directory entries had been corrupted (if you don't count a couple of netscape cache files), and even those are not lost: they will be restored from backup, maybe tomorrow... time was Dec 29, 21:14 EET
Data transfer: completed Dec 29, 21:14 EET
Restart of all systems: started Dec 29, 21:32 EET, estimated duration 2 hours
Restart of all systems: completed, all systems are up and running, Dec 29, 22:20 EET

But the story continues:
- After about an hour of successful operation, about Dec 29, 23:20 EET file corruption struck again
- This time we suspected that we have a hardware bug, we tried to change BIOS parameters, swapped the ECC memory with ordinary SDRAMs cannibalized from couple of other machines, tried to lower CPU clock speed, and lowered the SCSI bus speed from Ultra-SCSI to Fast-SCSI - no help
- About Dec 30, 02:30 we moved all RAID units to the same controller and surprise: everything started working again
- The disks appeared to be so corrupted that we decided to do a full restore from tapes (the DLT tape library sits again on the file server)
- It's now Dec 30, 03:07 EET and mkfs is running on the RAIDs, we hope that we can soon start an automated restore and go home to get some sleep
- Dec 30, 03:54 EET: automated restore started, estimated time of completion Dec 30, 13:00 EET
- Restore completed Dec 30, 10:24 EET
- Dec 30, 11:15 EET systems are coming up but are shut down very soon because of problems with home directory permissions
- Permission problems can be fixed, script started Dec 30, 12:03 EET, estimated time of completion Dec 30, 13:15 EET
- Script completed Dec 30, 12:59 EET, once again: systems are comping up...
Dec 30, 13:30 EET, operation completed, all systems are up and running
Special thanks to everyone involved, especially Kari, Mikael, Lea, and Miika

Petri Kutvonen
Petri.Kutvonen@cs.Helsinki.FI (but that of course doesn't reach me at the moment)
Mobile phone: +358 40 544 3020