subsystem crashes reboot system?

[Russell Miller]

Hi,

I have a feature request, I'm willing to hack away at it myself, but I want to 
know if there's any way of doing what I want to, or if there's a good 
technical reason why it would be impossible.

As I mentioned earlier, we had an ext3 subsystem crash, which a helpful person 
was nice enough to tell me that upgrading the kernel would fix.  All well and 
good.  But this crash left the system in a semi-functional state.  The 
networking stack was up and running, the kernel was running, but the 
filesystem was not functional and because of this the kernel was in a nearly 
unusable state.  Because the system was pingable, most tcp-stack level 
detectors would not have been able to tell that something serious was wrong.  
The machine (our main production machine that serves millions of hits a week) 
was down for three hours.

Since this was an assertion that failed, one would think that bringing the 
system down automatically in an orderly - then, if that fails, disorderly - 
fashion would be possible.  In particular, I would like for it to behave 
similar as with the panic sysctl.  If a subsystem crashes, reboot the 
machine, because the system is essentially worthless in that state.  I 
realize that this behavior isn't required for everyone, so a sysctl 
(panic_on_subsys_crash maybe) would be sufficient.

Since the machine was in a semi-usable state, one might ask why we just didn't 
have an automated process in place.  Two reasons:  a subsystem crashing 
happens rarely enough that I didn't see any reason to put the effort into it 
until now, and when the system is in a state like that it is impossible to 
tell what will work and what will not.  For example, when we did the three 
finger salute, the system would not go down all the way because one of the 
user space programs made an io call to the crashed filesystem.

In order of helpfulness, please tell me (only one of the following is more 
than enough):
- whether I can do this using the existing sysctl mechanism
- whether there is a patch available (or coming available) to do this
- whether there is a technical reason for me not to do this
- what would be a good place in the code to begin applying a patch.

Please CC me with any replies as I am not on the list.

Thanks.

--Russell