Hard rebooting

This is one of those bookmarks for future me, from https://infosec.exchange/@hal_pomeranz/115220370083388046, wherein it is stated that the following incantation against the /proc filesystem will trigger a hard reboot:

echo 1 > /proc/sys/kernel/sysrq
echo b > /proc/sysrq-trigger

It does require a root shell to write to those “files”, and you get none of the usual protections like file system sync taking place. So it’s a last resort. But sometimes this sort of thing really is necessary, like when you lose your filesystems, which has happened to me before.

Fun story:¹ while working at Sprint ATL, CA, we setup to do an experiment to see how VOIP traffic might fare at scale in the face of routing disruption. The plan was to take down an interface card in a router in Sprintlink on the path being taken by VOIP-like traffic between Switzerland and California, while monitoring everything in sight using Endace packet capture and the routing protocol capture code I’d written, PyRT, to grab the link states (IS-IS) and the BGP updates. It was to be done ~midnight to minimise disruption, with a live support call and active VP-level engagement. One of the more fun experiments I’ve done.

And then about 15 minutes before the card was to be taken down I discover that at least one of the hosts on which my routing capture code was running was failing. I had shells open but invoking commands such as ls (or anything else) was just reporting bad executables and suchlike unhelpful errors. Couldn’t do anything; presumably something had happened and shared libraries were no longer visible—never did pin it down. And then the IDRAC access failed as well.

The only thing left was a terminal I had with a running Python interpreter, and a loaded os module. Which happily proved enough to let me confirm that the routing capture processes I’d started earlier were still running and still seemed to be correctly writing to disk (the files existed, were being appended to, and were rolled over according to size as expected). So the experiment went ahead, the data was captured and retrieved, analysis was done and a paper produced: Analysis of Link Failures in an IP Backbone, Proc. ACM Internet Measurement Workshop 2002. Phew.