I’ve done more than a bit of embedded systems work of late, on gear that will wind up running in places that aren’t necessarily easy to access. As is usually the case in such projects, you have to think very far ahead to anticipate a variety of potential pitfalls that could cause at best downtime and at worst a site visit at a ridiculous hour to make a fix that isn't possible remotely. Those visits aren’t cheap, and they definitely aren’t convenient.
Instead, you plan for remote access to the device, arrange for remote software upgrades, and address the usual concerns, but more lifesaving steps can be taken when the picture goes pear shaped. These days, relatively cheap remote power control devices will not only let you power-cycle gear remotely, but even do it for you if they detect a network connectivity loss. You give the device an IP address to ping, and when those pings fail, it power-cycles an assigned outlet, which will hopefully restore connectivity.
These devices will generally provide a Web interface alongside SNMP and Telnet access. Frankly, I’d prefer to see SSH access with public key support instead of Telnet, but Telnet will get the job done. (Note that I haven’t used the Dataprobe devices in particular; I’ve only linked to them as an example. Raritan makes controllable PDUs as well, and I’ve used those for years, but they’re quite expensive.)
While you're deploying a remote power controller, you should also build in code in the device to monitor its own processes and at least attempt to restart critical access services if something goes wrong. For instance, if you’re running Dropbear outside of inetd, then write a tiny shell script in Cron to make sure it’s always running. Do the same with the DHCP client if it’s in use.
Obviously, you can do this for any processes the device is supposed to be running for its core functionality, but at the very least, make the remote access and control services as resilient as possible. You can fix nearly anything as long as you can get a shell prompt.
You should also have an SSH server running, or you should have the ability to trigger it to start with port knocking or a similar scheme. You might have a sweet Web-based UI that does everything you need, but if you think that remote SSH access is unnecessary, think again. You’ll always need to get to the underpinnings for one reason or another. Also, always use keys with SSH -- it’s the right way to do it and it makes scripting so much easier.
I’m also a fan of using as little as possible and making it as simple as possible for embedded systems. If a script needs to be written for one purpose or another, there’s no need to place more dependencies on the device by writing it in Python or Perl -- besides, Bash is really good at it. Unless your assignment is so complex that it would make for major headaches, do it in Bash (or Ash). And don’t immediately dismiss shell scripting as too primitive. You can do an absurd amount of higher-level stuff in modern shells.
When you are scripting, take extraspecial care to include error checking on every single action. It always seems like a pain to write those checks when you know the script will run for a single purpose on a single platform, but this is where gremlins love to hang out. Write a function and call it after every significant command.
The tools to build, maintain, and work with remote embedded systems are inherently simple. SSH, SNMP, and a functional network connection are all that are required in most cases. Toss in remote power control with automated network testing and you’re about as covered as you can be. Aim for the best, but plan for the worst -- you’ll thank yourself sooner than you think.