Our first remote-site build was, by office standards, a good one. Quality switching, a proper firewall, neat cabling. It failed within five weeks. Not because anything was wrong with the equipment, but because the design assumed conditions that do not exist at a site with generator power, fifty-degree summers, airborne dust and no permanent technical staff.
Here is what we learned, in the order the field taught us.
Assume you cannot get there
This is the principle everything else follows from. If a fix requires physical presence, the fix takes days, not minutes. So every design decision is judged by one question: can this be diagnosed and recovered remotely?
In practice that means out-of-band management on every site, without exception. An LTE-connected console server on the router and firewall serial ports costs very little and has saved us more site visits than everything else combined. When the primary link is down, out-of-band is the difference between a ten-minute fix and a two-day journey.
It also means remote power control. Switched PDUs that let you cycle a specific outlet turn "someone must drive out and reboot the modem" into a click.
Power is the actual problem
Connectivity gets the attention. Power causes the failures.
- Generator changeovers are violent. Sites transfer between grid, generator and back several times a day. Every transfer is a transient. Equipment plugged directly into site power will not survive long.
- Size UPS runtime for the changeover, not for a full outage. You need to ride through the gap and the instability either side. Thirty to forty-five minutes at real load is usually right.
- Batteries die fast in heat. See our note on UPS failures. In a container at forty-five degrees, plan on replacing them roughly twice as often as the datasheet suggests, and budget for it up front.
- Solar assist is worth costing. On sites where the generator is switched off overnight, a small solar and battery arrangement to keep the comms rack alive is often cheaper than the diesel to keep the generator running.
Heat and dust
Standard enclosures with fans pull dust in. Within months, fans clog, temperatures rise and equipment throttles or shuts down.
Use sealed IP-rated enclosures with a heat exchanger or an air conditioner rather than filtered ventilation, mount them out of direct sun, and monitor internal temperature separately from ambient. Choose industrial-temperature-rated switches where the budget allows. And accept that a rack in a portacabin is a different engineering problem from a rack in an office, even though it looks the same in the bill of materials.
Connectivity: plan for all three to fail
Our standard remote build now has a primary link, a secondary on a different medium, and out-of-band on a third.
- Primary: fibre or licensed microwave where available.
- Secondary: LTE from a different operator, with an external antenna properly mounted. The internal antenna on a modem inside a metal cabinet is not a plan.
- Deep fallback: VSAT on the sites where terrestrial coverage genuinely does not reach. Latency makes it unpleasant for interactive work but it keeps a camp reachable, which is the point.
- Out-of-band: a separate LTE SIM, ideally on a third operator, doing nothing but management.
Test failover by physically unplugging things during commissioning. Automatic failover that has never been proven is a belief, not a feature.
Build it in the lab, not on site
Every remote site kit we ship is racked, configured, labelled and burn-in tested in Erbil before it leaves. It arrives as a single enclosure that needs power, an uplink and an earth. A competent site electrician can bring it up without a network engineer present, and time-to-live is under a day.
Two things make this work: identical configuration across sites, so that any engineer can predict what they will find, and a printed one-page diagram inside the cabinet door showing what each port and cable does. That laminated sheet has resolved more incidents over the phone than any documentation portal.
Spares, on site
A spare switch, firewall, modem and set of patch leads live in the cabinet from day one. The cost of the spare is trivial next to the cost of a two-day courier journey plus the outage. For clusters of sites, hold a shared spares kit at the nearest camp rather than one per site.
The commissioning checklist we will not skip. Pull the primary link and confirm failover. Pull site power and confirm the UPS carries the load and the site stays reachable over out-of-band. Cycle a PDU outlet remotely. Confirm every alarm reaches the NOC. Photograph the finished installation and file it with the as-built. Then hand over.
Physical security is part of IT here
Remote sites lose equipment. Lockable enclosures, door contacts wired into monitoring, and cameras covering the cabinet are not paranoia. A door-open alert at three in the morning has prevented more than one loss, and it costs a single sensor.
The short version
- Out-of-band management on every site, always
- Remote-switchable power
- UPS sized for generator changeovers, batteries budgeted for heat
- Sealed and cooled enclosures, not filtered fans
- Three independent paths: primary, secondary on different media, out-of-band
- Pre-built and burn-in tested before shipping
- Identical configuration across sites
- A laminated diagram inside the cabinet door
- Spares on site from day one
- Failover proven by unplugging, at commissioning
None of this is exotic. It is just a different set of defaults from the ones that work in a city office, and the sites will find out which set you used.