ComoreTel — Distributed Private Cloud & Infrastructure Modernization: Build Process 3

Moved a telecom routing operation off unreliable commodity desktop hardware onto a geographically redundant private cloud, end to end — including the procurement, logistics, and physical build: How the implementation kept moving without adding avoidable complexity.

private-cloud-infrastructure-modernization Nov 10, 2016/4 min read
On this page

This build step focused on Geographic and network redundancy — multiple fiber providers, physically separate carrier routes, and distributed DNS (4 US, 2 UK servers) for both site- and equipment-level fault tolerance. That mattered because the project could not move forward until this part of the system was stable enough to trust. The implementation stayed close to the real stack and the actual workflow, so the result was something the business could keep using rather than a prototype that only worked in isolation.

Geographic and network redundancy — multiple fiber providers, physically separate carrier routes, and distributed DNS (4 US, 2 UK servers) for both site- and equipment-level fault tolerance. In practice, that meant the work touched the core path rather than the edges, whether the problem was data, infrastructure, automation, or user flow. The point was to remove the part of the system that kept forcing the same manual effort or failure mode back into the process.

The stack stayed aligned to the same constraint that showed up in the source material: qemu/libvirt · debian/rhel linux · cisco firewalls & switching · distributed dns · voip/telecom routing · elk stack · rsyslog · shell automation · vm templating · fiber redundancy engineering. That kept the build from drifting into unnecessary complexity.

Why it mattered

Supported 3+ million telecom minutes/month across 100+ countries.

The tradeoff was usually between speed, control, and maintenance overhead. The chosen step accepted one of those costs so the system could produce the actual business outcome instead of a temporary improvement that would fail under load.

That is why the build phase matters on these projects: it shows the specific mechanism that turns the earlier analysis into something operable. Without this step, the earlier design discussion would remain abstract.

The implementation stayed close to QEMU/libvirt, Debian/RHEL Linux, Cisco firewalls & switching, distributed DNS because the new system still had to live inside the same operating environment as the old one. That kept the work from drifting into a clean-room exercise that would look better on paper than it would in production. The practical question was always whether the implementation could hold up under the real workflow and the real users. If it could not do that, it was not finished.

The constraint behind the step was that Production ran on repurposed commodity desktops bought through secondary channels — unreliable, inconsistent, hard to scale, and space- and cooling-hungry, with recurring service-impacting outages. That is why the work had to trade one kind of cost for another instead of trying to eliminate cost altogether. In almost every case, the useful move was to spend a little more effort on clarity, validation, or control so the business would spend less effort on repeated manual work later. That is the pattern the project files keep pointing to.

The role in the work was Sole infrastructure architect, systems engineer, deployment lead, and procurement owner. That meant the implementation could not stop at the code boundary because the operating model, handoff, and support path were part of the outcome. The relevant outcome was Supported 3+ million telecom minutes/month across 100+ countries. The build only earns its place if the new result is visible in the way the business works after launch.

The specific step in this article was Geographic and network redundancy — multiple fiber providers, physically separate carrier routes, and distributed DNS (4 US, 2 UK servers) for both site- and equipment-level fault tolerance. That is the piece that moves the story from analysis into execution. It is also the part that shows the difference between a conceptual fix and a system people can actually use. That distinction matters more than style or novelty.