BGP communities, a few lessons from the field
Traffic engineering with BGP communities across two datacenters: what worked, which pager went off at night, and the checklists that came out of it.
Communities are the oldest trick in the BGP book, and still the most underused. A community is just a tag you attach to a route, but upstreams honor a growing vocabulary of them: prepend, local-preference pinning, blackhole, anycast region hints. Used consistently, they turn a dual-datacenter setup from “failover hope” into an actual steering wheel.
This post collects what we verified in production while engineering redundant peering for an operator environment, anonymized as always.
The baseline
The reference topology is two datacenters announcing the same /32 RIPE prefix through two transit providers each, connected by a private interconnect for state sync. The entire traffic steering policy reduces to four BGP community values: prepend x1, x2, x3, and a guarded blackhole, applied consistently at both sites.
The topology is deliberately boring: two datacenters, one /32 RIPE block announced from both, private interconnect for sync. Both sites receive full tables from two transits each. The interesting part is not the diagram, it is the policy attached to it.
Communities that earned their keep
Three BGP community conventions carried all the traffic steering weight: a canonical prepend ladder (x1, x2, x3) owned by a single ASN, inbound community rewriting at the edge into that canonical space, and a rate-limited blackhole community guarded like a firewall rule. The key insight is owning the vocabulary end-to-end.
Three conventions carried all the weight:
# Legible, stable, documented in the peering DB
65000:100 prepend x1 toward the tagged neighbor
65000:200 prepend x2
65000:300 prepend x3
65000:666 blackhole (URF-filtered, max-prefix protected)Two details matter more than the scheme itself:
- One ASN owns the vocabulary. Mixing your numbering with each transit’s own community table is how 3 a.m. pages are born. We rewrite inbound communities at the edge into our canonical space, and only the canonical space appears in route-maps.
- The blackhole community is guarded. It is advertised only to upstreams that filter it correctly, and it is rate-limited on our side. A community that drops traffic is powerful enough to deserve the same review as a firewall rule.
The checklists
Two operational artifacts emerged from production incidents: a per-service drain procedure that shifts traffic with BGP communities before touching sessions, with expected packet counters at each step, and a “prove it” cron of synthetic probes from outside both sites that alert when the winning path diverges from the intended one.
After the inevitable night page (a maintenance where one site’s announcements were withdrawn a whole minute before the other site’s were accepted, leaving a window of nothing), two artifacts emerged:
- A drain procedure: per-service, per-direction steps to shift traffic with communities before touching sessions, with expected packet counters at each step.
- A “prove it” cron: synthetic probes from outside both sites that alert when the winning path is not the intended one. Steering without observation is just latency with confidence.
What we would do differently
Start active-active from day one with community-based preference instead of primary/backup. Backup configurations ossify quickly: the day you need them, they describe a network that no longer exists. And treat your community policy as executable documentation: writing it down is part of shipping it.
Announce from both sites from day one with prepends, instead of starting primary/backup and evolving later. Backup configurations ossify: the day you need them, they describe a network that no longer exists. Active-active with community-based preference degrades more gracefully than active-anything.
If you take one thing from this: your community policy is not configuration, it is documentation that executes. Treat writing it down as part of shipping it.