BGP Fast Reroute

This chapter provides information about BGP Fast Reroute.

Topics in this chapter include:

Applicability

This chapter was initially written for SR OS Release 14.0.R7, but the MD-CLI in the current edition is based on SR OS Release 20.10.R1.

Overview

Border Gateway Protocol (BGP) is a key protocol for ISPs, supporting inter-Autonomous System (inter-AS) and intra-Autonomous System (intra-AS) applications with many address families. Additionally, ISPs need to maintain the service level agreements with their customers, even in case of network failures.

MPLS Fast Reroute (FRR) is often used to provide resiliency to intra-AS services, and relies on alternate label switched paths being established through the network. Traffic is switched to the alternate path in case of a failure of the primary path.

However, the traffic for inter-AS services crosses the boundaries of multiple ASs, so to provide resiliency, BGP FRR can be used. Before a network failure occurs, multiple paths must be received for a prefix to take advantage of this feature. When a prefix has a backup path and its primary paths fail, the affected traffic is rapidly diverted to the backup path without waiting for the control plane to reconverge. When many prefixes share the same primary paths, and in some cases also the same backup path, the time to divert traffic to the backup path is independent of the number of prefixes; this is also known as Prefix Independent Convergence (PIC). The traffic goes back to the primary paths when those paths are restored. Multiple primary paths can be active simultaneously when Equal Cost Multi Path (ECMP) applies.

Within SR OS, two BGP FRR functions are supported: Core PIC and Edge PIC. Core PIC describes a scenario where a link or node on the path to the BGP next-hop fails, but the BGP next-hop remains reachable; see Core PIC. Edge PIC describes a scenario where an edge node or edge link fails, which results in a change of the BGP next-hop; see Edge PIC.

Figure 1. Core PIC
Figure 2. Edge PIC

Within SR OS, Core PIC is enabled by default and cannot be disabled. Therefore, this chapter will describe the use of Edge PIC.

BGP FRR is supported for different BGP address families in the base router context or within a specific vprn context. This chapter will focus on the IPv4 address family within the base router context.

The following SR OS supported features can be used to allow BGP to maintain multiple paths through an autonomous system:

  • BGP best external

  • BGP add-paths

Convergence goes through several phases, which also apply to BGP:

  • detect the network failure

  • distribute updated routing information, and update the network topology

  • calculate new routes, and optionally change next-hops

  • update the forwarding plane

Several mechanisms are available to enhance BGP network convergence, such as:

  • Bidirectional Forwarding Detection (BFD)

  • Minimum Router Advertisement Interval (MRAI)

  • BGP peer tracking

This chapter describes the use of BFD and MRAI for faster network convergence.

Configuration

The example topology used in this chapter is shown in BGP FRR topology, and has the following characteristics:

  • iBGP sessions are established between AS 65537 routers using RR-6 as route reflector with P-2, P-3, P-4, and PE-5 as route reflector clients.

  • eBGP sessions are established between P-2 and P-3 of AS 65537 and PE-1 of AS 65536.

  • PE-1 advertises a BGP route for prefix 172.16.1.0/24 with community "1:0" to P-2 and P-3.

  • P-2 changes the local preference to 150 for the route advertised to its route reflector RR-6. Import policy "Import_LP150" on P-2 accepts routes with community "1:0" and modifies the local preference to 150; import policy "1:0" on P-3 accepts routes with community "1:0" without modifications.

  • PE-5 advertises a BGP route for prefix 172.16.5.0/24 with community "1:0" to RR-6. This route is re-advertised to the other RR clients and eventually, by P-2 and P-3 to PE-1.

Figure 3. BGP FRR topology

These characteristics enforce traffic for destination 172.16.1.0/24 to leave AS 65537 via P-2. P-2 (and also PE-5) learns the destination and the local preference via route reflector RR-6. But because P-3's own local preference is lower (default LP=100), it stops advertising prefix 172.16.1.0/24 toward RR-6, so that P-4 is aware of the path via P-2 only.

The objective is for P-4 to receive multiple copies of the 172.16.1.0/24 prefix with redundant next-hops, to provide for faster convergence under failure. Considering the characteristics previously listed for the topology, two features contribute for achieving this goal:

  1. Using BGP best external

  2. Using BGP add-paths

The BGP add-paths feature is required in scenarios with route-reflectors, possibly combined with the BGP best external feature. The BGP best external feature can be used without BGP add-paths in scenarios when the BGP peers are in a full mesh.

As a result, multiple exit paths for prefix 172.16.1.0/24 leaving AS 65537 are available, improving convergence time on the iBGP peers because they only need to update their FIBs if they lose the primary route.

BGP best external

P-3 is configured with the BGP best external feature, as follows:

# on P-3:
configure {
    router "Base" {
        autonomous-system 65537
        bgp {
            loop-detect discard-route
            advertise-inactive true
            split-horizon true
            advertise-external {
                ipv4 true
            }
            group "eBGP_AS65536" {
                peer-as 65536
                import {
                    policy ["1:0"]
                }
                export {
                    policy ["1:0"]
                }
            }
            group "iBGP_AS65537" {
                next-hop-self true
                peer-as 65537
            }
            neighbor "10.0.13.1" {
                group "eBGP_AS65536"
            }
            neighbor "192.0.2.6" {
                group "iBGP_AS65537"
            }

In this output, advertise-external is activated for the IPv4 address family only. It can also be activated for the IPv6, label-IPv4, and label-IPv6 address families.

Although it is not necessary to also enable BGP best external on P-2, it is not uncommon to also configure this feature on other autonomous system border routers.

P-3 advertises prefix 172.16.1.0/24 toward the route reflector RR-6, as follows:

[]
A:admin@P-3# show router bgp neighbor 192.0.2.6 advertised-routes
===============================================================================
 BGP Router ID:192.0.2.3        AS:65537       Local AS:65537
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP IPv4 Routes
===============================================================================
Flag  Network                                            LocalPref   MED
      Nexthop (Router)                                   Path-Id     IGP Cost
      As-Path                                                        Label
-------------------------------------------------------------------------------
i     172.16.1.0/24                                      100         None
      192.0.2.3                                          None        0
      65536                                                          -
-------------------------------------------------------------------------------
Routes : 1
===============================================================================

The BGP best external feature is sufficient for providing alternate paths in a fully meshed autonomous system, and could be used in conjunction with the BGP add-paths feature. The BGP add-paths feature is a requirement in scenarios with route reflectors.

BGP add-paths

P-2, P-3, P-4 and RR-6 are configured with the BGP add-paths feature. PE-5 does not require the add-paths feature, because the alternate path to 172.16.1.0/24 starts in P-4.

# on P-2, P-3, P-4, RR-6:
configure {
    router "Base" {
        bgp {
            group "iBGP_AS65537" {
                add-paths {
                    ipv4 {
                        send 2
                        receive true
                    }
                }
            } 

The BGP configuration on P-2 is as follows:

# on P-2:
configure {
    router "Base" {
        bgp {
            loop-detect discard-route
            advertise-inactive true
            split-horizon true
            group "eBGP_AS65536" {
                peer-as 65536
                import {
                    policy ["Import_LP150"]
                }
                export {
                    policy ["1:0"]
                }
            }
            group "iBGP_AS65537" {
                next-hop-self true
                peer-as 65537
                add-paths {
                    ipv4 {
                        send 2
                        receive true
                    }
                }
            }
            neighbor "10.0.12.1" {
                group "eBGP_AS65536"
            }
            neighbor "192.0.2.6" {
                group "iBGP_AS65537"
            }
        }

The BGP configuration for P-3 and P-4 is very similar and is not shown here.

The BGP configuration on RR-6 then is as follows:

# on RR-6:
configure {
    router "Base" {
        autonomous-system 65537
        bgp {
            loop-detect discard-route
            split-horizon true
            group "iBGP_AS65537" {
                peer-as 65537
                advertise-inactive true
                cluster {
                    cluster-id 6.6.6.6
                }
                add-paths {
                    ipv4 {
                        send 2
                        receive true
                    }
                }
            }
            neighbor "192.0.2.2" {
                group "iBGP_AS65537"
            }
            neighbor "192.0.2.3" {
                group "iBGP_AS65537"
            }
            neighbor "192.0.2.4" {
                group "iBGP_AS65537"
            }
            neighbor "192.0.2.5" {
                group "iBGP_AS65537"
            }

The default behavior of a route reflector is to only consider the best path. By enabling the add-paths feature on RR-6, multiple paths are considered.

Both P-2 and P-3 advertise route 172.16.1.0/24 to RR-6, as follows:

[]
A:admin@P-2# show router bgp neighbor 192.0.2.6 advertised-routes
===============================================================================
 BGP Router ID:192.0.2.2        AS:65537       Local AS:65537
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP IPv4 Routes
===============================================================================
Flag  Network                                            LocalPref   MED
      Nexthop (Router)                                   Path-Id     IGP Cost
      As-Path                                                        Label
-------------------------------------------------------------------------------
i     172.16.1.0/24                                      150         None
      192.0.2.2                                          1           0
      65536                                                          -
-------------------------------------------------------------------------------
Routes : 1
===============================================================================
[]
A:admin@P-3# show router bgp neighbor 192.0.2.6 advertised-routes
===============================================================================
 BGP Router ID:192.0.2.3        AS:65537       Local AS:65537
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP IPv4 Routes
===============================================================================
Flag  Network                                            LocalPref   MED
      Nexthop (Router)                                   Path-Id     IGP Cost
      As-Path                                                        Label
-------------------------------------------------------------------------------
i     172.16.1.0/24                                      100         None
      192.0.2.3                                          1           0
      65536                                                          -
-------------------------------------------------------------------------------
Routes : 1
===============================================================================

For more examples of the BGP add-paths feature, see BGP Add-Path and BGP Multipath in this guide.

Backup path

P-4 is the place in the topology where an alternate path is created. The data plane part of the Edge PIC configuration is performed by enabling the backup-path command within the bgp context. In the following, backup-paths are considered for the IPv4 address family only, but the IPv6, label-IPv4, and label-IPv6 address families are allowed too.

# on PE-1, P-4:
configure {
    router "Base" {
        bgp {
            backup-path {
                ipv4 true
            } 

In this way, BGP considers all alternate paths which are present through the BGP best external and BGP add-paths feature. The BGP configuration on P-4 is as follows:

# on P-4:
configure {
    router "Base" {
        autonomous-system 65537
        bgp {
            loop-detect discard-route
            split-horizon true
            backup-path {
                ipv4 true
            }
            group "iBGP_AS65537" {
                peer-as 65537
                add-paths {
                    ipv4 {
                        send 2
                        receive true
                    }
                }
            }
            neighbor "192.0.2.6" {
                group "iBGP_AS65537"
            }

In the default BGP behavior, without the backup-path command, two BGP routes exist. Both routes are valid, but only the first one is the best path (indicated by ">"), as follows:

[]
A:admin@P-4# show router bgp routes 172.16.1.0/24
===============================================================================
 BGP Router ID:192.0.2.4        AS:65537       Local AS:65537
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP IPv4 Routes
===============================================================================
Flag  Network                                            LocalPref   MED
      Nexthop (Router)                                   Path-Id     IGP Cost
      As-Path                                                        Label
-------------------------------------------------------------------------------
u*>i  172.16.1.0/24                                      150         None
      192.0.2.2                                          4           10
      65536                                                          -
*i    172.16.1.0/24                                      100         None
      192.0.2.3                                          5           10
      65536                                                          -
-------------------------------------------------------------------------------
Routes : 2
===============================================================================

The routing table then is as follows:

[]
A:admin@P-4# show router route-table protocol bgp

===============================================================================
Route Table (Router: Base)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
172.16.1.0/24                                 Remote  BGP       00h17m19s  170
       192.168.24.1                                                 0
-------------------------------------------------------------------------------
No. of Routes: 1
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================

With the backup-path command, again both BGP routes are valid; the first route is the best path, and now the second route is explicitly marked to be a backup path (indicated by "b"), as follows:

[]
A:admin@P-4# show router bgp routes 172.16.1.0/24
===============================================================================
 BGP Router ID:192.0.2.4        AS:65537       Local AS:65537
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP IPv4 Routes
===============================================================================
Flag  Network                                            LocalPref   MED
      Nexthop (Router)                                   Path-Id     IGP Cost
      As-Path                                                        Label
-------------------------------------------------------------------------------
u*>i  172.16.1.0/24                                      150         None
      192.0.2.2                                          4           10
      65536                                                          -
ub*i  172.16.1.0/24                                      100         None
      192.0.2.3                                          5           10
      65536                                                          -
-------------------------------------------------------------------------------
Routes : 2
===============================================================================

Now the routing table is as follows. The "B" flag indicates that a BGP backup path is available.

[]
A:admin@P-4# show router route-table protocol bgp
===============================================================================
Route Table (Router: Base)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
172.16.1.0/24 [B]                             Remote  BGP       00h00m16s  170
       192.168.24.1                                                 0
-------------------------------------------------------------------------------
No. of Routes: 1
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================

To show both routes, use the following command:

[]
A:admin@P-4# show router route-table protocol bgp alternative
===============================================================================
Route Table (Router: Base)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
      Alt-NextHop                                                Alt-
                                                                Metric
-------------------------------------------------------------------------------
172.16.1.0/24                                 Remote  BGP       00h01m36s  170
       192.168.24.1                                                 0
172.16.1.0/24 (Backup)                        Remote  BGP       00h01m36s  170
       192.168.34.1                                                 0
-------------------------------------------------------------------------------
No. of Routes: 2
Flags: n = Number of times nexthop is repeated
       Backup = BGP backup route
       LFA = Loop-Free Alternate nexthop
       S = Sticky ECMP requested
===============================================================================

The currently active next-hop in the forwarding path is 192.168.24.1, as follows:

[]
A:admin@P-4# show router fib 1 ip-prefix-prefix-length 172.16.1.0/24 all
===============================================================================
FIB Display
===============================================================================
Prefix [Flags]                                    Protocol            Installed
  NextHop
-------------------------------------------------------------------------------
172.16.1.0/24                                     BGP                 Y
  192.168.24.1 (int-P-4-P-2)
-------------------------------------------------------------------------------
Total Entries : 1
-------------------------------------------------------------------------------
===============================================================================

The active and standby next-hops are also programmed into the forwarding path, as follows:

[]
A:admin@P-4#  show router fib 1 ip-prefix-prefix-length 172.16.1.0/24 extensive
===============================================================================
FIB Display (Router: Base)
===============================================================================
Dest Prefix             : 172.16.1.0/24
  Protocol              : BGP
  Installed             : Y
  Indirect Next-Hop     : 192.0.2.2
    QoS                 : Priority=n/c, FC=n/c
    Source-Class        : 0
    Dest-Class          : 0
    ECMP-Weight         : 1
    Resolving Next-Hop  : 192.168.24.1
      Interface         : int-P-4-P-2
      ECMP-Weight       : 1
  Indirect Next-Hop     : 192.0.2.3
    QoS                 : Priority=n/c, FC=n/c
    Source-Class        : 0
    Dest-Class          : 0
    ECMP-Weight         : 1
    Backup-Path         : Yes
    Resolving Next-Hop  : 192.168.34.1
      Interface         : int-P-4-P-3
      ECMP-Weight       : 1
===============================================================================
Total Entries : 1
===============================================================================

In summary, two paths are available out of P-4 and leading to 172.16.1.0/24 in the remote AS, but only one is installed in the forwarding plane. The active route is P-4-P-2-PE-1; the backup route is P-4-P-3-PE-1. A traceroute command confirms the active path, as follows:

[]
A:admin@PE-5# traceroute 172.16.1.1 source-address 172.16.5.1 numeric
traceroute to 172.16.1.1 from 172.16.5.1, 30 hops max, 40 byte packets
  1  192.168.45.1    0.722 ms  0.662 ms  0.646 ms
  2  192.168.24.1    1.22 ms  1.21 ms  1.21 ms
  3  172.16.1.1    3.09 ms  1.78 ms  1.74 ms

Faster convergence through BFD

As already described, BFD can help speed up BGP convergence, mainly when detecting network failure. In the following, BFD is enabled on the eBGP sessions, and on the IS-IS protocol.

The BFD parameters are defined at interface level, enabling BFD for an application is done in the application context. Because PE-1 only has eBFD sessions toward P-2 and P-3, it is enabled at the global BGP level, but it can also be enabled at the group or neighbor level.

# on PE-1:
configure {
    router "Base" {
        interface "int-PE-1-P-2" {
            ipv4 {
                bfd {
                admin-state enable
                }
            }
        }
        interface "int-PE-1-P-3" {
            ipv4 {
                bfd {
                admin-state enable
                }
            }
        }
        bgp {
            bfd-liveness true
        }

Because the BFD configuration for P-2 and P-3 is very similar, it is only shown for P-2, as follows:

# for P-2:
configure {
    router "Base" {
        interface "int-P-2-PE-1" {
            ipv4 {
                bfd {
                    admin-state enable
                }
            }
        }
        interface "int-P-2-P-4" {
            ipv4 {
                bfd {
                    admin-state enable
                }
            }
        }

BFD is enabled for group eBGP_AS65536 only, at group level, as follows:

# on P-2:
configure {
    router "Base" {
        bgp {
            group "eBGP_AS65536" {
            bfd-liveness true
            }
        }

BFD for IS-IS is enabled at the IS-IS interface level, and is enabled for IPv4 only, as follows.

# on P-2:
configure {
    router "Base" {
        isis 0 {
            interface "int-P-3-P-4" {
                bfd-liveness {
                    ipv4 {
                    }
                }
            }

Faster convergence through MRAI

Adjusting the BGP MRAI also can help speed up network convergence, using the following command:

*[ex:configure router "Base" bgp]
A:admin@P-2# min-route-advertisement

 min-route-advertisement <number>
 <number>  - <1..255>
 Default   - 30

    Minimum time before a prefix can be advertised to peer

Lowering the MRAI puts a higher load on the CPM, so a trade-off must be made between convergence time and processing load.

Switchover

To demonstrate a switchover scenario, a failure is introduced by disabling port 1/1/1 on PE-1, as follows:

# on PE-1:
configure {
    port 1/1/1 {
        admin-state disable 

The path through the network is PE-5-P-4-P-3-PE-1, as follows:

[]
A:admin@PE-5# traceroute 172.16.1.1 source-address 172.16.5.1 numeric
traceroute to 172.16.1.1 from 172.16.5.1, 30 hops max, 40 byte packets
  1  192.168.45.1    0.698 ms  0.695 ms  0.698 ms
  2  192.168.34.1    1.21 ms  1.21 ms  1.15 ms
  3  172.16.1.1    1.73 ms  1.71 ms  1.70 ms

On P-4, traffic is now diverted to P-3, and the BGP routes are as follows:

[]
A:admin@P-4# show router bgp routes 172.16.1.0/24
===============================================================================
 BGP Router ID:192.0.2.4        AS:65537       Local AS:65537
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP IPv4 Routes
===============================================================================
Flag  Network                                            LocalPref   MED
      Nexthop (Router)                                   Path-Id     IGP Cost
      As-Path                                                        Label
-------------------------------------------------------------------------------
u*>i  172.16.1.0/24                                      100         None
      192.0.2.3                                          1           10
      65536                                                          -
-------------------------------------------------------------------------------
Routes : 1
===============================================================================

The route table is as follows:

[]
A:admin@P-4# show router route-table protocol bgp
===============================================================================
Route Table (Router: Base)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
172.16.1.0/24                                 Remote  BGP       00h01m41s  170
       192.168.34.1                                                 0
-------------------------------------------------------------------------------
No. of Routes: 1
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================

The forwarding plane is reprogrammed to send traffic for the 172.16.1.0/24 subnet to P-3, as follows:

[]
A:admin@P-4# show router fib 1 ip-prefix-prefix-length 172.16.1.0/24 extensive
===============================================================================
FIB Display (Router: Base)
===============================================================================
Dest Prefix             : 172.16.1.0/24
  Protocol              : BGP
  Installed             : Y
  Indirect Next-Hop     : 192.0.2.3
    QoS                 : Priority=n/c, FC=n/c
    Source-Class        : 0
    Dest-Class          : 0
    ECMP-Weight         : 1
    Resolving Next-Hop  : 192.168.34.1
      Interface         : int-P-4-P-3
      ECMP-Weight       : 1
===============================================================================
Total Entries : 1
===============================================================================

Bringing port 1/1/1 on PE-1 up again will result in the path PE-5-P-4-P-2-PE-1 being reactivated. Switchback takes longer, because the external BGP session needs to be re-established, and routes have to be relearned.

Conclusion

BGP FRR provides ISPs the means to offer backup paths with fast switchover times when used in combination with short failure detection times and short advertisement intervals. By guaranteeing service in case of network failures, ISPs can provide enhanced service offerings to their customers.