BGP Convergence - Delayed Route Advertisement

This chapter describes BGP Convergence - Delayed Route Advertisement.

Topics in this chapter include:

Applicability

The information and configuration in this chapter are based on SR OS Release 20.7.R1. BGP Delayed Route Advertisement is supported in SR OS Release 19.7.R1 and later.

Overview

When the BGP process on a router is starting up or restarting, BGP convergence is finished after the restarting router completes the following actions:

  • Reestablish the sessions with configured and discovered BGP neighbors.

  • Relearn BGP routes advertised by the direct BGP neighbors (their best paths plus potentially some additional paths).

  • Advertise to its direct neighbors the locally originated BGP routes plus the received routes from its set of best paths.

By default, the preceding steps are executed in parallel. After the first BGP session is reestablished, the restarting router starts advertising its own best paths to the BGP neighbor, even though it is still learning BGP routes and rebuilding its RIB-IN database. When more BGP sessions come up and more routes are learned, it is possible that routes previously considered best are no longer best, leading to multiple route advertisements for the same prefix, as shown in Default SR OS behavior when the BGP process restarts.

Figure 1. Default SR OS behavior when the BGP process restarts

Multiple route advertisements increase the processing workload on the restarting router and on its BGP neighbors. This lengthens the overall convergence time and it can cause short-term inefficiencies in traffic forwarding.

The BGP delayed route advertisement feature provides the following two convergence timers to offer the operator more control on the BGP convergence process when BGP is starting up or restarting: min-wait-to-advertise and max-wait-to-advertise. This feature applies to IPv4 unicast and IPv6 unicast routes of the base router BGP instance and VPRN BGP instances. BGP convergence tuning allows different timers in the base router and the VPRNs. Also, the max-wait-to-advertise timer can be different for IPv4 and IPv6 address families. BGP convergence tuning with delayed route advertisement shows the BGP convergence tuning.

Figure 2. BGP convergence tuning with delayed route advertisement

When a BGP peer has advertised all its routes for a specific address family, it sends an End of RIB (EOR) marker for each address family; for example, peer 192.0.2.4 sent the following EOR for IPv4:

159 2020/08/06 13:53:07.312 UTC MINOR: DEBUG #2001 Base Peer 1: 192.0.2.4
"Peer 1: 192.0.2.4: UPDATE
Peer 1: 192.0.2.4 - Received BGP UPDATE:
    Withdrawn Length = 0
    Total Path Attr Length = 0
      End-of-Rib marker (IPV4)
"

The restarting node will never advertise routes before the min-wait-to-advertise timer has expired. In BGP convergence tuning with delayed route advertisement, no routes were received at that time, but it is possible. Each peer advertises its routes followed by an EOR message per address family. When the restarting node receives all the expected EOR messages (and after the min-wait-to-advertise timer expires), it starts advertising its best routes. However, if the max-wait-to-advertise timer for the address family expires before all expected EORs have been received, it also starts advertising its best routes.

Note:

The timer values must be chosen well, because it is possible that the convergence degrades instead of improves with unsuitable timer values. The timer values depend on the BGP topology (number of peers, number of prefixes per peer, and BGP activeness of the peers). Timer values can be optimized by trial and error, and may have to be reviewed in case of network changes.

BGP convergence timers shows that the min-wait-to-advertise timer starts when the BGP process starts up or restarts, whereas the max-wait-to-advertise timer starts when the first peer (dynamic or configured) is established. It also shows that BGP convergence tuning does not apply to a new peer (peer-X) that is established after the min-wait-to-advertise timer has expired.

Figure 3. BGP convergence timers

The BGP convergence process can be monitored with the show router bgp convergence command. BGP convergence states shows the different BGP convergence states.

Figure 4. BGP convergence states

The BGP convergence states are:

  • Waiting: when BGP convergence timers are configured and no peer has reconnected yet.

  • Started: when the first peer (dynamic or configured) is established.

  • Partial: when the first EOR is received from a neighbor for a specific address family.

  • Converged: when the last EOR for an address family is received. If that occurs before the max-time-to-advertise timer expires, the restarting node starts advertising its RIB-OUT.

  • Timeout: when the max-wait-to-advertise timer expires before the last EOR for an address family is received. The restarting node advertises its RIB-OUT when the timer expires.

When the feature is implemented, BGP maintains information about the convergence process associated with the last startup.

Configuration

The following example shows the principles of SR OS BGP convergence, whereas real-life examples have much larger numbers of BGP sessions and routes. Example topology shows the example topology with one node in Autonomous System (AS) 64501 and three nodes in AS 64500. On all four nodes, VPRN 1 is configured in AS 64496.

Figure 5. Example topology

Example topology only shows the IPv4 addresses, but all interfaces also have IPv6 addresses.

The initial configuration on the nodes includes:

  • Cards, MDAs, ports

  • Router interfaces, with IPv4 and IPv6 addresses

  • SR-ISIS in the base router on the three nodes in AS 64500

  • IS-IS in VPRN 1 on all four nodes in AS 64496

In the base router, an eBGP session is established between PE-1 in AS 64501 and PE-2 in AS 64500. For the iBGP sessions in AS 64500, PE-2 acts as a Route Reflector (RR). The BGP configuration in the base router on PE-2 is as follows:

# on PE-2:
configure
    router Base
        autonomous-system 64500
        bgp
            split-horizon
            group "eBGP"
                local-as 64500
                peer-as 64501
                local-address "int-PE-2-PE-1"
                neighbor 192.168.12.1
                    family ipv4 
                    next-hop-self
                    export "export-10.21"
                exit
                neighbor 2001:db8::12:1
                    family ipv6
                    next-hop-self
                    export "export-10:21"
                exit
            exit
            group "iBGP-IPv4"
                family ipv4 
                cluster 192.0.2.2
                peer-as 64500
                neighbor 192.0.2.3
                    next-hop-self
                    export "export-10.21"
                exit
                neighbor 192.0.2.4
                    next-hop-self
                    export "export-10.21"
                exit
            exit
            group "iBGP-IPv6"
                family ipv6
                cluster 192.0.2.2
                peer-as 64500
                neighbor 2001:db8::2:3
                    next-hop-self
                    export "export-10:21"
                exit
                neighbor 2001:db8::2:4
                    next-hop-self
                    export "export-10:21"
                exit
            exit
            no shutdown

The export policies are the following:

# on PE-2:
configure
    router Base
        policy-options
            begin
            prefix-list "10.21.0.0/16"
                prefix 10.21.0.0/16 longer
            exit
            prefix-list "_::10:21_"
                prefix 2001:db8::10:21:0:0/120 longer
            exit
            policy-statement "export-10.21"
                entry 10
                    from
                        prefix-list "10.21.0.0/16"
                    exit
                    action accept
                    exit
                exit
            exit
            policy-statement "export-10:21"
                entry 10
                    from
                        prefix-list "_::10:21_"
                    exit
                    action accept
                    exit
                exit
            exi

The BGP configuration in the base router is similar on the other PEs, with similar export policies.

BGP is also configured in VPRN 1, with similar export policies. On RR PE-2, the BGP configuration in VPRN 1 is as follows:

# on PE-2
configure
    service
        vprn 1 name "VPRN 1" customer 1 create
            autonomous-system 64496
            bgp
                router-id 172.31.0.2
                split-horizon
                group "iBGP-VPRN1"
                    cluster 172.31.0.2
                    peer-as 64496
                    neighbor 172.31.0.1
                        family ipv4
                        local-address 172.31.0.2
                        export "export-10.1"
                    exit
                    neighbor 172.31.0.3
                        family ipv4
                        local-address 172.31.0.2
                        export "export-10.1"
                    exit
                    neighbor 172.31.0.4
                        family ipv4
                        local-address 172.31.0.2
                        export "export-10.1"
                    exit
                    neighbor 2001:db8::31:0:1
                        family ipv6
                        export "export-10:1"
                    exit
                    neighbor 2001:db8::31:0:3
                        family ipv6
                        export "export-10:1"
                    exit
                    neighbor 2001:db8::31:0:4
                        family ipv6
                        export "export-10:1"
                    exit
                exit
                no shutdown

The configuration is similar on the other nodes.

The following BGP summary on PE-2 shows the different sessions where PE-2 receives one IPv4 or IPv6 route per neighbor and advertises three IPv4 or IPv6 routes per neighbor, both in the base router (Def. Instance) and in VPRN 1 (Svc: 1):

*A:PE-2# show router bgp summary all 
===============================================================================
BGP Summary
===============================================================================
Legend : D - Dynamic Neighbor
===============================================================================
Neighbor
Description
ServiceId          AS PktRcvd InQ  Up/Down   State|Rcv/Act/Sent (Addr Family)
                      PktSent OutQ
-------------------------------------------------------------------------------
192.0.2.3
Def. Instance  64500        8    0 00h01m56s 1/1/3 (IPv4)
                           11    0           
192.0.2.4
Def. Instance  64500        8    0 00h01m43s 1/1/3 (IPv4)
                           11    0           
192.168.12.1
Def. Instance  64501        9    0 00h02m20s 1/1/3 (IPv4)
                           11    0           
2001:db8::2:3
Def. Instance  64500        8    0 00h01m56s 1/1/3 (IPv6)
                           11    0           
2001:db8::2:4
Def. Instance  64500        8    0 00h01m43s 1/1/3 (IPv6)
                           11    0           
2001:db8::12:1
Def. Instance  64501        9    0 00h02m14s 1/1/3 (IPv6)
                           11    0           
172.31.0.1
Svc: 1         64496        9    0 00h02m01s 1/1/3 (IPv4)
                           12    0           
172.31.0.3
Svc: 1         64496        9    0 00h02m02s 1/1/3 (IPv4)
                           12    0           
172.31.0.4
Svc: 1         64496        8    0 00h01m50s 1/1/3 (IPv4)
                           10    0           
2001:db8::31:0:1
Svc: 1         64496        8    0 00h01m50s 1/1/3 (IPv6)
                           10    0           
2001:db8::31:0:3
Svc: 1         64496        8    0 00h01m50s 1/1/3 (IPv6)
                           10    0           
2001:db8::31:0:4
Svc: 1         64496        8    0 00h01m50s 1/1/3 (IPv6)
                           10    0           
-------------------------------------------------------------------------------

By default, BGP does not delay route advertisement. The following show command on PE-2 shows that no min-wait-to-advertise timer and no max-wait-to-advertise timer is configured (the default value is 0). The number of established peers is 3 for IPv4 and IPv6 in the base router.

*A:PE-2# show router bgp convergence

===============================================================================
BGP IPv4 Convergence
===============================================================================
Min wait advertise timer                    : 0
Established peers at min wait timer expiry  : N/A
Current established peers                   : 3
First session established time              : N/A
Last session established time               : N/A
Max Wait advertise timer                    : 0
Converged peers                             : N/A
Converged state                             : N/A
Converged time                              : N/A
===============================================================================

===============================================================================
BGP IPv6 Convergence
===============================================================================
Min wait advertise timer                    : 0
Established peers at min wait timer expiry  : N/A
Current established peers                   : 3
First session established time              : N/A
Last session established time               : N/A
Max Wait advertise timer                    : 0
Converged peers                             : N/A
Converged state                             : N/A
Converged time                              : N/A
===============================================================================

A similar command can be launched for VPRN 1: show router 1 bgp convergence. The output is similar, but not shown here.

On PE-2, BGP delayed route advertisement is configured with min-wait-to-advertise equal to 20 seconds in the base router and min-wait-to-advertise equal to 60 seconds in VPRN 1. For both cases, the max-wait-to-advertise is three times as long as the min-wait-to-advertise, but it is possible to have different max-wait-to-advertise timers for IPv4 and IPv6.

In this example, BGP delayed route advertisement is only configured on PE-2, while the other nodes keep advertising their routes immediately after the BGP session is reestablished. PE-2 will accept these routes, but it will only advertise them after receiving the last expected EOR for IPv4 or IPv6 (for the base router or VPRN 1) and min-wait-to-advertise timer expires. If the max-wait-to-advertise timer expires before the last expected EOR is received for IPv4 or IPv6, PE-2 will start advertising the received routes.

# on PE-2:
configure
    router Base
        bgp
            convergence
                min-wait-to-advertise 20
                family ipv4
                    max-wait-to-advertise 30
                exit
                family ipv6
                    max-wait-to-advertise 30
                exit
            exit
            info
        exit
    exit
    service
        vprn "VPRN 1" 
            bgp
                convergence
                    min-wait-to-advertise 60
                    family ipv4
                        max-wait-to-advertise 180
                    exit
                    family ipv6
                        max-wait-to-advertise 180
                    exit
                exit

With this configuration, the BGP converged state on PE-2 changes to "waiting", because no BGP sessions are reestablished yet, so no BGP convergence tuning has taken place.

*A:PE-2# show router bgp convergence

===============================================================================
BGP IPv4 Convergence
===============================================================================
Min wait advertise timer                    : 20
Established peers at min wait timer expiry  : 0
Current established peers                   : 3
First session established time              : 00h00m00s
Last session established time               : 00h00m00s
Max Wait advertise timer                    : 30
Converged peers                             : 3
Converged state                             : waiting
Converged time                              : N/A
===============================================================================

===============================================================================
BGP IPv6 Convergence
===============================================================================
Min wait advertise timer                    : 20
Established peers at min wait timer expiry  : 0
Current established peers                   : 3
First session established time              : 00h00m00s
Last session established time               : 00h00m00s
Max Wait advertise timer                    : 30
Converged peers                             : 3
Converged state                             : waiting
Converged time                              : N/A
===============================================================================

The show router 1 bgp convergence command shows a similar output for VPRN 1, but is not shown here.

The following clear command on PE-2 causes BGP to restart:

# on PE-2:
clear router bgp protocol

When BGP restarts, the converged state remains "waiting" until the first peer is established.

With the first peer established, the converged state changes to "started", as follows:

*A:PE-2# show router bgp convergence

===============================================================================
BGP IPv4 Convergence
===============================================================================
Min wait advertise timer                    : 20
Established peers at min wait timer expiry  : 0
Current established peers                   : 3
First session established time              : 00h00m01s
Last session established time               : 00h00m01s
Max Wait advertise timer                    : 30
Converged peers                             : 0
Converged state                             : started
Converged time                              : N/A
===============================================================================

===============================================================================
BGP IPv6 Convergence
===============================================================================
Min wait advertise timer                    : 20
Established peers at min wait timer expiry  : 0
Current established peers                   : 3
First session established time              : 00h00m01s
Last session established time               : 00h00m01s
Max Wait advertise timer                    : 30
Converged peers                             : 0
Converged state                             : started
Converged time                              : N/A
===============================================================================

The show router 1 bgp convergence command shows a similar output for VPRN 1, but is not shown here.

After a few seconds, PE-2 receives IPv4 and IPv6 routes from PE-3 and PE-4, both in the base router and VPRN 1, as follows:

*A:PE-2# show router bgp summary all
===============================================================================
BGP Summary
===============================================================================
Legend : D - Dynamic Neighbor
===============================================================================
Neighbor
Description
ServiceId          AS PktRcvd InQ  Up/Down   State|Rcv/Act/Sent (Addr Family)
                      PktSent OutQ
-------------------------------------------------------------------------------
192.0.2.3
Def. Instance  64500        5    0 00h00m01s 1/1/0 (IPv4)
                            4    0
192.0.2.4
Def. Instance  64500        5    0 00h00m01s 1/1/0 (IPv4)
                            4    0
192.168.12.1
Def. Instance  64501        3    0 00h00m00s 0/0/0 (IPv4)
                            4    0
2001:db8::2:3
Def. Instance  64500        5    0 00h00m01s 1/1/0 (IPv6)
                            4    0
2001:db8::2:4
Def. Instance  64500        5    0 00h00m01s 1/1/0 (IPv6)
                            4    0
2001:db8::12:1
Def. Instance  64501        3    0 00h00m00s 0/0/0 (IPv6)
                            4    0
172.31.0.1
Svc: 1         64496        3    0 00h00m00s 0/0/0 (IPv4)
                            3    0
172.31.0.3
Svc: 1         64496        5    0 00h00m01s 1/1/0 (IPv4)
                            4    0
172.31.0.4
Svc: 1         64496        5    0 00h00m01s 1/1/0 (IPv4)
                            4    0
2001:db8::31:0:1
Svc: 1         64496        3    0 00h00m00s 0/0/0 (IPv6)
                            3    0
2001:db8::31:0:3
Svc: 1         64496        5    0 00h00m01s 1/1/0 (IPv6)
                            4    0
2001:db8::31:0:4
Svc: 1         64496        5    0 00h00m01s 1/1/0 (IPv6)
                            4    0
-------------------------------------------------------------------------------

PE-2 accepts the received routes, but does not advertise the routes because the min-wait-to-advertise timer has not expired yet, and PE-2 only received IPv4 and IPv6 routes and EORs from PE-3 and PE-4, not from PE-1, so the converged state is "partial", as follows:

*A:PE-2# show router bgp convergence
===============================================================================
BGP IPv4 Convergence
===============================================================================
Min wait advertise timer                    : 20
Established peers at min wait timer expiry  : 0
Current established peers                   : 3
First session established time              : 00h00m01s
Last session established time               : 00h00m02s
Max Wait advertise timer                    : 30
Converged peers                             : 2
Converged state                             : partial
Converged time                              : N/A
===============================================================================
===============================================================================
BGP IPv6 Convergence
===============================================================================
Min wait advertise timer                    : 20
Established peers at min wait timer expiry  : 0
Current established peers                   : 3
First session established time              : 00h00m01s
Last session established time               : 00h00m02s
Max Wait advertise timer                    : 30
Converged peers                             : 2
Converged state                             : partial
Converged time                              : N/A
===============================================================================

The show router 1 bgp convergence command shows a similar output for VPRN 1, but is not shown here.

After a few seconds, all IPv4 and IPv6 routes have been received in the base router. PE-2 has received an EOR message from each neighbor in the base router. The following BGP summary shows that PE-2 has received and advertised all IPv4 and IPv6 routes in the base router, whereas it only received IPv4 and IPv6 routes from two neighbors in VPRN 1, not yet from PE-1:

*A:PE-2# show router bgp summary all
===============================================================================
BGP Summary
===============================================================================
Legend : D - Dynamic Neighbor
===============================================================================
Neighbor
Description
ServiceId          AS PktRcvd InQ  Up/Down   State|Rcv/Act/Sent (Addr Family)
                      PktSent OutQ
-------------------------------------------------------------------------------
192.0.2.3
Def. Instance  64500        5    0 00h00m12s 1/1/3 (IPv4)
                            7    0
192.0.2.4
Def. Instance  64500        5    0 00h00m11s 1/1/3 (IPv4)
                            7    0
192.168.12.1
Def. Instance  64501        5    0 00h00m14s 1/1/3 (IPv4)
                            5    0
2001:db8::2:3
Def. Instance  64500        5    0 00h00m12s 1/1/3 (IPv6)
                            7    0
2001:db8::2:4
Def. Instance  64500        5    0 00h00m11s 1/1/3 (IPv6)
                            7    0
2001:db8::12:1
Def. Instance  64501        5    0 00h00m13s 1/1/3 (IPv6)
                            5    0
172.31.0.1
Svc: 1         64496        4    0 00h00m13s 0/0/0 (IPv4)
                            4    0
172.31.0.3
Svc: 1         64496        5    0 00h00m12s 1/1/0 (IPv4)
                            4    0
172.31.0.4
Svc: 1         64496        5    0 00h00m11s 1/1/0 (IPv4)
                            4    0
2001:db8::31:0:1
Svc: 1         64496        4    0 00h00m13s 0/0/0 (IPv6)
                            4    0
2001:db8::31:0:3
Svc: 1         64496        5    0 00h00m12s 1/1/0 (IPv6)
                            4    0
2001:db8::31:0:4
Svc: 1         64496        5    0 00h00m11s 1/1/0 (IPv6)
                            4    0
-------------------------------------------------------------------------------

As a result of this, BGP is in the "converged" state in the base router, both for IPv4 and IPv6, as follows:

*A:PE-2# show router bgp convergence
===============================================================================
BGP IPv4 Convergence
===============================================================================
Min wait advertise timer                    : 20
Established peers at min wait timer expiry  : 3
Current established peers                   : 3
First session established time              : 00h00m01s
Last session established time               : 00h00m03s
Max Wait advertise timer                    : 30
Converged peers                             : 3
Converged state                             : converged
Converged time                              : 00h00m20s
===============================================================================
===============================================================================
BGP IPv6 Convergence
===============================================================================
Min wait advertise timer                    : 20
Established peers at min wait timer expiry  : 3
Current established peers                   : 3
First session established time              : 00h00m01s
Last session established time               : 00h00m03s
Max Wait advertise timer                    : 30
Converged peers                             : 3
Converged state                             : converged
Converged time                              : 00h00m20s
===============================================================================

The converged time is only applicable in the "converged" state and is measured relative to BGP instance restart at time T=0.

BGP is still in the "partial" state within the VPRN 1 context, both for IPv4 and IPv6, as follows:

*A:PE-2# show router 1 bgp convergence
===============================================================================
BGP IPv4 Convergence
===============================================================================
Min wait advertise timer                    : 60
Established peers at min wait timer expiry  : 0
Current established peers                   : 3
First session established time              : 00h00m01s
Last session established time               : 00h00m03s
Max Wait advertise timer                    : 180
Converged peers                             : 3
Converged state                             : partial
Converged time                              : N/A
===============================================================================
===============================================================================
BGP IPv6 Convergence
===============================================================================
Min wait advertise timer                    : 60
Established peers at min wait timer expiry  : 0
Current established peers                   : 3
First session established time              : 00h00m01s
Last session established time               : 00h00m03s
Max Wait advertise timer                    : 180
Converged peers                             : 3
Converged state                             : partial
Converged time                              : N/A
===============================================================================

After a while, PE-1 also advertises its routes for VPRN 1, followed by EORs for IPv4 and IPv6. BGP converges for VPRN 1, as follows:

*A:PE-2# show router 1 bgp convergence
===============================================================================
BGP IPv4 Convergence
===============================================================================
Min wait advertise timer                    : 60
Established peers at min wait timer expiry  : 3
Current established peers                   : 3
First session established time              : 00h00m01s
Last session established time               : 00h00m03s
Max Wait advertise timer                    : 180
Converged peers                             : 3
Converged state                             : converged
Converged time                              : 00h01m00s
===============================================================================
===============================================================================
BGP IPv6 Convergence
===============================================================================
Min wait advertise timer                    : 60
Established peers at min wait timer expiry  : 3
Current established peers                   : 3
First session established time              : 00h00m01s
Last session established time               : 00h00m03s
Max Wait advertise timer                    : 180
Converged peers                             : 3
Converged state                             : converged
Converged time                              : 00h01m00s
===============================================================================

Conclusion

With BGP convergence tuning (by means of delaying route advertisements using two timers), less path churn and fewer advertisements can result in faster convergence. BGP convergence is mainly important in scaled environments (high number of BGP sessions and routes). As a result, the advertised paths are more optimal. The BGP convergence process can be monitored using a show command.