BGP Convergence - Delayed Route Advertisement
This chapter describes BGP Convergence - Delayed Route Advertisement.
Topics in this chapter include:
Applicability
The information and configuration in this chapter are based on SR OS Release 20.7.R1. BGP Delayed Route Advertisement is supported in SR OS Release 19.7.R1 and later.
Overview
When the BGP process on a router is starting up or restarting, BGP convergence is finished after the restarting router completes the following actions:
Reestablish the sessions with configured and discovered BGP neighbors.
Relearn BGP routes advertised by the direct BGP neighbors (their best paths plus potentially some additional paths).
Advertise to its direct neighbors the locally originated BGP routes plus the received routes from its set of best paths.
By default, the preceding steps are executed in parallel. After the first BGP session is reestablished, the restarting router starts advertising its own best paths to the BGP neighbor, even though it is still learning BGP routes and rebuilding its RIB-IN database. When more BGP sessions come up and more routes are learned, it is possible that routes previously considered best are no longer best, leading to multiple route advertisements for the same prefix, as shown in Default SR OS behavior when the BGP process restarts.
Multiple route advertisements increase the processing workload on the restarting router and on its BGP neighbors. This lengthens the overall convergence time and it can cause short-term inefficiencies in traffic forwarding.
The BGP delayed route advertisement feature provides the following two convergence timers to offer the operator more control on the BGP convergence process when BGP is starting up or restarting: min-wait-to-advertise and max-wait-to-advertise. This feature applies to IPv4 unicast and IPv6 unicast routes of the base router BGP instance and VPRN BGP instances. BGP convergence tuning allows different timers in the base router and the VPRNs. Also, the max-wait-to-advertise timer can be different for IPv4 and IPv6 address families. BGP convergence tuning with delayed route advertisement shows the BGP convergence tuning.
When a BGP peer has advertised all its routes for a specific address family, it sends an End of RIB (EOR) marker for each address family; for example, peer 192.0.2.4 sent the following EOR for IPv4:
159 2020/08/06 13:53:07.312 UTC MINOR: DEBUG #2001 Base Peer 1: 192.0.2.4
"Peer 1: 192.0.2.4: UPDATE
Peer 1: 192.0.2.4 - Received BGP UPDATE:
Withdrawn Length = 0
Total Path Attr Length = 0
End-of-Rib marker (IPV4)
"
The restarting node will never advertise routes before the min-wait-to-advertise timer has expired. In BGP convergence tuning with delayed route advertisement, no routes were received at that time, but it is possible. Each peer advertises its routes followed by an EOR message per address family. When the restarting node receives all the expected EOR messages (and after the min-wait-to-advertise timer expires), it starts advertising its best routes. However, if the max-wait-to-advertise timer for the address family expires before all expected EORs have been received, it also starts advertising its best routes.
The timer values must be chosen well, because it is possible that the convergence degrades instead of improves with unsuitable timer values. The timer values depend on the BGP topology (number of peers, number of prefixes per peer, and BGP activeness of the peers). Timer values can be optimized by trial and error, and may have to be reviewed in case of network changes.
BGP convergence timers shows that the min-wait-to-advertise timer starts when the BGP process starts up or restarts, whereas the max-wait-to-advertise timer starts when the first peer (dynamic or configured) is established. It also shows that BGP convergence tuning does not apply to a new peer (peer-X) that is established after the min-wait-to-advertise timer has expired.
The BGP convergence process can be monitored with the show router bgp convergence command. BGP convergence states shows the different BGP convergence states.
The BGP convergence states are:
Waiting: when BGP convergence timers are configured and no peer has reconnected yet.
Started: when the first peer (dynamic or configured) is established.
Partial: when the first EOR is received from a neighbor for a specific address family.
Converged: when the last EOR for an address family is received. If that occurs before the max-time-to-advertise timer expires, the restarting node starts advertising its RIB-OUT.
Timeout: when the max-wait-to-advertise timer expires before the last EOR for an address family is received. The restarting node advertises its RIB-OUT when the timer expires.
When the feature is implemented, BGP maintains information about the convergence process associated with the last startup.
Configuration
The following example shows the principles of SR OS BGP convergence, whereas real-life examples have much larger numbers of BGP sessions and routes. Example topology shows the example topology with one node in Autonomous System (AS) 64501 and three nodes in AS 64500. On all four nodes, VPRN 1 is configured in AS 64496.
Example topology only shows the IPv4 addresses, but all interfaces also have IPv6 addresses.
The initial configuration on the nodes includes:
Cards, MDAs, ports
Router interfaces, with IPv4 and IPv6 addresses
SR-ISIS in the base router on the three nodes in AS 64500
IS-IS in VPRN 1 on all four nodes in AS 64496
In the base router, an eBGP session is established between PE-1 in AS 64501 and PE-2 in AS 64500. For the iBGP sessions in AS 64500, PE-2 acts as a Route Reflector (RR). The BGP configuration in the base router on PE-2 is as follows:
# on PE-2:
configure
router Base
autonomous-system 64500
bgp
split-horizon
group "eBGP"
local-as 64500
peer-as 64501
local-address "int-PE-2-PE-1"
neighbor 192.168.12.1
family ipv4
next-hop-self
export "export-10.21"
exit
neighbor 2001:db8::12:1
family ipv6
next-hop-self
export "export-10:21"
exit
exit
group "iBGP-IPv4"
family ipv4
cluster 192.0.2.2
peer-as 64500
neighbor 192.0.2.3
next-hop-self
export "export-10.21"
exit
neighbor 192.0.2.4
next-hop-self
export "export-10.21"
exit
exit
group "iBGP-IPv6"
family ipv6
cluster 192.0.2.2
peer-as 64500
neighbor 2001:db8::2:3
next-hop-self
export "export-10:21"
exit
neighbor 2001:db8::2:4
next-hop-self
export "export-10:21"
exit
exit
no shutdown
The export policies are the following:
# on PE-2:
configure
router Base
policy-options
begin
prefix-list "10.21.0.0/16"
prefix 10.21.0.0/16 longer
exit
prefix-list "_::10:21_"
prefix 2001:db8::10:21:0:0/120 longer
exit
policy-statement "export-10.21"
entry 10
from
prefix-list "10.21.0.0/16"
exit
action accept
exit
exit
exit
policy-statement "export-10:21"
entry 10
from
prefix-list "_::10:21_"
exit
action accept
exit
exit
exi
The BGP configuration in the base router is similar on the other PEs, with similar export policies.
BGP is also configured in VPRN 1, with similar export policies. On RR PE-2, the BGP configuration in VPRN 1 is as follows:
# on PE-2
configure
service
vprn 1 name "VPRN 1" customer 1 create
autonomous-system 64496
bgp
router-id 172.31.0.2
split-horizon
group "iBGP-VPRN1"
cluster 172.31.0.2
peer-as 64496
neighbor 172.31.0.1
family ipv4
local-address 172.31.0.2
export "export-10.1"
exit
neighbor 172.31.0.3
family ipv4
local-address 172.31.0.2
export "export-10.1"
exit
neighbor 172.31.0.4
family ipv4
local-address 172.31.0.2
export "export-10.1"
exit
neighbor 2001:db8::31:0:1
family ipv6
export "export-10:1"
exit
neighbor 2001:db8::31:0:3
family ipv6
export "export-10:1"
exit
neighbor 2001:db8::31:0:4
family ipv6
export "export-10:1"
exit
exit
no shutdown
The configuration is similar on the other nodes.
The following BGP summary on PE-2 shows the different sessions where PE-2 receives one IPv4 or IPv6 route per neighbor and advertises three IPv4 or IPv6 routes per neighbor, both in the base router (Def. Instance) and in VPRN 1 (Svc: 1):
*A:PE-2# show router bgp summary all
===============================================================================
BGP Summary
===============================================================================
Legend : D - Dynamic Neighbor
===============================================================================
Neighbor
Description
ServiceId AS PktRcvd InQ Up/Down State|Rcv/Act/Sent (Addr Family)
PktSent OutQ
-------------------------------------------------------------------------------
192.0.2.3
Def. Instance 64500 8 0 00h01m56s 1/1/3 (IPv4)
11 0
192.0.2.4
Def. Instance 64500 8 0 00h01m43s 1/1/3 (IPv4)
11 0
192.168.12.1
Def. Instance 64501 9 0 00h02m20s 1/1/3 (IPv4)
11 0
2001:db8::2:3
Def. Instance 64500 8 0 00h01m56s 1/1/3 (IPv6)
11 0
2001:db8::2:4
Def. Instance 64500 8 0 00h01m43s 1/1/3 (IPv6)
11 0
2001:db8::12:1
Def. Instance 64501 9 0 00h02m14s 1/1/3 (IPv6)
11 0
172.31.0.1
Svc: 1 64496 9 0 00h02m01s 1/1/3 (IPv4)
12 0
172.31.0.3
Svc: 1 64496 9 0 00h02m02s 1/1/3 (IPv4)
12 0
172.31.0.4
Svc: 1 64496 8 0 00h01m50s 1/1/3 (IPv4)
10 0
2001:db8::31:0:1
Svc: 1 64496 8 0 00h01m50s 1/1/3 (IPv6)
10 0
2001:db8::31:0:3
Svc: 1 64496 8 0 00h01m50s 1/1/3 (IPv6)
10 0
2001:db8::31:0:4
Svc: 1 64496 8 0 00h01m50s 1/1/3 (IPv6)
10 0
-------------------------------------------------------------------------------
By default, BGP does not delay route advertisement. The following show command on PE-2 shows that no min-wait-to-advertise timer and no max-wait-to-advertise timer is configured (the default value is 0). The number of established peers is 3 for IPv4 and IPv6 in the base router.
*A:PE-2# show router bgp convergence
===============================================================================
BGP IPv4 Convergence
===============================================================================
Min wait advertise timer : 0
Established peers at min wait timer expiry : N/A
Current established peers : 3
First session established time : N/A
Last session established time : N/A
Max Wait advertise timer : 0
Converged peers : N/A
Converged state : N/A
Converged time : N/A
===============================================================================
===============================================================================
BGP IPv6 Convergence
===============================================================================
Min wait advertise timer : 0
Established peers at min wait timer expiry : N/A
Current established peers : 3
First session established time : N/A
Last session established time : N/A
Max Wait advertise timer : 0
Converged peers : N/A
Converged state : N/A
Converged time : N/A
===============================================================================
A similar command can be launched for VPRN 1: show router 1 bgp convergence. The output is similar, but not shown here.
On PE-2, BGP delayed route advertisement is configured with min-wait-to-advertise equal to 20 seconds in the base router and min-wait-to-advertise equal to 60 seconds in VPRN 1. For both cases, the max-wait-to-advertise is three times as long as the min-wait-to-advertise, but it is possible to have different max-wait-to-advertise timers for IPv4 and IPv6.
In this example, BGP delayed route advertisement is only configured on PE-2, while the other nodes keep advertising their routes immediately after the BGP session is reestablished. PE-2 will accept these routes, but it will only advertise them after receiving the last expected EOR for IPv4 or IPv6 (for the base router or VPRN 1) and min-wait-to-advertise timer expires. If the max-wait-to-advertise timer expires before the last expected EOR is received for IPv4 or IPv6, PE-2 will start advertising the received routes.
# on PE-2:
configure
router Base
bgp
convergence
min-wait-to-advertise 20
family ipv4
max-wait-to-advertise 30
exit
family ipv6
max-wait-to-advertise 30
exit
exit
info
exit
exit
service
vprn "VPRN 1"
bgp
convergence
min-wait-to-advertise 60
family ipv4
max-wait-to-advertise 180
exit
family ipv6
max-wait-to-advertise 180
exit
exit
With this configuration, the BGP converged state on PE-2 changes to "waiting", because no BGP sessions are reestablished yet, so no BGP convergence tuning has taken place.
*A:PE-2# show router bgp convergence
===============================================================================
BGP IPv4 Convergence
===============================================================================
Min wait advertise timer : 20
Established peers at min wait timer expiry : 0
Current established peers : 3
First session established time : 00h00m00s
Last session established time : 00h00m00s
Max Wait advertise timer : 30
Converged peers : 3
Converged state : waiting
Converged time : N/A
===============================================================================
===============================================================================
BGP IPv6 Convergence
===============================================================================
Min wait advertise timer : 20
Established peers at min wait timer expiry : 0
Current established peers : 3
First session established time : 00h00m00s
Last session established time : 00h00m00s
Max Wait advertise timer : 30
Converged peers : 3
Converged state : waiting
Converged time : N/A
===============================================================================
The show router 1 bgp convergence command shows a similar output for VPRN 1, but is not shown here.
The following clear command on PE-2 causes BGP to restart:
# on PE-2:
clear router bgp protocol
When BGP restarts, the converged state remains "waiting" until the first peer is established.
With the first peer established, the converged state changes to "started", as follows:
*A:PE-2# show router bgp convergence
===============================================================================
BGP IPv4 Convergence
===============================================================================
Min wait advertise timer : 20
Established peers at min wait timer expiry : 0
Current established peers : 3
First session established time : 00h00m01s
Last session established time : 00h00m01s
Max Wait advertise timer : 30
Converged peers : 0
Converged state : started
Converged time : N/A
===============================================================================
===============================================================================
BGP IPv6 Convergence
===============================================================================
Min wait advertise timer : 20
Established peers at min wait timer expiry : 0
Current established peers : 3
First session established time : 00h00m01s
Last session established time : 00h00m01s
Max Wait advertise timer : 30
Converged peers : 0
Converged state : started
Converged time : N/A
===============================================================================
The show router 1 bgp convergence command shows a similar output for VPRN 1, but is not shown here.
After a few seconds, PE-2 receives IPv4 and IPv6 routes from PE-3 and PE-4, both in the base router and VPRN 1, as follows:
*A:PE-2# show router bgp summary all
===============================================================================
BGP Summary
===============================================================================
Legend : D - Dynamic Neighbor
===============================================================================
Neighbor
Description
ServiceId AS PktRcvd InQ Up/Down State|Rcv/Act/Sent (Addr Family)
PktSent OutQ
-------------------------------------------------------------------------------
192.0.2.3
Def. Instance 64500 5 0 00h00m01s 1/1/0 (IPv4)
4 0
192.0.2.4
Def. Instance 64500 5 0 00h00m01s 1/1/0 (IPv4)
4 0
192.168.12.1
Def. Instance 64501 3 0 00h00m00s 0/0/0 (IPv4)
4 0
2001:db8::2:3
Def. Instance 64500 5 0 00h00m01s 1/1/0 (IPv6)
4 0
2001:db8::2:4
Def. Instance 64500 5 0 00h00m01s 1/1/0 (IPv6)
4 0
2001:db8::12:1
Def. Instance 64501 3 0 00h00m00s 0/0/0 (IPv6)
4 0
172.31.0.1
Svc: 1 64496 3 0 00h00m00s 0/0/0 (IPv4)
3 0
172.31.0.3
Svc: 1 64496 5 0 00h00m01s 1/1/0 (IPv4)
4 0
172.31.0.4
Svc: 1 64496 5 0 00h00m01s 1/1/0 (IPv4)
4 0
2001:db8::31:0:1
Svc: 1 64496 3 0 00h00m00s 0/0/0 (IPv6)
3 0
2001:db8::31:0:3
Svc: 1 64496 5 0 00h00m01s 1/1/0 (IPv6)
4 0
2001:db8::31:0:4
Svc: 1 64496 5 0 00h00m01s 1/1/0 (IPv6)
4 0
-------------------------------------------------------------------------------
PE-2 accepts the received routes, but does not advertise the routes because the min-wait-to-advertise timer has not expired yet, and PE-2 only received IPv4 and IPv6 routes and EORs from PE-3 and PE-4, not from PE-1, so the converged state is "partial", as follows:
*A:PE-2# show router bgp convergence
===============================================================================
BGP IPv4 Convergence
===============================================================================
Min wait advertise timer : 20
Established peers at min wait timer expiry : 0
Current established peers : 3
First session established time : 00h00m01s
Last session established time : 00h00m02s
Max Wait advertise timer : 30
Converged peers : 2
Converged state : partial
Converged time : N/A
===============================================================================
===============================================================================
BGP IPv6 Convergence
===============================================================================
Min wait advertise timer : 20
Established peers at min wait timer expiry : 0
Current established peers : 3
First session established time : 00h00m01s
Last session established time : 00h00m02s
Max Wait advertise timer : 30
Converged peers : 2
Converged state : partial
Converged time : N/A
===============================================================================
The show router 1 bgp convergence command shows a similar output for VPRN 1, but is not shown here.
After a few seconds, all IPv4 and IPv6 routes have been received in the base router. PE-2 has received an EOR message from each neighbor in the base router. The following BGP summary shows that PE-2 has received and advertised all IPv4 and IPv6 routes in the base router, whereas it only received IPv4 and IPv6 routes from two neighbors in VPRN 1, not yet from PE-1:
*A:PE-2# show router bgp summary all
===============================================================================
BGP Summary
===============================================================================
Legend : D - Dynamic Neighbor
===============================================================================
Neighbor
Description
ServiceId AS PktRcvd InQ Up/Down State|Rcv/Act/Sent (Addr Family)
PktSent OutQ
-------------------------------------------------------------------------------
192.0.2.3
Def. Instance 64500 5 0 00h00m12s 1/1/3 (IPv4)
7 0
192.0.2.4
Def. Instance 64500 5 0 00h00m11s 1/1/3 (IPv4)
7 0
192.168.12.1
Def. Instance 64501 5 0 00h00m14s 1/1/3 (IPv4)
5 0
2001:db8::2:3
Def. Instance 64500 5 0 00h00m12s 1/1/3 (IPv6)
7 0
2001:db8::2:4
Def. Instance 64500 5 0 00h00m11s 1/1/3 (IPv6)
7 0
2001:db8::12:1
Def. Instance 64501 5 0 00h00m13s 1/1/3 (IPv6)
5 0
172.31.0.1
Svc: 1 64496 4 0 00h00m13s 0/0/0 (IPv4)
4 0
172.31.0.3
Svc: 1 64496 5 0 00h00m12s 1/1/0 (IPv4)
4 0
172.31.0.4
Svc: 1 64496 5 0 00h00m11s 1/1/0 (IPv4)
4 0
2001:db8::31:0:1
Svc: 1 64496 4 0 00h00m13s 0/0/0 (IPv6)
4 0
2001:db8::31:0:3
Svc: 1 64496 5 0 00h00m12s 1/1/0 (IPv6)
4 0
2001:db8::31:0:4
Svc: 1 64496 5 0 00h00m11s 1/1/0 (IPv6)
4 0
-------------------------------------------------------------------------------
As a result of this, BGP is in the "converged" state in the base router, both for IPv4 and IPv6, as follows:
*A:PE-2# show router bgp convergence
===============================================================================
BGP IPv4 Convergence
===============================================================================
Min wait advertise timer : 20
Established peers at min wait timer expiry : 3
Current established peers : 3
First session established time : 00h00m01s
Last session established time : 00h00m03s
Max Wait advertise timer : 30
Converged peers : 3
Converged state : converged
Converged time : 00h00m20s
===============================================================================
===============================================================================
BGP IPv6 Convergence
===============================================================================
Min wait advertise timer : 20
Established peers at min wait timer expiry : 3
Current established peers : 3
First session established time : 00h00m01s
Last session established time : 00h00m03s
Max Wait advertise timer : 30
Converged peers : 3
Converged state : converged
Converged time : 00h00m20s
===============================================================================
The converged time is only applicable in the "converged" state and is measured relative to BGP instance restart at time T=0.
BGP is still in the "partial" state within the VPRN 1 context, both for IPv4 and IPv6, as follows:
*A:PE-2# show router 1 bgp convergence
===============================================================================
BGP IPv4 Convergence
===============================================================================
Min wait advertise timer : 60
Established peers at min wait timer expiry : 0
Current established peers : 3
First session established time : 00h00m01s
Last session established time : 00h00m03s
Max Wait advertise timer : 180
Converged peers : 3
Converged state : partial
Converged time : N/A
===============================================================================
===============================================================================
BGP IPv6 Convergence
===============================================================================
Min wait advertise timer : 60
Established peers at min wait timer expiry : 0
Current established peers : 3
First session established time : 00h00m01s
Last session established time : 00h00m03s
Max Wait advertise timer : 180
Converged peers : 3
Converged state : partial
Converged time : N/A
===============================================================================
After a while, PE-1 also advertises its routes for VPRN 1, followed by EORs for IPv4 and IPv6. BGP converges for VPRN 1, as follows:
*A:PE-2# show router 1 bgp convergence
===============================================================================
BGP IPv4 Convergence
===============================================================================
Min wait advertise timer : 60
Established peers at min wait timer expiry : 3
Current established peers : 3
First session established time : 00h00m01s
Last session established time : 00h00m03s
Max Wait advertise timer : 180
Converged peers : 3
Converged state : converged
Converged time : 00h01m00s
===============================================================================
===============================================================================
BGP IPv6 Convergence
===============================================================================
Min wait advertise timer : 60
Established peers at min wait timer expiry : 3
Current established peers : 3
First session established time : 00h00m01s
Last session established time : 00h00m03s
Max Wait advertise timer : 180
Converged peers : 3
Converged state : converged
Converged time : 00h01m00s
===============================================================================
Conclusion
With BGP convergence tuning (by means of delaying route advertisements using two timers), less path churn and fewer advertisements can result in faster convergence. BGP convergence is mainly important in scaled environments (high number of BGP sessions and routes). As a result, the advertised paths are more optimal. The BGP convergence process can be monitored using a show command.