This post is simply to discuss Cisco Best Practice regarding vPC design on 7K, differentiating your vPC vlans, (to downstream peers 5Ks), from your non vPC vlans ( routed layer 3 OSPF between 7K ).
Cisco recommends an addtional layer 2 link between 7K to carry non-vPC between vPC members, such as those vlans which OSPF uses. In a RAPID PVST+ environment, you can trunk non-vPC vlans on a separate port-channel connecting the vPC peers.
In the production environment in which I work, this additional link between 7Ks for OSPF was done as a Layer 3 link, (ip ospf point-to point) no vlans:
7K# show run int po 6
!Command: show running-config interface port-channel2
!Time: Mon Apr 25 13:51:48 2011
version 5.0
interface port-channel6
description ospf link
mtu 9216
ip address 10.1.1.1/30
ip ospf network point-to-point
ip router ospf 100 area 0.0.0.0
7K# show run int po 6 membership
!Command: show running-config interface port-channel2 membership
!Time: Mon Apr 25 13:59:21 2011
version 5.0(
interface port-channel6
description ospf link
mtu 9216
ip address 10.1.1.1/30
ip ospf network point-to-point
ip router ospf 100 area 0.0.0.0
interface Ethernet2/5
description to N7K-2
mtu 9216
channel-group 6 mode active
no shutdown
interface Ethernet1/5
description to N7K-2
mtu 9216
channel-group 6 mode active
no shutdown
In this configuration, OSPF stays up regardless of a vPC peer link fail. Cisco points out that if you must run non-vPC vlans over a vPC peer link, that you use the following command:
dual-active exclude interface-vlan <non-vPC vlans>
which will disconnect the SVI vlans from a vPC link failure.
On your CORE 7K's, for normal RSTP, Cisco recommends the pathcost method long as enabled. When I saw that, I was like, what the hell is pathcost method long? Then I found Scott Hogg's blog on Network World, and he does a fine job of explaining it:
http://www.networkworld.com/community/node/57598
but for the short of it, to quote Scott, " This command changes the path cost to increase it from a 16-bit value to a 32-bit value. More bits in the path cost value increases the range of possible link speeds. You can confirm the path cost method being used on your Cisco switch with "show spanning-tree summary" command."
Per Cisco, "After configuring the vPCs, the two Cisco Nexus 7000 Series Switches appear as a single switching device from a spanning-tree perspective (remember that the vPC primary is the Cisco Nexus 7000 Series Switch that processes BPDUs). The following output illustrates the Spanning-Tree topology after the vPC configuration. "
Monday, April 25, 2011
Validating VPC / Port Channels in NX-OS
Let's say you are coming into an environment fresh, and you want to learn the vPC topology - or just troubleshoot . . . let's scope this out with some handy NX-OS commands.
N7K# show vpc
vPC Peer-link status
---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ --------------------------------------------------
1 Po1 up
We see that the vPC peer link is Port-Channel 1. What members are in that port channel?
N7K# show run int Po1 membership
!Command: show running-config interface port-channel1 membership
!Time: Mon Apr 25 10:26:00 2011
version 5.0
interface port-channel1
description vPC PEERLINK
switchport
switchport mode trunk
vpc peer-link
spanning-tree port type network
mtu 9216
interface Ethernet1/1
description N7K-2 Eth1/1
switchport
switchport mode trunk
rate-mode dedicated force
mtu 9216
channel-group 1 mode active
no shutdown
interface Ethernet3/1
description N7K-2 Eth3/1
switchport
switchport mode trunk
rate-mode dedicated force
mtu 9216
channel-group 1 mode active
no shutdown
Cool. No what vlans are riding over the Peer link?
Nice. Okay. What is this 7K's vPC role?
N7K# show interface po1 switchport
Name: port-channel1
Switchport: Enabled
Switchport Monitor: Not enabled
Operational Mode: trunk
Access Mode VLAN: 1 (default)
Trunking Native Mode VLAN: 1 (default)
Trunking VLANs Enabled: 1-3967,4048-4093
Administrative private-vlan primary host-association: none
Administrative private-vlan secondary host-association: none
Administrative private-vlan primary mapping: none
Administrative private-vlan secondary mapping: none
Administrative private-vlan trunk native VLAN: none
Administrative private-vlan trunk encapsulation: dot1q
Administrative private-vlan trunk normal VLANs: none
Administrative private-vlan trunk private VLANs: none
Operational private-vlan: none
N7K# show vpc role vPC Role status
----------------------------------------------------
vPC role : primary
Dual Active Detection Status : 0
vPC system-mac : 00:23:04:ee:be:0a
vPC system-priority : 32667
vPC local system-mac : a8:b1:d4:55:4e:42
vPC local role-priority : 16332
DENVWSW0001-INTERNAL#
Right then. What about the keep alive link? what are the IPs and the interfaces?
N7K# show run vpc
!Command: show running-config vpc
!Time: Mon Apr 25 10:37:16 2011
version 5.0
feature vpc
vpc domain 10
peer-switch
role priority 16332
peer-keepalive destination 10.1.1.2 source 10.1.1.1 vrf keepalive
peer-gateway
reload restore
N7K# show ip int br vrf keepalive
IP Interface Status for VRF "VPC-KEEPALIVE"(5)
Interface IP Address Interface Status
port-channel5 10.1.1.1 protocol-up/link-up/admin-up
N7K# show run int po5 memebership
!Command: show running-config interface port-channel3 membership
!Time: Mon Apr 25 10:42:36 2011
version 5.0(3E4)
interface port-channel5
description keepalive link
vrf member keepalive
ip address 10.1.1.1/30
interface Ethernet2/1
description to N7K-2
channel-group 5 mode active
no shutdown
With this handful of handy commands, you can see what is going on with vPC and port-channels in a Jiffy. I do notice in NX-OS that the port-channel interfaces are missing when you want to view your 'logical' nieghbors with cdp.
N7K# show cdp neigh interface ?
ethernet Ethernet IEEE 802.3z
mgmt Management interface
So you have to use the membership command (referenced above) to find out the physical interfaces that are bound to the port-channel and issue the show cdp neigh on those interfaces.
N7K# show vpc
vPC Peer-link status
---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ --------------------------------------------------
1 Po1 up
We see that the vPC peer link is Port-Channel 1. What members are in that port channel?
N7K# show run int Po1 membership
!Command: show running-config interface port-channel1 membership
!Time: Mon Apr 25 10:26:00 2011
version 5.0
interface port-channel1
description vPC PEERLINK
switchport
switchport mode trunk
vpc peer-link
spanning-tree port type network
mtu 9216
interface Ethernet1/1
description N7K-2 Eth1/1
switchport
switchport mode trunk
rate-mode dedicated force
mtu 9216
channel-group 1 mode active
no shutdown
interface Ethernet3/1
description N7K-2 Eth3/1
switchport
switchport mode trunk
rate-mode dedicated force
mtu 9216
channel-group 1 mode active
no shutdown
Cool. No what vlans are riding over the Peer link?
Nice. Okay. What is this 7K's vPC role?
N7K# show interface po1 switchport
Name: port-channel1
Switchport: Enabled
Switchport Monitor: Not enabled
Operational Mode: trunk
Access Mode VLAN: 1 (default)
Trunking Native Mode VLAN: 1 (default)
Trunking VLANs Enabled: 1-3967,4048-4093
Administrative private-vlan primary host-association: none
Administrative private-vlan secondary host-association: none
Administrative private-vlan primary mapping: none
Administrative private-vlan secondary mapping: none
Administrative private-vlan trunk native VLAN: none
Administrative private-vlan trunk encapsulation: dot1q
Administrative private-vlan trunk normal VLANs: none
Administrative private-vlan trunk private VLANs: none
Operational private-vlan: none
N7K# show vpc role vPC Role status
----------------------------------------------------
vPC role : primary
Dual Active Detection Status : 0
vPC system-mac : 00:23:04:ee:be:0a
vPC system-priority : 32667
vPC local system-mac : a8:b1:d4:55:4e:42
vPC local role-priority : 16332
DENVWSW0001-INTERNAL#
Right then. What about the keep alive link? what are the IPs and the interfaces?
N7K# show run vpc
!Command: show running-config vpc
!Time: Mon Apr 25 10:37:16 2011
version 5.0
feature vpc
vpc domain 10
peer-switch
role priority 16332
peer-keepalive destination 10.1.1.2 source 10.1.1.1 vrf keepalive
peer-gateway
reload restore
N7K# show ip int br vrf keepalive
IP Interface Status for VRF "VPC-KEEPALIVE"(5)
Interface IP Address Interface Status
port-channel5 10.1.1.1 protocol-up/link-up/admin-up
N7K# show run int po5 memebership
!Command: show running-config interface port-channel3 membership
!Time: Mon Apr 25 10:42:36 2011
version 5.0(3E4)
interface port-channel5
description keepalive link
vrf member keepalive
ip address 10.1.1.1/30
interface Ethernet2/1
description to N7K-2
channel-group 5 mode active
no shutdown
With this handful of handy commands, you can see what is going on with vPC and port-channels in a Jiffy. I do notice in NX-OS that the port-channel interfaces are missing when you want to view your 'logical' nieghbors with cdp.
N7K# show cdp neigh interface ?
ethernet Ethernet IEEE 802.3z
mgmt Management interface
So you have to use the membership command (referenced above) to find out the physical interfaces that are bound to the port-channel and issue the show cdp neigh on those interfaces.
VPC External links and resources
Good things to know!
Routing over Nexus 7000 vPC peer-link
http://bradhedlund.com/2010/12/16/routing-over-nexus-7000-vpc-peer-link-yes-and-no/
Cisco vPC Design Guide
http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/C07-572830-00_Agg_Dsgn_Config_DG.pdf
vPC Configuration Guide
http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/layer2/Cisco_Nexus_5000_Series_NX-OS__chapter8.pdf
http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/layer2/421_n2_1/Cisco_n5k_layer2_config_gd_rel_421_N2_1_chapter8.html
Nexus caveats: vPC to FEX
http://www.howfunky.com/2010/04/cisco-nexus-5k-and-2148t-caveats.html
Cisco Nexus 7000 Series NX-OS Troubleshooting Guide -- Troubleshooting vPCs:
Vlans on a vPC moved to suspend state
http://docwiki.cisco.com/wiki/Cisco_Nexus_7000_Series_NX-OS_Troubleshooting_Guide_--_Troubleshooting_vPCs#VLANs_on_a_vPC_moved_to_suspend_state
Routing over Nexus 7000 vPC peer-link
http://bradhedlund.com/2010/12/16/routing-over-nexus-7000-vpc-peer-link-yes-and-no/
Cisco vPC Design Guide
http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/C07-572830-00_Agg_Dsgn_Config_DG.pdf
vPC Configuration Guide
http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/layer2/Cisco_Nexus_5000_Series_NX-OS__chapter8.pdf
http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/layer2/421_n2_1/Cisco_n5k_layer2_config_gd_rel_421_N2_1_chapter8.html
Nexus caveats: vPC to FEX
http://www.howfunky.com/2010/04/cisco-nexus-5k-and-2148t-caveats.html
Cisco Nexus 7000 Series NX-OS Troubleshooting Guide -- Troubleshooting vPCs:
Vlans on a vPC moved to suspend state
http://docwiki.cisco.com/wiki/Cisco_Nexus_7000_Series_NX-OS_Troubleshooting_Guide_--_Troubleshooting_vPCs#VLANs_on_a_vPC_moved_to_suspend_state
Nexus VPC: Auto recovery vs. peer-config-check-bypass
It's critical to understand how your vPC will behave in different fail scenarios. In my own effort to understand this behavior thoroughly, I will explain the two vPC domain commands: Auto recovery vs. peer-config-check-bypass.
Before talking about Auto recovery vs. peer-config-check-bypass, we need to understand what happenes before two Nexus switches can bring up a vPC. First, the two switches in the same vPC domain exchange configuration parameters, ( a consistency check), to ensure both switches have compatible vPC configrations to enable thier vPC topology and begin forwarding traffic.
All checks that match a "type 1" inconsistency, will behave like this: When * graceful consistency check* is on ( by default ),, the primary switch keeps the vPC up while the secondary bring it down. when graceful consistency check is disabled, both peer switches suspend VLANS on the vPC ports.
* show vpc consistency-parameters global
* show vpc consistency-paramenters interface port-channel 200
* show vpc brief
Full Cisco Paper on this: http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/operations/n5k_vpc_ops.html
Scenario 1: The vPC check message is sent via the vPC peer link, and does not function if the vPC peer link is down. In the situation, the secondary vPC switch suspends all of its vPC member ports, while the Primary vPC switch's member ports remain forwarding.
Scenario 2: The vPC check message is sent via the vPC peer link, and does not function if the vPC peer link is down. If one of the vPC MEMBER ports on the vPC primary switch flaps while the vPC peer link is down, the ports remain down to to the vPC consistency check,
hence enter the feature . . Auto recovery
I have not seen this work live, but according to Cisco . . . Cisco NX-OS Release 5.0(2)N2 the auto-recover feature brings up the vPC member ports, (after a member port flap on the primary vPC switch), when one peer is down. Cisco outlines some scenarios and what Auto-recover does in each:
* If both switches reload, and only one switch boots up, auto-recovery allows that switch to assume the role of the primary switch. The vPC links come up after a configurable period of time if the vPC peer-link and the peer-keepalive fail to become operational within that time.
[ NOTE: I am not sure what Cisco means here that the auto-recovery feature "allows the switch to assume the role of the primary switch", as the role of the vPC primary is clearly determined by the 'role-priority' command ].
* If both switches reload, both switches come up, the vPC peer link comes up, but the peer-keepalive does not come up, both peer switches keep the vPC links down.
* When you disable vPCs on a secondary vPC switch because of a peer-link failure and then the primary vPC switch fails, the secondary switch reenables the vPCs. In this scenario, the vPC waits for three consecutive keepalive failures before recovering the vPC links.
Okay, thats 'auto-recovery', now let's look at peer-config-check-bypass
The above referenced URL makes no reference to peer-config-check-bypass, although it appears they do pretty much the same thing, except this is solely for the Primary vPC switch.
With this command, Cisco says that we can modify the default, (above mentioned) default vPC beahvior when a peer link is down on the vPC Primary switch. [ NOTE: Cisco does state that there is no behavior change on the secondary switch with this command, whereas with auto-recovery, there is mention of behavior change on the secondary switch. ]. Specifically with this command, Cisco states " This command allows newly configured vPCs and existing vPCs that have flapped to be brought up when a peer link is down and the vPC switch role has been determined to be primary"
Here's Cisco's paper on it:
http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/layer2/421_n2_1/Cisco_n5k_layer2_config_gd_rel_421_N2_1_chapter8.html
On our config here, we are using peer-config-check-bypass in production. Please add your thoughts on these two commands!
Before talking about Auto recovery vs. peer-config-check-bypass, we need to understand what happenes before two Nexus switches can bring up a vPC. First, the two switches in the same vPC domain exchange configuration parameters, ( a consistency check), to ensure both switches have compatible vPC configrations to enable thier vPC topology and begin forwarding traffic.
All checks that match a "type 1" inconsistency, will behave like this: When * graceful consistency check* is on ( by default ),, the primary switch keeps the vPC up while the secondary bring it down. when graceful consistency check is disabled, both peer switches suspend VLANS on the vPC ports.
* show vpc consistency-parameters global
* show vpc consistency-paramenters interface port-channel 200
* show vpc brief
Full Cisco Paper on this: http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/operations/n5k_vpc_ops.html
Scenario 1: The vPC check message is sent via the vPC peer link, and does not function if the vPC peer link is down. In the situation, the secondary vPC switch suspends all of its vPC member ports, while the Primary vPC switch's member ports remain forwarding.
Scenario 2: The vPC check message is sent via the vPC peer link, and does not function if the vPC peer link is down. If one of the vPC MEMBER ports on the vPC primary switch flaps while the vPC peer link is down, the ports remain down to to the vPC consistency check,
hence enter the feature . . Auto recovery
I have not seen this work live, but according to Cisco . . . Cisco NX-OS Release 5.0(2)N2 the auto-recover feature brings up the vPC member ports, (after a member port flap on the primary vPC switch), when one peer is down. Cisco outlines some scenarios and what Auto-recover does in each:
* If both switches reload, and only one switch boots up, auto-recovery allows that switch to assume the role of the primary switch. The vPC links come up after a configurable period of time if the vPC peer-link and the peer-keepalive fail to become operational within that time.
[ NOTE: I am not sure what Cisco means here that the auto-recovery feature "allows the switch to assume the role of the primary switch", as the role of the vPC primary is clearly determined by the 'role-priority' command ].
* If both switches reload, both switches come up, the vPC peer link comes up, but the peer-keepalive does not come up, both peer switches keep the vPC links down.
* When you disable vPCs on a secondary vPC switch because of a peer-link failure and then the primary vPC switch fails, the secondary switch reenables the vPCs. In this scenario, the vPC waits for three consecutive keepalive failures before recovering the vPC links.
This example shows how to enable the auto-recovery feature and to set the reload delay period:
switch(config)# vpc domain 10
switch(config-vpc-domain)# auto-recovery ?
<CR>
reload-delay Duration to wait after reload to recover vPCs
switch(config-vpc-domain)# auto-recovery reload-delay ?
<240-3600> Time-out for restoring vPC links (in seconds)
switch(config-vpc-domain)# auto-recovery reload-delay 240Okay, thats 'auto-recovery', now let's look at peer-config-check-bypass
The above referenced URL makes no reference to peer-config-check-bypass, although it appears they do pretty much the same thing, except this is solely for the Primary vPC switch.
With this command, Cisco says that we can modify the default, (above mentioned) default vPC beahvior when a peer link is down on the vPC Primary switch. [ NOTE: Cisco does state that there is no behavior change on the secondary switch with this command, whereas with auto-recovery, there is mention of behavior change on the secondary switch. ]. Specifically with this command, Cisco states " This command allows newly configured vPCs and existing vPCs that have flapped to be brought up when a peer link is down and the vPC switch role has been determined to be primary"
Here's Cisco's paper on it:
http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/layer2/421_n2_1/Cisco_n5k_layer2_config_gd_rel_421_N2_1_chapter8.html
On our config here, we are using peer-config-check-bypass in production. Please add your thoughts on these two commands!
Subscribe to:
Posts (Atom)