Friday, September 11, 2015

Cisco, Native VLAN's, and PVST+ (What a hot mess...)

This adventure began when I was called in to help triage a customer outage caused by spanning tree instability between a Stack of Cisco Catalyst and an EX4500 Virtual Chassis.  I was not the first to delve in here, and I had been warned that, to quote a colleague, "Cisco uses a non-standard destination MAC for PVST+ on the native VLAN."

So with that bit of caution, I dove into the customer problem on-site.  Confident in my knowledge about Cisco PVST+ and native VLAN's, I suggested the customer avoid using the native VLAN for anything, and instead re-map the native VLAN to something not in use.  That way we could simply tag VLAN 1 on both the Catalyst and EX and move forward with a proper PVST+/VSTP solution.  Except it didn't work.  And furthermore, it seemed to make the situation worse.  So after a couple of hours of troubleshooting, we reached a quasi-stable point where the Catalyst and EX switches in the customer network all agreed on STP topology for all VLAN's except VLAN 1, and in VLAN 1 they disagreed in a way that didn't impact forwarding.

Time to draw the line for the day and retreat to lab testing...

So I broke out an old Catalyst 2950 and an EX2200-C at home, cabled up a link between them, and started testing.  Here is a summary of what I found.  It's a long read, but hopefully worth the time.

Test 1:  Basic Setup
  •     Single trunk link between EX2200-C (ge-0/1/0) and Cat2950 (g0/1)
  •     Trunk supports VLANs 1, 10, 11
  •     Native VLAN not set on EX2200-C
  •     Rapid PVST+ enabled on 2950
  •     VSTP enabled on "vlan all" on EX2200
  •     EX2200 set with bridge-priority 4k on vlan all

Before connecting the switches together, I see this on the EX2200:

root@ex2200-c> show spanning-tree bridge
STP bridge parameters
Context ID                          : 1
Enabled protocol                    : RSTP

STP bridge parameters for VLAN 10

  Root ID                           : 4106.54:e0:32:37:85:41
  Hello time                        : 2 seconds
  Maximum age                       : 20 seconds
  Forward delay                     : 15 seconds
  Message age                       : 0
  Number of topology changes        : 0
  Local parameters
    Bridge ID                       : 4106.54:e0:32:37:85:41
    Extended system ID              : 1
    Internal instance ID            : 0

STP bridge parameters
Context ID                          : 2
Enabled protocol                    : RSTP

STP bridge parameters for VLAN 11
  Root ID                           : 4107.54:e0:32:37:85:41
  Hello time                        : 2 seconds
  Maximum age                       : 20 seconds
  Forward delay                     : 15 seconds
  Message age                       : 0
  Number of topology changes        : 0
  Local parameters
    Bridge ID                       : 4107.54:e0:32:37:85:41
    Extended system ID              : 2
    Internal instance ID            : 0

And I see this on the Cat2950:

cat2950#sh spanning-tree vlan 1-11

Spanning tree instance(s) for vlan 1-11 does not exist.

Next I cable the switches together via their respective trunk ports.  Now the EX2200 reports this:

root@ex2200-c> show spanning-tree bridge

STP bridge parameters
Context ID                          : 1
Enabled protocol                    : RSTP

STP bridge parameters for VLAN 10
  Root ID                           : 4106.54:e0:32:37:85:41
  Hello time                        : 2 seconds
  Maximum age                       : 20 seconds
  Forward delay                     : 15 seconds
  Message age                       : 0
  Number of topology changes        : 2
  Time since last topology change   : 52 seconds
  Topology change initiator         : ge-0/1/0.0
  Topology change last recvd. from  : 00:13:7f:09:0b:59
  Local parameters
    Bridge ID                       : 4106.54:e0:32:37:85:41
    Extended system ID              : 1
    Internal instance ID            : 0

STP bridge parameters
Context ID                          : 2
Enabled protocol                    : RSTP

STP bridge parameters for VLAN 11
  Root ID                           : 4107.54:e0:32:37:85:41
  Hello time                        : 2 seconds
  Maximum age                       : 20 seconds
  Forward delay                     : 15 seconds
  Message age                       : 0
  Number of topology changes        : 2
  Time since last topology change   : 52 seconds
  Topology change initiator         : ge-0/1/0.0
  Topology change last recvd. from  : 00:13:7f:09:0b:59
  Local parameters
    Bridge ID                       : 4107.54:e0:32:37:85:41
    Extended system ID              : 2
    Internal instance ID            : 0


And the Cat2950 reports this:

cat2950#sh spanning-tree vlan 1-11

VLAN0001
  Spanning tree enabled protocol rstp
  Root ID    Priority    32769
             Address     0013.7f09.0b40
             This bridge is the root
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32769  (priority 32768 sys-id-ext 1)
             Address     0013.7f09.0b40
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time 300

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------
Gi0/1            Desg BLK 4         128.25   P2p

VLAN0010
  Spanning tree enabled protocol rstp
  Root ID    Priority    4106
             Address     54e0.3237.8541
             Cost        4
             Port        25 (GigabitEthernet0/1)
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32778  (priority 32768 sys-id-ext 10)
             Address     0013.7f09.0b40
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time 300

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------
Gi0/1            Root FWD 4         128.25   P2p

VLAN0011
  Spanning tree enabled protocol rstp
  Root ID    Priority    4107
             Address     54e0.3237.8541
             Cost        4
             Port        25 (GigabitEthernet0/1)
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32779  (priority 32768 sys-id-ext 11)
             Address     0013.7f09.0b40
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time 300

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------
Gi0/1            Root FWD 4         128.25   P2p


This is actually what we expect.  Junos requires that if we want VSTP on VLAN 1, we must
call it out explicitly.  So let's do that.

Test 2:  Add VLAN-1 to EX2200-C VSTP Config
Let's start by adding VLAN-1 to the VSTP config on the EX2200.

root@ex2200-c# set protocols vstp vlan VLAN-1 bridge-priority 4k

And let's see how that changes things.  First on the EX2200:

root@ex2200-c> show spanning-tree bridge

STP bridge parameters
Context ID                          : 1
Enabled protocol                    : RSTP

STP bridge parameters for VLAN 10
  Root ID                           : 4106.54:e0:32:37:85:41
  Hello time                        : 2 seconds
  Maximum age                       : 20 seconds
  Forward delay                     : 15 seconds
  Message age                       : 0
  Number of topology changes        : 2
  Time since last topology change   : 214 seconds
  Topology change initiator         : ge-0/1/0.0
  Topology change last recvd. from  : 00:13:7f:09:0b:59
  Local parameters
    Bridge ID                       : 4106.54:e0:32:37:85:41
    Extended system ID              : 1
    Internal instance ID            : 0

STP bridge parameters
Context ID                          : 2
Enabled protocol                    : RSTP

STP bridge parameters for VLAN 11
  Root ID                           : 4107.54:e0:32:37:85:41
  Hello time                        : 2 seconds
  Maximum age                       : 20 seconds
  Forward delay                     : 15 seconds
  Message age                       : 0
  Number of topology changes        : 2
  Time since last topology change   : 214 seconds
  Topology change initiator         : ge-0/1/0.0
  Topology change last recvd. from  : 00:13:7f:09:0b:59
  Local parameters
    Bridge ID                       : 4107.54:e0:32:37:85:41
    Extended system ID              : 2
    Internal instance ID            : 0

STP bridge parameters
Context ID                          : 3
Enabled protocol                    : RSTP

STP bridge parameters for VLAN 1
  Root ID                           : 4097.54:e0:32:37:85:41
  Hello time                        : 2 seconds
  Maximum age                       : 20 seconds
  Forward delay                     : 15 seconds
  Message age                       : 0
  Number of topology changes        : 0
  Local parameters
    Bridge ID                       : 4097.54:e0:32:37:85:41
    Extended system ID              : 3
    Internal instance ID            : 0


And next on the Cat2950:

cat2950#sh spanning-tree vlan 1-11

VLAN0001
  Spanning tree enabled protocol rstp
  Root ID    Priority    32769
             Address     0013.7f09.0b40
             This bridge is the root
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32769  (priority 32768 sys-id-ext 1)
             Address     0013.7f09.0b40
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time 300

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------
Gi0/1            Desg FWD 4         128.25   P2p

VLAN0010
  Spanning tree enabled protocol rstp
  Root ID    Priority    4106
             Address     54e0.3237.8541
             Cost        4
             Port        25 (GigabitEthernet0/1)
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32778  (priority 32768 sys-id-ext 10)
             Address     0013.7f09.0b40
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time 300

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------
Gi0/1            Root FWD 4         128.25   P2p

VLAN0011
  Spanning tree enabled protocol rstp
  Root ID    Priority    4107
             Address     54e0.3237.8541
             Cost        4
             Port        25 (GigabitEthernet0/1)
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32779  (priority 32768 sys-id-ext 11)
             Address     0013.7f09.0b40
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time 300

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------
Gi0/1            Root FWD 4         128.25   P2p


Note that VLAN-1 still shows a disconnect.  The EX2200 believes it is the root bridge on VLAN-1, but the Cat2950 believes it is the root bridge.  It appears clear that the Cat2950 is not seeing the BPDU's from the EX2200 on VLAN-1.  That would make sense considering that we have not set the Native VLAN on the EX2200 side, and Cisco defaults to VLAN-1 as native, but let's check the BPDU's.  Click the image below for a larger view:



Note how the 2950 is outputting two BPDU's on the native VLAN -- one in 802.1D/RSTP format (Destination MAC = 0180:C200:0000) and one in PVST+ format (Destination MAC = 0100:0CCC:CCCD).  Why?  What does it expect back?  Does it prefer one over the other?

Now let's make VLAN-1 the native VLAN on the EX2200 side of the trunk:

Test #3: EX2200 Configured for Native VLAN 1
We configure the trunk port on the EX2200 for native-vlan 1 like so:

root@ex2200-c# show interfaces ge-0/1/0
unit 0 {
    family ethernet-switching {
        port-mode trunk;
        vlan {
            members [ VLAN-10 VLAN-11 ];
        }
        native-vlan-id 1;
    }
}


And we re-check the spanning tree state on the EX2200:

root@ex2200-c> show spanning-tree bridge

STP bridge parameters
Context ID                          : 1
Enabled protocol                    : RSTP

STP bridge parameters for VLAN 10
  Root ID                           : 4106.54:e0:32:37:85:41
  Hello time                        : 2 seconds
  Maximum age                       : 20 seconds
  Forward delay                     : 15 seconds
  Message age                       : 0
  Number of topology changes        : 2
  Time since last topology change   : 1728 seconds
  Topology change initiator         : ge-0/1/0.0
  Topology change last recvd. from  : 00:13:7f:09:0b:59
  Local parameters
    Bridge ID                       : 4106.54:e0:32:37:85:41
    Extended system ID              : 1
    Internal instance ID            : 0

STP bridge parameters
Context ID                          : 2
Enabled protocol                    : RSTP

STP bridge parameters for VLAN 11
  Root ID                           : 4107.54:e0:32:37:85:41
  Hello time                        : 2 seconds
  Maximum age                       : 20 seconds
  Forward delay                     : 15 seconds
  Message age                       : 0
  Number of topology changes        : 2
  Time since last topology change   : 1728 seconds
  Topology change initiator         : ge-0/1/0.0
  Topology change last recvd. from  : 00:13:7f:09:0b:59
  Local parameters
    Bridge ID                       : 4107.54:e0:32:37:85:41
    Extended system ID              : 2
    Internal instance ID            : 0

STP bridge parameters
Context ID                          : 3
Enabled protocol                    : RSTP

STP bridge parameters for VLAN 1
  Root ID                           : 4097.54:e0:32:37:85:41
  Hello time                        : 2 seconds
  Maximum age                       : 20 seconds
  Forward delay                     : 15 seconds
  Message age                       : 0
  Number of topology changes        : 1
  Time since last topology change   : 2 seconds
  Topology change initiator         : ge-0/1/0.0
  Local parameters
    Bridge ID                       : 4097.54:e0:32:37:85:41
    Extended system ID              : 3
    Internal instance ID            : 0



Well, we just took a topology change on VLAN 1!  Let's check out the Cat2950 to see if it agrees:

cat2950#show spanning-tree vlan 1-11

VLAN0001
  Spanning tree enabled protocol rstp
  Root ID    Priority    32769
             Address     0013.7f09.0b40
             This bridge is the root
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32769  (priority 32768 sys-id-ext 1)
             Address     0013.7f09.0b40
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time 300

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------
Gi0/1            Desg FWD 4         128.25   P2p

VLAN0010
  Spanning tree enabled protocol rstp
  Root ID    Priority    4106
             Address     54e0.3237.8541
             Cost        4
             Port        25 (GigabitEthernet0/1)
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32778  (priority 32768 sys-id-ext 10)
             Address     0013.7f09.0b40
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time 300

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------
Gi0/1            Root FWD 4         128.25   P2p

VLAN0011
  Spanning tree enabled protocol rstp
  Root ID    Priority    4107
             Address     54e0.3237.8541
             Cost        4
             Port        25 (GigabitEthernet0/1)
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32779  (priority 32768 sys-id-ext 11)
             Address     0013.7f09.0b40
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time 300

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------
Gi0/1            Root FWD 4         128.25   P2p



No change on the 2950.  It still thinks it's the root bridge in VLAN 1.  Let's look at the BPDU's on the wire again:


You can see here that the EX2200 is sending out untagged BPDU's to the PVST+ Destination MAC of 0100:0CCC:CCCD now that we have set VLAN 1 as the native VLAN on the EX2200 side.  And you can see that the Cat2950 is still sending out untagged BPDU's to both the standard RSTP and the PVST+
Destination MAC.  But it is clearly not listening for PVST+ BPDU's on the native VLAN, or else it would see the EX2200 as the root bridge in VLAN 1.  Why does the switch send BPDU's that it doesn't listen for?  Weird...

So if the Catalyst is not listening for PVST+ BPDU's on the native VLAN, let's make the EX2200 send out standard RSTP BPDU's on the native VLAN instead.

Test 4:  EX2200 configured for RSTP on VLAN 1
Now we take the VSTP configuration off of VLAN 1 and add some RSTP config instead.  We will just add the trunk port in this example and set the RSTP bridge priority to 4k, just as we have in the VSTP config:

root@ex2200-c# show protocols
rstp {
    bridge-priority 4k;
    interface ge-0/1/0.0 {
        mode point-to-point;
    }
}
vstp {
    vlan all {
        bridge-priority 4k;
    }
}

Once more, to the spanning tree state on the EX2200:

root@ex2200-c> show spanning-tree bridge

STP bridge parameters
Context ID                          : 0
Enabled protocol                    : RSTP
  Root ID                           : 4096.54:e0:32:37:85:41
  Hello time                        : 2 seconds
  Maximum age                       : 20 seconds
  Forward delay                     : 15 seconds
  Message age                       : 0
  Number of topology changes        : 1
  Time since last topology change   : 55 seconds
  Topology change initiator         : ge-0/1/0.0
  Topology change last recvd. from  : 00:13:7f:09:0b:59
  Local parameters
    Bridge ID                       : 4096.54:e0:32:37:85:41
    Extended system ID              : 0
    Internal instance ID            : 0

STP bridge parameters
Context ID                          : 1
Enabled protocol                    : RSTP

STP bridge parameters for VLAN 10
  Root ID                           : 4106.54:e0:32:37:85:41
  Hello time                        : 2 seconds
  Maximum age                       : 20 seconds
  Forward delay                     : 15 seconds
  Message age                       : 0
  Number of topology changes        : 2
  Time since last topology change   : 2740 seconds
  Topology change initiator         : ge-0/1/0.0
  Topology change last recvd. from  : 00:13:7f:09:0b:59
  Local parameters
    Bridge ID                       : 4106.54:e0:32:37:85:41
    Extended system ID              : 1
    Internal instance ID            : 0

STP bridge parameters
Context ID                          : 2
Enabled protocol                    : RSTP

STP bridge parameters for VLAN 11
  Root ID                           : 4107.54:e0:32:37:85:41
  Hello time                        : 2 seconds
  Maximum age                       : 20 seconds
  Forward delay                     : 15 seconds
  Message age                       : 0
  Number of topology changes        : 2
  Time since last topology change   : 2740 seconds
  Topology change initiator         : ge-0/1/0.0
  Topology change last recvd. from  : 00:13:7f:09:0b:59
  Local parameters
    Bridge ID                       : 4107.54:e0:32:37:85:41
    Extended system ID              : 2
    Internal instance ID            : 0



So you can see that the EX2200 has pulled out of VSTP on VLAN 1 and is instead participating in RSTP on the untagged (VLAN 1) topology.  Let's see what the Cat2950 thinks of this:

cat2950#show spanning-tree vlan 1-11

VLAN0001
  Spanning tree enabled protocol rstp
  Root ID    Priority    4096
             Address     54e0.3237.8541
             Cost        4
             Port        25 (GigabitEthernet0/1)
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32769  (priority 32768 sys-id-ext 1)
             Address     0013.7f09.0b40
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time 300

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------
Gi0/1            Root FWD 4         128.25   P2p

VLAN0010
  Spanning tree enabled protocol rstp
  Root ID    Priority    4106
             Address     54e0.3237.8541
             Cost        4
             Port        25 (GigabitEthernet0/1)
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32778  (priority 32768 sys-id-ext 10)
             Address     0013.7f09.0b40
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time 300

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------
Gi0/1            Root FWD 4         128.25   P2p

VLAN0011
  Spanning tree enabled protocol rstp
  Root ID    Priority    4107
             Address     54e0.3237.8541
             Cost        4
             Port        25 (GigabitEthernet0/1)
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32779  (priority 32768 sys-id-ext 11)
             Address     0013.7f09.0b40
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time 300

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------
Gi0/1            Root FWD 4         128.25   P2p



Et voila!  Now the Cat2950 and EX2200 agree on the root bridge for all VLAN's.  So it seems that the Cat2950, even though it sends both flavors of BPDU's on the native VLAN, will itself only listen for standards-based, RSTP BPDU's on the native VLAN.

And of course, let's check the BPDU's to see what's going on:



Now you can see that the EX2200 is sending standard RSTP BPDU's untagged, and the Cat2950 is quiet.  That the Cisco is no longer sending any BPDU's is what exactly we'd expect a switch doing when monitoring its root port.  Switches participating in a spanning tree only send BPDU's downstream, away from the root bridge.

The question still remains: Why send PVST+ BPDU's on the native VLAN if the switch is not going to listen back?

So now let's go completely mad.  We're going to add VLAN 2 to both switches and make it the native VLAN on both...

Test 5:  Add VLAN 2 to both switches as the native VLAN
Step 1 is to simply add VLAN 2 as the native VLAN on both sides of the trunk.  On each platform, I have already created the VLAN.  It's simply a matter of adding it as the native VLAN.  First on the EX2200:

root@ex2200-c> show configuration interfaces ge-0/1/0
unit 0 {
    family ethernet-switching {
        port-mode trunk;
        vlan {
            members [ VLAN-10 VLAN-11 VLAN-1 ];
        }
        native-vlan-id 2;
    }
}


And then on the Cat2950:

cat2950#sh run interface g0/1
Building configuration...

Current configuration : 132 bytes
!
interface GigabitEthernet0/1
 switchport trunk native vlan 2
 switchport trunk allowed vlan 1,2,10,11

 switchport mode trunk
end


No changes to either switches STP config.  Remember that we have the VSTP configuration on the EX2200 set for "vlan all", and we have RSTP running on the native VLAN.  So what do we see?  On the EX2200:

STP bridge parameters
Context ID                          : 0
Enabled protocol                    : RSTP
  Root ID                           : 4096.54:e0:32:37:85:41
  Hello time                        : 2 seconds
  Maximum age                       : 20 seconds
  Forward delay                     : 15 seconds
  Message age                       : 0
  Number of topology changes        : 1
  Time since last topology change   : 2750 seconds
  Topology change initiator         : ge-0/1/0.0
  Topology change last recvd. from  : 00:13:7f:09:0b:59
  Local parameters
    Bridge ID                       : 4096.54:e0:32:37:85:41
    Extended system ID              : 0
    Internal instance ID            : 0

STP bridge parameters
Context ID                          : 1
Enabled protocol                    : RSTP

STP bridge parameters for VLAN 10
  Root ID                           : 4106.54:e0:32:37:85:41
  Hello time                        : 2 seconds
  Maximum age                       : 20 seconds
  Forward delay                     : 15 seconds
  Message age                       : 0
  Number of topology changes        : 2
  Time since last topology change   : 5435 seconds
  Topology change initiator         : ge-0/1/0.0
  Topology change last recvd. from  : 00:13:7f:09:0b:59
  Local parameters
    Bridge ID                       : 4106.54:e0:32:37:85:41
    Extended system ID              : 1
    Internal instance ID            : 0

STP bridge parameters
Context ID                          : 2
Enabled protocol                    : RSTP

STP bridge parameters for VLAN 11
  Root ID                           : 4107.54:e0:32:37:85:41
  Hello time                        : 2 seconds
  Maximum age                       : 20 seconds
  Forward delay                     : 15 seconds
  Message age                       : 0
  Number of topology changes        : 2
  Time since last topology change   : 5435 seconds
  Topology change initiator         : ge-0/1/0.0
  Topology change last recvd. from  : 00:13:7f:09:0b:59
  Local parameters
    Bridge ID                       : 4107.54:e0:32:37:85:41
    Extended system ID              : 2
    Internal instance ID            : 0

STP bridge parameters
Context ID                          : 3
Enabled protocol                    : RSTP

STP bridge parameters for VLAN 2
  Root ID                           : 4098.54:e0:32:37:85:41
  Hello time                        : 2 seconds
  Maximum age                       : 20 seconds
  Forward delay                     : 15 seconds
  Message age                       : 0
  Number of topology changes        : 1
  Time since last topology change   : 305 seconds
  Topology change initiator         : ge-0/1/0.0
  Topology change last recvd. from  : 00:13:7f:09:0b:59
  Local parameters
    Bridge ID                       : 4098.54:e0:32:37:85:41
    Extended system ID              : 3
    Internal instance ID            : 0



So the first bit of weirdness we see is that RSTP is running on the native VLAN, and VSTP is running on VLAN 2 (the native VLAN).  And there is no STP running on tagged VLAN 1.  That's really not good,
so we'll add vlan 1 back to VSTP and remove vlan 2:

root@ex2200-c&gt show configuration protocols
rstp {
    bridge-priority 4k;
    interface ge-0/1/0.0 {
        mode point-to-point;
    }
}
vstp {
    vlan-group {
        group TEN-ELEVEN {
            vlan [ VLAN-10 VLAN-11 ];
            bridge-priority 4k;
        }
    }
    vlan VLAN-1 {
        bridge-priority 4k;
    }
}


So spanning tree on (native) VLAN 2 is handled by the RSTP config.  Spanning tree on VLAN's 1, 10, and 11 is handled by the VSTP config.  Remember that Junos will not run VSTP on VLAN 1 unless it is explicitly configured.  It doesn't matter if VLAN 1 is tagged or native.  Now looking at the spanning tree state on the EX2200, we see:

juniper@ex2200-c> show spanning-tree bridge

STP bridge parameters
Context ID                          : 0
Enabled protocol                    : RSTP
  Root ID                           : 4096.54:e0:32:37:85:41
  Hello time                        : 2 seconds
  Maximum age                       : 20 seconds
  Forward delay                     : 15 seconds
  Message age                       : 0
  Number of topology changes        : 1
  Time since last topology change   : 110 seconds
  Topology change initiator         : ge-0/1/0.0
  Topology change last recvd. from  : 00:13:7f:09:0b:59
  Local parameters
    Bridge ID                       : 4096.54:e0:32:37:85:41
    Extended system ID              : 0
    Internal instance ID            : 0

STP bridge parameters
Context ID                          : 1
Enabled protocol                    : RSTP

STP bridge parameters for VLAN 10
  Root ID                           : 4106.54:e0:32:37:85:41
  Hello time                        : 2 seconds
  Maximum age                       : 20 seconds
  Forward delay                     : 15 seconds
  Message age                       : 0
  Number of topology changes        : 1
  Time since last topology change   : 110 seconds
  Topology change initiator         : ge-0/1/0.0
  Topology change last recvd. from  : 00:13:7f:09:0b:59
  Local parameters
    Bridge ID                       : 4106.54:e0:32:37:85:41
    Extended system ID              : 1
    Internal instance ID            : 0

STP bridge parameters
Context ID                          : 2
Enabled protocol                    : RSTP

STP bridge parameters for VLAN 11
  Root ID                           : 4107.54:e0:32:37:85:41
  Hello time                        : 2 seconds
  Maximum age                       : 20 seconds
  Forward delay                     : 15 seconds
  Message age                       : 0
  Number of topology changes        : 1
  Time since last topology change   : 110 seconds
  Topology change initiator         : ge-0/1/0.0
  Topology change last recvd. from  : 00:13:7f:09:0b:59
  Local parameters
    Bridge ID                       : 4107.54:e0:32:37:85:41
    Extended system ID              : 2
    Internal instance ID            : 0

STP bridge parameters
Context ID                          : 3
Enabled protocol                    : RSTP

STP bridge parameters for VLAN 1
  Root ID                           : 4097.54:e0:32:37:85:41
  Hello time                        : 2 seconds
  Maximum age                       : 20 seconds
  Forward delay                     : 15 seconds
  Message age                       : 0
  Number of topology changes        : 1
  Time since last topology change   : 110 seconds
  Topology change initiator         : ge-0/1/0.0
  Local parameters
    Bridge ID                       : 4097.54:e0:32:37:85:41
    Extended system ID              : 3
    Internal instance ID            : 0



Much better.  Now RSTP is running over the native VLAN (VLAN 2) and VSTP is running on VLAN's 1, 10, and 11.  So what does the Cat2950 see?

cat2950#sh spanning-tree vlan 1-11

VLAN0001
  Spanning tree enabled protocol rstp
  Root ID    Priority    4096
             Address     54e0.3237.8541
             Cost        4
             Port        25 (GigabitEthernet0/1)
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32769  (priority 32768 sys-id-ext 1)
             Address     0013.7f09.0b40
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time 300

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------
Gi0/1            Root FWD 4         128.25   P2p

VLAN0002
  Spanning tree enabled protocol rstp
  Root ID    Priority    32770
             Address     0013.7f09.0b40
             This bridge is the root
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32770  (priority 32768 sys-id-ext 2)
             Address     0013.7f09.0b40
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time 300

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------
Gi0/1            Desg FWD 4         128.25   P2p

VLAN0010
  Spanning tree enabled protocol rstp
  Root ID    Priority    4106
             Address     54e0.3237.8541
             Cost        4
             Port        25 (GigabitEthernet0/1)
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32778  (priority 32768 sys-id-ext 10)
             Address     0013.7f09.0b40
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time 300

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------
Gi0/1            Root FWD 4         128.25   P2p

VLAN0011
  Spanning tree enabled protocol rstp
  Root ID    Priority    4107
             Address     54e0.3237.8541
             Cost        4
             Port        25 (GigabitEthernet0/1)
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32779  (priority 32768 sys-id-ext 11)
             Address     0013.7f09.0b40
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time 300

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------
Gi0/1            Root FWD 4         128.25   P2p



So we've broken things again.  We're looking for the EX2200 to be the root everywhere.  But the Cat2950 sees itself as the root for (untagged) VLAN 2.  What's on the wire now:



Here things get really weird.  You can see in the trace that the EX2200 is sending PVST+ BPDU's on VLAN's 1, 10, and 11.  You can also see the EX2200 is sending RSTP BPDU's on the native VLAN (VLAN 2 in this case).

The Cat2950 seems thoroughly confused, however.  We see that the Cat2950 is sending PVST+ BPDU's on the native VLAN (again VLAN 2), but it is not sending any RSTP BPDU's.  Because VLAN 2 is the native VLAN, and because the Cat2950 sees itself as the root bridge on VLAN 2, it should be sending
both types of BPDU's -- untagged RSTP and tagged PVST+ -- for VLAN 2, just as it did in Test 2 when we had native VLAN 1 on the Cat2950 and no native VLAN on the EX2200.  But it's not.

Furthermore, look at the other protocol traffic originating from the Cat2950 in the trace.  The Dynamic Trunk Protocol PDU's are sent on the native VLAN, which makes sense because this sort of protocol activity really addresses link-specific (and not VLAN-specific) control plane maintenance.  But the Cat2950 is also sending Loop Reply packets on tagged VLAN 1.  Why is the Loop protocol bound to VLAN 1, while DTP is not?  And I have seen in later traces that CDP is also running as a tagged protocol on VLAN 1.  Why?

From what I have observed, it appears that within IOS there are some inextricable linkages between the native VLAN and VLAN 1.  The question is why?  It's been 15 years since SANS published David Taylor's piece on VLAN hopping attacks against Cisco's VLAN implementation.  And from that, there's been a fair amount of industry best-practice to set the native VLAN on Cisco switches to some otherwise unused VLAN (or to simply disable the native VLAN on vendor implementations that don't mandate a native VLAN on every trunk.)  And yet doing so in IOS, particularly in multi-vendor environments, has the potential to completely wreck stability and interoperation.

Conclusion
This example, to me anyway, clearly demonstrates one important reason why you want to avoid vendor-proprietary protocols in all cases -- even when you plan on only deploying a single vendor in your environment.  It's not just about interoperation between vendors, it's also on some level about peer review.  If a team of engineers at any company work together to develop something like a new protocol, it's very likely that they are going to be hampered (at least a bit) by corporate group-think.  They've all been indoctrinated together, they are all working towards a goal and a vision laid out by their organization, and they have easy access to their tools (routers, switches, etc.), while access to other vendor's platforms is likely to be more expensive and less convenient.  And if there's no concern about interoperation, then there's not even a reason to look beyond the corporate walls.

And this is the problem.  Without the input from individuals and organizations that have a different view of the world, you get solutions that are likely to crumble in a whole bunch of corner cases.  When you engage across the community, you are much more likely to benefit from the perspectives and experiences of others.  Yes, there is going to be a fair amount of gamesmanship and politicking, and the process is going to slow, but generally you can expect that the number of corner cases that pose a risk to your implementation will be much lower.

Vendor proprietary protocols can be a good thing -- they occasionally push the envelope of current technology.  But they carry important operational risks as well.  They make transition to multi-vendor environments more difficult, and they are likely to have more hidden time bombs than standards-based solutions -- even in single-vendor environments.


No comments:

Post a Comment