Nexus 5k NTP Failure

Ran into this bug today. Went to a pair of Nexus 5500s to debug a vPC link and noticed the timestamps were off. I thought that was odd. I tried show ntp peer-status and received no output in return, which I thought was even more odd. I poked at NTP for a while and decided it had to be a bug. I found the bug in the release notes and it was fixed in 5.2(1)N1(6). Here’s the Cisco bug report:

Nexus 5k acting as an NTP Client can’t sync with any NTP server(s).
when issuing a “show ntp peer-status” or a “show ntp peers” it does not display any of the servers/peers configured.

Nexus 5500/5000 running 5.2(1)N1(5).

Proactive workaround to prevent from this issue is none.
Reactive workaround to recover this issue is below. However, after reloading system, same issue may happen again.

#conf t
#clock protocol none
#clock protocol ntp
#copy run start

Fun. At least it can be fixed without reloading, which is a good thing in a data center switch.


No More VLANs Jumping on the Trunk

ThinkGeek Cable MonkeyNo more VLANs jumping on the trunk. That’s what I wanted. Instead, I got all the VLANs and a nice supervisor debilitating loop in the process… From my after action report: “the issue was a configuration error that caused the CPU to peg at 99%, which caused a variety of issues.”

Due to a legacy design that we have been working to retire, we have a layer 2 Ethernet WAN that can’t run STP. Yep, it’s a loop waiting to happen.

We were migrating sites to a new WAN. The circuits come in over a trunk with each site getting a VLAN tag. I was moving the last site from an interface. The procedure I wrote had the VLAN being removed from the interface with “switchport trunk allowed vlan remove ###”. Worked great until we hit the last VLAN on the trunk. When the last VLAN is removed from an interface, you might expect no VLANs to be allowed. If that’s what you expected, you are wrong. When you remove the last VLAN with “switchport trunk allowed vlan”, the port reverts to allowing ALL VLANs.

Unfortunately, since all VLANs were suddenly available on that port, it allowed a bridging loop to be created through the sites that were still connected to both the old and new WANs. This caused a variety of different bits of havoc that resulted in connectivity issues. Yeah, something of an understatement.

So, here’s some lessons I took from this:

  • Dual connected sites should have their legacy WAN ports shut down once they are stable. (This shouldn’t be a problem in a proper L3 design, but is important with our broken L2 design.)
  • Designs that prevent you from using STP to prevent loops need to be fixed sooner rather than later.
  • Entering explicit commands is better than implicit. In this case, “switchport trunk allowed vlan none” would have been better than “switchport trunk allowed vlan remove <last vlan>” and assuming the result will be what you expect.
  • Don’t make assumptions on how a device will behave. Lab it and know.
  • Always verify that changes had the desired effect. Had I verified the interface was configured as expected, the problem would have been discovered immediately and had little or no impact.


Your old network gear is EoL, too…

3560V2 Switches

Cisco 3560V2 Switches

Cisco 3650 Unified Access Switches

So yesterday, I pointed out that Your break room phones are EoL. Continuing on that EoL theme for Turkey week, I’m going to complicate your life by letting you know that the ancient network design you have been using with the 3560V2 or 3750V2 switches is going to have to change because those switches have seen their day. So sayeth Cisco. Cisco knows it’s harder to change your infrastructure than your endpoints, so they are generously giving you two years to stop buying those. The End of Sale date isn’t until November 14, 2015. Of course, if you have it on a service contract you’ll be able to stretch those out to 2020. I hope you don’t have to resort to that. Cisco recommends you get modern with a nice 3650 Unified Access switch, and really, why wouldn’t you?

They are a lot more powerful for the same money. To be honest, I was surprised to discover you could even still order the V2. I mean, just look at the 3650 paint job. Who doesn’t what that hidden away in a rack where only you can enjoy it?

2960X Series Switches

Cisco 2960X Series Switches

Of course, I would be remiss if I didn’t remind you to upgrade your 2960 switches to the 2960X Series while you were at it. I mean, those old 2960 switches are EoL, too. You’ll have to start buying newer models, anyway. You may as well make sure they match your shiny new L3 kit, right?

Again, as has been the pattern for the last few generations on these, same price, more power and features. And these have the Enhanced Limited Lifetime Warranty. Buy a couple extras to keep on the shelf and skip the SMARTnet. You’ll save a ton of money. Just don’t tell Cisco I said that.


Password required, but none set

I had a strange thing happen today when I upgraded a 2960G from IOS 12.2 to 15.0. After booting the upgraded IOS, I logged in, entered the enable command, and was surprised to get this error:


Password required, but none set

Everything seemed to work fine, but it was a little odd. I did discover that the following command resolves the error:

aaa authentication enable default enable

Which is also odd, because the documentation states:

If the default list is not set, only the enable password is checked. This has the same effect as the following command:

aaa authentication enable default enable

I’ll go with the explicit statement because the error message, while it may be spurious, makes me uncomfortable.


The Mythical Sup2T

The mythical Supervisor Engine 2T for the Catalyst 6500 is no longer a myth. Cisco has finally officially announced the product and if you just bought Sup 720’s, you might be wishing you could have waited. I haven’t seen the pricing yet, but it might make a recent Sup720 purchase more palatable. We shall see…

Here’s some of the details that I’ve discovered here at Cisco Live:

  • It will only run in the -E chassis. Older chassis lack the traces required.
  • Doubles the switch fabric speed from 20Gb to 40Gb.
  • It has a USB console (shocking, I know)
  • It has a dual 40Gb line card available. There will be an adapter reminiscent of the TwinGig that will allow a slot to hold 4 10Gb SFPs.
  • It is theoretically possible to support 100Gb Ethernet in the chassis, but no commitment has been made. It doesn’t help that the optics for 100Gb are $100k each and not many customers are up for that.
  • It has enough channels to the switch fabric for a 6513-E to get full performance out of every slot.
  • It uses a PFC4 and an MSFC5. I’ve no idea why the numbering is no longer in sync.
    • The MSFC5 has a Connectivity Management Processor (CMP), similar to the N7k boxes. I believe it’s essentially a LOM.
  • It will ship with 12.2SY. An IOS 15SY release will be coming. I don’t know the time frame. Sup720 will also get updated to an IOS 15 train.
  • There will be 6800 and 6900 series line cards. 6800 are apparently identical to the 6700 series, but with DFC4. 6900 will support the 80Gb fabric.
  • The 2T removes a lot of (the sometimes odd) limitations with the Sup720 platform. Particularly around IPv6 and MPLS.
  • No more Switch Processor; there is only a Route Processor.

What’s new in the PFC4:

  • Larger hardware tables (TCAM, MAC, etc)
  • A variety of IPv6 related improvements
  • VPLS without a SIP
  • Egress Netflow and Flexible Netflow
  • TrustSec
  • Lots of MPLS upgrades and new related features
  • Significantly higher performance numbers (obviously) than the PFC3C
  • The XL variant continues to up our FIB TCAM from 256k to 1M.
  • You can continue to use CFC line cards. Only DFC4(XL) cards will work with the 2T.
    • The 6708 line card will not be upgradable to a DFC4. It’s missing a critical chip.
    • Several of the 6700 series cards can be upgraded with a DFC4. There will be two part numbers for this, depending on the card. I believe the 6704, 6716, and 6748 cards were specifically mentioned, but I have no documentation. YMMV.
  • Many improvements to CoPP.

That’s the quick rundown of what seemed the highlights to me. I suspect there are MPLS people who will be extremely excited by this bit of kit, but I don’t know enough about MPLS to know which parts are really exciting. The architecture of the 2T has not changed radically from the 720-10G platform. It’s mostly the MSFC, but the architecture drawing is otherwise nearly identical. The PFC4 is a bit different, but mostly in more and different TCAM tables. Hopefully this gives you quick answers to some of the questions you had about the 2T. Let me know if you have others and I’ll try to find out the answer while I have Cisco’s ear.


CCNP SWITCH Study Materials

There’s a ton of materials out there to help you study for Cisco exams. I just finished spending several months studying for the SWITCH exam and spending a lot of time with the Cisco Press materials. I wanted to share my thoughts on some of the materials available.

The first book I started studying was the CCNP SWITCH 642-813 OCG [current version is CCNP Routing and Switching SWITCH 300-115 Official Cert Guide Premium Edition eBook and Practice Test] from Cisco Press. The current editions of this book do not include the test material covering planning, syslog, SNMPv3, or ip sla. These are included as downloads on the web site, so make sure you grab the addendum for chapter 1 and appendix B from the book’s web page. These downloads cover those topics. This book is a fairly technical explanation of all the topics. It’s a pretty dry read and while it can be useful, I don’t think it covers the topics in a way that left me understanding them as well as I could have.

I then read through the CCNP SWITCH 642-813 Quick Reference [current version is CCNP Routing and Switching SWITCH 300-115 Quick Reference]. This eBook is laid out to be nicely printable or viewable on a screen (an iPad works well). This is a reference guide to the various topics you need to know. Particularly good for memorizing the details of how the various flavors of STP and FHRP’s differ and other memorization bits like that.

After I had gone through these, I started using the Boson NetSim product to practice a few things for which my lab at work was not equipped and which GNS3 can’t do (at least not well). I also used the Boson ExSim Max practice tests to get a feel for the types of questions I would see on the test and to find areas where I needed work. The first practice test was an eye opener and I decided to hit the books again.

This time I decided to read the Implementing Cisco IP Switched Networks (SWITCH) FLG [current version is Implementing Cisco IP Switched Networks (SWITCH) Foundation Learning Guide]. I like this book a lot more than the official cert guide. I found it more interesting because it does a better job of helping you understand when to use the various technologies and their purpose. It’s a bit more scenario focussed, as opposed to just telling you the facts. That said, I’m not sure I would have gotten as much out of this one if I had started with it. It’s entirely possibly (if not probable) that there was benefit in reading the different explanations of the same topics in order to really internalize them.

Check the errata on both of the study guides. Both books have errors in them, most are not a big deal, but definitely worth checking. I did not use the practice tests these books come with, as I had digital copies of them and they do not include the CD content. By the way, I recommend checking out the eBook Deal of the Week at the Cisco Press site. These books regularly show up there.

There are CCNP study forums on the Cisco Learning Network. The forums aren’t really my style and I don’t use them much. That said, it is sometimes worthwhile to do a search on a given topic there if something isn’t making sense. It’s also a good place to have questions about the exam answered.

If I had not had the Boson practice exams, I would have failed the first time I took this exam. I credit Boson with helping me pass this on the first try and I would recommend their products to anyone who wasn’t completely sure of their knowledge.

The best part about passing this is now I can quit reading about STP. I can’t tell you how tired I am of STP. I’m ready to move on to routing!

[Since originally writing this post, I’ve found a good comparison of OCG vs FLG is over at CCNP ROUTE: Official Study Guide or Foundation Learning Guide? His comments regarding the ROUTE books are equally applicable to the SWITCH books.]


Recovering a Cisco Fixed Switch from the Boot Loader

Wait, that’s not right…

Let’s say you have Cisco fixed switch (2960, 3560, etc) and you copied over the tar file with an IOS upgrade, removed the old IOS, and rebooted the switch. Let’s say you forgot to install the new IOS and now you’re at the boot loader because the tar file isn’t bootable, it’s just a container for the IOS bin and the web based device manager. If you don’t grab the version with the web based device manager, you just have the IOS binary and you’re good to go, no install necessary. If you’re smart, you don’t remove the old IOS, but in my case this was for a fresh deployment so I didn’t see the need to hang on to the old IOS. Now you’re at the boot loader “switch:” prompt.

If you have an old switch/doorstop

You may have been to this movie before and been at the rommon of a newer router. The newer routers allow you to get the router onto the network and TFTP a new IOS in. So do the 3560X and 3750X, but I’ll get to that in a moment. If you have an older switch, you get to transfer the IOS image to your switch with xmodem.

There are two things that can be done to make this go a bit faster.

  • The big one is to change your serial console speed from 9600bps to 115200bps. For a 14MB IOS image, that takes your transfer time down from over 3 hours to around 30 minutes. At least, that’s what it would be without overhead.
  • Reduce your protocol overhead! Basic xmodem uses 128-byte packets. That gives you 40% overhead at 9600bps. Now you are well over 4 hours at 9600bps and closer to 45 minutes at 115000bps. Using xmodem-1k drops you to 19% overhead. That saves about 10 minutes at 115kbps and around 40 minutes at the snail’s pace of 9600bps.

To set your terminal speed to 115200, enter the command:

switch: set BAUD 115200

This takes effect immediately and your terminal will now give you garbage until you set it’s rate to 115200 as well.

Now you can start your xmodem transfer. Don’t forget to make sure your xmodem is using xmodem-1k. Also, make sure you copy over a .bin IOS image and not a .tar, or you’re wasting a lot of time.

switch: copy xmodem flash:c2960-lanbasek9-tar.122-55.SE1.bin

Once it’s done, you can either boot if you want run the .bin you copied over or you can reset if you want the system to do a hard reset (similar to a power cycle). Don’t forget to switch back to 9600bps…

If you have a 3560X/3750X

If you have a 3560X/3750X, you are in luck. TFTP is now available to you via the management FastEthernet interface. Set the IP_ADDR variable (the variable names may be case sensitive, depending on your platform) with your IP/netmask and DEFAULT_ROUTER with your default gateway. For example:

switch: set IP_ADDR
switch: set DEFAULT_ROUTER
switch: copy tftp:// flash:c3560e-universalk9-mz.122-55.SE1.bin

If things don’t work, it’ll sit there for a while and reward you with:

tftp:// connection timed out

If things are working correctly, the screen will fill with periods and you’ll have to wait a couple minutes to be rewarded with something like this:

File "tftp://" successfully copied to "flash:c3560e-universalk9-mz.122-55.SE1.bin"


Now you can either boot if you want to just fire up the .bin you copied over or you can reset if you want the system to do a hard reset (similar to a power cycle).