#vBrownBag Book Contest (not) Still Going!

The contest is still going until June 30, 2014 over! There hasn’t been a huge number of entries, so your odds are very good. Get your entries in!

Contest Details

ShowCover.aspI have copies of both the CCNA Routing and Switching 200-120 Official Cert Guide Library and Networking for VMware Administrators to give away. These are courtesy of Cisco Press and VMware Press, so a big thanks goes to them!

Here’s how you can win one:
Send out a tweet with your CCNA R&S question, including the hashtag #vBrownBag and @scottm32768. If your question too long for twitter, you can post it as a comment here, then link to it on twitter. The best questions by the end of the month (June 30 2014 23:59) will win. Myself and others related to the #vBrownBag podcast will make this decision. In the event we cannot reach a decision, we will use the contents of a hermetically sealed envelope kept in a #2 mayonnaise jar on Funk and Wagnall’s back porch. Or maybe just choose winners at random.

FIN

Cisco Live 2014 Day 1

Pedometer Stats

Todays stats for the Official Unofficial Cisco Live Pedometer were:

Steps: 7611
KCals: 411
Miles: 3.6

Not as much as I expected for today. Maybe things are closer than I at first thought.

VIRL & CML Update

I had a chance to talk with some of the CML/VIRL team today and learned a few things of note:

  • VIRL (Virtual Internet Routing Labs) is back as the personal/developer edition and will be available through the Cisco DevNet. For free. Yes, you read that right!
    • I believe this will support up to 15 Cisco VMs and as many other VMs (Linux guests, etc) as fit in your RAM, but I need to verify the 15 number is still accurate.
    • The quid pro quo for this deal is they will be collecting data. They will be completely up front about this, the data they are collecting, and will transmit it in clear text. The data they will be collecting is going to be things like what kinds of virtual devices you launch, how many, what specific features you use, and that sort of general statistics. They will not collect any configuration information.
    • No NX-OS at this time. The virtual NX-OS is just not ready, yet.
    • Still no L2, but they have a prototype already for adding this in the future. It will be coming, just not right away.
    • This will be community supported.
  • CML (Cisco Modeling Labs) is the corporate version. This is licensed by the node, can scale much larger, and will have Cisco support. I have not seen pricing for this, but expect it to be priced for corporate use.
  • I have no release dates for either of these, but it does sound like they are pretty close to being able to ship it.

I’m really looking forward to this. It’s going to be a great tool for both validation and learning.

There was more to the day, but those were the bits that are probably most interesting for the moment.

FIN

Cisco Live 2014 Day 0

The Official Unofficial #CLUS Pedometer

The Official Unofficial #CLUS Pedometer

So It Begins

It’s that time of year again when I break out the Official Unofficial #CLUS Pedometer. Today wasn’t a very busy day and I didn’t actually have any technical sessions, so the count is pretty low.

Steps: 5272

KCals: 285

Miles: 2.499

Not stellar numbers, but I assure you they will be bigger tomorrow!

Today consisted of a morning of last minute studying for the CCIE R&S written, followed by failing the CCIE R&S written, and in turn followed by regret for wasting my “free” exam on a test that I was pretty sure I wasn’t going to pass. Oh well, hind sight is 20/20. If I can find some time later this week, I have a couple other exams I’m thinking of taking that I’m much more likely to pass. I don’t think I’ve left a Cisco Live without passing a test, yet.

Social Media Hub

I spent most of my afternoon catching up with old friends and meeting new ones at the Social Media Hub as people started to arrive for the conference in force. It was a good time talking about all kinds of mostly geeky stuff. Later in the afternoon was the official tweet up with probably a couple hundred people powered by caffeine and sugar. Meeting more people and doing more geek talk.

Cisco Empowered Women’s Network

After meeting with people for a bit, a group of went over to the Cisco Empowered Women’s Network session. We went over primarily to see Amy Lewis (@commsninja) speak, but ended up staying to hear Carlos Dominguez (@carlosdominguez) speak as well. Both were very good talks. Amy spoke about social and how it can be used for you and your career. Carlos’ talk was about how these tools are both disruptors and enablers and how they can be used for the greater good and to create change. Both were humorous and informative. I suspect some of the earlier talks must have been pretty good, too. If you want to check out the video, it should show up at this URL sometime in the next few weeks: CEWN-1000 – Cisco Empowered Women’s Network – Today. Tomorrow. Transformed.

On Monday, the real learning begins.

FIN

Prime Infrastructure 2.1 Upgrade

Cisco Prime Infrastructure Login Screen

Cisco Prime Infrastructure Login Screen

Cisco recently released Cisco Prime Infrastructure 2.1. Here’s a link to the release notes. I run my instance as a VMware appliance. That helps to make upgrades relatively painless and risk free because you can take a snapshot before you start. If everything goes heinously wrong, you can restore the snapshot and everything is hunky dory again. Then you can go figure out why it didn’t work without worrying about a service being down.

The Repository

With ADE-OS, you can’t just reference the URL for the download like you can with IOS. There’s no command like copy tftp://192.0.2.1/upgrade.tgz flash:. You must pre-configure a “repository” and reference that. For some reason, I almost always seem to have issues using a tftp repository. It may just be something about the way I have my machine setup, but it seems to only happen with these ADE-OS appliances. In my case, I have configured a repository for my workstation:

repository httpRepo
  url http://192.0.2.1:8000/

When I want to transfer an upgrade or patch to an ADE-OS appliance, I just fire up a python HTTP server with python -m SimpleHTTP in the directory where the upgrade file lies.

The repository will also let you configure FTP with a username and password. I use this method for my backup repository that points at an appropriate server. You configure that something like this:

repository backupRepo
  url ftp://backupserver.example.com/
  user ncs password plain <password>

The password will be stored as a hash in the config.

SimpleHTTPServer

So here’s what my Python http server looks like:

scottm@scottm-mac:~/tftp$ python -m SimpleHTTPServer
Serving HTTP on 0.0.0.0 port 8000 ...

It’s pretty exciting. When you’re ready for it to quit, hit ^C in that window. The same thing works if you’re running on Windows. Don’t forget that you may need to allow the port through your host firewall.

The Upgrade

Before you start the upgrade, verify that you are running a version that us supported. As of this writing, you can upgrade from 2.0.0.294 or 1.3.0.20 with the point patch (see the Quick Start Guide for more info). Sorry, still no path from 1.4. Here’s how you check your version:

prime/admin# show application version NCS

Cisco Prime Infrastructure
------------------------------------------
Version : 2.0.0.0.294

All good, so now we stop the NCS processes:

prime/admin# ncs stop

Stopping Network Control System...

This may take a few minutes...

Network Control System successfully shutdown.
Plug and Play Gateway is being shut down..... Please wait!!!

Stop of Plug and Play Gateway Completed!!
SAM daemon process id does not exist
DA daemon process id does not exist
DA syslog daemon process id does not exist

prime/admin#

Here’s where the real stuff starts. You took a snapshot of your VM and/or have a valid backup, right?

prime/admin# application upgrade PI-Upgrade-2.1.0.0.87.tar.gz httpRepo
Save the current ADE-OS running configuration? (yes/no) [yes] ?
Generating configuration...
Saved the ADE-OS running configuration to startup successfully

Please ensure you have a backup of the system before proceeding.
Proceed with the application upgrade ? (yes/no) [yes] ?

DO NOT press ^C while the upgrade is in progress
Aborting upgrade with a ^C may leave the system in a unrecoverable state

Initiating Application Upgrade...
Stage 1 of 7: Transferring file ...

At this point, you should see it hit your HTTP server if you used the Python SimpleHTTPServer:

scottm@scottm-mac:~/tftp$ python -m SimpleHTTPServer
Serving HTTP on 0.0.0.0 port 8000 ...
192.0.0.10 - - [05/May/2014 13:49:55] "GET /PI-Upgrade-2.1.0.0.87.tar.gz HTTP/1.1" 200 -

This is what the rest of the install looks like. This takes some time… The time stamps in the logs I am posting are from an actual upgrade, so it gives you some sense of the time involved when running the upgrade on a lightly loaded C210 M2 host.

Initiating Application Upgrade...
Stage 1 of 7: Transferring file ...
-- complete.
Stage 2 of 7: Unpacking file ...
-- complete.

*** System will reboot after a successful installation of this package ***
After reboot, please login again into the server to check status

Stage 3 of 7: Executing pre-install ...

[WARNING] System will reboot after a successful installation of this package (after Stage 7).
After reboot, please login again into the server to check status.
No action required at this time. Continuing with Stage 3.

-- complete.
Stage 4 of 7: Upgrading binaries ...
-- complete.
Prime Infrastructure Application installation completed
Stage 5 of 7: Retrieving system version ...
-- complete.
Stage 6 of 7: Updating Database Schema ...
: This could take long time based on the existing data size.

Stage 1 of 5: Pre Migration Schema Upgrade ...
-- completed at: 2014-05-05 13:58:01.026, Time Taken : 0 hr, 0 min, 19 sec
Stage 2 of 5: Schema Upgrade ...
: This could take long time based on the existing data size.
-- completed at: 2014-05-05 14:02:07.149, Time Taken : 0 hr, 4 min, 6 sec
Stage 3 of 5: Post Migration Schema Upgrade ...
-- completed at: 2014-05-05 14:03:34.82, Time Taken : 0 hr, 1 min, 27 sec
Stage 4 of 5: Enabling DB Constraints ...
-- completed at: 2014-05-05 14:07:02.317, Time Taken : 0 hr, 3 min, 25 sec
Stage 5 of 5: Finishing Up ...
-- completed at: 2014-05-05 14:07:20.315, Time Taken : 0 hr, 0 min, 17 sec
-- complete.
Stage 7 of 7: Re-enabling Database Settings ...
-- complete.
Upgrade Finished. Server is restarting . Please wait ..

% This application Install or Upgrade requires reboot, rebooting now...

Broadcast message from root (pts/0) (Mon May 5 14:10:31 2014):

The system is going down for reboot NOW!

Application upgrade successful
prime/admin#

This takes quite a while to start up after the reboot, but eventually you’ll be able to login and use the new features. Enjoy!

FIN

Nexus 5k NTP Failure

Ran into this bug today. Went to a pair of Nexus 5500s to debug a vPC link and noticed the timestamps were off. I thought that was odd. I tried show ntp peer-status and received no output in return, which I thought was even more odd. I poked at NTP for a while and decided it had to be a bug. I found the bug in the release notes and it was fixed in 5.2(1)N1(6). Here’s the Cisco bug report:

Symptom:
Nexus 5k acting as an NTP Client can’t sync with any NTP server(s).
when issuing a “show ntp peer-status” or a “show ntp peers” it does not display any of the servers/peers configured.

Conditions:
Nexus 5500/5000 running 5.2(1)N1(5).

Workaround:
Proactive workaround to prevent from this issue is none.
Reactive workaround to recover this issue is below. However, after reloading system, same issue may happen again.

#conf t
#clock protocol none
#clock protocol ntp
#copy run start

Fun. At least it can be fixed without reloading, which is a good thing in a data center switch.

FIN

Networking Podcasts

I have a long commute and I like to make good use of this time. I used to listen to a lot of audiobooks (Audible.com was great!), but now I listen mostly to podcasts. Here are some podcasts I recommend:

Packet Pushers LogoPacket Pushers

 “The Packet Pushers Podcast is an audio program published multiple times per month covering the data networking industry. Co-hosts Greg Ferro and Ethan Banks are professional network architects and writers with many years of network engineering and design experience in a variety of industries, as well as being Cisco Certified Internetwork Experts.”

This is a highly technical podcast. I listen to the “Fat Pipe” feed, which is all their podcasts. If that’s too much content for you, there’s several feeds that you can use to limit the content. There’s a great back catalog of episodes here. Really good stuff!

iTunes LinkRSS Link

Wireless LAN Weekly LogoWireless LAN Weekly

“A weekly audio podcast designed to educate, inform, entertain, and inspire Wireless LAN Professionals. Those folks dedicated to the craft of designing, installing, configuring, maintaining, securing, troubleshooting and managing Wireless Networks.”

This is another podcast that has a great back catalog. If you are into wireless, you might want to listen to the whole back catalog. You’ll learn a lot! You might also want to check out the videos recorded at the WLAN Pros Summit.

iTunes LinkRSS Link

No Strings Attached Show LogoNo Strings Attached

Another WLAN focussed podcast. They do a combination of news/discussion episodes and sponsored episodes covering new equipment and software from leading vendors.

iTunes LinkRSS Link

Cisco TAC Security Podcast LogoCisco TAC Security Podcast

“The Cisco TAC Security Podcast Series is created by Cisco TAC engineers. Each episode provides an in-depth technical discussion of Cisco product security features, with emphasis on troubleshooting.”

iTunes LinkRSS Link

Cisco Champion LogoCisco Champions Radio

“#CiscoChampion Radio is a podcast series by Cisco Champions as technologists and I.T. professionals, hosted by Cisco’s Amy Lewis (@CommsNinja).”

This podcast covers a variety of Cisco-centric networking topics.

iTunes LinkRSS Link

Class-C Block LogoThe Class-C Block

“The Class-C Block is a show dedicated to all things networking and nerdy. It is the most awesomest place – it is where you want to be. It is the brain child of Matthew Stone and CJ Infantino. The show discusses a variety of networking topics, and occasionally drifts into other nerdy topics.”

iTunes LinkRSS Link

Other IT Podcasts
These podcasts are not necessarily networking related, but they are IT related.

AdaptingIT Logo AdaptingIT

“The idea for this podcast came about when Mike Laverick, Jane Rimmer, and Lauren Malhoit, were discussing how to encourage more women to get involved in the technical community.  This podcast is not necessarily about women in tech, but rather women talking about tech.”

I often don’t listen to the entire show, but I normally listen to at least the beginning and end. Often the middle talks about areas that I either don’t grok or don’t have much interest in.

iTunes Link - RSS Link

Geek Whisperers Logo Geek Whisperers

“The Geek Whisperers came to be in 2013 based on one too many good conversations we could no longer keep private. Focused on Social Media and Community for Enterprise, our home base is High Tech, but we all look far beyond our field and current communities for inspiration.”

This podcast is more about social media and marketing-ish topics. I find it valuable (it’s full of unicorns and bacon), but it might be a bit too inside baseball for those not active in social media.

iTunes LinkRSS Link

FIN

No More VLANs Jumping on the Trunk

ThinkGeek Cable MonkeyNo more VLANs jumping on the trunk. That’s what I wanted. Instead, I got all the VLANs and a nice supervisor debilitating loop in the process… From my after action report: “the issue was a configuration error that caused the CPU to peg at 99%, which caused a variety of issues.”

Due to a legacy design that we have been working to retire, we have a layer 2 Ethernet WAN that can’t run STP. Yep, it’s a loop waiting to happen.

We were migrating sites to a new WAN. The circuits come in over a trunk with each site getting a VLAN tag. I was moving the last site from an interface. The procedure I wrote had the VLAN being removed from the interface with “switchport trunk allowed vlan remove ###”. Worked great until we hit the last VLAN on the trunk. When the last VLAN is removed from an interface, you might expect no VLANs to be allowed. If that’s what you expected, you are wrong. When you remove the last VLAN with “switchport trunk allowed vlan”, the port reverts to allowing ALL VLANs.

Unfortunately, since all VLANs were suddenly available on that port, it allowed a bridging loop to be created through the sites that were still connected to both the old and new WANs. This caused a variety of different bits of havoc that resulted in connectivity issues. Yeah, something of an understatement.

So, here’s some lessons I took from this:

  • Dual connected sites should have their legacy WAN ports shut down once they are stable. (This shouldn’t be a problem in a proper L3 design, but is important with our broken L2 design.)
  • Designs that prevent you from using STP to prevent loops need to be fixed sooner rather than later.
  • Entering explicit commands is better than implicit. In this case, “switchport trunk allowed vlan none” would have been better than “switchport trunk allowed vlan remove <last vlan>” and assuming the result will be what you expect.
  • Don’t make assumptions on how a device will behave. Lab it and know.
  • Always verify that changes had the desired effect. Had I verified the interface was configured as expected, the problem would have been discovered immediately and had little or no impact.

FIN