A Strange, Unsolved WLAN Problem

I’m seeing strange behavior on the WLAN at one location. This location is one of many that are on the same controller and identically configured, save for AP locations and IP addresses. No other site is reporting this problem. Here’s my problem description:

Users report that many devices are unable to access “any site that requires a login”. From what I’ve seen, this really means most (but not all) SSL protected URLs. HTTP URLs work fine, HTTPS URLs timeout. This only happens on the open guest SSID. If one connects to the secured corporate SSID, everything works normally. Reports indicate that many, but not all devices are impacted. We couldn’t find a single Apple device that was impacted, but on-site staff believes it hits some of them, too. One of the staff owns an Android phone on which the problem is reproducible. I’m heading out there tomorrow with a suite of test devices to see if I can duplicate it with any of them. There is a possibility it only hits 802.11ac devices, but this is not the only 802.11ac site and it is the only one reporting this problem. I have connected with an 802.11ac laptop and had no issues. The 2.4GHz RF environment leaves something to be desired (and was the source of the Spectrum Analysis as Art post), but this problem also occurs on 5GHz. The APs are in FlexConnect mode, so I tried switching them to local mode and that did not change the behavior.

Does this sound like anything someone has seen? Any ideas what is going on?

FIN

Spectrum Analysis as Art

SpecAn Art

Spectrum Analysis as Art

I was looking into a wireless problem and ran across this interesting view in 2.4GHz in my spectrum analyzer. Believe it or not, this wasn’t related to the problem I was troubleshooting, because the problem I was fixing also impacted 5GHz. However, this struck me as artistic. I see it as oceans, trees, and mountains. I believe it’s a combination of some sort of analog frequency hopper (the peaks in back), a narrow band frequency hopper that may not be bluetooth (the “trees”), and a hodgepodge of wireless networks, not all of which were on 1,6, or 11. I posted it on Twitter and quite a few people thought it was interesting and wanted a copy, so here it is. We had a little fun parodying Bob Ross, too.

In case you want the version straight from Airmagnet Spectrum XT, I’ve included that below.

As saved from AirMagnet Spectrum XT

As saved from AirMagnet Spectrum XT

FIN

Cisco Live 2015, Antennas, & Mike Rowe

Last month I attended Cisco Live! 2015. I’m a big fan of this conference and always feel like it’s a great event. This year was a large show with around 25,000 attendees, well over 700 sessions, hundreds of vendors, and only four days to take in as much as you can.  It is summer camp for geeks, where we get to learn new things and talk to everyone we haven’t seen since last year’s conference.

The Keynote

This year’s conference was historic as the last Cisco Live with John Chambers as the CEO of Cisco. At the end of his keynote he brought out his replacement, Chuck Robbins. They said all the things you expect them to say, but we’ll have to see how everything shakes out with the changes this brings. After 25 years with Chambers at the helm, this is going to be quite the transition for Cisco. It will be interesting to watch over the next few years, but I’m optimistic that things will go well.

The Antennas

One of the more interesting things I was able to get a close look at is the new Hyperlocation Module for Cisco APs. It uses a new version of the WSM (WSM2, I believe) and wraps around the AP. They made a cool version of the module that allows you to see the antenna arrays inside.

Cisco Hyperlocation Module, Transparent Edition

Cisco Hyperlocation Module, Transparent Edition

The idea here is that with this array of antennas, they can determine the Angle of Arrival of a Wi-Fi signal. This allows a much more precise calculation of location and with these you can improve from the previous best case of about 3m of accuracy to about 1m of accuracy. That’s some pretty precise location information. Potentially more important, this will give more flexibility in design. You no longer will need to have APs all the way out in the corners of a building to get good location information. They also said that the module is where they do their research and try out new things before including them in the AP. There is an implication that they will try to get this technology inside the next generation of APs. Imagine if all your APs just had something like this built in. Designing a wireless deployment for 5GHz might naturally be a location capable design if you choose the right APs.

I also learned about a new patch antenna. The 2513P stadium patch antenna available from Cisco has 30 degree beam width and the 2566P patch has a beam width between 105 and 120 degrees, depending on the band. They wanted something in between so there is now a 2566D that has a 60 degree beam width. If you’ve ever worked with the 2566P, you know about dealing with the cables. There just are not many good ways to install that antenna in an aesthetically pleasing manner. The 2566D helps with that. The antenna will mount flush to the wall with the cables either going straight out the bottom or straight out the back. This gives you options for a much cleaner installation.

Cisco AIR-ANT2566D4M-R Antenna

Cisco AIR-ANT2566D4M-R Antenna

The Dirty Jobs

The conference ended with a closing keynote from Mike Rowe, who told the story of what led to the creation of the Mike Rowe Works Foundation. Mike started the foundation to provide scholarships to those who wanted to work hard and learn a skilled trade. He was very entertaining, but had a message that things are out of whack in the US when many people are out of work, yet jobs remain unfilled. Many of those jobs are from the skilled trades such welders, plumbers, and electricians. Jobs that don’t require a college degree, yet people are racking up huge debt from student loans for an education to get a job that may not exist. Mike Rowe seems like a down to earth guy and he’s leveraged his position to do some good work. You might consider checking out http://profoundlydisconnected.com if you like the sound of what he’s doing. You can also view his keynote online at the Cisco Live on-demand library (free account required).

I have more to say, but that’s enough for now. I’m already looking forward to next year’s conference. In fact, I’m already registered!

FIN

Cisco TSHOOT v2 (300-135)

My CCNP certification (Story here, if interested) was expiring at the end of June. I’d taken a number of professional and expert level exams at Cisco Live over the years, but hadn’t passed anything that would renew my CCNP. It was time for the (almost) sure thing. TSHOOT.

The TSHOOT exam is my favorite Cisco written exam because it is all hands on. You don’t actually configure anything; you are just looking and analyzing. This exam is unique in several ways, including:

  • The network topology is available for download.
  • You really can score 1000 (that’s a perfect score).
  • It’s actually fun.

The exam is set up as a series of trouble tickets in the topology mentioned above. There is a problem description and you have access to the consoles of simulated equipment to try and determine the cause. After you have investigated, you answer three questions to narrow down the cause and the solution, then you move on to the next ticket. If you know your stuff, it’s a pretty easy exam. That said, don’t get too comfortable. I started running out of time because I was taking too relaxed of an approach and ended up using almost the entire time to complete the exam.

If you’ve taken the previous version:

  • The topology is the same (as best I can tell)
  • The question style for the tickets is the same.
  • You are no longer allowed to abort a ticket. You must finish once started and you must answer in the order delivered.
  • The simple multiple choice questions are spread throughout the test instead of all being at the beginning.
  • The interface is different. There are no tabs for the tickets. You just click “next” when you are finished with your answers.

I’m pleased to report that I did pass and even had a perfect score. I’ve never done that before and I doubt I could do it on any of the conventional multiple choice exams, but it was satisfying to achieve on this one. With my CCNP R&S recertification complete, I can concentrate on other studies without worrying about it!

In case you are looking for study materials, you may want to investigate the CCNP Routing and Switching TSHOOT 300-135 Official Cert Guide Premium Edition eBook and Practice Test or the Troubleshooting and Maintaining Cisco IP Networks (TSHOOT) Foundation Learning Guide.

FIN

What’s New In Cisco Wireless Software 8.0.110 – 8.1.102

Cisco Wireless Software 8.1.102 has been released and it’s time to bring my notes up to date.

These are abridged notes covering high points. Read the release notes for yourself and test your chosen build before deploying it in production.

8.0.110 (Release Notes)

Note: If you need to run 8.0.110 (or 7.6), please read this post: TAC Recommended AireOS 7.6 and 8.0 – 2Q CY15

  • If you have 3700P APs, don’t install this release. Contact TAC. This warning doesn’t apply to 3700i or 3700e.
  • Support added for the 1570 AP
    • Handful of features added to support 1570 specific features.
  • Support for priming universal APs (APs not locked to a regulatory domain) and auto setting the regulatory domain based on location. See Cisco Aironet Universal AP Priming and Cisco AirProvision User Guide for more info.
  • Enhancements to Express Setup for the 2500 WLC.
  • SSLv3 is now disabled by default.
  • Lots of resolved caveats. Lots of open ones, too…

8.0.115 (Release Notes)

  • Nothing new. Very short list of bug fixes, mainly for the 3700P.
  • The 8.0.110 special build (mentioned in the TAC Recommended link above) may be a better bet right now.

8.1.102 (Release Notes)

  • Virtual WLC now supported on KVM.
  • These APs retain feature parity with 8.0 and do not gain new features: 1050, 1140, 1260. Nice to see support not completely vanish, yet.
  • Support for WLC 5520 and 8540 added.
  • Dynamic Bandwidth Selection (DBS)
    • Chooses 20/40/80MHz channel width automatically. Tries to balance client needs with RF needs. I’ll be interested to hear what others think. I’m sticking with 40MHz.
  • Flexible DFS
    • Automatically adjust channel and width to avoid radar for more efficient channel usage. I presume this leverages DBS.
  • Enhanced Interference Mitigation
    • ED-RRM now also takes Wi-Fi interferers into account.
  • Optimized Roaming Extensions (802.11v BSS Transition Management)
    • Infrastructure helps clients make better roaming choices. Not sure what clients actually support this.
  • Defaults now implement best practices.
  • AVC added to FlexConnect APs
    • I’ve been told this isn’t supported on the 2504, but the release notes don’t say. YMMV.
  • SNMP MIB enhanced to allow monitoring of an HA WLC.
  • Support for Lync SDN API.
    • In short, Lync tells the WLC when a call is happening so the WLC can take QoS actions.
  • AVC updates
    • Per app, per client rate limiting
    • AVC based QoS markings
  • Inter controller roaming across IOS-XE and AireOS based controllers (8500 series, 5520, 5760)
  • AAA can override FlexConnect VLAN.
  • Stateful client switchover for mesh APs (RAPs and MAPs)
  • There is a decent list of caveats. Please go read them for yourself if you find this release of interest.

Notes at the end remind that 7.6 is still the recommended release for 802.11ac deployments, with 7.4 for 802.11n deployments.

Also, Field Upgradeable Software (FUS) 1.9 is recommended. You can do a “show sysinfo” to see what you have installed. Look for the Firmware Version line and the Field Recovery Image Version. If you have FUS 1.9, it will look like this:

Bootloader Version............................... 1.0.20
Field Recovery Image Version..................... 7.6.101.1

FIN

In BYOD, Client Devices Manage You

I wrote another post over at the Aruba Airheads Community. Here’s a taste…

With the recent release of OS X and iOS updates, I have been reminded (again) of how subject we are to the manufacturers of our client devices. In this particular example, I’m contemplating that since the release of iOS 8 and OS X Yosemite, reports of Wi-Fi problems in my organization have skyrocketed. Not that I’m trying to pick on Apple, they just happen to be the current source of trouble…

If you’d like to read the rest of it, you can check it out at the Airheads Technology Blog.

My Cisco Live Schedule

First of all, if you are a Cisco Live NetVet, the scheduler is open at https://www.ciscolive2015.com/connect/mySchedule.ww. If you are not a NetVet, it will open for you on March 31st.

I’ve spent a bit of time tweaking my schedule and here’s my current plan:

My schedule for Cisco Live

Scott’s schedule for Cisco Live

I suspect I may want to change things up a little after the technical seminar on Sunday. I might also change things after the slides are released and I can see what the plan is for those sessions. Focus this year is obviously on wireless. :) I’m looking to learn more about are CMX and all the changes that have come with MSE 10. I’m not using CMX right now, but the need for it and the value are starting to rise. My current architecture is centralized controllers and FlexConnect APs. This has some limitations, so I’ll be looking to learn more about the converged access solution to see if I can leverage the 3650s we’ve been deploying to get the APs back to local mode, but still have the branch traffic stay in the branch. I might drop a session to spend some quality time in the WoS without a huge crowd, but I’ll have to see what the session slides say first. I haven’t looked to see what recent recordings are available in the Cisco Live On-demand Library, either. Might be able to free up a session or two that way, as well.

I dropped the industry keynote and replaced it with the CCIE Wireless written exam. I’ve taken the R&S exam a couple times before and didn’t quite make the cut, but I’ve been doing a lot more wireless work than routing work, so I’m optimistic that I can get a better score here. We’ll see how I do in a few months. :)

FIN