Cisco WLC 7.0.220.0 and WCS 7.0.220.0 Released

A few notes regarding the recently released WLC and WCS software updates.

WLC 7.0.220.0
Release Notes: http://www.cisco.com/en/US/docs/wireless/controller/release/notes/crn7_0_220_0.html

• There are only a couple of new features with the rest of the update comprised of bug fixes:
• Online Certificate Status Protocol (OCSP) support. OCSP is an updated method of certificate revocation over the older Certificate Revocation Lists (CRL).
• Ability to view AP images stored in WLC flash by way of the new CLI command show ap bundle {all | primary | secondary} 
• Band Select and Load Balancing Support on Cisco Aironet 1040 series Access Points
• Support for AIR-CAP1552SA-x-K9 & AIR-CAP1552SD-x-K9 mesh APs
• I had heard that support for the 2100 and 4400 WLCs was meant to be dropped in this release but the release notes still show them as supported
• There is a note detailing that WCS will loose connectivity if you are upgrading from 7.0.116.0 to 7.0.220.0 AND using SNMPv3. Workarounds are provided.
• The rest is comprised of the usual assortment of bugs fixes

WCS 7.0.220.0
Release Notes: http://www.cisco.com/en/US/docs/wireless/wcs/release/notes/WCS_RN7_0_220.html

• There are no new features however support for a few WLCs has been dropped. It seems strange that WCS has dropped support for these WLCs but support has not been dropped in the paired WLC 7.0.220.0 image. Perhaps this is an oversight in the WLC 7.0.220.0 release notes. The following Location Appliances and WLCs have been dropped:
• Cisco 2700 Series Location Appliance
• Cisco 2000 Series Wireless LAN Controllers
• Cisco 4400 Series Wireless LAN Controllers
• There is now version consistency between WLC and WCS images
• The rest is comprised of the usual assortment of bugs fixes

FCC crackdown on RF jammers

I just read an article concerning the FCC cracking down on US websites selling RF jammers. This made me wonder whether the Wi-Fi-nerd-pleasing NutsAboutNets AirHORN jammer... ahem, signal generator, is affected. Well, from looking at the list of websites in the 'Citation and Order', the AirHORN is safe.

To be fair, whilst the AirHORN could be classed as a wideband jammer, it doesn't take out the whole 2.4 GHz or 5 GHz band at once. It therefore isn't the same as the traditional jammers that the FCC is targeting which are all-band jamming, affecting the entire 2.4 GHz band for example. When you use the AirHORN, you can see that it is a modulated 802.11 signal. It is really a glorified Wi-Fi client which is why you have to pick the 802.11 channel to start the signal generation or allow it to sweep through the channels. I can't help but think that the AirHORN is just using a modified version of the Queensland Attack. This shows that it really isn't intended for jamming and is meant to be used as a signal generator - therefore it wouldn't be affected by this FCC crackdown.

I have used the AirHORN during training to demonstrate spectrum analysis. It (should) also be useful for testing device classification in the big three Spectrum Analysers (Cisco SpectrumExpert, AirMagnet SpectrumXT and Metageek Channelyzer Pro). In addition, it (should) be useful for testing / demonstrating a high duty-cycle interferer in the new breed of AP-integrated spectrum analysis implementations (Cisco CleanAir, Aruba RFProtect, Aerohive Spectrum Analysis, Motorola AirDefense Spectrum Analysis, etc).

AirHORN aide, this citation from the FCC will have little effect on those in the US wishing to purchase all-band jammers because apparently you can now buy goods on-line... from overseas no less! What an age to be alive! Additionally, publicising this issue may well bring about the Streisand Effect. I am not advocating malicious use of a jammer however getting a hold of one will remain easy. Globalisation biting the US in the ass, yet again! One of the great ironies of those advocating, selling or actually using all-band jammers maliciously is their total ignorance as to their operation. Many people who would use these jammers would inflict more damage on their own Wi-Fi communications without realising it. From one of the sellers websites - "Place in your friends home as a prank or use it to secure you own Wi-Fi.". Well I guess if even you can't access your Wi-Fi, it could be deemed secure!

If the goverment can't help me, who can?
So if the FCC crackdown has little effect in helping reduce the market for jammers what are your options? The key for those operating Wi-Fi networks is to have the right tools on hand to address the issue - namely, a Spectrum Analyser. Metageek has made this particularly easy in recent years with their affordable analysers as have Wi-Fi vendors rolling out their AP-integrated offerings. Also, this is yet another reason to purchase 5 GHz capable Wi-Fi clients, where possible. The vast majority of all-band jammers transmit at 2.4 GHz as do the majority of other high duty cycle transmitters (baby monitors, A/V transmitters, wireless video cameras, etc). Whilst 5 GHz all-band jammers exist, they are few and far between and no where near as cheap.

EOF.... almost
As for the FCC crack-down, the Streisand Effect has worked and reminded me that I have been meaning to buy a cheap all-band jammer for (isolated) demonstrations and spectrum analysis testing. I've thrown an analogue video camera and A/V transmitter into the cart for good measure. Thank you FCC!

WLAN Controller Discovery Best Practices

The methods that APs use to discovery the WLAN controller(s) in a Cisco environment are relatively well known. Andrew vonNagy has documented them quite thoroughly. Take a look at his post if you are not familiar with these methods.

Whilst Andrew has covered the details of how these discovery methods work I thought I would provide my best practices as to when you may use one method for controller discovery over another. As I have referenced Andrew's post, I will copy his list of discovery methods verbatim in order to maintain a blog post SOE ;).

WLAN controller discovery methods
1.       Broadcast on the local subnet
2.       Local NVRAM list of the previously joined controller, previous mobility group members, and administrator primed controller through the console
3.       Over the Air Provisioning (OTAP) (subsequently removed in version 6.0.170.0 code)
4.       DHCP Option 43 returned from the DHCP server
5.       DNS lookup for "CISCO-CAPWAP-CONTROLLER.localdomain"

Broadcast
I never rely on this method nor would I recommend it. I have encountered issues with its use though these may be bugs long since squashed, none the less I prefer not to rely on it. The only time I would consider it is if I the server team was not willing or able to modify DNS or DHCP in order to implement the DNS or DHCP discovery methods. Though you can bypass the server team and run DHCP on a router, this means you are likely not maintaining consistency with your SOE and additionally Option 43 on a router is quite messy. If the controller is on a different subnet to the APs you will need to configure an ip-helper. This will result in the WLAN controller receiving other forwarded protocols (by default this includes, TFTP, DNS, NETBIOS, DHCP, TACACS amongst a few others however these defaults can be disabled using 'no ip forward-protocol up <protocol>').

Local NVRAM or Primed
Whilst the APs can store details in NVRAM of controllers they have previously associated to, they have to get this information to begin with so this method is not going to work as your only method of discovery. In other words, know that it exists but use another method.

APs can also be primed which involves manually configuring them with an IP, Mask, Gateway and WLAN controller address. The APs IP address could in fact come via DHCP and then you would just prime it with the controllers IP address. But if you have a DHCP server available why not configure DHCP Option 43? Perhaps such a scenario exists but it wouldn't be common to do this in any enterprise WLAN deployment. But then, priming more than half a handful of APs in any enterprise deployment would likewise be a pain. The only time I prime an AP is when I have a one or two APs to be deployed for use in a temporary office or when I have done WLAN training and needed to get an AP up and running before the server-side infrastructure was in place.

I find that sometimes a company will associate an AP with a local controller in order to download the appropriate code and then ship the AP to site for connection to the remote sites controller. Once on-site the AP is powered up and attempts to connect to a controller. The issue with this comes with the AP having saved the local controllers details to NVRAM and may result in the AP connecting to the local controller if the primary controller configuration is not performed on the AP, instead of the remote sites own controller. I am unsure what the logic is here as the code upgrade will be performed over the sites LAN so no time or WAN bandwidth is saved. Basically, don't bother as there is no benefit and it will just result in more trouble-shooting on your part. The number of times I have seen APs connect to the wrong controller due to a poorly thought out design...

Keep in mind that APs can also find a controller based on the mobility group of any controller they have already associated with. I was recently trouble-shooting an issue that involved an AP joining the wrong controller. After verifying DHCP, DNS and priming methods I checked the mobility group of both controllers. Sure enough they were in the same mobility group - the local AP had learned about the remote controller by means of the local controller passing the details on. A typo in the primary and secondary controller configuration meant that it disregarded these controllers and used the information it learnt via the mobility group configuration. The secondary issue here was a design issue - the controllers should not have been in the same mobility group. 

Over the Air Provisioning (OTAP)
No longer available nor should it have been in my opinion - it was always a bad idea.

DHCP Option 43
OK, now we get to the first of the two recommended methods of controller discovery. Option 43 seems to be the most common method used for discovery. I recommend its use whenever you have a controller at each of your sites and therefore you want each of the 'Site XX APs' to connect to the 'Site XX WLAN controller'. In each of the DHCP scopes for each site, configure Option 43 for the particular models of APs at that site, pointing the APs to the appropriate sites controller.

The problem with the DHCP discovery method can come when you need to configure Option 43 for each model of AP at each of your sites. Not an issue with one site and one model of AP. But take 30 sites, 30 DHCP scopes, potentially 30 DHCP servers and a few models of AP at each site. In addition, in 18 months when you have left the business, is someone going to know that some DHCP modifications are required when a new model of AP is rolled out? If you have a separate WLAN controller at each site however, then it really is the best method despite the extra admin overhead. On the other hand, if you have either a centralised controller or a single / pair of controllers at a single site, I would recommend using the DNS method.

Another time to look at the DNS method is if you happen to be hosting DHCP on one of your routers. The Option 43 router configuration is messy compared with the ease of configuration involved in getting DHCP Option 43 up and running on a Windows Sever 2003 / 2008 box so stick with DNS in this case.

DNS
The time to use DNS is when all of your APs connect back to the same WLAN controller(s). This could either be remote sites using a centralised controller or perhaps you just have a single site. Let’s say you have 30 sites. Instead of configuring 30 DHCP scopes or 30 DHCP servers with Option 43, just configure an A or CNAME record for CISCO-CAPWAP-CONTROLLER.localdomain pointed at your centralised controller(s). If any new model of AP is rolled out, you don't have to change anything regarding your discovery method, unlike with the DHCP method. This assumes of course that all of your sites fall under the same .localdomain. You can a create global scope option in Windows Server 2003 / 2008 but then Option 43 will apply to all your scopes - not just the AP scopes. This may or may not be of concern.

If however your 30 sites each have their own controller and assuming you DON'T have a separate sub-domain for each site you are going to have problems with APs associating to the wrong controller. You would need to create 30 A or CNAME records for the 30 WLAN controllers but how do you direct APs to the correct controller? You can't do it in any reasonable fashion. Yes, you can specify primary and secondary controllers but this occurs after the AP has associated. You could also configure the master controller option but this is messy. Someone will without a doubt forget to specify the primary controller at some point and you will end up with APs associated to the wrong controller. Inexperienced engineers trouble-shooting the issue will scratch their head as to why the AP has not associated - not realising that it has but is associated to another controller.

Logging
Though not directly related to controller discovery, I will discuss a little know DHCP option that can greatly aid in trouble-shooting AP join issues.

Out-of-the box APs transmit syslogging messages as limited broadcasts (255.255.255.255). When you have an AP that is receiving a DHCP address but is not associating to the controller you may need to console in to see where the issue lies - not very convenient as the AP is typically mounted or often at another site. You cannot telnet or ssh as this is disabled by default and you cannot enable it until after the AP has associated.

The solution - configure DHCP Option 7. This will cause the APs to transmit their logging messages to your syslog server. You can now grep your logs to try and establish why the AP cannot associate.

EOF... almost

Priming - use it when you need to temporarily setup a small number of APs
DHCP - use it when DHCP Option 43 can easily be configured and when you have multiple sites each with their own controller and a global domain used throughout all of your sites.
DNS - use it when you have a centralised controller, a single site or controllers at each site and a sub-domain for each of the sites. This is my preferred method where possible.

Rogue AP Classification

When I configure a Cisco WLAN controller, one of the configuration steps I normally take is to configure a Rogue Classification Rule that marks as malicious any rogue AP detected with a signal of -70 dBm or greater (I used to configure the rule for -69 but got sick of the giggles when explaining the implementation). This figure can obviously be tweaked - somewhere between -65 dBm and -75 dBm would be appropriate for most environments. So what is the significance of -70 dBm? Well it is the signal strength that I feel is a good middle-ground in locating rogue APs that are probably on the premises whilst eliminating most of the false positives from surrounding homes and businesses. Sure, a small number of false positives will occur depending how close surrounding businesses and homes are but without this rule, locating which rogue APs are actually within the business is like finding a needle in a haystack. Likewise, in most modern WLANs designed for capacity, the vast majority of rogue APs which are in fact on the businesses premises should be detected with of signal of -70 dBm or greater due to the high density of APs.

The configuration required for the -70 dBm rule

Without this rule...
If you do not have Cisco WCS running, you would have to click on every single rogue on the WLAN controller GUI to find the strongest signal strength heard by one of your APs. Unless you have only a handful of rogues, this is unlikely to happen as it is far too time consuming - looking on my current work WLAN controller shows close to 400 rogue APs. Things are a little better if you have WCS as you can see a large number of rogues and their associated signal strengths on a single screen. None the less, having to constantly check this list of rogues should not be required.

With this rule...
Creation of this rogue rule eases this pain. Once the rule is configured, on the WLAN controller GUI you can navigate to MONITOR --> Rogues --> Malicious APs and all rogues detected at -70 dBm or greater will be listed. Some of the rogues I see listed have a signal strength lower than -70 dBm, even from the detecting AP with the strongest strength. I am unsure if this is a bug (running 7.0.116 code) or that this rogue was previously detected with a stronger signal but has not yet been removed from the malicious list. Regardless, the list of rogues is far more manageable.

In WCS, things are even easier. Modify your general tab so that one of the components you see is 'Recent Malicious Rogue AP Alarms' or 'Malicious Rogue APs'. Now every time you log into WCS you will see if you have any potential rogues worth investigating..

Configuring this rule on a recent WLAN deployment caused 5 of the 86 APs to be marked as malicious. One of these was in fact on the premises with the other four false positives. In this case, the figure of -70 dBm could be tweaked to around -65dBm to reduce the number of false positives and due to the high density of APs installed (the WLAN was surveyed for 5 GHz VoWiFi) I would still be confident of detecting any 'on-premises' rogues. Unfortunately malicious is not an accurate term but it cannot be changed on the WLAN controller.

Are these APs really malicious?
Usually not, as I explain below. I classify rogues into one of three categories:

1. Real malicious rogue APs
2. Employee supplied or otherwise 'on-the-premises' APs
3. False positives from nearby businesses or homes

Real malicious rogue APs
The term 'rogue AP' and certainly 'malicious' exist for this category of AP but ironically it would be under rare circumstances that you would actually catch a hacker with the default rules. To have some chance of doing so you would have needed to have configured this -70 rule, actively (on a daily basis) monitor your WLAN controller, WCS or other WNMS, actively track down each malicious rogue so as to eliminate category two and three rogues and finally the hacker would need to have placed this rogue on your premises in an easy to find location or actually still be on the premises. Hello hacker from several kilometres away and hello 24dB parabolic antenna! How many WLAN security breaches have actually taken place via these circumstances? I realise an evil-twin attack could occur but you would have to be very conscientious to capture one by this method.

Your best bet here is to rely on your (hopefully) well-designed and therefore well-secured WLAN alongside your wIDS or wIPS for signs of spoofed deauthentication or disassociation frames. Cisco's Management Frame Protection (MFP) would he helpful here but unfortunately requires CCXv5 clients. Your only hope of enabling it would therefore be on a voice WLAN where you had all Cisco 7925G phones which are CCXv5 compliant. I find most non-Cisco clients these days are CCXv4 and CCXv5 clients are uncommon. The standardised version of MFP is 802.11w and I did see a reference that there may be a Wi-Fi Alliance certification for clients in the works which would be great, if I interpreted correctly.

Employee supplied or otherwise 'on-the-premises' APs
This is the category that we are trying to identify with this -70 dBm rule. This is also the category that is mentioned in every article or book that discusses rogues. I have found that these APs typically fall into one of three categories:

Employee-supplied APs
Occasionally I have found employees to bring in a SOHO Wi-Fi router or AP so that they may access Wi-Fi at work. Jennifer points out that this is increasingly unlikely and I agree to some degree. Western Australia is not the US however. We actually still have an economy and many very highly paid workers where the cost of an AP purchased for work would be considered chump-change. In addition, in many regional areas 3G coverage is poor or non-existent unless your carrier is Telstra where you have traditionally paid through the nose for data. Consequently, I feel you are more likely to see user-supplied rogues here. I have certainly seen them but it is not the most common type within this category in my experience.

Business-supplied APs
I find these to be much more common than employee-supplied APs. Often in a large multi-site enterprise, the local site will use their own funds to purchase a SOHO Wi-Fi solution where a business-supplied WLAN does not exist, regardless of your businesses polices on the matter. In addition, many businesses do not have WLAN policies in place, particularly if they are yet to deploy their own WLAN. These rogues are typically detected upon roll-out of the new WLAN and should be eliminated at this time.

Business within the business
Ahh! The ol' business within the business trick! Many of the rogues I see are from a small business that operates from the main businesses premises. A school book-shop within the school or a GP / dentist within the hospital. These businesses usually operate with some autonomy, often have their own ADSL connection and consequently have a SOHO ADSL Wi-Fi router. This can present a security issue as often this associated business shares the main businesses wired LAN and of course, rarely is the SOHO Wi-FI router going to be sufficiently secure. It can also present performance issues - 802.11b enabled, adjacent channels configured, 40 MHz 2.4 GHz channels configured, etc. Many times the Wi-Fi isn't even used, so getting it disabled is not much of an issue.

False positives from nearby businesses or homes
This is the category that we are trying to eliminate with the -70 dBm rule and these make up the vast majority of detected rogues thanks to the pervasiveness of Wi-Fi. Whilst these can generally be ignored I do find them interesting to look at to get a gauge of 'APs in the area'. What percentage are 2.4 GHz vs. 5 GHz, what are the most common vendors, how much WEP is still in use - you know, all of those things that seemed cool when Netstumbler and War-driving was all the rage back in the early 2000s! I should mention, Western Australia via the WAFreeNet was the first community wireless network to perform WarFlying back in the day... or at least the first to get slashdotted for it!

Whilst it is commonly said that you have no control over these types of rogues I tend to disagree. If a rogue is being a bad neighbour by exhibiting any of the following behavior, it may be worthwhile getting in contact with the businesses IT person and explaining that these issues are negatively affecting both his or her business and your own. It is in their interest to fix the issue. Of course if they don't comply, you could always try gentle persuasion via the AirHORN 2.4 GHz jammer + high-gain antenna - just make sure it has very small side and back-lobes!... I'm kidding of course ;)

The following are reasons to hit up your 'bad neighbour' with some advice:

• 802.11b enabled where client support is not required (you would have to take a stab here and assume they have no 802.11b-only clients)
• Adjacent channels in use at 2.4 GHz (At 2.4 GHz, in most circumstances only channels 1, 6 and 11 should be used)
• Overly high transmit power from APs (this may be difficult to determine however)
• 2.4 GHz 40 MHz channel usage

EOF... almost
Whilst rogue detection rules may not be particularly useful at actually finding real malicious APs, creating this rogue rule can still help you improve the security and performance of your WLAN by identifying employee-supplied, business-supplied, business within the business and bad neighbour rogues.

Eliminating 802.11a clients from your WLAN

Several years ago I was working on a large-scale Cisco WLAN deployment where my number one technical goal was achieving a high performance WLAN design. This was due to the high user density and high bandwidth requirements. In other words, I was trying to wring every last piece of bandwidth out of the WLAN!

One of tweaks I looked at was disabling 802.11a support. Disabling 802.11b support is well understood and an easy way to reclaim a large portion of 'lost' 2.4 GHz bandwidth. Disabling 802.11a support, on Cisco infrastructure at least, is not nearly as easy. Trying to disable all 802.11a data rates does not work on a Cisco WLAN controller as works when disabling 802.11b rate support. Attempting it results in the following:

The error even comes complete with a missing whack of the space bar. Wait... should there be a space between space and bar?!?

Perhaps Cisco will allow the disabling of 802.11a rates when they finally support the 802.11n Greenfield protection mechanism. A few posts on the Cisco forum suggest that a newer chipset is required, though these posts are from around 2009. Fair enough, but there has been plenty of 802.11n APs since the Cisco 1250 series APs but alas... still no support. Certainly, there would be very few environments were Greenfield could be enabled at 2.4 GHz currently due to the massive installed base of legacy 802.11g clients. 5 GHz is a different story however. Due to the remoteness and low population density of much of Western Australia, I can think of many WLANs where it could be enabled with few concerns for the 'bad neighbour' issues that enabling it can apparently present. To be fair, there seem to be few, if any, enterprise vendors that support Greenfield.

So, Greenfield - not an option. I next looked at disabling all 802.11a data rates apart from 54 Mbps. The logic was that I may at least be able to limit 802.11a client associations to only those that could achieve a 54 Mbps data rate. You have to be careful when disabling data rates as it can cause some clients to break... and that was exactly the result. A particular a/b/g client could authenticate (802.11 authentication) and associate but broke somewhere during the 802.1X authentication.

The Final Solution 
My last attempt was to disable 802.11a support on a best effort basis. Unfortunately, low-level client configuration of laptops on an enterprise scale has not yet been worked out by vendors. Disabling 802.11a in Windows can be done from the advanced tab of the WLAN client driver.

This is a an 802.11a/b/g/n client but the same principle applies.

This is a workable solution in a very small WLAN deployment. This particular deployment was designed for roll-out to between 100 to 200 sites covering several 10s of thousands of machines - manually disabling 802.11a support in the driver would not scale.

The solution? I got a list of NIC chipsets from the software deployment / device inventory tool. I culled the non-WLAN chipsets from the list and then pulled out all of the a/b/g chipsets from the list. There were no 802.11a-only chipsets as is to be expected in the vast majority of enterprise environments. Clients with a/b/g chipsets made up around 10% of the total number of WLAN clients - this was a high enough percentage to make this worth the effort. From this list, I pulled out the chipsets that accounted for the majority of a/b/g clients. Around 90% of the clients contained either a particular Atheros a/b/g or one of two Intel a/b/g chipsets.

For each of the three chipsets, I then found the registry key that contained all of the settings that the advanced driver tab modifies. I found a machine with each of these chipsets, RDP'ed into the machine and opened the registry key. I changed the setting under the advanced tab of the driver from the default of a/b/g to b/g, refreshed the registry key and noted the new value. 802.11b support was disabled on the WLAN controller so b/g instead of just 'g' was not an issue.

With these registry values in hand, I had a startup script written to detect the client chipset and if one of these three chipsets were detected, modify the registry value in order to disable 802.11a support. This worked very well and resulted in the vast majority of clients on the 5 GHz band being 802.11n clients. Not a 100% perfect solution but pretty close without needing to spend time disabling the small number of a/b/g chipsets that didn't contain one of these three chipsets.

EOF... almost
This is a solution that is worth the effort where performance is key and you have a detailed insight into the client population. In your average office WLAN where there are far easier wins to be had and performance isn't nearly as critical, this would not be worth the effort. I have suggested this solution to a few clients since but as it requires a fairly solid device inventory tool and a willingness to get the hands dirty, few clients would have implemented it themselves.