High Channel Utilisation: Part One

This is part one in a gripping two part series!

Yesterday I was performing a WLAN audit for a client. As part of this audit, I noticed that APs were reporting a high level of channel utilisation whilst the Tx and Rx utilisation was either very low (<5%) or in most cases 0%. Most of the APs I looked at had 0 clients associated which explains the Tx and Rx utilisation of 0%. The channel utilisation generally ranged from 20% to 60% and in the case of one AP, was 92% with 0 clients associated. In the case of the 92% utilised AP, this means that 92% of the airtime is unavailable before a single client even associates. This directly correlates to how much bandwidth is available for clients once they do associate. If I associated a client to this AP and ran iperf, instead of the ~20 Mbps of real throughput at a 54 Mbps data rate, I would achieve a maximum of a few Mbps. The most common cause of this high utilisation is an excessive number of low data rate frames - almost always 1 Mbps frames. This was suspected but I passed through the motions to determine the cause.

Spectrum Analysis was performed to rule out any Layer 1 (non-Wi-Fi) interference. The spectral density view in Metageek's Channelyzer Pro showed three clearly defined 802.11 channels but nothing else of significance. The integrated spectrum analysis functionality in the Cisco CleanAir APs was also checked and showed the usual low duty cycle interferers (Bluetooth and DECT phones) but nothing to accountant for such high channel utilisation.

Next, layer 2 analysis was performed by way of packet sniffing. It was immediately obvious that the excessive 1 Mbps frame hypothesis was correct. These frames accounted for between 70% and 80% of all frames. There are a number of factors that can contribute to an excessive number of 1 Mbps frames:

1 Mbps frames allowed on the WLAN
Obviously enabling low data rates such as 1 Mbps on the WLAN will account for some of the frames seen. A frame transmitted at 1 Mbps takes significantly longer than even the slowest 802.11g frame (6 Mbps) and therefore many 1 Mbps frames can take up a significant portion of available airtime. Common management frames such as beacons, probe requests and probe responses are sent at the lowest configured manatory data rate (1 Mbps in this case). These frames are sent even with no clients associated to the AP. In addition because 1 Mbps is an 802.11b data rate, a protection mechanism such as RTS/CTS or CTS-to-Self is needed to allow 802.11b clients and APs to interoperate with 802.11g clients and APs. RTS/CTS results in around a 30% to 40% reduction in throughput and this does not include the reduction in throughput caused by the actual 1 Mbps data frames. RTS and CTS frames are usually also transmitted at the lowest mandatory data rate.

All 802.11b data rates were enabled on this clients’ WLAN, which includes the 1 Mbps data rate. This accounts for some of the 1 Mbps frames seen but it was not the largest contributing factor.

Excessive co-channel interference (CCI) caused by a high density of APs within the WLAN
The higher the density of APs within the WLAN, the more beacons, probes and control frames (RTS and CTS) will be transmitted within the environment. As detailed above, these frames are typically transmitted at the lowest mandatory data rate.

The AP density of this clients’ WLAN was low and although CCI at 2.4 GHz is usually present to some degree in most WLANs, it wasn't a major contributing factor in this case due to the low AP density.
     
Number of SSIDs broadcast by the WLAN
The more SSIDs supported by the WLAN, the more management frames will be transmitted (beacons, probes, etc) to support these additional SSIDs. A maximum of four to six SSIDs per WLAN is typically recommended. 1 Mbps data rate support exacerbates this problem significantly. Hiding the SSID does not help.

This was not a contributing factor as the client is currently only supporting a single SSID.

Rogue APs within the businesses premises
SOHO APs are often brought in by employees or purchased by departments within the organisation without the knowledge of the IT department. Employee's using smart-phone tethering is becoming increasingly common also. Essentially this turns the smart-phone into a Wi-Fi hot-spot. These APs almost always support the 1 Mbps data rate.

This was a small contributing factor in the case of this client.

Rogue APs outside the businesses premises 
Essentially a neighbouring businesses’ poorly designed or maintained WLAN can affect your WLAN. In this context, the definition of poorly designed or maintained is support for 802.11b / 1 Mbps data rates. Very few WLANs in late 2011 should support 802.11b and 1 Mbps data rates but unfortunately many still do. Within a WLAN context, these businesses are often referred to as bad neighbours.

This was the major cause of high channel utilisation in the case of this client.Further details of this can be found in part two detailed at the end of this post.
   
Solutions
There are a number of changes that can be made to help mitigate the low data rate problem.

1 Mbps frames allowed on the WLAN
Where possible, disabled 802.11b entirely which will drop support for 1 Mbps, 2 Mbps, 5.5 Mbps and 11 Mbps data rates. However there may be cases where you have to continue supporting 802.11b. I can see two such scenarios.

1. You have a significant 802.11b-only client-base. For example, if 10 of your 1000 laptops are 802.11b-only, drop support and if they really require WLAN access, provide them with a PCMCIA or USB 802.11a/g/n WLAN client. If however, 300 of your 1000 laptops are 802.11b-only, you will likely need to maintain support. This is particularly uncommon as of late 2011 or late 2008 for that matter!
2. You have some very expensive 802.11b clients. For example, you have 10 pieces of 802.11b-only medical equipment but each piece of kit costs six-to-seven figures.

If 802.11b cannot be disabled entirely, consider disabling 1 Mbps, 2 Mbps and 5.5 Mbps data rates so that you are left with only 11 Mbps in addition to your 802.11g data rates.
  
Excessive co-channel interference (CCI) caused by a high density of APs within the WLAN
One option is to re-survey and reposition your APs to lower the density although this is unlikely to be feasible in most cases. A better option is to consider disabling the 2.4 GHz radio on dual-band APs or entirely removing single-band APs.

Number of SSIDs broadcast by the WLAN
Consolidate your SSIDs to ensure a maximum of four to six. Dynamic VLAN allocation can help with this.

Rogue APs within the businesses premises
Locate and either remove or disable rogue APs. I have previously posted about how to narrow down which Rogue's are within the businesses premises, when running Cisco kit.

Rogue APs outside the businesses premises
It is typically stated that you have little control over bad neighbours however this discounts the usefulness of social engineering. If the neighbouring businesses WLAN is compliant with the Australian Radiocommunications (Low Interference Potential Devices) Class License 2000 or your countries equivalent then your best option is a little social engineering. You may have some luck approaching either the relevant technical contact responsible for the WLAN or otherwise you might need to speak with the big cheese. Explaining that 802.11b support is hurting not only your but also their WLAN is recommended. They may have a legitimate use for 1 Mbps data rates but this is rare as of late 2011. This approach is useful when you only have one or two bad neighbours. My client however has a large building in the middle of the Perth CBD. The layer 2 analysis revealed a large number of 802.11b-supporting bad neighbours and approaching all of them is unlikely to be feasible. The worst of these bad neighbours is actually a well-known municipal Wi-Fi implementation in the Perth CDB which will be detailed in part two.
 
EOF... almost
There are many causes of high channel utilisation and most have relatively easy solutions. The one over which you have minimal control is that of rogue APs outside the businesses premises. One particular bad neighbour is having a severe impact on my clients WLAN and this will be detailed in the gripping conclusion to this two part series!

Cisco WCS HA and WCS Navigator HA

A few gotcha's I have noticed regarding WCS High Availability and WCS Navigator High Availability, in no particular order.

WCS

• During installation, you specify that the instance of WCS you are installing is the primary.

• Likewise, you specify the secondary WCS instance during installation on the second box / VM or blade.

• You cannot browse to the secondary WCS web interface until you have failed-over to it. In other words, during normal operation, you cannot browse to the secondary instance.

• You can however browse to the WCS Health Monitor on the default port of 8082 on the secondary WCS, even when the primary WCS is active (non-failover state) - e.g. https://wcs02.deadbeef.com.au:8082/

• The WCS v7 configuration guide states that you must purchase a HA license. This is not required if you are using the Enterprise version of WCS. If you buy WCS with a 1000 AP (or greater) license, you are using the Enterprise license and so automatically get the HA capability.

• Consequently, only a single WCS Enterprise license is required for HA - you do not have to split your license in two (e.g., a 1000 x AP license can remain that way and does not need to be split into 2 x 500 AP licenses).

• In Windows, you cannot test failover by using the GUI tool to shutdown the primary instance. You either need to use the CLI command "nmsadmin.bat -switchover stop" or simulate it through more realistic means (shutdown the primary WCS, yank the cable or shutdown the switchport, etc). Personally I prefer a closer-to-the-real-world test by shutting down the server or switchport but the CLI command method is useful if these methods are not practical. More details at http://www.cisco.com/en/US/docs/wireless/wcs/7.0/configuration/guide/7_0main.html#wp1072955.

• One secondary WCS instance can provide HA for multiple primary WCS instances. This point does not seem to be particularly well-known. The box / VM or blade providing your secondary HA instance may need to be beefier than the primary WCS boxes if you want to allow for multiple primary failover's. That is, two primary's requiring 4 GB of RAM each may need 8 GB in the secondary in the case of both primary's failing.

• You must either manually point (your browser) to the secondary in a failover scenario, employ some DNS redirection tricks or similar redirection bizness.

• When you first log in to the WCS Health Monitor, the authentication key requested as your login credential is not the key you have specified during HA configuration but in fact the root password for WCS. Later you can use the key to login.

• Do not have the WCS directory or WCS log files open when testing failover. This will prevent failover from... failing over, presumably due to database file access issues. This is stated in the doco but is easy to forget when trouble-shooting. Ironically, I had log files open to troubleshoot a separate failover issue, which caused this (second) failover issue. Doh!

WCS Navigator

• WCS Navigator is not available in a HA configuration. I don't see this as a major issue as it is really just a WNMS of WNMS's. Your WLCs and WCS instances will not be affected if Navigator goes down.

• In the case of a WCS failover from primary to secondary, the WCS v7 configuration guide states that Any Navigators in the network start monitoring the secondary WCS - this is not the case. WCS Navigator does not support WCS HA. In a failover scenario, the primary WCS is shown as down and WCS Navigator has no knowledge of the secondary WCS. If you add the secondary WCS to Navigator, it shows as a completely separate WCS instance and therefore it appears that you have double the number of WLCs and APs. Apparently a feature request is in.

Cisco 3602i and Cisco 3602e APs

Edit (6th Nov 2011): Cisco has officially announced the 3600 series APs - the data sheet is pretty much identical to that detailed below. In addition a few more details of ClientLink 2.0 are available as is a white paper discussing the 3600 series.

This PDF shows that the antennas to be used on the 3602e are dual-band as speculated below. Ironically the gain on the external antennas for the 3602e are slightly lower than the internals at 5 GHz and equal at 2.4 GHz - still you might require the 3602e for ruggedised deployments or of course, other antennas.

Edit (17th Nov 2011):  Cheers to Marcus Burton for correcting me regarding dual-band antennas.That is, no antenna magic is needed for an antenna to utilise two frequencies simultaneously..

For the last few months there has been murmurs in the WiFi-Nerd... err, underground... yer, let's go with that, regarding the pending release of Cisco's new 3600 series access points. These APs have not been officially announced on the Cisco website as yet but 900-odd Google results for the model numbers, mostly from retailers, must mean they are no longer under NDA. In fact, the 3600 series was listed on Cisco's website however they have pulled the details but you know... this is the Internet's and ahh, Google cache exists. A few months ago, waking up in the middle of the night, thinking about wireless as I often do, I speculated that the soon to be released WLC code 7.2 would likely coincide with the 3600 series being released. Well 7.2 has not been released but the 3600 series data sheet lists it as required. Instead we have the recently released WLC code 7.0.220.0. There was no mention in the release notes of 3600 series support however. Google cache again to the rescue; this time for a "no longer listed on cisco.com" 7.1.91.0 WLC code release. The release notes mention that this release is a restricted limited lifetime release specifically for the 3600 series. The rest of the details are uneventful and in-line with the 7.0.220.0 more or less. Finally, a snazzy photo exists on this Ingram Micro hosted PDF data sheet.

The Cisco 3602e. Now featuring less antennas!

Initially, I had heard that this AP would support four MIMO streams. I thought this was logical considering Cisco is late to the game with their 'better-than-two-stream' offering. Sure, you're almost 12 months behind the earliest three-stream enterprise offerings, but you'll jump straight to four MIMO streams... kind of like Gillette's five-blade razor. However, since hearing this news I began to doubt that this AP would support four streams. Specifically, speculation from Cisco and Aerohive that enterprise vendors may skip four streams and jump straight to 802.11ac, though this really only makes sense if 802.11ac arrives on-time which is unlikely, given the history. 802.11ac delays aside, another issue with a four stream AP arriving now is the lack of client support; specifically there are none. Lastly, the Wi-Fi Alliance does not have a certification program to test four stream APs or clients.

So, a Cisco four stream in 2011 seemed unlikely... and the data sheet indicates exactly that. The 3600 series supports three streams - 4x4:3 but three streams none-the-less. An "in-no particular-order", "assuming the data sheet is accurate" list of my 3600 musings:

4x4:3: I imagine the reason I had been told four-streams was because of some confusion originating from the 4x4 nature of the AP. 4x4:3 means that the AP has four transmit antennas, four receive antennas but only supports three spatial streams. Keeping in mind, this is per band. More correctly, radio chains instead of antennas but antennas are easier to picture so we'll stick with that! So if you are only transmitting or receiving a maximum of three streams concurrently, what are the fourth antennas for? The extra antennas are used in 802.11n MIMO devices for similar reasons to those used in 802.11a/g devices - diversity and in the case of 802.11n, some particularly advanced diversity techniques. Space Time Block Coding (STBC), Cyclic Shift Diversity (CSD), Transmit Beam-forming (TxBF), Maximal Ratio Combining (MRC) are all 802.11n techniques which help to increase the signal strength or reliability available to the client or AP which can result in increased link reliability and throughput. Both the 3500 and 3600 series of APs support CSD, TxBF and MRC however the 3600 series now supports TxBF for a larger range of clients, detailed below. So 4x4:3 gives us three streams (450 Mbps maximum data rate) but with the benefit of 4x4. For reference, the 3500 series does 2x3:2. 

Three spatial streams: Three streams with 40 MHz channels and a 400ns Guard Interval tops out at 450 Mbps. One of the major advantages of three streams is the ability to move back to 20 MHz channels at 5 GHz whilst still achieving a maximum data rate of 216 Mbps. Not quite the 300 Mbps afforded by using 40 MHz channels but much better than the 144 Mbps from 20 MHz channels. These extra channels are particularly important in Australia where we currently only have access to six 40 MHz channels and potentially four, if radar starts causing you significant grief. Whilst there aren't many three-stream clients available, your APs are going to be in place for several years at which point there will hopefully be a significant portion of three-stream enabled laptops - I doubt we'll ever see a three or four stream smart-phone on the other hand. 450 meg clients, where art thou!

ClinkLink 2.0: Cisco's beefed up transmit beam-forming (TxBF) offering. For reference, the 3500 series (amongst others) offers TxBF in the form of ClinkLink (1.0). A few weeks back, I was pondering why Cisco only offered ClientLink to legacy (802.11a/g) clients. I thought that 802.11n single-stream devices (smart-phones, tablets, etc) could benefit greatly from beam-forming - these are precisely the sorts of devices that could see a benefit from a few extra dB due to their generally weak receive sensitivity. To support two or three stream clients though you need an extra antenna and the 3500 series only does 2x3:3 (two transmit antennas) and therefore does not have a spare transmit antenna free when servicing a two stream client. This is where the 4x4:3 comes in handy - that fourth transmit antenna can be used to provide beam-forming to one, two and three-stream 802.11n clients. So now we have TxBF for legacy, one, two and three stream 802.11n clients. Hopefully ClientLink 2.0 implements some antenna beam-forming action over the ClientLink 1.0 on-chip beam-forming where marginal gains were, at least originally, achieved. Count-down to a Ruckus blog post dissing this TxBF implementation as half-assed ;). To be fair, Ruckus are the bomb when it comes to antenna magic!

200 mW (23 dBm) of transmit power: The 3500 series only offered 200 mW for 802.11b data rates which should be disabled in almost every new deployment anyway. As it is best practice to match the APs maximum transmit power to that of the poorest client, I can't see where this extra power would come in handy. If you were using the 3602e for a bridge link perhaps but a higher gain antenna would be a better bet. I can't help but think that this power may be spread between the antennas when performing one of the supported types of transmit diversity though... hmm.

Antenna gain: Down from 4 dbi on the 3500 series to 2 dbi on the 3600 series at 2.4 GHz. Up from 3 dbi on the 3500 series to 5 dbi on the 3600 series. With only four external antennas on the 3602e my guess is that they are dual-band antennas. Whilst this works fine on a client as it is only associated to one band at a time, pretty much all decent APs must be able to use both bands concurrently. For an AP to transmit or receive on both bands simultaneously I imagine some sort of smart antenna technology is being employed - pure speculation on my part and all based on the photo! We will see.


PoE: Powered by 802.3af like all other two and three stream enterprise APs (the EOL Cisco 1250 not with-standing). Surely it is in the various Cisco switching business units interests to require 802.3at for the AP ;). Of course, customers would not be happy knowing that all other three stream APs run off a mere 12.95W!. At this rate we will have to wait for 802.11ac to see some 30W AP lovin!

Band support: The 3600 series appears to only be available in a dual-band version. About time really, as I find the single-band APs to be a waste of silicon!

Weight: Light! Like the 3500 series, due to the CleanAir capabilities, this AP is light-weight only. Now the question is, which autonomous AP IOS can be used to survey with the 3600 series!

The LED: It's bigger - surely its most impressive feature, by far! ;) 

EOF... almost
Like most vendors with a large market share, Cisco receives their fair share of flack regarding a lack of innovation, resting on ones laurels and so on. As the first enterprise vendor to offer 4x4:3, up to three stream beam-forming and the 'not-really-that-long-ago' CleanAir (AP spectrum analayser integration) I think at least the Cisco Wireless Business Unit still has the ability knock out an innovative product. Now I wait for the official release to get my hands on one and do some testing!

Since the first three-stream enterprise AP was released I kept seeing Cisco down-play the importance of three-streams. I assume that downplay will end right about ... ;)