NetOptimizer

Announcing: The IPTV-Analyzer

2011-05-11T16:47:00.004+02:00

I'm happy to announce the first official release of the IPTV-Analyzer project, as an Open Source project.

Git tree: https://github.com/netoptimizer/IPTV-Analyzer

The IPTV-Analyzer is a continuous/real-time tool for analyzing the contents of MPEG2 Transport Stream (TS) packets, which is commonly used for IPTV multicast signals. The main purpose is continuous quality measurement, with a focus on detecting MPEG2 TS/CC packet drops.

The core component is an iptables (Linux) kernel module, named "mpeg2ts". This kernel module performs the real-time Deep Packet Inspection of the MPEG2-TS packets. Its highly performance optimized, written for parallel processing across CPU cores (via RCU locking) and hash tables are used for handling large number of streams. Statistics are exported via the proc filesystem (scalability is achieved via use of the seq_file proc API). It scales to hundreds of IPTV channels, even on small ATOM based CPUs.

Please send bugreports, patches, improvement, comments or insults to: hawk@comx.dk

Bufferbloat: Wireless is worse than expected

2011-01-10T15:11:00.002+01:00

Jim Gettys also pointed out, that bufferbloat also exists on Wifi wireless connections[1].

I didn't take his word for it, but tested it my self. And the result was far worse than I expected! In optimal conditions sitting next to the Wifi AP, I can easily introduce a 600 ms delay (0.6 sec), and moving further away I quickly see latency approaching 1 sec. See Gettys tests[1] for more info.

This bufferbloat issue is going to get worse, as we get more bandwidth on our broadband connections. This means that we have not see problem at its full extend yet.

Thus, lets fix it before it gets out of hand!

An interesting property of Wifi bufferbloat is, that the queue/bloat happens on you own machine (when uploading).

Linux is to blame, big time!

An some point we/the Linux kernel developers, increased the default transmit queue length (txqueuelen), from 100 packets to 1000 packets (happend when the netcards went from 100Mbit/s to 1000Mbit/s).

The major problem here is, that wireless devices also inherited this default txqueuelen setting of 1000 packets.

This is fortunately easily fixed/adjusted via e.g the command:

ifconfig wlan0 txqueuelen 10

But unfortunately, this does not remove all the TX bufferbloat in the system. The Wifi driver and hardware also have significant TX buffering, as Getty also describes[2].

Setting the txqueuelen to 1, I still experienced an average delay of 96ms and max 116ms, when linking 24 Mbit/s.

24 Mbit/s * 116 ms = 348000 bytes
= 232 packets (with 1500 bytes packets)

Thus, a hardware bufferbloat on minimum 232 packets. Gettys report his wifi hardware has 255 packets in hardware.

Fixing the hardware Wifi bufferbloat, is harder at we need to fix each individual driver and most of them not exported the functionality to tools like ethtool).

We need to at least mitigate the wifi bufferbloat, to some sane level.

Next we can talk about choosing a queueing stategy AQM for wifi networks.

Links:
[1] http://gettys.wordpress.com/2010/12/02/home-router-puzzle-piece-two-fun-with-wireless/
[2] http://gettys.wordpress.com/2011/01/03/aggregate-bufferbloat-802-11-and-3g-networks/

Buffer Bloat: The calculations

2010-12-30T14:40:00.001+01:00

The buffer-bloat blog posts by Jim Gettys [1][2], are very interesting, and relevant for everybody with a broadband connection (especially asymmetric links). They are well written, but also too long and too many posts.

In this blog post, I'll explain what is going on, by showing how you can calculate your own latency issues.

The issue raised by Gettys, is that buffer-bloat (too big buffers on the network path) has fundamentally broken Internet broadband connections [2].
As buffer-bloat can introduce enough latency to cripple your line. Basically killing the possibility of interactive and realtime services, being delivered (e.g. by companies) over your broadband connection.

I'm very happy to see that Gettys is bring this issue up again.

Back in 2005, I discovered the same issues as Gettys. I wrote my masters thesis[3] about the issue, and even created an Open Source "mitigation" solution the ADSL-optimizer[4]. It seems my solution has not gained a wider use.

The major contribution for my side, is to take the ADSL overhead into account when doing QoS packet shaping. (Everything is in mainline, just use/add the TC options "linklayer adsl" and "overhead", if you already have a Linux box doing QoS on your line.)

The issue is that:

A single TCP upload cause a delay of 1.2 seconds (on a 512 Kbit/s ADSL line)

(see thesis[3] page 21).

Lets calculate what is happening, without going into details of why TCP/IP miss-behaves and cause queues to build (details are in my thesis[3]).

Before starting the calculations, here is a beautiful cite by Jim Gettys[1]:

"Large network buffers can be thought of as 'dark buffers', analogous to 'dark matter' in the universe; they are undetectable under many/most circumstances, and you can detect them only by indirect means. Buffers do not cause problems when they are empty. But when they fill they introduce additional latency (and create other problems, possibly very severe) to other traffic sharing the link."

Given the line speed and the delay, we can calculate the buffer size
(this is the bandwidth-delay product). Due to ADSL overhead the
effective bandwidth is actually 454 Kbit/s of the 512 Kbit/s line,
and the measured delay was 1138 ms.

454 Kbit/s * 1138 ms = 64581 bytes

This, corresponds to the TCP window-size. Thus, this is not the maximum buffer-size of the modem. (Use several TCP connection or UDP to find your maximum ping RTT, and calc your buffer size).

Where does the delay come from?!

The delay consists of different components, the important one in our case is the transmission delay (combined with the packets in queue).

The transmission delay of a 1500 bytes (MTU) packet is:

1500 bytes / 454 Kbit/s = 26.34 ms

Thus, the experienced delay is the time it takes to empty the packets in the queue, which is greatly dependend on the line speed.

E.g. 64000 bytes / 454 Kbit/s = 1127 ms.

With a RTT delay of 150 ms, your interactive SSH connection will feel sluggish.
(A side note on ADSL is that; the processing delay in the ADSL modem can get as large as 60 ms, and is caused by the interleaving depth, but its "fortunately" a constant fixed delay on the path)

Increasing the bandwidth, will reduce the latency, but its not the
solution. Besides, ADSL technology is often limited to a 1024 Kbit/s
upstream link.

The Point:

"ISPs SHOULD configure the buffer size based upon the link bandwidth"

I have a feeling that the ISP just configure a default queue size, and tune the queue size based upon max throughput on their largest product.

The line I did my measurements on, I could see delay on 3.3 sec, thus a buffer-bloat size of 187332 bytes (454 Kbit/s * 3300 ms), or 125 packets at 1500 bytes (MTU). Simply crazy!

For more details on formulas and calculation see thesis page 19 to 27.

Links:
[1] Jim Gettys: introducing-the-criminal-mastermind-bufferbloat
[2] Jim Gettys: whose-house-is-of-glasse-must-not-throw-stones-at-another
[3] http://www.adsl-optimizer.dk/thesis/
[4] http://www.adsl-optimizer.dk/
[5] http://en.wikipedia.org/wiki/Jim_Gettys

tcpdump vs VLAN tags

2010-09-29T22:00:00.000+02:00

In this post I will explain why tcpdump sometimes doesn't show the VLAN (802.1q) tags, when dumping network traffic.

The issue is related to libpcap and VLAN hardware acceleration in the Linux kernel.

Patrick McHardy, fixed the kernel in version 2.6.27, and also fixed libpcap.
His patches for libpcap has the subject:

"libpcap: VLAN acceleration support".

Which got accepted, back in Aug 2008.

So whats the problem?

The problem is that tcpdumping VLAN IDs works under Ubuntu, but NOT under Debian.

I'm running a new kernel 2.6.35 on the Debian machine (which has the support >2.6.27). So I downloaded the libpcap and tcpdump source and compiled it manually, BUT it still didn't work!?! (even with the latest libpcap 1.1.1).

This was strange!

Digging into the cause of the issue, I found that is related to an old include file on the Debian system.

/usr/include/linux/if_packet.h

This old header file, cause the libpcap configure command to disable the libpcap VLAN feature. When running configure for libpcap, look for:

"checking if tpacket_auxdata struct has tp_vlan_tci member... no"

Remember this issue related to all programs using libpcap, which also includes Wireshark.

I hope confused VLAN dump users, will find this blogpost...

Update

The binary tcpdump on Debian Squeeze actually works with hardware accelerated VLAN, although you have specify option '-e' to make it show the VLAN tags.

BUT here it gets really strange...

It will NOT work if you compile libpcap and tcpdump you self, on Debian Squeeze (GCC 4.4.5).

The header file if_packet.h contains the correct structs, but libpcap's configure command will refuse to enable the VLAN code, as the check (of if_packet.h) fails.

The problem this time is related to the compiler version.

GCC 4.4 does not indirectly include the type of u_int, which the configure code choose to use for this test. Thus, the if_packet check fails due to the error: ‘u_int’ undeclared, which it interpreted as no VLAN support. Thus, the reason the binary works on Squeeze is that the maintainer has compiled it with an older version of GCC. Argh!

When I find the time, I'll post some patches to the maintainer of libpcap.