Disabling shaping in one direction with CeroWRT

Another test Dave Taht asked for was to disable the downstream shaper and see what happened.  It wasn't pretty.  So I also disabled the upstream shaper.  Also not pretty.  Then I went back to using the shaper enabled in both directions.

Baseline using fq_codel and 54.5Mbps downstream limit and 10Mbps upstream limit:

Now, turning off the downstream shaper yields:

This indicates (to me), that there's very excessive buffering at the CMTS, and the downstream shaper is badly needed to keep it under control.

So I re-enabled downstream, and disabled the upstream shaper:

Again, not ideal.  It's interesting seeing how the shapers interact with the traffic in the other direction as well.  Because their ACKs are going through the other path, disabling shaping on one clearly effects the other.

I don't like this PIE (more bufferbloat testing)

One of the other algorithms for dealing with bufferbloat is PIE.  This is Cisco's answer to the problem, and the one standardized on by Cablelabs for inclusion in DOCSIS 3.1.  While it's an improvement over the big buffers and a FIFO queue that exists now, I don't think it compares to what fq_codel can do.

Here's the "before", my raw, unmanaged, Comcast "Blast" 50 down, 10Mbps up connection:

Lots of bandwidth, and when loaded, lots of latency.  Baseline latency to the test server is ~14ms.  Bufferbloat is adding over 250ms of latency.  This is why your connection is crap during big downloads.

Now, PIE does make this better.  LOTS better:

From 250ms down to 60-80ms..  1/4 the latency.  Not bad, but not great, either.  It's still 3-4x the latency that it could have:

fq_codel simply erases all the latency on a connection like this.  And this was over a 4 minute sustained test.  Able to push over 50Mbps downstream, and over 10Mbps upstream as the numbers in the middle graph are achieved data rates, not including the TCP ACK packets or other overhead.

So, yeah, PIE.

It's ok.  But I like fq_codel a LOT better.

Fixing Bufferbloat on Comcast's "Blast" 50/10Mbps Service

At our new apartment here in the Bay Area, I ended up going with Comcast after realizing that it was about my only option for high speeds.

But I knew it was going to need to love to make it work.  It turned out to take very little effort to get pretty fantastic results.

First, latency under full load (upstream and downstream), to a server that's very close:

After a bit, it settles into 350ms of buffer-induced latency.  OTOH, this is pretty screaming fast on downlink (58Mbps!), but the uplink is suspicious looking.  It gets the advertised 10Mbps for about 10 seconds, and then falls apart.

I put in my WNDR3800 running CeroWRT (a bleeding-edge OpenWRT variant:  http://www.bufferbloat.net/projects/cerowrt).  Then I set the qos parameters based on the above measurements:

ingress rate limit:  54.5Mbps
egress rate limit:  10Mbps

using simple.qos, and with ecn enabled (both directions).

Results:

I was amazed.  Ruler-flat throughput, at the advertised rates, good sharing between the various buckets (first time I've had enough bandwidth to see that actually work), and about 5ms of buffer-induced latency.  Unfortunately, the WNDR3800 is at it's CPU limit (load factor of 3-4).  OTOH, I don't push full bandwidth all that often (on the downstream side, I do on the upstream when I'm uploading photos).

This is what the Arris modem/router should have done out of the box.  Hopefully PIE will help with that in DOCSIS 3.1.  Maybe.  We'll see.

Measured bufferbloat on Orange.fr DSL (Villefranche-sur-Mer)

Yes, I'm doing bufferbloat experiments on a beach vacation.  The rest of the family is napping after time at the beach, and the light isn't good enough yet for photography, so I might as well.

Summary:

- lots of bloat (60-100ms or more of additional latency when loaded)
- but traffic classification actually works

I've setup a new test in netperf wrapper that's a bit gentler, specifically for low-bandwidth DSL setups (like I've had for the last year).  Instead of the 4 streams up, down, and 4 UDP streams with ICMP, it's just 2 in each direction (dropping CS5 and EF streams from TCP, and BK and BE from UDP).

In the 2-stream test (rrul_lowbw), the EF UDP shows a clear front-of-the-queue advantage over ICMP, which was surprising to see.

Then the full 4 stream rrul test really, really added latency:

But, interestingly, the EF-marked UDP packets were clearly getting some sort of priority treatment.

==========

Qualitative thoughts.  It's bursty/chunky.  Throughput isn't bad, when it gets around to it, but it likes to stall and then burp back a lot of data.  When it's not stalled out, it feels pretty fast, but then everything halts for a few seconds, and then runs again.

I don't think that the DSL modem/router's local DNS server is caching .  I'm seeing 50ms lookups for addresses (the second time and follow-on times), and 100+ms for the first.  Google DNS (8.8.8.8) is ~40ms away, which is why I think that the local DNS server isn't caching, just forwarding.

===========

I dug my WNDR3800 running CeroWRT out of my luggage (long story), and set it up behind the Livebox from Orange.  I tuned it for 12000 down and 1000 up, and got the following results with 2 streams and with 4:

Latency is far, far better.  And the TCP streams seem to be smoother.  It feels a bit snappier, and no slower, even though it's testing out at about 10% slower than when going direct to the Livebox.

===========

Interestingly, while this is distinctly slower than the Free.fr service I had in Paris, this feels subjectively faster.  Google servers are about the same distance away (40ms), but it's much snappier, and uploads in particular are performing better, even though they're testing worse...  When loaded, the DSL upload really doesn't handle multitasking very well.

Also, this is the first time I've actually seen the diffserv classes make a difference, and a clear difference at that.  I'm not sure why I wasn't seeing this with Free.fr, but here with the Orange Livebox, it's clearly working.

Possibly, as I'm only here for a week, I didn't care about maximizing throughput, and I'm imposing a larger bottleneck than I was with Free.fr?

===========

Next test (in a week or two):  Comcast 50Mbps cable.