I don't like this PIE (more bufferbloat testing)

One of the other algorithms for dealing with bufferbloat is PIE.  This is Cisco's answer to the problem, and the one standardized on by Cablelabs for inclusion in DOCSIS 3.1.  While it's an improvement over the big buffers and a FIFO queue that exists now, I don't think it compares to what fq_codel can do.

Here's the "before", my raw, unmanaged, Comcast "Blast" 50 down, 10Mbps up connection:

Lots of bandwidth, and when loaded, lots of latency.  Baseline latency to the test server is ~14ms.  Bufferbloat is adding over 250ms of latency.  This is why your connection is crap during big downloads.

Now, PIE does make this better.  LOTS better:

From 250ms down to 60-80ms..  1/4 the latency.  Not bad, but not great, either.  It's still 3-4x the latency that it could have:

fq_codel simply erases all the latency on a connection like this.  And this was over a 4 minute sustained test.  Able to push over 50Mbps downstream, and over 10Mbps upstream as the numbers in the middle graph are achieved data rates, not including the TCP ACK packets or other overhead.

So, yeah, PIE.

It's ok.  But I like fq_codel a LOT better.

Fixing Bufferbloat on Comcast's "Blast" 50/10Mbps Service

At our new apartment here in the Bay Area, I ended up going with Comcast after realizing that it was about my only option for high speeds.

But I knew it was going to need to love to make it work.  It turned out to take very little effort to get pretty fantastic results.

First, latency under full load (upstream and downstream), to a server that's very close:

After a bit, it settles into 350ms of buffer-induced latency.  OTOH, this is pretty screaming fast on downlink (58Mbps!), but the uplink is suspicious looking.  It gets the advertised 10Mbps for about 10 seconds, and then falls apart.

I put in my WNDR3800 running CeroWRT (a bleeding-edge OpenWRT variant:  http://www.bufferbloat.net/projects/cerowrt).  Then I set the qos parameters based on the above measurements:

ingress rate limit:  54.5Mbps
egress rate limit:  10Mbps

using simple.qos, and with ecn enabled (both directions).

Results:

I was amazed.  Ruler-flat throughput, at the advertised rates, good sharing between the various buckets (first time I've had enough bandwidth to see that actually work), and about 5ms of buffer-induced latency.  Unfortunately, the WNDR3800 is at it's CPU limit (load factor of 3-4).  OTOH, I don't push full bandwidth all that often (on the downstream side, I do on the upstream when I'm uploading photos).

This is what the Arris modem/router should have done out of the box.  Hopefully PIE will help with that in DOCSIS 3.1.  Maybe.  We'll see.

Measured bufferbloat on Orange.fr DSL (Villefranche-sur-Mer)

Yes, I'm doing bufferbloat experiments on a beach vacation.  The rest of the family is napping after time at the beach, and the light isn't good enough yet for photography, so I might as well.

Summary:

- lots of bloat (60-100ms or more of additional latency when loaded)
- but traffic classification actually works

I've setup a new test in netperf wrapper that's a bit gentler, specifically for low-bandwidth DSL setups (like I've had for the last year).  Instead of the 4 streams up, down, and 4 UDP streams with ICMP, it's just 2 in each direction (dropping CS5 and EF streams from TCP, and BK and BE from UDP).

In the 2-stream test (rrul_lowbw), the EF UDP shows a clear front-of-the-queue advantage over ICMP, which was surprising to see.

Then the full 4 stream rrul test really, really added latency:

But, interestingly, the EF-marked UDP packets were clearly getting some sort of priority treatment.

==========

Qualitative thoughts.  It's bursty/chunky.  Throughput isn't bad, when it gets around to it, but it likes to stall and then burp back a lot of data.  When it's not stalled out, it feels pretty fast, but then everything halts for a few seconds, and then runs again.

I don't think that the DSL modem/router's local DNS server is caching .  I'm seeing 50ms lookups for addresses (the second time and follow-on times), and 100+ms for the first.  Google DNS (8.8.8.8) is ~40ms away, which is why I think that the local DNS server isn't caching, just forwarding.

===========

I dug my WNDR3800 running CeroWRT out of my luggage (long story), and set it up behind the Livebox from Orange.  I tuned it for 12000 down and 1000 up, and got the following results with 2 streams and with 4:

Latency is far, far better.  And the TCP streams seem to be smoother.  It feels a bit snappier, and no slower, even though it's testing out at about 10% slower than when going direct to the Livebox.

===========

Interestingly, while this is distinctly slower than the Free.fr service I had in Paris, this feels subjectively faster.  Google servers are about the same distance away (40ms), but it's much snappier, and uploads in particular are performing better, even though they're testing worse...  When loaded, the DSL upload really doesn't handle multitasking very well.

Also, this is the first time I've actually seen the diffserv classes make a difference, and a clear difference at that.  I'm not sure why I wasn't seeing this with Free.fr, but here with the Orange Livebox, it's clearly working.

Possibly, as I'm only here for a week, I didn't care about maximizing throughput, and I'm imposing a larger bottleneck than I was with Free.fr?

===========

Next test (in a week or two):  Comcast 50Mbps cable.

CeroWRT 3.10.32-12 - SQM comparison on Free.fr

Earlier this week I updated to CeroWRT 3.10.32-12 on my WNDR3800.  This was to pick up the many changes since 3.10.24-8, primarily DNSSEC.  SQM also seems to be behaving better.

Setup:
ADSL2+ from Free.fr (75002 Paris, France)
v6 Freebox Server
Netgear WNDR3800 running CeroWRT v3.10.32-12
17" Macbook Pro (OSX 10.8) (wired directly to either wndr or freebox depending on the test)

fq_codel results

I'm pretty happy with these results.  I'd like induced latency to be less, but 10-15ms isn't bad on DSL (I guess).  What I'm happiest about is that the traffic seems to be sharing pretty fairly.  However I'm not seeing the distinct tiering of bandwidth that I see on the results from higher-bandwidth setups (where the BK traffic is distinctly less than the average, with BE and EF above average.

SQM settings are as follows:

- 21000 down / 1100 up
- fq_codel (ingress ECN, no egress ECN)  -- defaults
- link-layer adaptation set for ADSL at 40 bytes

Base ping is ~55ms, so this is showing 10-20 ms of induced latency under load.  Not fantastic, but not something I'm going to worry about excessively.  However, what I have noticed is that the traffic shaping seems more stable in .32-12 than it was in .24-8:

Results from Rich Brown's speed test script are:

Testing against demo.tohojo.dk while pinging 8.8.8.8 (60 seconds in each direction)

Download:  14.09 Mbps
  Latency: (in msec, 62 pings, 0.00% packet loss)
      Min: 30.157
    10pct: 30.691
   Median: 33.412
      Avg: 34.044
    90pct: 36.970
      Max: 48.250

Upload:  0.87 Mbps
  Latency: (in msec, 57 pings, 8.06% packet loss)
      Min: 30.655
    10pct: 30.744
   Median: 36.658
      Avg: 36.379
    90pct: 41.414
      Max: 46.451

ADSL line rates are ~21Mb down and 1160Kb up.

---

Dave Täht had asked about the performance of pie on dsl (my experiences have not been positive), and also it was mentioned that nfq_codel should perform better on low-bandwidth networks.  So I both both of those to the test.

nfq_codel results

Testing against demo.tohojo.dk while pinging 8.8.8.8 (120 seconds in each direction)

Download:  13.75 Mbps
  Latency: (in msec, 121 pings, 0.00% packet loss)
      Min: 30.059
    10pct: 30.378
   Median: 32.644
      Avg: 33.379
    90pct: 35.773
      Max: 66.983

Upload:  0.87 Mbps
  Latency: (in msec, 104 pings, 14.75% packet loss)
      Min: 29.981
    10pct: 30.714
   Median: 36.427
      Avg: 37.557
    90pct: 43.589
      Max: 49.973

I think I prefer the fq_codel results over the nfq_codel results.

---

pie

Pie continues to not perform well at all on my setup.  The traffic classes separate out nicely, but the latency can only be described as awful.

Yes, that's log-scale, and it needs it.

Testing against demo.tohojo.dk while pinging 8.8.8.8 (120 seconds in each direction)

Download:  14.95 Mbps

  Latency: (in msec, 121 pings, 0.00% packet loss)

      Min: 29.602 

    10pct: 30.532 

   Median: 34.861 

      Avg: 37.178 

    90pct: 46.831 

      Max: 67.668

Upload:  0.86 Mbps

  Latency: (in msec, 118 pings, 3.28% packet loss)

      Min: 30.322 

    10pct: 49.952 

   Median: 82.397 

      Avg: 83.052 

    90pct: 120.814 

      Max: 170.018

---

fq_codel

A check against the first results (same setup as first run).

Testing against demo.tohojo.dk while pinging 8.8.8.8 (30 seconds in each direction)

Download:  13.17 Mbps

  Latency: (in msec, 31 pings, 0.00% packet loss)

      Min: 30.446 

    10pct: 30.620 

   Median: 33.115 

      Avg: 33.193 

    90pct: 35.233 

      Max: 42.143

Upload:  0.88 Mbps

  Latency: (in msec, 30 pings, 6.25% packet loss)

      Min: 30.262 

    10pct: 30.915 

   Median: 34.738 

      Avg: 36.437 

    90pct: 42.182 

      Max: 45.980

Parting Thoughts

Something that I find curious is the ping packet-loss during upload.  It seems to be endemic to my setup (when using the wndr with CeroWRT), and doesn't happen when running directly behind the Freebox Server itself.  This loss wreaks havoc with the netperf UDP pings (and their results are conspicuously absent from all the results where I was testing from behind the CeroWRT router).

simple.qos vs. simplest.qos

This is a series of tests exploring more of the options in CeroWRT, in particular, this was to see what was causing the dropped out UDP ping data.  Spoiler:  dropped packets.

I view this as a bug with netperf and it's UDP_RR test.  Later, I may look into a patch for that, or see if I can find a workaround for it in netperf-wrapper, but it certainly clouds the results.

Simplest.qos certainly is behaving different from a ping CDF plot, and also is making more progress through the UDP pings before they drop out.

022 - simplest.qos, fq_codel, ecn

Switching to simplest.qos seems to have improved things.  UDP pings are lasting longer, and the latencies are better, especially at the "slow" end of things.

023 - simplest.qos, efq_codel, ecn

efq continues to act oddly, ramping up in bandwidth over time (as the UDP flows fall off?)

ECN or no?

The other question was what ECN options I have enabled, and how that might be factoring into things.

First, the previous settings, but with ECN disabled in the router on upstream (but still enabled on the client and server hosts...):

021 - simple.qos, fq_codel, no ecn on router upstream

No improvement in UDP ping longevity, no improvement in latency.

020 - simplest.qos, fq_codel, no ecn on router upstream

Better UDP longevity, but the tcp upload results look suspicious.

Conclusions?

I'm not sure how much stock to put in the ending of the UDP pings in these tests, except that they make it hard to make comparisons against other results.  The fact that the netperf UDP_RR test ends on dropped packets is frustrating in this case, I think that changing that would perhaps improve things.