Bufferbloat with Comcast gigabit with Arris SB8200

With the working from home due to COVID-19, I decided it was finally time to upgrade my service.  I moved up to the gigabit plan (mostly for the extra upload bandwidth), and that also required a new modem, so bought an Arris SB8200.  It's DOCSIS 3.1, but is apparently synch'ing with my CMTS via DOCSIS 3.0, so these tests are with it in that mode.

I ran a couple raw wired performance tests, and saw the expected ~940Mbps download, but upload was seesawing all over the place.  Per http://fast.com, download latency was negligible, but the upload was up well over 200ms.

So I moved behind my home router (WRT1900AC running OpenWRT), and ran a set of tests with https://flent.org.

Cake: the latest in sqm (QoS) schedulers

Today I finally had the opportunity to try out Cake, the new replacement for the combination of HTB+fq_codel that the bufferbloat project developed as part of CeroWRT's sqm-scripts package.

Background

The bufferbloat project is tackling overbloated systems in two ways:

1. Removing the bloat everywhere that we can
2. Moving bottlenecks to places where we can control the queues, and keep them from getting bloated

sqm-scripts, and now cake, are part of the latter.  They work by restricting the bandwidth that flows through an interface (ingress, egress, or both), and then carefully managing the queue so that it doesn't add any (or much) latency.

More details on how cake works can be read HERE.

The WNDR3800

Cake was meant to perform well on lower-end CPUs like those in home routers.  So the test results that follow are all on a Netgear WNDR3800.  This was a fairly high-end router, 5 years ago when it was new.  Now, it's dual 802.11n radios are falling behind the times, and it's 680MHz MIPS CPU is distinctly slow compared to the >1GHz multi-core ARM CPUs that are currently in many home routers.

All the tests that follow were taken using the same piece of hardware.

Final Results

I'm starting with the final results, and then we'll compare the various revisions of settings and software that led to this.

Comcast Service Speeds:

180Mbps download

12Mbps upload

100s of ms of latency

Cake's shaping limits (before the CPU is maxed out):

~135 Mbps download speed

12Mbps upload

no additional latency vs idle conditions

What's really impressive is how smooth the incoming streams are.  They really are doing well.  Upstream is also pretty good (although not great, this is the edge of what the CPU can manage).  But what's simply amazing is the latency graph.  It doesn't change between an idle or fully-in-use connection.

And the CDF plot really shows that.  There's no step between the idle and loaded operation, just a near-vertical line around the link latency (which is almost entirely between the modem and the head-end).

Gear Review: Brevite Camera Backpack

As I mentioned in my previous post, while the Crumpler bag is great for travel, it's not so great for day-to-day use, and in particular isn't great on a hike.  But then, neither are most hiking back-packs, being designed to be out of the way when you're hiking.

So when I saw the the Brevite kickstarter, I decided to take a chance and back it, and see if it would work out better for me.

Unfortunately, it's not well suited to me and my cameras.  But I think it would be fine for many other people.  An important clue in all of this was in the kickstarter campaign, and had I been paying closer attention, would have realized the issue I ran into in advance.

But first let's talk about the bag, and what it does well.

Gear Review: Crumpler Proper Roady Photo Full Backpack

3 years ago, when living in Paris, I needed something for carrying camera gear on planes that was better than my 10yo daypack.

So I ordered this, having seen the Amazon Basics knock-off version of it:

Proper Roady Photo Full Backpack, by Crumpler

It's been utterly fantastic for carting camera gear and a latop on planes (or any other time I need to travel with the "full kit".  It holds a ton of camera gear and a 15" laptop, fits under-seat if it needs to, and is fairly comfortable once all the straps are adjusted right.

Here it is:

iperf3 and microbursts, part II

Previously, I talked about iPerf3's microbursts, and how changing the frequency of the timer could smoothen things out.

One of the first feedback items that I got, and lined up with some musings of my own, was if it might be better to calculate the timer based on the smoothest possible packet rate:

PPS = Rate / Packet Size

This is, btw, what iPerf3 does automatically on Linux, when the fq socket scheduler is available.  So this is really just seeing if we can fake it from user-land on systems that don't have it (like my OSX system)

Luckily, adding this to iPerf's code is pretty easy.

iperf3 and microbursts

This is in the "Always know how your tools work" category.

Background

We were doing some end-to-end network testing at work using iperf3.  While TCP would happily fill the link, UDP was giving us just miserable results.  It wouldn't achieve more than 8 mbps of 'goodput' on a 50mbps link (which we'd just verified using TCP).  Extra packets over 8mbps were getting dropped.

The setup at the traffic generation end was:

PC -> switch1 -> switch2 -> 50mbps link to ISP

• PC to switch1 was 1Gbps
• switch1 to switch2 was 100Mbps

If we connected the PC running iperf3 directly to switch2, we'd get much better throughput (or, rather, much less packet loss for the same traffic load).  But then the PC was transmitting into a 100Mbps port, not a 1Gbps port...

I thought that this sounded like perhaps packet bursts were exceeding some buffers, and got curious as to how iperf3 actually generates traffic and performs pacing with the -b option.

Packet Bursts

I personally didn't have access to the test setup, so I did some quick code inspection.  What I found was that it turns transmission on using a 100ms timer, turning transmission back off when it's achieved the right long-term average transmit rate (specified with -b ).

What this ends up looking like for a 100Mbps rate transmitting into a much higher rate interface, using 1KB packets, is below.  This is a graph of MB send per 1ms period, scaled into Mbps.

Here's the first second of that, using the same Y scale (1Gbps):

So yeah, bursty.  Not exactly "micro" bursts, either.  More like millibursts.

Sharpness Comparison: Nikon DX 18-200 vs. FX 24-120 f/4

A friend of mine and I have each owned the Nikon 18-200 DX zoom.  Which, for being an 11x zoom, is a pretty impressive lens.  But we both felt that it never delivered what we really wanted out at 200mm.

Especially as we moved up to the FX bodies (he's on a D750, I'm on a D600), and started shooting with prime lenses (the Nikon 50/1.8G and the Tokina 100/2.8 are my main lenses, both are stupendously sharp, even wide open).

In comparison to the primes, shooting with the 18-200 was just disappointing.  Obviously, an 11x superzoom has to make a lot of compromises, and the test data backs that up (DxOMark, dpreview, photographylife.com).

My friend pointed me at the 24-120/4, and he's picked up a copy (kitted with the D750), and I rented one from +BorrowLenses.com over this last weekend.  It's a fantastic walkabout lens, and is super sharp for what it is.  It's not the 28-70/2.8  But then, it costs 2/3 as much, and zooms nearly twice as far in trade for the loss from f/2.8 to f/4.0.

What I noticed the day I picked it up is that at 120mm, it felt sharper than the 200mm.  Maybe it's better VR, the much shorter length reducing camera shake (I have horrible hand-shake for a photographer).

And so after shooting with it for the last 6 days, and loving it, I decided that I needed to shoot some benchmark photos in controlled circumstances, with the same subject, in a short timeframe.

I don't have a snazzy photo target and lab space for this, so I reparked my motorcycle and used it's fairing graphics and bolts as my sharpness indicators.

The first test was to setup the tripod and shoot the bike from a distance, and simulate a situation where I'm short on lens, and going to be cropping down to a subject from too far away.  This is my "zoo" and "motorsports" scenario.

24-120 @ FX 120mm f/6.3 1/125sec ISO110

18-200 @ FX 200mm f/6.3 1/200 ISO400

 I think that in this case, the 18-200 edges out the 24-120, but you need to be looking at a fairly large image to tell that (the thumbnails on the screen are equivalent to me).  Pixel-peeping gives the edge to the superzoom, but not by a huge margin.

And the advantage comes in when you scale the larger 200mm image down to the same pixel dimensions as the 120mm (after cropping down to the same subject):

24-120 @ 120mm, cropped

18-200 @ 200mm, cropped and scaled down to same pixel dimensions as the 120mm

But again, you need to be looking much closer than this to really see the difference:

24-120 @ 120mm

18-200 @ 200mm

 In this case, the 18-200 clearly edges out the 24-120, but it took a bunch of zooming in to make it apparent.

And even then, I'm not sure that I care that much, as at normal viewing sizes, they're pretty comparable.

---

But then I realized that there's another use-case, one where you're not stuck trying to photograph something with what's really too short of a lens for what you want.  If you can put the whole image to use, and therefore get closer or farther from the subject, so that when the picture is taken, it's filling the frame, what can you get?  (ie, perform the same composition through the viewfinder, and just use that).

24-120 @ 120mm

18-200 @ 200mm (FX)

18-200 @ 200mm (DX, for 300mm equivalent)

Here, there's about 50% more lines of resolution in the two FX-composed photos (as is expected), and they make a huge difference (although again, you need to be viewing at full screen to see it).

After scaling them all down to the same pixel dimensions, and zooming in, however, the different is stark, and the 24-120 handily out-does the 18-200.

24-120 @ 120mm

18-200 @ 200mm (FX)

18-200 @ 200 (DX for 300mm equivalent)

However, it's also _much_ closer (10m @120 vs 16m @ 200 vs 25m @ 300 for the same field of view).  So perhaps that's just cheating...

But, overall?  I'm definitely going to upgrade to the 24-120.  It's far better than the 18-200 over their overlapping areas that are equivalent (18-80mm on the DX lens).  The measurements are that you have about 50% more resolution on the FX zoom than on the DX zoom.  For the range that the 24-120 doesn't cover, but needs to be cropped down for, it's close enough.

And that's before taking into consideration the constant f/4.0 aperture.  The 18-200 is only close to that at 18mm, and there it has far worse vignetting in the corners, and can't be used at FX, and it very quickly drops to only f/6.3.

---

The 24-120 is going to give me an FX lens that covers the entire range that the 18-200 did, and almost always do a better job, and only when I've reached a point where I should be using a 300+mm lens am I really going to be giving anything up.  And in those cases, I should just be using a big lens, anyway.  At least for me, I just don't need that long of a lens very often.  And when I do, I know it well enough in advance that I can go rent the exceptionally sharp 70-200/2.8 or 300/4 for about $120/week.