I ran a couple raw wired performance tests, and saw the expected ~940Mbps download, but upload was seesawing all over the place. Per http://fast.com, download latency was negligible, but the upload was up well over 200ms.
So I moved behind my home router (WRT1900AC running OpenWRT), and ran a set of tests with https://flent.org.
All tests were performed on my laptop (2015 MacBook Pro), using the Apple Thunderbolt 2 gigabit ethernet adapter, through my home network. So there's a bit of noise from that aspect of the setup (there are two simple GbE switches between my desk and the router). But this is also the setup I mainly work with, so it's the one that I wanted to test.
I started by turning off SQM on the WAN side, and running a flent "rrul" test and seeing how this was going to turn out:
Definitely not pretty. But the download bandwidth was constrained by the router, so I guessed that the latency was all on the upload side, so I ran an upstream-only test
Which pretty much confirmed that it was primarily an upload issue.
A Piece of Cake to the Rescue
Skipping ahead to the end, here's the final result, using "piece_of_cake.qos" and limiting uploads to 35Mbps, and not restricting downloads at all (letting cake and fq_codel do their work since the router was already the bottleneck).
I need to sort out what's going on with the poor sharing. That could be a few different things (including the upload competition for bandwidth for acks).
But the latency difference is astonishing. I've increased overall throughput, and there's no additional latency from the load.
I also ran a 12-up and 12-down test, to see how those would do.
Download (all BE):
Again, very poor sharing.
Upload (all BE):
Cake, however, has made the upload look amazing.
I'm leaving a little bit of upload on the table, I think, but overall, not a lot, especially for the gains in latency control.