HTB rate limiting not quite lining up

I've noticed this off/on with first the WNDR3800, and now the WRT1900AC.  The rates I enter for the sqm_scripts aren't being met, and not, I think, because of CPU load issues, but something about the book-keeping.

Here's a set of tcp_download tests on the WNDR3800, the ingress rate limits are in the legend:

The WNDR holds up linearly until 90Mbps, and then it's clear that everything's come apart.  With the measured "good-put" at an eyeballed 95% of the rate that's setup in the limiter.  This is likely to be expected TCP overhead vs. the raw line bit-rate (which is where the limiter is running).

However, on the WRT1900AC, it's off rather significantly:

Maybe 80% of the target ingress rate?

+Dave Taht suggested I turn off TCP offloads, and it got less linear, worse on the low-end, better on the higher end.

This is definitely going to take some more testing (and longer test runs) to map out what the issue(s) might be.

**

Corrections:  This post previously stated that the WNDR3800 was falling short, but after talking with some other people, I think that's likely just the expected overhead of TCP, which becomes a more-obvious difference between the raw line rate and the "good-put" as bandwidths go up (5Mbps is easier to see than 500Kbps).

sqm_scripts: before and after at 160Mbps+

Apparently I've been upgraded.  I did a baseline test today (with sqm off), and saw that the new download rate was up around 160-175Mbps from 120-130.  That's some very impressive over-provisioning from Comcast.

Unfortunately, it also includes some rather unfortunate bufferbloat.  That's a surprising change for the worse, as the service, when initially installed with the same modem, was actually quite good by "retail" standards.  But still awful vs. what it should be.

The ugly (but fast):

Classic bufferbloat.  At idle, the target endpoint is maybe 10-12ms away.  200+ms of latency is pretty awful, and drags the "effective" performance of the service from >150Mbps down to what "feels" like a couple Mbps.

After upping the limits in the sqm, and turning off the tcp stack offloads, I ended up with this:

So, total bandwidth available has dropped to about 140-150Mbps (still more than the 120Mbps the service claims to be).  But latency is basically gone.  fq_codel holds the 5ms target rather nicely.

To make that latency difference more apparent:

Settings:

200Mbps ingress limit (something is odd with the math on this, clearly)

12Mbps egress limit

ethtool -k eth1 tso off gso off gro off

TCP Offloads: more harm than good

+Dave Taht has been saying for a while that TCP offloads do more harm than good, especially when mixed with fq_codel, and the ingress rate limiter that the sqm_scripts package uses to replace the large inbound buffers in the modem and CMTS with a much smaller buffer (but nearly as fast bandwidth), under the control of the router.

I finally put some numbers on that tonight.

The first dataset (green plots) are without gro, tso, and gso.  The second plots are with those offloads all re-enabled.  So enabling offloads:

1) slows it down
2) increases latency

??

Yeah, I'm keeping all the offloads turned off (and adjusted my router startup scripts to keep them off each time the simple.qos script runs).

sqm-scripts on Linksys WRT1900AC (part 1)

more actual context later, for now, I just wanted to make a quick post with some numbers from tonights tests.

• Comcast Blast! (120Mbps/12Mbps) internet service
• Arris AB6141 modem
• Linksys WRT1900AC router

Stock firmware:

full speed, but a fair bit of latency (although honestly, 120ms is pretty good compared to most stock setups)

After:

And what's even better?  Wireless is pretty awesome, too:

Thoughts on Iot: Products and Novelties

To be successful, you need a product

The Internet Refrigerator is a meme that gets a lot of jeers, but I think at the core of those jeers is the fact that it seemed like a solution looking for a problem.  To me, most of IoT feels like this (especially most of the stuff at CES).

Early devices like the toaster were done because they could be done.  Dares and fun side projects.  But they weren't real products that solved someone else's problems or filled a need.

I've seen a lot of kickstarters and startup companies go after home automation without really having a product or service that solved a real problem.  And those have mostly faded away.  The compelling reasons to use the products aren't there, leaving them as no more than novelties.

This is, perhaps, the genius of the Nest.  It's a thermostat (boring), made to look beautiful (yes, a novelty), but with remote access, and the smarts to learn your schedule instead of you telling it your schedule.  The goal is clear, the execution is beautiful, and now that they've had time to refine the results, everyone I know with one loves it.

I have an internet-enabled bathroom scale from Withings.  Yes, really.  What it offers is that it remembers my weight, every time I step on it.  And it gives me that data later, graphed over time.  And it does the same for my heart rate and some other health-related data.  The problem it solves is that I hate data entry, which is why I never was very good at tracking my weight in the past.  Now it's tracked for me, automatically.  And now I have a handy reference for some of my vital stats.

These are useful products, even if still overly expensive and luxury items.  But the electronics are just going to keep getting cheaper.  But if the business proposition doesn't fill a need, just is "X plus the internet", I don't see it ever really being more than a novelty today, and tomorrow's humorous internet meme.

Thoughts on IoT: Introduction and Hurdles

The Internet of Things is about connecting devices.  To what?  To everything.  It's not about making toasters or refrigerators that are "internet-enabled", it's about connecting devices to everything else.  And the internet is the doorway to "everything else".

Nearly a decade ago, when I entered this space, I hadn't realized quite what it was going to turn into.  Since then, I've seen a bunch of ideas come and go (and some come back again).  The core problems really haven't changed, and the solutions are getting better every year.

This is the first in a series of posts about the Internet of Things: lessons I've learned from it, the hurdles for anyone in the space, problems that I think we need to solve, and where I'd like to see things go in the future.

These aren't likely to be in any particular order, and the list below isn't in one, either.  Of these, most of them are the hurdles I mentioned in the title.  These are the problems that need to be solved in IoT.

---

Successful products and services vs. novelties - Most IoT devices seem like novelties, and that's a fair argument to make.  I think we're still in a phase where we're still exploring what we can do, and trying to build solutions to real problems.

Communication within the home - This is mostly about engineering the solution, but also about standards and communications.  This is a really hard problem to solve neatly, in a way that works in the "real world".

Communication from the device to the cloud - Talking out can be easy (if you have TCP/IP), but trying to communicate back to the devices gets a lot more complicated in a hurry, given the current state of home networking.

Security - Security is always hard, and IoT marries some things that expose new problems:  embedded device security, network security, and data security (let alone issues of data privacy)

Data - As we connect more devices to the internet, and start trying to do things with their data, we're going to need to deal with that data:  ownership, storage, search, privacy, analytics, automation, etc.

Interoperability - It's not really an internet of things if the things can't talk with each other (or their clouds can't).  The Hue is a very expensive, smartphone-controlled lightbulb by itself.  But it could be 

so much more if other things could talk to it.

---

I don't see any one company solving all of these, or any one consortium, either.  But as an industry, as builders of the internet and the devices that we connect to it, we'll need to be solving these issues for the Internet of Things to really bloom in it's full potential.

Comcast IPv6 rrul tests

These are just some raw results, I haven't done any tuning (need to do other things today), but I finally got around to contacting Comcast support, and getting the modem switched to full bridge mode.  Since it's their combo modem/AP/router/voip unit, support has to do it on the backend.  But live-chat support was able to do it (and able stay operational through all the ip address changes, which was nice).

But the WNDR3800 is certainly at (past) the edge of what it can handle at these data-rates when using the simple.qos scripts at:

download (ingress):  100000
upload (egress):  12000
squash ingress DSCP
ECN enabled in both directions (but not enabled on the local end of the test)


So, here's a bunch of graphs: