Bufferbloat.net

Bufferbloat FAQs

bloat@lists.bufferbloat.net (The Bufferbloat community) — Sun, 01 Dec 2024 09:10:12 +0000

Bufferbloat FAQs

We hear these questions all the time, so we collect the answers here for reference.

Answers are grouped into solutions for bufferbloat, then a review of Bufferbloat and its causes, then the science behind the fixes, then “objections”, and finally actual questions from various forums.

TL;DR - Solutions for Bufferbloat

Question #1.1: Is there a fix for bufferbloat?

Yes. The science for solving bufferbloat is well understood. Routers generally employ fq_codel or CAKE algorithms for ISP connections and Tx Queue limits, AQL, and ATF algorithms to address bufferbloat in Wi-Fi. The cake-autorate algorithm handles links with varying rates, such as 4G/5G cell phones, cable modems, etc.

The OpenWrt Project - where all these algorithms were developed - gives them the umbrella term “Smart Queue Management” (SQM).

Question #1.2: Are there any commercial solutions I can just buy?

Yes. The What Can I Do About Bufferbloat? page list several vendors who have figured out how to include these algorithms into their products.

Question #1.3: Are there other solutions?

There are a number of open-source projects listed on the What Can I Do… page above. As noted above, the OpenWrt Project developed and tested all these algorithms over the last decade.

Question #1.4: Where can I read more?

Check out How OpenWrt Vanquishes Bufferbloat for a list of the techniques a router can employ, including ISP-bloat, WiFi-bloat, and bloat caused by variable speed links.

About Bufferbloat…

Question #2.1: What is Bufferbloat?

Wikipedia says, “Bufferbloat is the undesirable latency that comes from a router or other network equipment buffering too many data packets.”

Question #2.2: What does that mean? How could that happen?

If a router doesn’t use a better algorithm, it will happily place every newly-arriving packet at the end of a single FIFO queue waiting to be sent to your ISP.

If packets arrive at the router faster than they can be transmitted to the ISP, a queue builds up. From time to time, the queue might hold dozens (or hundreds) of packets, potentially causing multiple seconds of latency or lag. Those buffered packets are “the bloat” in Bufferbloat.

Question #2.3: Where can bufferbloat occur?

Bufferbloat can occur anywhere there’s a bottleneck - a place where a fast link feeds into a slow link. When many packets arrive at the bottleneck, the router queues those packets. One or two queued packets can be beneficial so the slow link never starves. But queueing more packets only adds latency (delay) to the transit time of those packets.

Question #2.4: Where does bufferbloat happen in the real world?

These large queues can build up in your router’s connection to the ISP (because the outbound link tends to be slower than the local LAN interfaces) and also in Wifi interfaces (again, computers can create packets faster than the wireless link can carry them).

Question #2.5: How can I tell whether I’m experiencing bufferbloat?

There are a number of web-based tests that measure latency during the download and upload:

If the test shows an increase of latency under load of less than 15-25 msec, then the latency is well under control.

To make more repeatable tests, consider Flent or Crusader.

Question #2.6: You say there can be high latency in Wi-fi?

Yes. Wi-Fi drivers can often queue hundreds of milliseconds of packets, which adds additional delay to packet transit time (round-trip time).

Question #2.7: How can I measure Wi-Fi latency?

Check out the Crusader application. You’ll need two computers: connect the first by Ethernet to a LAN switch port on your router and start the Crusader Server; run the Crusader Client from a second computer on Wi-Fi.

The Science

Question #3.1: How do those algorithms minimize latency?

A router can control latency using one or more of the SQM algorithms. In general, these algorithms use a variation of this technique:

Place the arriving packets of each traffic flow (each individual connection from each computer on the local network) into their own queue.
In a round-robin fashion, remove a small batch of packets from a queue and transmit those packets through the (slow) bottleneck link. When they have been fully sent, move to the next queue, remove and send a batch, and so on.
Offer back pressure to flows that are sending “more than their share” of data.

Each turn, the round-robin process doles out a few packets from a queue, generally 5-15 msec worth of traffic. Low-traffic flows empty their queue “right away” - in their next turn. A single high-traffic flow gets to use the entire capacity of the bottleneck, because no other queues have data to send. When there are competing high-traffic flows, the round-robin process cycles between all the queues that hold packets, sending a “fair amount” from each in every turn.

Question #3.3: These descriptions make it sound as if bufferbloat only happens on upload (from my local network toward my ISP). Does Bufferbloat ever happen on download?

Absolutely. There’s also a bottleneck at your ISP. Their high-speed lines feed traffic to the (slower) link coming toward you. That fast-to-slow transition within the ISP equipment can also build up significant queues.

Question #3.4: How do the bufferbloat algorithms work for the download direction?

The SQM algorithm creates a new “download interface” within the router to act as the bottleneck. As with the upload direction, this internal download interface is configured to be slightly slower than the ISP link (typically 5%-10% slower). That lets the queue build up within the local router, where the bufferbloat algorithm can control it.

Question #3.5: How can latency be controlled in Wi-Fi?

WiFi bufferbloat can easily exceed hundreds of milliseconds. See the Ending the Anomaly and Bufferbloat mitigation in the WiFi stack talks from 2017 that document this.

Fortunately, that paper also presents a solution involving a) individual transmit queues for each station, b) AirTime Fairness, and c) Airtime Queue Lengths to drop latency by an order of magnitude. These techniques are described in the papers cited above and How OpenWrt Vanquishes Bufferbloat.

Question #3.6: How can latency be controlled on variable-speed links?

My cable connection speed varies from daytime to evening. And my 5G cell connection is even worse - changing from minute to minute. How can I choose a setting for the CAKE download and upload speed parameters?

The cake-autorate algorithm continually monitors traffic and latency, and adjusts the CAKE parameters up and down to give the highest throughput while minimizing latency.

Question #3.7: Won’t all those queues “clog up” my router’s memory?

No more than happens with a single FIFO queue. (Those packets are already being buffered now.) In fact, these algorithms can decrease memory use: queues for low-traffic flows are almost always empty; high-traffic flows - if they begin to build up a queue - get back pressure either using ECN or by dropping packets.

Question #3.8: Isn’t dropping packets bad?

No. Packet loss is always required by the TCP protocol to signal that there is congestion, or that the sender is “sending too fast”. Anytime a TCP sender detects that signal, they must decrease their rate of sending.

When a router using SQM notices that a significant queue is building for one of its flows, it implies that the sender is sending too fast for current conditions. If the router didn’t control it, the sender would attempt to use more than its share of the limited bandwidth. Consequently, the router occasionally drops a packet from the head of that queue to signal the sender to slow down.

Question #3.9: Can’t I just tweak the QoS settings of my router?

Maybe. It’s a fiddly process, and doesn’t always work. See What’s wrong with simply configuring QoS? for more information.

Question #3.10: How do I put a router “in front of” my ISP’s router?

In a normal setting, your ISP’s router is connected by cable/phone/fiber to their equipment in their offices. Your local devices connect either through Ethernet or WiFi to the ISP router.

But if the ISP gear doesn’t control Bufferbloat, you need to take control of your network. A very common solution is to put a router with SQM “in front of” the ISP gear, making it the primary router. To do this:

Turn off WiFi in the ISP router
Connect the WAN/Internet port of the new router to one of the LAN ports of the ISP router
Configure the new router’s SQM or other latency control options
Connect all your equipment to the new router. (Do not leave any equipment connected to the old device - it will throw off the anti-bufferbloat algorithms.)
As always, test the new connection to ensure that it’s working.

Objections We Hear

Question #4.1: But, that’s not “bufferbloat” - it’s just ordinary behavior. Of course things will get slower when there’s more traffic…

It seems you’re ignoring the order of magnitude increase in latency. That’s far more than expected if the network were “just busy”.

Question #4.2: But I’m the only one using the internet…

It may be true that you’re the only human in the house. But does your phone ever upload photos? Does your computer fire off any automated process? Does your Tesla (or your refrigerator) get updates? Do you browse the web, or otherwise use the internet? Any of those can generate traffic that, in turn, induces latency.

Question #4.3: It only happens some of the time…

Exactly - bufferbloat is transitory. You probably notice it when someone’s uploading photos, or your computer is doing something in the background.

Question #4.4: Those bufferbloat tests you hear about are bogus. They artificially add load, which isn’t a realistic test.

Yes, the tests do add load. But what would you expect to happen to your network’s performance if you actually were uploading or downloading a file?

Question #4.5: Bufferbloat only happens when the network is 100% loaded.

This is related to the previous answer. When you open a web page or open an email attachment, your computer - by design - briefly uses 100% of the link. Is this enough to cause momentary lag?

Question #4.6: It’s OK. I just tell my kids/spouse not to use the internet when I’m gaming.

Really?

Question #4.7: But, I have gigabit service from my ISP

That helps, but if you’re reading this because you’re worried about a “slow network” you still have to rule out bufferbloat.

Question #4.8: I can’t believe that router manufacturers would ever allow such a thing to happen.

In the 2010 time period, no one understood this phenomenon. In 2011, Jim Gettys reported on his work with other network experts (see Dark Buffers in the Internet) to show how surprising it was that routers would queue far more data than they could send in a reasonable time.

In 2012, CoDel was invented in response the newly-named “bufferbloat”. In the decade since, the fq_codel, CAKE, and cake-autorate open-source algorithms have been proven to minimize latency.

Today, there’s no excuse for router vendors not to incorporate this technology. But still, many have not done it.

Question #4.9: I mean… wouldn’t router vendors want to provide the best for their customers?

Not necessarily – implementing any new code requires engineering effort. They’re selling plenty of routers using their decade-old software. The Boss asks, “Would we sell more routers if we make those changes?” (Probably not, so the vendors don’t change.)

More to the point, if you complain that “It’s too slow” to your router vendor (or your ISP), they have an incentive to encourage you to buy a faster device or ISP plan. It may not solve the problem, but they’re happy to take your money.

But a round of reviews saying Vendor X has bufferbloat (and games are unplayable), but Vendor Y works great! might change the game.

Question #4.10: Why would they sell me a router that gave crappy service? They’re a big company - they must know about this stuff.

Maybe. We have reached out to lots of vendors. But remember they profit if you decide to upgrade to a higher capacity device/plan.

Question #4.11: Besides, I just spent $300 on a “gaming router”. It was the most expensive solution on the market…

Maybe that router’s not as good as their advertising says…

Question #4.12: I can’t believe you’re telling me that a bunch of academics have come up with a better algorithm than commercial router developers - that company who sold me that $300 router?

Well, the SQM algorithms seem to solve the problem when they replace the vendor firmware…

Question #4.13: And then you say that I should just install some “open source firmware”? What the heck is that? And why should I believe you?

Same answer as above.

Question #4.14: What if it doesn’t solve the problem? Who will give me support? And how will I get back to a vendor-supported system?

This is a valid point - see What Can I Do About Bufferbloat? for a list of commercial products.

Real-world questions

Question #5.1: Traffic Shaper Queue Limiters not helping with Bufferbloat

… Wondering if anyone could help me out with this configuration, again the whole purpose of this is to have the lowest latency as possible when gaming. I understand that the bufferbloat test is designed to see how the network handles high stress loads and when considering for online gaming (eg. COD MW3) I understand it doesn’t use anywhere near by max bandwidth. Even under lower loads I still get the same latency values. (Original post on Reddit)

Despite the confident assurances from other posters that it isn’t bufferbloat, it sounds as if you’re seeing latency/lag when gaming. Here’s what could be going on.

Bufferbloat is transitory. You can have very low ping times when the link is idle, but if someone else starts using the network (reading the web, watching a movie, uploading photos from their phone), their bursts of traffic can momentarily load the network to 100%. Could that be enough to make you miss your shot?
You’re right - bufferbloat tests “artificially load the network”. They do this to see how your network performs during those moments of 100% load.
You didn’t say, but at ISP speeds above 300-500mbps, the bufferbloat in the Wi-Fi system can become important. This is a solved problem (see Ending the Anomaly), but not universally deployed in routers.

Bufferbloat is not your problem on a gig symmetrical link unless you are smashing the upload.

This is exactly the definition of bufferbloat. If sending a lot of traffic causes your latency to increase significantly, something is wrong, likely bufferbloat.

Question #5.2: Extreme Bufferbloat on fibre connection

… I’m playing competitive games and it feels like im desynced to the server 70 % of my matches. First image is with no QOS in the router, the second one is when im limiting my bandwith to 85 % up and down. With QOS its ok, but i just dont understand why jitter and latency is that high without QOS on a fibre connection. (Original post on Reddit)

Garden variety commercial routers (even “gaming routers”) frequently don’t have guard rails to prevent them from queueing too much data. All the traffic goes “into the queue” (technically, they use a FIFO). A burst of bulk packets (photos from your phone, reading a web page, etc.) delay smaller packets such as gaming updates, voice and videoconference traffic.

QoS helps (as you’ve seen), but it won’t entirely solve the problem.

However, this is a solved problem if you have a good router. See What can I do about Bufferbloat? for more details

Question #5.3: Bufferbloat on Wifi

Given the focus of latency in the Wi-Fi 6 and Wi-Fi 7 standards, has anyone tested the bufferbloat behavior of these AP? I’m particularly interested in U6+.

… The amount of packet buffer they will have on board will be trivial. (Original post on Reddit)

And yet, WiFi bufferbloat can easily exceed hundreds of milliseconds. See the Ending the Anomaly and Bufferbloat mitigation in the WiFi stack talks from 2017 that document this.

… An easy way to test that would be to use Crusader.

The Crusader network tester is terrific. See the question above about using it to test Wifi latency.

Question #5.4: An Edgerouter solves latency, but seems not to handle the speed…

… Whenever my Tesla started updating, my ping went to crap… Finally, I threw my hands up in the air, configured an edgerouter I had here and enabled smart queue. Instantly I was getting A+ scores, with +0ms on both incoming and outcoming.

Do I need new hardware? If so what is recommended? Should I just give up and use the edge router. (Original post on Reddit)

A couple thoughts. My mentor once said, “If you can’t tell the difference, it doesn’t make a difference”. How does that apply to the situation you describe?

If you’re happy with the way your network performs, you can declare victory. If the Edgerouter gives near zero additional latency, it seems like a win.
You didn’t mention the rated speed for your ISP, but do you ever notice that it’s “not fast enough” when using the Edgerouter?
If not, see the next question for more information…

Question #5.5: Consider a lower tier plan from your ISP

… I’d like a plug and play QoS router with a 5gb/s WAN port and at least 1 5gb/s LAN port (future proofing) … (Original post on Reddit)

A 5Gbps-capable router is going to cost serious money. (I don’t know your budget, but it’ll be a lot.)

Your ISP has probably been offering a higher speed plan, especially if you say “it’s not fast enough”. But if your router is adding latency, then everything will feel slow, regardless of the “speed” of the link.

Surprisingly, the “speed to deliver” normal data doesn’t increase by much for a faster link. See the charts in The Latency Effect that show that increasing ISP speed doesn’t make web pages load much faster. Decreasing latency always made pages load faster.

So 250mbps or 300mbps could be plenty unless you want bragging rights.

Here is a bit of contrarian advice. If you don’t actually need such high-speed service, consider these options:

Get a lower tier plan from your ISP
Get a router with the SQM algorithms for minimizing latency/bufferbloat. See What can I do about Bufferbloat? for a list of suitable routers
You’ll see the latency drop without much difference in your perceived speed
… and save a bunch of money per month on your ISP bill

Question #5.6: I don’t need SQM…

@WWicketW writes:

I’m on Flint2 also …

With SQM mine max speed was 1,2 ÷ 1,3 Gbps in download (and A+ on waveform), with HO+PS (Hardware offloading and Packet steer on all CPU) I’ve reach 2,2Gbps on download and A on waveform.

I definitely can survive without SQM 😅 (Original post on Reddit)

That sounds like a terrific solution. No one asserts you need SQM. It’s only useful if lag/latency is affecting you. (In other words, “If you’re happy, I’m happy.”)

It’s also valuable to know that Flint 2 (less than US$150 with coupon on Amazon) can feed a 2Gbps+ link and keep up with the rated speed. Congratulations!

Question #5.7: My ISP’s router gives terrible bufferbloat…

I have been struggling with an incredible sluggish and inconsistent video game experience on my computer. I have payed for a pc, I’ve paid to have it tweaked professionally, I’ve bought the best peripherals. I have done everything to fix this issue and none of it has worked, this has been going on for months… (Original post on Reddit)

You are going to need to take control of your network. See What Can I Do About Bufferbloat? for more information.

TL;DR - you’ll probably need to put another, smarter, router in front of their router that can control the queueing/latency. Consider the OpenWrt One - it’s reasonably priced, can handle your data rate, and it’s the platform where all these SQM algorithms were developed.

cake-autorate

bloat@lists.bufferbloat.net (The Bufferbloat community) — Sun, 21 Jul 2024 08:30:13 +0000

cake-autorate for variable rate links

Background: The CAKE algorithm controls bufferbloat well on constant rate links by using fixed upload and download rate parameters to control the amount of queued data. But many links, such as LTE cell-phones, cable modems, and Starlink have rates that vary from morning to evening, or even from minute to minute. Because CAKE uses fixed parameters, it forces an unpalatable compromise because it can’t adapt to these varying-rate links.

Solution: The cake-autorate algorithm continually measures the current latency and adjusts the parameters of CAKE to minimize latency.

The cake-autorate repo is on Github with an active community on the OpenWrt forum. Here’s an excerpt from the README

cake-autorate is a script that automatically adjusts CAKE bandwidth settings based on traffic load and one-way-delay or round-trip-time measurements. cake-autorate is intended for variable bandwidth connections such as LTE, Starlink, and cable modems and is not generally required for use on connections that have a stable, fixed bandwidth.

CAKE is an algorithm that manages the buffering of data being sent/received by a device such as an OpenWrt router or an Asus Merlin router so that no more data is queued than is necessary, minimizing the latency (“bufferbloat”) and improving the responsiveness of a network. An instance of cake on an interface is set up with a certain bandwidth. Although this bandwidth can be changed, the cake algorithm itself has no reliable means to adjust the bandwidth on the fly. cake-autorate bridges this gap.

cake-autorate presently supports installation on devices running OpenWrt and Asus Merlin.

Status

This is the development (master) branch. New work on cake-autorate appears here. It is not guaranteed to be stable.

The stable version for production/every day use is 3.2.1 available from the v3.2 branch.

Toward a Consumer Responsiveness Metric

bloat@lists.bufferbloat.net (The Bufferbloat community) — Fri, 10 Sep 2021 09:10:12 +0000

Toward a Consumer Responsiveness Metric

At a recent videoconference, I advocated strongly for a consumer-facing measurement of latency/responsiveness. I had not planned to speak, so I gave off-the-cuff comments. This is an organized explanation of my position. I offer these thoughts for consideration at the IAB Workshop “Measuring Network Quality for End-Users, 2021” - Rich Brown

I hunger for a day when vendors (router manufacturers and service providers) compete on the basis of “responsiveness” in the same way that they compete on speed - “Up to X megabits per second, and Y responsiveness!”

I have been working on the “Bufferbloat Project” [1] since 2011, trying to find layman’s terms for what was happening, and what to do about it. [2] [3] The delay goes by the name “lag”, “latency under load”, or “bufferbloat”. At first, the effects seemed mysterious and non-intuitive. Even to knowledgeable individuals, the magnitude of the delay caused by queueing was astonishing. No matter what name you use, it makes people say, “the internet is slow today”.

My router at home has solved this problem. I enjoy the fruits of the intense research from the mid 2010’s that led to well-understood solutions such as fq_codel, cake, PIE, and airtime fairness. Even using 7 mbps DSL, my network was quite usable, and very responsive.

My frustration in 2021 is that this remains a problem for nearly everyone else. The market has not provided solutions. Every day, people purchase brand name equipment that happily queues hundreds of msec of traffic.

I postulate that vendors have not considered responsiveness to be an important characteristic of their offerings. Consequently, they have not prioritized the engineering resources to incorporate the well-tested solutions listed above.

My hope, from this note, and from our on-going efforts, is that we can come up with a test tool that consumers can use to raise awareness of the problem of bad responsiveness.

Characteristics of a Responsiveness Tool

I seek a “responsiveness tool” with these characteristics:

Easy to use. People need an easy way to measure responsiveness so they can give feedback to their vendors.
A single number, so it’s easy to report and compare.
Bigger must be better. High latency means bad responsiveness. People have no intuitive feel for a millisecond: “Is 100 msec bad? Isn’t that really short…?”
An approximate measure is OK. Consumers won’t mind separate runs varying 20% or 30%, especially since poor responsiveness could be an order of magnitude different from good.
Resistant to cheating. Vendors sometimes optimize pings to make latency look lower. But real people’s traffic doesn’t use pings. The responsiveness test must use protocols that match actual traffic patterns.
Vendor and technology independent. People should use and get similar results from their phone, their desktop, on the web, or using an app.
“Good enough”. A widely implemented and promoted metric that substantially matches people’s real experience is vastly superior to a host of competing metrics that muddy the waters in consumer’s minds.

A Proposed Metric - RPM

Apple has produced an Internet Draft “Responsiveness under Working Conditions” [4] and implementation. It defines a procedure for continually making short HTTPS transactions on a path to a server that has been fully loaded in both directions. The number of transactions in a fixed time is expressed as the number of “round-trips per minute”, which is given the name “RPM”, a wink to the “revolutions per minute” that we use for cars.

The RPM measurement satisfies all my concerns.

Non-requirements

It is not a requirement for the responsiveness test to provide:

Strict reproducibility. The wider internet has widely varying conditions, with bottlenecks moving around by time of day or adjacent traffic. It is not reasonable/feasible to expect that any measure used by consumers will be exactly reproducible.
Detailed statistics or distributions of measurements. This is not a diagnostic tool. A nuanced data set with medians and percentiles may excite techies, but for others, it’s hard to understand the implications.
Performance of any particular protocol. The responsiveness tool must measure a broad variety of typical traffic.
Data to be used as input for vendors to design solutions. The responsiveness measure needs to be used the same way we say to our mechanic, “The car makes a funny noise when I …”. I expect the specialist to work to reproduce the symptom, using the provided equipment, and come up with an appropriate solution.

Summary

The research of the last decade has developed a wide variety of solutions. There are plenty of corner-cases where these solutions aren’t perfect. I encourage vendors and researchers to study the field and advance our knowledge further. I would be delighted if they found practices even better than the current state of the art.

But “the rest of the internet” (including my neighbors and family members, for whom I’m the support person) would all benefit from a world where off-the-shelf equipment already incorporated well-known, best practice solutions.

References

[1] Bufferbloat Project https://bufferbloat.net

[2] Bufferbloat and the Ski Shop https://randomneuronsfiring.com/bufferbloat-and-the-ski-shop/

[3] Best Bufferbloat Analogy - Ever https://randomneuronsfiring.com/best-bufferbloat-analogy-ever/

[4] Responsiveness under Working Conditions - Internet-Draft at: https://datatracker.ietf.org/doc/draft-cpaasch-ippm-responsiveness/ Full disclosure: I am one of the editors of the “Responsiveness Under Working Conditions I-D”

[5] This note was first published at https://randomneuronsfiring.com/toward-a-consumer-responsiveness-metric/

CakeRecipes

bloat@lists.bufferbloat.net (The Bufferbloat community) — Wed, 07 Oct 2020 14:10:00 +0000

Cake Recipes

Cake is the comprehensive queue management system the bufferbloat project has been working on since 2013. It has been included in the Linux kernel since v4.19.

These recipes are annotated combinations of settings that have been used for particular kinds of networks, organized from most common to most unusual. The detailed reference for all the parameters is tc-cake(8).

Additional recipes invited! To edit this page, submit a pull request to the Github repository.

Internet over TV Cable

Outbound, General Case

tc qdisc replace dev eth0 root cake docsis ack-filter bandwidth 17mbit

dev is the device, eth0 in this case.
root means this is the “top” qdisc.
docsis says tune for a cable-tv uplink’s overheads: cable TV follows the docsis standards.
ack-filter skips sending redundant acknowledgements.
bandwidth is the upload bandwidth of your link, often taken from speed tests like Waveform.

Inbound, General case

This applies to most inbound cases, not just cable.

ip link add name ifb4eth0 type ifb
tc qdisc del dev eth0 ingress
tc qdisc add dev eth0 handle ffff: ingress
tc qdisc del dev ifb4eth0 root
tc qdisc add dev ifb4eth0 root cake bandwidth 170mbit besteffort
ip link set ifb4eth0 up # important 
tc filter add dev eth0 parent ffff:  matchall action mirred egress redirect dev ifb4eth0

This creates a named link for download/ingress and applies CAKE to it.

bandwidth is the download bandwidth of your link, often from a speed test.

Internet over Telephone Lines

Outbound

tc qdisc replace dev eth0 root cake pppoe-ptm ack-filter bandwidth 61mbit

pppoe-ptm is one of the DSL (digital subscriber line) variants used by telcos such as British Telecommunications.

There are numerous dsl options: if your ISP doesn’t publicise which they use, there is also conservative which sets the expected overhead to a safely high value.

Internet from a Preexisting Ethernet

Outbound

tc qdisc replace dev eth0 root cake ethernet bandwidth 1gbit

ethernet says to tune for ethernet’s overheads.
bandwidth 1GBit is substantially faster than is commonly seen in offices. The example is from an office connected upstream to a datacenter with a faster 10Gbit network.
bandwidth unlimited suits an upstream which is the same speed as the office network.

Dave Taht's Take on TCP

bloat@lists.bufferbloat.net (The Bufferbloat community) — Thu, 21 Mar 2019 15:38:14 +0000

Dave Taht’s take on TCP

TCP is done. It’s baked. It’s finished. With the arrival of BQL, TSQ, and pacing, there is very little left we can do to improve it, and we should move on to improving new transports such as QUIC which have option space left. However, a recent development has caused me to change my mind.

Ever since RFC6013 failed in favor of tcp fast open, I’d given up on tcp. RFC6013 was a lousy rfc in that it didn’t make clear its best use case was in giving dns servers a safe and fast way to use tcp, which would have helped reduce the amount of DDOS and reflection attacks, speed things up, and so on. It wasn’t until I had a long discussion with Paul Vixie about this use case and worldwide problem with dns that I understood it’s intent to add a good stable 3way handshake to dns was so good…. and by then it was too late.

Instead, tcp fast open was standardized for the limited (IMHO) use case of making web traffic better. Web traffic has a terrible interaction with TCP, in that it tends to start up 6 or more simultaneous connections and slam the link with stuff in slow start simultaneously. Other standards that I opposed, like IW10 , also got adopted, and we (as part of the cake project) tried to get an AQM (cobalt) that responded faster to stuff in slow start. Which we succeeded at, and that paper is progress, but it’s still not good enough.

It makes me really crazy that seemingly all the other TCP researchers in the world tend to focus on improving TCP behavior in congestion avoidance mode - because the statistics are easy to measure! - instead of focusing on the 95% of flows that never manage to get out of slow start. Yea, it’s hard to look at slow start. That’s why we’ve been looking at it hard for 5+ years in the bufferbloat project - trying to get linux, flent, irtt, to be able to look in detail at sub 4ms intervals, among other things.

There are so many other problems with TCP as a transport - it requires a stateful firewall for ipv4 + nat, and more stuff than I have time to go into today…

One item off that long list:

QUIC and Wireguard have a really nice 1 RTT reconnect over crypto time. I like it a lot. I have not had time to poke much into the DOH working group at the ietf, but my take on it was that we needed to make dns better, not replace it.

[1] Up until about 6 months ago, I really felt that we couldn’t improve tcp anymore. DCTCP was a dead end. However the SCE idea makes it possible to have selectable behaviors on the receiver side - notably, a low priority background transport application (for backups/bittorrent) can merely overreact to SCE markings by sending back ECE to the tcp sender thus getting them to back off faster and be invisible to other applications. Or something more complicated (in slow start phase) could be used. ACCUECN also seemed feasible. And dctcp like approaches to another transport than tcp seemed very feasible.

But to me, the idea was that we’d improve low latency applications such as gaming and videoconferencing and VR/AR with SCE, not “fix” tcp, overall. Goal in life was to have 0 latency for all flows - if it cost a little bandwidth, fine - 0 latency. The world is evolving to “enough” bandwidth for everything, but still has too much latency. The whole l4s thing conflating the benefits low latency with an ecn-enabled tcp has makes me crazy because it isn’t true, as loss is just fine on most paths - lordy I don’t want to go into that here, today. loss hurts gaming and videoconferencing more.

Another ietf idea that makes me crazy is the motto of “no host changes” in homenet, and “dumb endpoints” - when we live in an age where we have quad cores and AI coprocessors in everybody’s hands.

The whole QUIC experiment shows what can be done when you have smart endpoints, along with a network that is as dumb as possible, but no dumber.

Jake Holland's Stance on ECN

bloat@lists.bufferbloat.net (The Bufferbloat community) — Sun, 17 Mar 2019 01:14:22 +0000

Jake’s ECN Position=Blue (+Yellow Stripes)

I think of myself as BLUE, with big yellow racing stripes.

I think ECN has a lot of important potential for improving the way the internet operates.

In general, I think the sender can do great things with rate tuning when it has just a little more information about the state of the network than “there might have been some loss, I’m not quite sure”.

(I worked on FastTCP for years and saw it there, and I’m seeing it again from more of a distance in BBR.)

I mean, it’s complicated. No argument here. The response depends on the RTT in uncomfortable ways, and analyzing the performance characteristics of competing flows with different parameters gets you out into the weeds of insanity really fast, and that’s before you try to worry about the possibility of corner case issues in the details of different implementations.

But the advantages of getting an explicit signal out of the network are huge, compared to trying to infer guesses about the network out of the ack patterns. Night vs. day. Forensics vs. surveillance cameras. Just because today’s congestion controls don’t nail the optimal response yet doesn’t mean the information isn’t worth its weight in bitcoins.

I think that to the extent we can get better at this, we can have a better internet. I’d love to see it out there and working. I’m glad Apple and Linux have done their parts to get deployment closer to a real possibility, and I’m glad this list is here and the people on it are digging into the issues. I hope other groups working on middle boxes are doing so as well.

With a little luck and a lot more work, maybe we’ll end up with a more efficient internet one day. My biggest fear is getting it unfixably wrong. But I have a pretty strong belief there’s a good answer in there somewhere.

Mailing list post for reference.

Pete Heist's Thoughts on ECN

bloat@lists.bufferbloat.net (The Bufferbloat community) — Thu, 06 Sep 2018 23:28:00 +0000

Pete Heist’s Thoughts on ECN

Although I feel the usual pull of team Grey, I think I’ll call myself a Purple for this, possibly overly simplistic reason:

In theory, the ECN bits only give congestion control algorithms more information than they would otherwise have, and in a perfect world, dropping packets would be a measure of last resort.

Purple should lose if it can be shown that nothing can (practically) be done to come up with an agreed upon treatment of ECN bits that results in “better” congestion control with less packet loss, most of the time. Or if there are commonly observed, demonstrably harmful pathologies that are not reasonably possible to overcome.

Presently, I’m more interested in what happens to traffic for regular people and small to medium sized ISPs, say 5-200ms RTTs and 1-100Mbit bitrates, than what happens in the data centers of trillion dollar companies.

Lastly, my interest in ECN is to understand better what tools and metrics can be used on real-world traffic to evaluate its impacts. If I can contribute anything to this project, I hope it will be to bring some unsettled contradictions to slightly earlier resolutions…

Dave Taht's Stance on ECN

bloat@lists.bufferbloat.net (The Bufferbloat community) — Wed, 05 Sep 2018 15:38:14 +0000

Dave Taht’s take on ECN

When it comes to ECN, I am firmly on the yellow team: I’m chicken.

I just used rigorous argument and moral authority on what is now probably the worlds largest active ecn in the edge deployment, to block systemd’s attempt at universal enablement in Linux. As an accidental co-author of what is now the largest edge-user ecn-enabled fq + aqm deployment in fq_codel, the sqm-scripts, and the newest implementation in the ath9k and athk wifi code… and now sch_cake… I lose sleep over the ecn component only.

I have a ton of data on it. It’s mixed.

What I see with lots of ECN traffic on a fq_codel’d link is that other valuable packets are delayed (slightly) or lost and am always saying “ECN has mass” to anyone that will listen. This is made up for by nearly eliminating retransmits, however ECN’d flows can bloat up the link and inflate RTTs enormously when I feel they shouldn’t. ECN is a huge win for interactive tcp traffic, which is why apple adopted it (and - helpfully - their reno tcp is far less aggressive than linux cubic). I worry about lockout at low speeds and that one day we’d have to mark all packets of all types of flows (including voip and dns) as ecn capable, unless tcps evolve appropriately towards an agreed upon response curve.

One really interesting side-effect of ecn on is that fq_codel, running locally on the server, can self congest and start marking packets locally, thus regulating the behavior of the server better after a rtt.(try 128 flows coming out of a short path at a gbit). But local fq_codel induced loss (currently) on the other hand is sometimes not lost there but signals the local stack to immediately to reduce cwnd without actually losing the packet. Others might view either behavior as a problem and prefer that that server serve 100s more flows at ever increasing self inflicted local delay (as sch_fq does) until you run out of cpu.

I certainly wish we’d come up with a more robust response to overload in fq_codel for ecn, as currently a malicious ecn sender, or too many ECN’d flows can push fq_codel to its memory limit before being dropped robustly. (pie drops ecn at overload). fq_codel (and now sch_cake) can continue to evolve as can everything else of course!

In my mind ECN MAY be a good idea at very short rtts(sub 2ms), and IS good very long ones(>150ms), for interactive traffic. It’s good for doing things like protecting video iframes from loss. I use it to protect routing babel protocol packets from being dropped. I helped put it into mosh and wireguard. Etc. I think the use cases where it is actually needed generally are far more limited than most think.

Others (in the bbr, l4s, dctcp communities) want to change the definition of ECN to mean a multi-bit rate reduction and obsolete rfc3168, where a loss is equivalent to a mark and the recommended rate reduction is ¹⁄₂. But: fq_codel, pie, red, and all other deployed ecn capable aqm systems essentially implement rfc3168 behavior and it’s what apple’s tcp - and linux cubic - and bsd’s - and windows - general deployment expects. I had hoped with wider deployment of AQMS that dealt with ECN at all properly that we’d see more servers also enabling it… and we’d see TCPs evolve to treat aqms doing multiple ecn marks per rtt yet per rfc3168 more sanely than they do today as it is a stronger signal of congestion than loss.

Instead… well, see BBR, which currently more or less ignores packet loss on its quest to own the link, and currently has no ecn response. There’s a thread on the bbr list that talks about how they are leaning towards not respecting that rfc. The L4S folk vehemently defend the idea that some form of dctcp can run outside the datacenter, based on bigbuckbunny based demos, combined with a custom and patented AQM, never tested against wifi or 3g, who create a lot of noise in the IETF, where I am dubious of the accuracy required in ACKs in asymmetric networks, and against other forms of real traffic, among other things.

I’ve run most of the the DCTCP based stuff over simulations and over the internet, and thus far, fq-codel’s basic response has been adaquate for low numbers of flows. I’ve thus spent the last several years avoiding thinking about ECN at all.

Nearly every time I’ve quit smoking, an ecn debate started me up again, and instead of continuing to deal with it, I left the ietf, leaving the folk there to theorize amongst themselves with no actual deployment to worry about. I’m so frustrated with the “make tcp go fast at any cost” people that periodically I fiddle with something called tcp-fu (for users), that has an adjustable response curve to fq_codel’s ecn marks from background (torrent-like) to “gentle”, rather than “rabid”. Despite the enormous success of fq_codel and BQL in eliminating bufferbloat and network latency - I feel bad about essentially obsoleting the entire field of LPCC with our work on fq_codel and would like to do something about it.

I used to take glee in how fq techniques generally beat dctcp’s, even in the datacenter… at how even the guy that invented dctcp moved onto fq…

I really resented repeated attempts to closely couple AQM deployment to ECN deployment. DOCSIS-PIE shipped without it. And FQ+AQM alone led to nearly two orders of magnitude reduction in latency under load across the internet, and I wished everyone could get behind that. Instead the IETF AQM mailing list devolved into an endless bikeshed, and I ended up calling for the closure of that working group rather than continue to deal with it. It still took 6 years to get fq_codel - the first solution proposed - through the IETF process to become RFC8290, and I still approve - even after all that time stuck within the IETF process - to consider that and all the other products of the working group - to be “experimental” in scope until we saw more widespread deployment. fq_codel is at ~100% penetration now and it is time to roll up the very minor changes to it into an internet standard.

I once got so frustrated with the ECN advocates that I suggested they get a sysadmin drunk enough at their university to deploy it fully. None ever did. I believe in early adoptors, in knowledgable users and limited deployments along the edge to get real world data, in a careful deployment.

ECN is the wet paint of the congestion control universe. You have to touch it, feel it, use it, explore it, day and day out - as I have for over 5 years - to worry about the long term effects on the health of the Internet, as I do.

Jonathan Morton's Take on ECN

bloat@lists.bufferbloat.net (The Bufferbloat community) — Tue, 04 Sep 2018 15:38:14 +0000

Jonathan Morton’s Take on ECN

I would characterise the problem not as “should we use ECN or not”, but “how does ECN need to evolve to reach its potential”. This likely means changes in both routers and endpoints, but since there is already some deployment, changes have to be backwards and forwards compatible.

The DCTCP approach fails primarily because it is not backwards compatible. DCTCP endpoints do not react as intended to ECN signals given by Codel, which was designed very specifically around RFC-compliant TCP behaviour. That’s why there is concern over BBR2’s proposed handling of ECN, which follows the DCTCP model instead of the RFC-compliant model. For similar reasons, other proposals to “soften” the response to individual CE marks are Bad Ideas.

In my view, ECN is essential for modern congestion control. Without it, there is no way to separate random loss and re-ordering from congestion signals, and signalling congestion incurs application latency penalties due to HoL blocking in TCP while the lost packets are retransmitted. With ECN, congestion can be signalled unambiguously as congestion, and without incurring retransmits; network engineers who rely on packet loss as a primary metric should also be pleased by its deployment.

The evidence against ECN chiefly consists of its present effect on inter-flow latency with exemplary-standard flow isolation, under particular measurement techniques in which the latency-measuring flows are treated as saturating flows and thus equal to the rest of the saturating traffic. This is a remarkably specific and unusual set of circumstances, which seems unlikely to be replicated by real traffic. Nevertheless, there are improvements that can sensibly be made without ditching ECN entirely.

Principal among these is enabling TCP senders to operate at very small cwnds. Presently, most appear to operate on the principle that four packets must be kept in flight at all times (unless application limited), so that a single lost packet can be reliably detected and retransmitted within one RTT. Additionally, some widely-deployed TCP stacks define the cwnd only in terms of whole MTU-sized packets. The net result is that the effective cwnd cannot shrink below 4xMSS, when sometimes 1xMSS (perhaps still as two, three, or four distinct packets) would be more appropriate. In such cases, Codel devolves to continuously marking all packets in these flows, ensuring that they remain at their minimum cwnd.

With the advent of TCP pacing, fractional cwnds are theoretically practical to implement, such that on average there is less than one packet in flight. I remain uncertain that such extreme measures are necessary or desirable, given the re-engineering required (ie. changing cwnd from a packet to a byte basis) to implement them. However, pacing out one MSS over an RTT via four separate packets should be easier to implement and would help the aforementioned case, by reducing the serialisation delay multiplier in the overall inter-flow latency equation. An individual sender implementing this incurs a slight throughput cost due to greater relative packet overhead, but may immediately see an application latency improvement of a few milliseconds in the given scenario - or a larger latency benefit at very low bandwidths.

As an alternative to the DCTCP model, using the distinction between ECT(0) and ECT(1) could be a backward-compatible method of providing “softer” congestion signals to endpoints. Existing endpoints and routers would ignore the distinction, treating both as merely indicating an ECN Capable Transport. An old RFC suggested using the distinction to protect the integrity of the ECN signal itself, but this was never deployed and there are no plans to do so.

Modified routers could detect incipient congestion - which has not yet merited a CE mark but warrants caution on the part of TCP senders - and convert some proportion of ECT(0) marks to ECT(1) to give a fine-grained control signal. It should be feasible to reflect that information back to the sender, which can then perform a greater variety of cwnd evolutions, not just slow-start followed by AIMD. In particular, these signals could instruct the sender to drop out of exponential or polynomial growth in favour of additive-increase, or to hold cwnd steady instead of oscillating, or to perform additive-decrease to correct a slight overshoot. Only if the latter was not sufficient would a CE mark be sent, with the RFC-compliant response expected.

I certainly don’t want to write off ECN before the above measures are at least tried.

ECN-Sane Project

bloat@lists.bufferbloat.net (The Bufferbloat community) — Mon, 03 Sep 2018 15:38:14 +0000

About the ecn-sane project

As the accidental co-authors of what may well become the world’s largest ECN-enabled AQM deployment, our ecn-sane project is started in the hope that a clear, safe, sane path forward for ECN enablement across the edge of the Internet will emerge.

What is ECN?

Explicit Congestion Notification is a means to do network congestion control without dropping packets.

Why are we concerned about it?

Neither the original codel or pie AQM research covered ECN.

However, the fair queueing variants of these algorithms, the ‘fq_pie’ and ‘fq_codel’ qdiscs enabled ECN by default, because, in very limited tests by bufferbloat.net members in 2012, it seemed to work well. FQ_Codel, in particular, is in increasingly wide deployment. We’ve long encouraged individual users to try it out… and then, in 2017… Apple enabled it universally across their devices and stacks.

ECN’d behaviors are observably different than drop, and different AQMs treat it differently. The pie algorithm starts dropping ecn enabled packets once the drop probability cracks 10. RED behaves similarly. Codel does not drop until it runs out of packet space.

Our limited tests showed codel alone to be somewhat ineffective against ECN, and in both pie and codel’s single queue implementations in Linux we left it disabled by default. Cake developed a more refined approach to ECN management. The sqm-scripts enable ECN for inbound universally and disable it for outbound. Our FQ_Codel implementation for WiFi, now shipping in quantity millions, enables it universally. Inconsistencies in ECN behavior abound in both AQMs and TCPs.

Much of this project will be focused on analyzing and reducing any additional congestion caused by modern protocols with ecn enabled, as well as examining potential side-effects on other protocols.

We expect much work to take place on the mailing list. Like all bufferbloat.net lists, ecn-sane is an open mailing list, however, given the level of religious advocacy of ecn elsewhere on the Internet, it has several policies that are new to bufferbloat.net.

People will be banned, after 3 warnings, from the email list, for the following reasons:

Unwillingness or inability to repeat experiments against modern versions of cubic, bbr and reno
Non-publication of sufficient code required for others to repeat your experiments
Unbridled advocacy OR outright hatred of ecn

Additionally:

Project members should identify themselves and flag experiments as part of the red, blue, yellow and purple teams, but it is not required.

Scientific skepticism of both negative and positive results is utterly required here.

We are hoping that these rules here are sufficient to keep the noise level low, and to make progress forward on this sensitive topic.

ECN-Sane Project Plans

Explore the possible negative effects of a partially or fully ecn-enabled internet on

Gaming, voice, and DNS traffic
Explore improvements to voip (higher bit resolutions, 2.5ms sample rates, etc)
Analyze Unresponsive senders
ECN-enabled DDOS attacks
What does it do VOIP MOS scores?
Side effects on routing protocols (such as Babel, ISIS and ARP)
Effects of deploying it on protocols such as bittorrent
Discover what of rfcXXXXes holds in today’s IW10 environment

What new can be accomplished by constructive use of ECN?

QUIC
MFTP
DCCP
ECN for iframes (NADA, redux)
Can low priority congestion control be made to work again?

Explore constructive means of coping with ECN for traditional UDP applications

defensive measures for an overwhelmingly ECN’d network (an example would be to ecn mark two out of five voip packets)

Explore fixes for tcp cubic, reno, and BBR and what other TCPs may apply

Recognising “loss and mark” as an even stronger signal than either
Dynamically reducing mss size at cwnd 2
Treating an ecn mark even more strongly than loss or reordering (reducing cwnd growth as either would)
Exponential & cubic backoff verses strict AIMD
Initial spreading for IW10 based ecn transports
Improving packet pacing to cope with sub cwnds
Explore dctcp behaviors on unreliable, jittery networks such as wifi
Explore behaviors against ack-filters and random ack drop

Other sub-projects

A setsocket option to selectively enable ecn
Explore improvements to existing deployed AQMs that more or less follow rfc3168
Create repeatable experiments
Create dual-licensed implementations of alternate AQM proposals such as l4s and dualQ
Attempt to incorporate working ideas from these proposals in pie, codel, fq_codel, and fq_pie.

ECN-Sane Project

bloat@lists.bufferbloat.net (The Bufferbloat community) — Fri, 24 Aug 2018 15:38:14 +0000

ECN-Sane’s rules of operation

This project intends to thoroughly evaluate the effects of ECN across the edge of the Internet.

We expect much work to take place on the mailing list. Like all bufferbloat.net lists, ecn-sane is an open mailing list, however, given the level of religious advocacy of ECN elsewhere on the Internet, it has several policies that are new to bufferbloat.net.

People will be banned, after 3 warnings, from the email list, for the following reasons:

Unwillingness or inability to repeat experiments against modern versions of cubic, bbr and reno.
Non-publication of sufficient code required for others to repeat your experiments.
Unbridled advocacy OR outright hatred of ECN.

Additionally:

Project members should identify themselves as part of the grey, yellow, pink, red, blue, purple or green teams, but it’s not required.

With a scientific approach there is no need to argue over anything with stress, or to give others coronaries, as it were. There is no shame in being “wrong”, nor everlasting glory in being “right”, about anything, if one yields to well-gathered evidence. There’s just the scientific process. The bulldozer of science.

We suggest this - and a good sense of humor! - as a part of any policy to avoid logical fallacies, and to at least admit or recognize them when we do.

We don’t see too many logical fallacies from this crowd. What we see more often is that we sometimes lack the data to draw certain conclusions, because we don’t always have the time and resources to collect it. So we use our intuition when we have to (and there are some pretty darn good ones in this group), but we do have to be careful about what final conclusions we draw. Assertions and hypotheses for sake of discussion starters should be fine, and we shouldn’t be afraid to be wrong with those, lest we freeze before saying or trying anything. We will try to avoid Argumentum ad baculum, Proof by intimidation, Thought terminating cliche’s, Single cause fallacies, Regression fallacies, Proof by repeated Assertion, and Argument from authority

Prior to now, the 21 one year long bikeshed around ECN has been around - those that favor it, those that favor it with modifications, those that are scared of it, and the vast majority are those that haven’t heard of it and/or don’t care.

TEAMS

GREY - Don’t care about ECN one way or another

Toke Høiland-Jørgensen’s position, at least until he finishes his thesis.

RED - Opposed to wide deployment of ECN

At the moment there is no-one on the red team.

PINK - ECN’s definition needs to be totally changed to deploy

This group is well populated currently outside this project. Members of that group are welcome here so long as they play by the rules.

YELLOW - Chicken about wide deployment of ECN

The PI for this is Dave Taht, as explained in his position paper.

SEERSUCKER - With a particular reservation

“I think ECN needs to be demonstrated to be needed and useful, especially given BBR.”

– Vint Cerf

PURPLE - How does ECN need to evolve to reach its potential?

The color might change. This classification can be changed. Jonathon Morton’s position and Pete Heist’s position fall in this category, as does Loganaden Velvindron.

BLUE - we are GO for ECN on EVERYTHING!

This group is well populated currently outside this project. In addition Jake Holland is blue with yellow stripes.

GREEN - ECN is safe for universal deployment

At the moment there should be nobody on the green team.

Obligatory FutureRama Snippet

Make-Wifi-Fast Project

bloat@lists.bufferbloat.net (The Bufferbloat community) — Wed, 14 Feb 2018 15:38:14 +0000

Make Wi-Fi Fast Project

This project focuses on reducing latency throughout the wifi stack, firmware, and hardware.

The Make Wi-Fi Fast Manifesto - Wi-Fi does not need to be slow!

The hardware now available for Wi-Fi can accomplish tremendous performance, but it is hobbled by software designs that guarantee high latency under load. This, in turn, dramatically lowers performance in real-world settings (multiple users, home routers, commercial access points) leading to the myth that “Wi-Fi is slow.”

We believe that the same sort of systems thinking that went on in the Bufferbloat Project can lead to performance improvements of an order of magnitude or more in Wi-Fi.

Current Status

In late 2024, we have decreased Wi-Fi latency by at least an order of magnitude, with fair sharing of airtime across fast and slow devices.

Working software is available in the Linux kernel, as well as the OpenWrt firmware that runs on off-the-shelf routers, x86 boxes, and embedded systems.
An academic paper describing the mechanism has been published at the 2017 USENIX Annual Technical Conference: Ending the Anomaly: Achieving Low Latency and Airtime Fairness in Wifi.
Ongoing work was discussed at the Linux network developers conference (NetDev 2.2) in November 2017. The session was recorded and is available from the NetDev web site.

Rationale

WiFi may be the single most successful internet access technology. It is used by over a billion people. Unregulated use has enabled an explosion of products and deployments using WiFi. Individuals can take immediate action as no network operator has to be asked to install or extend a WiFi network, and this contrasts strongly with centrally managed and deployed systems such as the cellular telephone based communications systems. One WiFi hop is between any company and a large fraction of its users; yet we have paid scant attention to how well WiFi functions, and nearly none at all at how it will continue to scale to the next 300 million hotspots, and 10 billion new users and devices in the next 4 years. WiFi devices now cost as little as US$3, Linux WiFi devices as little as US$9. A large fraction of these devices run/will run Linux, and the current Linux WiFi stack and drivers are far from optimal.

There has been little cross-fertilization between the participants of the IETF, who understand how the Internet’s end to end protocols function, and the IEEE participants in the 802.11 standards process who are primarily radio and hardware engineers. WiFi downward compatibility constraints causes increasing complexity and problems with every succeeding generation of the technology. We must attack the problems in today’s WiFi as it is between us and almost all devices, much or all of the time.

Goals of the Project

Reduce latency on a single AP, single station connected at the lowest rate (6mbits) to under 30ms under load, down from the commonly observed 600ms or more, while not sacrificing peak throughput under real world conditions
Develop new packet scheduling and AQM techniques applicable to aggregated, parking lot network types
Improve the stack sufficiently for 802.11ac MU-MIMO to actually work
Save the world

The Make Wi-Fi Fast Plan

The current working draft is at: https://docs.google.com/document/d/1Se36svYE1Uzpppe1HWnEyat_sAGghB3kE285LElJBW4/edit

What Can I Do About Bufferbloat?

bloat@lists.bufferbloat.net (The Bufferbloat community) — Fri, 10 Mar 2017 09:10:12 +0000

What Can I Do About Bufferbloat?

Bufferbloat is high latency (or lag) that occurs when there’s other traffic on your network. This means that your network isn’t always responsive - it’s wasting your time.

How does bufferbloat apply to me?

Watch the Home Internet Connections Are Unfair! (Bufferbloat) video which gives an intuitive description of Bufferbloat. Or read the more detailed Best Bufferbloat Analogy - Ever blog post.

OK - How do I get rid of Bufferbloat?

1. Measure the Bufferbloat: Use any of the tests below that measure latency both when the line is idle and during upload or download traffic.

If the test shows latency below 15-25 msec, it means that bufferbloat is under control. If a test shows higher latency, you likely have bufferbloat. For more details about testing, read the Tests for Bufferbloat page.

2. Possible Solutions: There are lots of ways to throw time or money at this problem. Most won’t work.

Your ISP would love to sell you a faster connection, but link speed isn’t the problem - it’s your router buffering more data than necessary. This adds delay that can never be cured by faster transmission rates.
Buying an expensive router (even one for “gaming”) won’t necessarily help, since many commercial, off-the-shelf router manufacturers are clueless about excess buffering in their routers.
Twiddling the router’s QoS might make a difference, but it’s a hassle, and only helps a bit.

Instead…

3. Take Control of Your Network: No one else (not your router manufacturer, not your ISP) has a strong incentive to fix Bufferbloat. But once you take control, the network will stay fixed for all time, and you can adapt to changing practices at your ISP or other vendors.

You need to find a router vendor that understands the relationship between latency/responsiveness and bufferbloat, and has firmware that uses one of the Smart Queue Management algorithms such as cake, fq_codel, PIE, or others to eliminate it. Here are some options, from easy to harder:

Enable SQM settings if your router already has them.

First, measure the link speed without SQM using one of the speed tests above. Then turn on SQM, setting the up and down speed to the measured values above. Keep running your speed test and adjusting the SQM speed settings until the latency remains low while achieving good speeds. See, for example, this description of a tuning session.
Install an off-the-shelf router with SQM Several router vendors have a clue. Here is a list of those we have found:
- OpenWrt One In November 2024, the OpenWrt project released its own router. It incorporates the fq_codel/CAKE algorithms as well those to minimize latency in Wifi. It is now available from AliExpress in many countries.
- Ubiquiti gear has fq_codel settings. People say its EdgeRouter will handle over 400 mbps.
- The eero mesh routers have a setting to “optimize for conferencing and gaming” (their term for SQM.) Their third generation devices support SQM at speeds up to a gigabit/second.
- All the GL.iNet routers run a modified version of OpenWrt that contains SQM. (Available at Amazon.)
- All Comcast/Xfinity DOCSIS 3.1 RDK-B-based gateway models have now been updated with DOCSIS-PIE AQM and all are achieving dramatically improved working latency. (See Footnote 59 of Improving Latency with Active Queue Management (AQM) During COVID-19 for model numbers.)
- Many other mesh router vendors claim to solve bufferbloat. Check their spec’s or ask them about latency.
- Untangle NG Firewall has fq_codel settings.
- ipfire.org has fq_codel settings.
- pfsense and OPNsense have fq_codel and fq_PIE settings, courtesy of FreeBSD and ipfw/dummynet
- (Regrettably, the IQrouter from Evenroute.com is no longer on the market.)
Install custom firmware. All the projects below support some kind of queue management based on FQ-CoDel and/or Cake.
- OpenWrt (version 22.03 or newer, supported device list). The Smart Queue Management guide tells how to configure the luci-app-sqm package.
- Asuswrt-Merlin (ASUS routers only). In Web GUI follow to Adaptive QoS → QoS. More customizations via Web GUI is available with CakeQOS-Merlin.
- DD-WRT
- Gargoyle
- Tomato
- Padavan (TW641’s CAKE Port) (142 legacy MediaTek/Ralink models). Brings CAKE to Padavan firmware (Linux 3.4 / 4.4) while maintaining HWNAT and SFE acceleration.
- Raspberry Pi 4 - This link provides “some assembly required” instructions for installing OpenWrt on a RPi4 and connecting it between your ISP modem and your existing router to control latency.
Call your router vendor’s support line if none of the above are possible. You have the information from the latency tests. Mention that the ping times get really high when someone is up/downloading files, and that it really hurts your network performance. Ask if they’re working on the problem. Ask when they’re going to release a firmware update that solves it.
Consider cake-autorate for variable-rate ISP links. LTE, cable modems, and Starlink can all change throughput rates from morning to evening, or even from minute to minute. The cake-autorate algorithm adapts to the current network conditions to minimize latency.

Your network’s responsiveness is in your hands…

Read More…

More about Bufferbloat

bloat@lists.bufferbloat.net (The Bufferbloat community) — Fri, 10 Feb 2017 09:16:12 +0000

More about Bufferbloat

What is Bufferbloat?

Bufferbloat is the undesirable latency that comes from a router or other network equipment buffering too much data. For a simple description of bufferbloat, read Best Bufferbloat Analogy - Ever.

For lots more details about the CoDel (and fq_codel) algorithm, see the Codel wiki at: http://www.bufferbloat.net/projects/codel/wiki/Wiki

Why does SQM work so well?

Why do cake, fq_codel, PIE, etc. and other qdisc’s work so well? These smart queue management (SQM) algorithms put each flow’s traffic into its own queue. (A “flow” is typically defined as traffic from a single IP addresses/port to another address/port.) Then the qdisc makes sure that none of the queues get “too long”. To do this, the qdisc looks at all the queues, and preferentially chooses to send packets from flows that have no/small queue. If the queue for a flow gets large, the qdisc can mark traffic with ECN, or drop a certain percentage of those packets to allow congestion avoidance to kick in for that flow. (The various queue management algorithms use different metrics to make these transmit/drop decisions, avoid starvation, etc.)

What’s wrong with simply configuring QoS?

Quality of Service (QoS) settings will help, but won’t solve bufferbloat completely. Why not? Any prioritization scheme works by pushing certain packets to the head of the queue, so they’re transmitted first. Packets farther back in the queue still must be sent eventually. New traffic that hasn’t been prioritized gets added to the end of the queue, and waits behind those previously queued packets. QoS settings don’t have any way to inform the big senders that they’re sending too fast/too much, so packets from those flows simply accumulate, increasing delay for all.

Furthermore, you can spend a lot of time updating priorities, setting up new filters, and checking to see whether VoIP, gaming, ssh, netflix, torrent, etc. are “balanced”. (There is a whole cottage industry in updating WonderShaper rule sets. They all have terrible flaws, and they don’t help a lot.) Worst of all, these rules create a maintenance hassle. Each new rule has to be adjusted in the face of new kinds of traffic. And if the router changes, or speed changes, or there’s new traffic in the mix, then they need to be adjusted again.

Setting up a Router Manually

If you can’t get SQM/fq_codel in your router, your strategy should be to adjust the settings to control queue lengths first, then think about QoS. To do this:

Remove all the QoS/Prioritization rules.
Control the amount of data queued. If your router supports BQL in the kernel and some kind of SQM/qdisc, such as fq_codel, PIE, etc., make sure they are enabled. In general, no configuration is required at all. These changes are available in modern Linux kernels, the OpenWrt and CeroWrt routers, and a growing number of other devices. This one change automatically sets up the router to work well:
- Small flows of data (pings, DNS, ssh sessions, gaming, VoIP, SYN/Ack messages for TCP/web traffic, etc.) go right through with minimal delay
- Large flows (Netflix, file uploads/downloads, filesharing, etc) automatically adjust their rates
- All traffic gets a fair share of the bottleneck traffic capacity
Measure. Try to detect if all your data types/flows are as responsive as you like. We often find that there is no need for further configuration because the fq_codel algorithm does such a good job of giving priority to the flows that aren’t sending much data.
If you can determine that some traffic needs to priority, then set up some QoS rules. The number of rules will probably be small, perhaps only applying to a couple specific traffic types.
And finally… If prioritization/QoS doesn’t solve the problem, it may be necessary to get more bandwidth. When SQM is in place, the need for prioritization typically arises when there’s already too much data to send on a long-term basis. Creating the rules simply determines which packets go to the head of the queue, and which will be sent later. If there is regularly more data than traffic capacity, QoS doesn’t really help.

RRUL Chart Explanation

bloat@lists.bufferbloat.net (The Bufferbloat community) — Tue, 27 Dec 2016 11:00:00 -0400

Explaining RRUL Charts

The Realtime Response Under Load (RRUL) test performs accurate, automated, and repeatable measurements of network performance. RRUL is especially well suited to measuring induced latency when heavy traffic is present. This latency is often called “bufferbloat”.

The charts on this page were created by Flent, a Python program that runs the RRUL (and other) tests and creates graphs of the result. This page describes the charts created by the Flent program.

An RRUL test run saturates the network by initiating eight data connections to/from a netperf server at a remote location (four simultaneous connections sending data in each direction). Each connection has a different diffserv class to compare their performance and fairness. In addition, the RRUL test measures latency across the network with ICMP or UDP pings.

The Charts

Flent produces three major types of charts:

RRUL Three-plot Charts, showing download, upload, latency in three plots
Cumulative Distribution Function (CDF) charts that show the distribution of latency readings
Box Plots show a comparison between multiple Flent test runs

RRUL Three-Plot Charts

The RRUL test runs for 70 seconds by default: 5 seconds of idle (to give a baseline), 60 seconds of full-rate data transfer, and 5 more seconds of idle. A three-plot RRUL Chart shows:

Download plot showing the average download speed (Mbits/sec, in black) of the four diffserv connections (individual colors.) Multiply the average by four to get the actual throughput/link speed.
Upload plot showing the average upload speed (Mbits/sec, black) of the four diffserv class connections (individual colors.) Multiply the average by four to get the actual throughput/link speed.
Latency plot showing the average latency (msec, in black) of all the connections.
Horizontal axis for all plots is seconds, across the test duration.

The image above shows two Flent RRUL runs, with good (on the left) and poor latency control (on the right). Here’s what the charts show:

Good latency control Notice how the black (average) plot shows constant traffic at 4x31mbps for download; upload is constant at 4x2.8mbps, and the latency is very steady around 15 msec. (Full size)
Poor latency control Although the black download and upload rates are roughly the same, each of the four connections (in color) show far more variability. But most important, the latency jumps up from ~15 msec when idle to 250 msec during the test. This indicates bufferbloat. (Full size)

Note: The measured download and upload rates tend to be lower than the full rated speed of the ISP connection. This is because the TCP Ack traffic uses a significant fraction (often as much as 20%) of the capacity in the opposite direction. This does not skew RRUL results, because all tests will have the same testing conditions.

Cumulative Distribution Function (CDF) Plots

The Cumulative Distribution Function charts (CDF) show the percentage of readings with latency below a certain time. The two charts below show the same test runs as the charts above: one (on the left) showing good latency control, while the one on the right shows poor control.

Good latency control (on the left) Notice that most latency samples are below 15 msec, and nearly 100% are less than 20 msec. (Full size)
Poor latency control (on the right) A small fraction (a few percent) of the samples are less than 50 msec; 50% are > 250 msec (12x longer); with a long tail that’s greater than a third of a second. (Full size)

Flent Box Plots

In addition to displaying the results of the current run, Flent can save those results for later analysis. The Box Plot displays the result of multiple test runs to provide a comparison.

The chart above shows four test runs (Full size):

No SQM control (green): The (total) download speed averaged a bit above 120 mbps (4 connections at 30 mbps), but the green rectangle indicated that the values between the 25th and 75th percentile varied between 110 and 140 mbps. The dotted lines show the full range of measured values were between 75 and 180 mbps.
HTB+fq_codel qdisc (orange): The average download was lower: about 75 mbps, but with less variability and latency. (The lowered speed came from the router’s inability to process packets fast enough using HTB+fq_codel.)
layer cake qdisc (purple): The more efficient layer cake algorithm allowed the router CPU to keep the download rate high while minimizing variability.
piece of cake qdisc (red): This gave a bit larger download speed bump (~130 mbps), with similar (low) variability.

Note that all four plots show that upload speeds perform at the rated ISP link speed, and the last three qdisc’s (2-4) gave very good control over latency, with “piece of cake” being a slight winner.

Many thanks to Aaron Wood for permission to use images from his blog post about setting up SQM, QoS, fq_codel, and cake. Read more at: http://burntchrome.blogspot.com/2016/12/cake-latest-in-sqm-qos-schedulers.html

Bufferbloat.net

Bufferbloat FAQs

Bufferbloat FAQs

TL;DR - Solutions for Bufferbloat

Question #1.1: Is there a fix for bufferbloat?

Question #1.2: Are there any commercial solutions I can just buy?

Question #1.3: Are there other solutions?

Question #1.4: Where can I read more?

About Bufferbloat…

Question #2.1: What is Bufferbloat?

Question #2.2: What does that mean? How could that happen?

Question #2.3: Where can bufferbloat occur?

Question #2.4: Where does bufferbloat happen in the real world?

Question #2.5: How can I tell whether I’m experiencing bufferbloat?

Question #2.6: You say there can be high latency in Wi-fi?

Question #2.7: How can I measure Wi-Fi latency?

The Science

Question #3.1: How do those algorithms minimize latency?

Question #3.2: How does this lead to fair sharing of the bottleneck?

Question #3.3: These descriptions make it sound as if bufferbloat only happens on upload (from my local network toward my ISP). Does Bufferbloat ever happen on download?

Question #3.4: How do the bufferbloat algorithms work for the download direction?

Question #3.5: How can latency be controlled in Wi-Fi?

Question #3.6: How can latency be controlled on variable-speed links?

Question #3.7: Won’t all those queues “clog up” my router’s memory?

Question #3.8: Isn’t dropping packets bad?

Question #3.9: Can’t I just tweak the QoS settings of my router?

Question #3.10: How do I put a router “in front of” my ISP’s router?

Objections We Hear

Question #4.1: But, that’s not “bufferbloat” - it’s just ordinary behavior. Of course things will get slower when there’s more traffic…

Question #4.2: But I’m the only one using the internet…

Question #4.3: It only happens some of the time…

Question #4.4: Those bufferbloat tests you hear about are bogus. They artificially add load, which isn’t a realistic test.

Question #4.5: Bufferbloat only happens when the network is 100% loaded.

Question #4.6: It’s OK. I just tell my kids/spouse not to use the internet when I’m gaming.

Question #4.7: But, I have gigabit service from my ISP

Question #4.8: I can’t believe that router manufacturers would ever allow such a thing to happen.

Question #4.9: I mean… wouldn’t router vendors want to provide the best for their customers?

Question #4.10: Why would they sell me a router that gave crappy service? They’re a big company - they must know about this stuff.

Question #4.11: Besides, I just spent $300 on a “gaming router”. It was the most expensive solution on the market…

Question #4.12: I can’t believe you’re telling me that a bunch of academics have come up with a better algorithm than commercial router developers - that company who sold me that $300 router?

Question #4.13: And then you say that I should just install some “open source firmware”? What the heck is that? And why should I believe you?

Question #4.14: What if it doesn’t solve the problem? Who will give me support? And how will I get back to a vendor-supported system?

Real-world questions

Question #5.1: Traffic Shaper Queue Limiters not helping with Bufferbloat

Question #5.2: Extreme Bufferbloat on fibre connection

Question #5.3: Bufferbloat on Wifi

Question #5.4: An Edgerouter solves latency, but seems not to handle the speed…

Question #5.5: Consider a lower tier plan from your ISP

Question #5.6: I don’t need SQM…

Question #5.7: My ISP’s router gives terrible bufferbloat…

cake-autorate

cake-autorate for variable rate links

Status

Toward a Consumer Responsiveness Metric

Toward a Consumer Responsiveness Metric

Characteristics of a Responsiveness Tool

A Proposed Metric - RPM

Non-requirements

Summary

References

CakeRecipes

Cake Recipes

Internet over TV Cable

Outbound, General Case

Inbound, General case

Internet over Telephone Lines

Outbound

Internet from a Preexisting Ethernet

Outbound

Dave Taht's Take on TCP

Dave Taht’s take on TCP

Jake Holland's Stance on ECN

Jake’s ECN Position=Blue (+Yellow Stripes)

Pete Heist's Thoughts on ECN

Pete Heist’s Thoughts on ECN

Dave Taht's Stance on ECN

Dave Taht’s take on ECN

Jonathan Morton's Take on ECN

Jonathan Morton’s Take on ECN

ECN-Sane Project