Discussion:
Win10 Updates vs cake
(too old to reply)
Rich Brown
2017-12-22 02:01:49 UTC
Permalink
I'm using LEDE 17.01.4 on my Archer C7v2. I have a 7mbps/768kbps ADSL2+ connection through Fairpoint. The modem stats page shows its "attainable rates" (kbps): 13330/1272 and Rates: 8271/1181. My SQM settings are:

Download: 7000 (kbps)
Upload: 925
Queue Disc: Cake/piece_of_cake.qos
Link Layer: ATM/44 bytes overhead
Advanced Options: default

I have noticed that Win10 updates cause the network connection to become unusable for other services/people, as if I had bufferbloat. But ping times remain stable - they jump from ~20-22 msec unloaded to 40-50 msec.

Experiments I have tried:

- Setting download speed to 5000 makes the connection usable for other people, although the ping times remain about the same (40-50 msec)

- Setting the download speed to 8600 still keeps ping times down, but that really harms other people's performance.

- The link rates (download and upload) seem to track the SQM setting, measured with both YAMon and the built-in real-time graphs. I get ~6,000 kbps with a 7000 download setting, I got ~3,000kbps at the 5000 setting. I get ~8500 kbps after setting download to 8600.

- This doesn't seem to happen when I'm downloading other kinds of files (I haven't tried torrenting files...) Downloading non-Win10 update files seems to leave the connection in a fairly responsive state.

Any thoughts? What other experiments should I make? Thanks!

Rich
Ryan Mounce
2017-12-22 02:55:03 UTC
Permalink
I've experienced this recently myself. In my case I have a 100Mbps
link from my ISP and their shaper will queue up to about 120ms worth
of packets (on top of the ~10ms baseline latency). I run cake in
ingress mode at 99.2Mbps, which has normally been enough to keep
everything in check and keep my ISP's queue empty at least in the
steady state.

Boot up a Windows 10 PC that's been unused for a few months, let it
update and bam! 100-130ms RTT, family member's Netflix stream in the
next room stalls completely, and running a quick speedtest on a
different machine (that should get ~50% share of the link under normal
circumstances with dual-dsthost) yields about 1.2Mbps on a 100Mbps
link! I performed a quick pcap while this was happening and determined
that Windows Update had started on the order of 120 parallel HTTP
downloads from 2 different Akamai cache IPs (within my ISP's network
20ms away).

The 20ms jump in latency in your case just indicates that there is a
small buffer in your DSLAM, however it is still being flooded by the
parallel transfers from the CDN.

Windows 10 probably deserves most of the blame for opening so many
parallel connections, however I think there is also some concern here
with Akamai's FastTCP not responding to congestion signals.

Regards,
Ryan Mounce
Post by Rich Brown
Download: 7000 (kbps)
Upload: 925
Queue Disc: Cake/piece_of_cake.qos
Link Layer: ATM/44 bytes overhead
Advanced Options: default
I have noticed that Win10 updates cause the network connection to become unusable for other services/people, as if I had bufferbloat. But ping times remain stable - they jump from ~20-22 msec unloaded to 40-50 msec.
- Setting download speed to 5000 makes the connection usable for other people, although the ping times remain about the same (40-50 msec)
- Setting the download speed to 8600 still keeps ping times down, but that really harms other people's performance.
- The link rates (download and upload) seem to track the SQM setting, measured with both YAMon and the built-in real-time graphs. I get ~6,000 kbps with a 7000 download setting, I got ~3,000kbps at the 5000 setting. I get ~8500 kbps after setting download to 8600.
- This doesn't seem to happen when I'm downloading other kinds of files (I haven't tried torrenting files...) Downloading non-Win10 update files seems to leave the connection in a fairly responsive state.
Any thoughts? What other experiments should I make? Thanks!
Rich
_______________________________________________
Bloat mailing list
https://lists.bufferbloat.net/listinfo/bloat
Ryan Mounce
2017-12-22 03:57:32 UTC
Permalink
Some further reading on the issue

https://www.mail-archive.com/***@nanog.org/msg87442.html
https://forums.whirlpool.net.au/archive/2530363

Regards,
Ryan Mounce
Post by Ryan Mounce
I've experienced this recently myself. In my case I have a 100Mbps
link from my ISP and their shaper will queue up to about 120ms worth
of packets (on top of the ~10ms baseline latency). I run cake in
ingress mode at 99.2Mbps, which has normally been enough to keep
everything in check and keep my ISP's queue empty at least in the
steady state.
Boot up a Windows 10 PC that's been unused for a few months, let it
update and bam! 100-130ms RTT, family member's Netflix stream in the
next room stalls completely, and running a quick speedtest on a
different machine (that should get ~50% share of the link under normal
circumstances with dual-dsthost) yields about 1.2Mbps on a 100Mbps
link! I performed a quick pcap while this was happening and determined
that Windows Update had started on the order of 120 parallel HTTP
downloads from 2 different Akamai cache IPs (within my ISP's network
20ms away).
The 20ms jump in latency in your case just indicates that there is a
small buffer in your DSLAM, however it is still being flooded by the
parallel transfers from the CDN.
Windows 10 probably deserves most of the blame for opening so many
parallel connections, however I think there is also some concern here
with Akamai's FastTCP not responding to congestion signals.
Regards,
Ryan Mounce
Post by Rich Brown
Download: 7000 (kbps)
Upload: 925
Queue Disc: Cake/piece_of_cake.qos
Link Layer: ATM/44 bytes overhead
Advanced Options: default
I have noticed that Win10 updates cause the network connection to become unusable for other services/people, as if I had bufferbloat. But ping times remain stable - they jump from ~20-22 msec unloaded to 40-50 msec.
- Setting download speed to 5000 makes the connection usable for other people, although the ping times remain about the same (40-50 msec)
- Setting the download speed to 8600 still keeps ping times down, but that really harms other people's performance.
- The link rates (download and upload) seem to track the SQM setting, measured with both YAMon and the built-in real-time graphs. I get ~6,000 kbps with a 7000 download setting, I got ~3,000kbps at the 5000 setting. I get ~8500 kbps after setting download to 8600.
- This doesn't seem to happen when I'm downloading other kinds of files (I haven't tried torrenting files...) Downloading non-Win10 update files seems to leave the connection in a fairly responsive state.
Any thoughts? What other experiments should I make? Thanks!
Rich
_______________________________________________
Bloat mailing list
https://lists.bufferbloat.net/listinfo/bloat
Jonathan Morton
2017-12-22 04:03:06 UTC
Permalink
Using the latest version of cake (cobalt branch) and - most importantly -
turning on ingress mode should help here.

- Jonathan Morton
Jonathan Morton
2017-12-22 05:26:08 UTC
Permalink
Those threads are about 18 months old. Is it still happening the same way,
or has it become a large number of well-behaved TCP streams? Cake in
ingress mode can cope with the latter pretty well.

- Jonathan Morton
Ryan Mounce
2017-12-29 04:49:01 UTC
Permalink
Post by Jonathan Morton
Those threads are about 18 months old. Is it still happening the same way,
or has it become a large number of well-behaved TCP streams? Cake in
ingress mode can cope with the latter pretty well.
At least in my case I am using the latest revision of the cobalt
branch and running cake in ingress mode.
Post by Jonathan Morton
- Jonathan Morton
Kevin Darbyshire-Bryant
2017-12-22 09:17:23 UTC
Permalink
Post by Ryan Mounce
Some further reading on the issue
https://forums.whirlpool.net.au/archive/2530363
Regards,
Ryan Mounce
Chaps,

As a result of the following conversation between Dave & myself.
Post by Ryan Mounce
Opinions on doing a PR for LEDE so they can enjoy(/crash) the fruits of our latest cobalt. They’re currently missing ingress mode & ack filtering filtering toys.
Nah. Let's get it into net-next first this time. I would hope for some
good comments.
LEDE master until very recently (a week) still had old versions of cake & tc. Since the upstreaming appears to have stalled on a technical issue, I took the opportunity to update LEDE master. I had submitted patches for 17.01 yesterday, after having let master ’stew’ a bit in case of problems, but subsequently pulled them this morning due to new arrivals (and discrepancies) in the cake repo.

As of this email - LEDE master has a full featured cake/cobalt but with a double overhead bug. LEDE 17.01 has a much older, pre ingress mode, pre ack filter version.

Once the cake repo is sorted out I can redo & resubmit the patches for both master & 1701 - and with a bit of luck we’ll all be in an even place again.

Cheers,

Kevin D-B
+44 7947 355344

012C ACB2 28C6 C53E 9775 9123 B3A2 389B 9DE2 334A
Rich Brown
2017-12-22 13:07:21 UTC
Permalink
Kevin,
Post by Kevin Darbyshire-Bryant
Once the cake repo is sorted out I can redo & resubmit the patches for both master & 1701 - and with a bit of luck we’ll all be in an even place again.
Thanks for this update. I'm subscribed to the cake github stream, so I hear about changes there.

If I understand correctly, after those patches are resubmitted, 17.01 will have the newest cake with all bells/whistles.

Would you post a note here to a) let us know when they're available, and b) how "normal people" (that is, non-developers) can get this newest code? (I'm happy to post instructions to the forum if no one beats me to it.) Thanks.

Rich
Benjamin Cronce
2017-12-28 17:48:24 UTC
Permalink
Post by Ryan Mounce
I've experienced this recently myself. In my case I have a 100Mbps
link from my ISP and their shaper will queue up to about 120ms worth
of packets (on top of the ~10ms baseline latency). I run cake in
ingress mode at 99.2Mbps, which has normally been enough to keep
everything in check and keep my ISP's queue empty at least in the
steady state.
Boot up a Windows 10 PC that's been unused for a few months, let it
update and bam! 100-130ms RTT, family member's Netflix stream in the
next room stalls completely, and running a quick speedtest on a
different machine (that should get ~50% share of the link under normal
circumstances with dual-dsthost) yields about 1.2Mbps on a 100Mbps
link! I performed a quick pcap while this was happening and determined
that Windows Update had started on the order of 120 parallel HTTP
downloads from 2 different Akamai cache IPs (within my ISP's network
20ms away).
The 20ms jump in latency in your case just indicates that there is a
small buffer in your DSLAM, however it is still being flooded by the
parallel transfers from the CDN.
Windows 10 probably deserves most of the blame for opening so many
parallel connections, however I think there is also some concern here
with Akamai's FastTCP not responding to congestion signals.
Without packet pacing, this is impossible. All TCP implementations have a
minimum window size of 2 segments. Assuming 1500 bytes per segment, that's
3,000 bytes. Divide by RTT of 20ms for 1.2Mb/s minimum bandwidth. Because
there is no pacing, aka delaying packets, the packets are sent as soon as
the ACK hits. With 120 connections, that's 144Mb/s. It can't respond to
congestion by slowing itself down because it's already as slow as it can
go. Packet pacing is very new and is slowly being standardized. Windows
just needs to reduce the number of connections.
Post by Ryan Mounce
Regards,
Ryan Mounce
Post by Rich Brown
I'm using LEDE 17.01.4 on my Archer C7v2. I have a 7mbps/768kbps ADSL2+
connection through Fairpoint. The modem stats page shows its "attainable
Post by Rich Brown
Download: 7000 (kbps)
Upload: 925
Queue Disc: Cake/piece_of_cake.qos
Link Layer: ATM/44 bytes overhead
Advanced Options: default
I have noticed that Win10 updates cause the network connection to become
unusable for other services/people, as if I had bufferbloat. But ping times
remain stable - they jump from ~20-22 msec unloaded to 40-50 msec.
Post by Rich Brown
- Setting download speed to 5000 makes the connection usable for other
people, although the ping times remain about the same (40-50 msec)
Post by Rich Brown
- Setting the download speed to 8600 still keeps ping times down, but
that really harms other people's performance.
Post by Rich Brown
- The link rates (download and upload) seem to track the SQM setting,
measured with both YAMon and the built-in real-time graphs. I get ~6,000
kbps with a 7000 download setting, I got ~3,000kbps at the 5000 setting. I
get ~8500 kbps after setting download to 8600.
Post by Rich Brown
- This doesn't seem to happen when I'm downloading other kinds of files
(I haven't tried torrenting files...) Downloading non-Win10 update files
seems to leave the connection in a fairly responsive state.
Post by Rich Brown
Any thoughts? What other experiments should I make? Thanks!
Rich
_______________________________________________
Bloat mailing list
https://lists.bufferbloat.net/listinfo/bloat
_______________________________________________
Bloat mailing list
https://lists.bufferbloat.net/listinfo/bloat
Ryan Mounce
2017-12-29 04:44:06 UTC
Permalink
Post by Benjamin Cronce
Post by Ryan Mounce
I've experienced this recently myself. In my case I have a 100Mbps
link from my ISP and their shaper will queue up to about 120ms worth
of packets (on top of the ~10ms baseline latency). I run cake in
ingress mode at 99.2Mbps, which has normally been enough to keep
everything in check and keep my ISP's queue empty at least in the
steady state.
Boot up a Windows 10 PC that's been unused for a few months, let it
update and bam! 100-130ms RTT, family member's Netflix stream in the
next room stalls completely, and running a quick speedtest on a
different machine (that should get ~50% share of the link under normal
circumstances with dual-dsthost) yields about 1.2Mbps on a 100Mbps
link! I performed a quick pcap while this was happening and determined
that Windows Update had started on the order of 120 parallel HTTP
downloads from 2 different Akamai cache IPs (within my ISP's network
20ms away).
The 20ms jump in latency in your case just indicates that there is a
small buffer in your DSLAM, however it is still being flooded by the
parallel transfers from the CDN.
Windows 10 probably deserves most of the blame for opening so many
parallel connections, however I think there is also some concern here
with Akamai's FastTCP not responding to congestion signals.
Without packet pacing, this is impossible. All TCP implementations have a
minimum window size of 2 segments. Assuming 1500 bytes per segment, that's
3,000 bytes. Divide by RTT of 20ms for 1.2Mb/s minimum bandwidth. Because
there is no pacing, aka delaying packets, the packets are sent as soon as
the ACK hits. With 120 connections, that's 144Mb/s. It can't respond to
congestion by slowing itself down because it's already as slow as it can go.
Packet pacing is very new and is slowly being standardized. Windows just
needs to reduce the number of connections.
This can't be the reason, under this extreme load the queue of my
ISP's downstream shaper is permanently full and inducing at least an
additional 100ms of latency. Assuming an RTT to Akamai of 100ms this
gives a minimum bandwidth of 0.24Mbps, or 28.8Mbps for 120
connections. That is less than 1/3 of my link capacity.

Regards,
Ryan Mounce
Post by Benjamin Cronce
Post by Ryan Mounce
Regards,
Ryan Mounce
Post by Rich Brown
I'm using LEDE 17.01.4 on my Archer C7v2. I have a 7mbps/768kbps ADSL2+
connection through Fairpoint. The modem stats page shows its "attainable
Download: 7000 (kbps)
Upload: 925
Queue Disc: Cake/piece_of_cake.qos
Link Layer: ATM/44 bytes overhead
Advanced Options: default
I have noticed that Win10 updates cause the network connection to become
unusable for other services/people, as if I had bufferbloat. But ping times
remain stable - they jump from ~20-22 msec unloaded to 40-50 msec.
- Setting download speed to 5000 makes the connection usable for other
people, although the ping times remain about the same (40-50 msec)
- Setting the download speed to 8600 still keeps ping times down, but
that really harms other people's performance.
- The link rates (download and upload) seem to track the SQM setting,
measured with both YAMon and the built-in real-time graphs. I get ~6,000
kbps with a 7000 download setting, I got ~3,000kbps at the 5000 setting. I
get ~8500 kbps after setting download to 8600.
- This doesn't seem to happen when I'm downloading other kinds of files
(I haven't tried torrenting files...) Downloading non-Win10 update files
seems to leave the connection in a fairly responsive state.
Any thoughts? What other experiments should I make? Thanks!
Rich
_______________________________________________
Bloat mailing list
https://lists.bufferbloat.net/listinfo/bloat
_______________________________________________
Bloat mailing list
https://lists.bufferbloat.net/listinfo/bloat
Mario Hock
2017-12-22 09:12:45 UTC
Permalink
Hi,

except from buffer bloat performance can also be degraded by an
increased packet loss probability.

Can you also track packet loss rates / packet loss probability? And also
throughput/progress of the Windows update connections?

If the reason actually is that FastTCP does not respond to congestion in
a fair manner (as Ryan Mounce suggested) than you should see high packet
loss (for all connections), high throughput for the Windows update
connections and very low throughput for regular TCP connections.

Also, it would be interesting to see what happens to fixed bit rate UDP
streams.

Best, Mario
Post by Rich Brown
Download: 7000 (kbps)
Upload: 925
Queue Disc: Cake/piece_of_cake.qos
Link Layer: ATM/44 bytes overhead
Advanced Options: default
I have noticed that Win10 updates cause the network connection to become unusable for other services/people, as if I had bufferbloat. But ping times remain stable - they jump from ~20-22 msec unloaded to 40-50 msec.
- Setting download speed to 5000 makes the connection usable for other people, although the ping times remain about the same (40-50 msec)
- Setting the download speed to 8600 still keeps ping times down, but that really harms other people's performance.
- The link rates (download and upload) seem to track the SQM setting, measured with both YAMon and the built-in real-time graphs. I get ~6,000 kbps with a 7000 download setting, I got ~3,000kbps at the 5000 setting. I get ~8500 kbps after setting download to 8600.
- This doesn't seem to happen when I'm downloading other kinds of files (I haven't tried torrenting files...) Downloading non-Win10 update files seems to leave the connection in a fairly responsive state.
Any thoughts? What other experiments should I make? Thanks!
Rich
_______________________________________________
Bloat mailing list
https://lists.bufferbloat.net/listinfo/bloat
Rich Brown
2017-12-22 13:13:21 UTC
Permalink
Post by Mario Hock
Can you also track packet loss rates / packet loss probability? And also
throughput/progress of the Windows update connections?
Oh, yes... I meant to add in the initial report that during times of Win10 downloads, packet loss for all services went to pieces.

I run Intermapper at home to monitor uptime/packet loss/performance of lots of devices across the net. During those downloads, packet loss frequently jumped to 5-6% for most devices.

Thanks again,

Rich
Loading...