| View previous topic :: View next topic |
| Author |
Message |
fls
Tux's lil' helper
Joined: 18 Apr 2003
Posts: 111
Location: Germany
|
 Posted: Sun Sep 05, 2004 9:19 am Post subject: HTB not keeping correct ratio for its classes [UNRESOLVED] |
 |
|
Hi people,
I´ve set up traffic shaping on our company´s router with HTB classes.
They are working as expected with one glitch on which I´m stuck since a few days. 
Here´s my (simplified) setup and the problem:
| Code: | tc qdisc add dev eth0 root handle 1: htb default 20 r2q 1
tc class add dev eth0 parent 1: classid 1:1 htb rate 360kbit ceil 360kbit [uplink from ISP is 384kbit]
tc class add dev eth0 parent 1:1 classid 1:10 htb rate 340kbit ceil 360kbit prio 1
tc class add dev eth0 parent 1:1 classid 1:20 htb rate 20kbit ceil 180kbit prio 2 |
The packets are classified with the iptables CLASSIFY target and they properly arrive at the correct chain.
The class 1:20 is constantly filled with traffic (gentoo package CD uploads on bittorrent  ) and as long as there´s no other traffic it gets those 180kbit as it should be.
But as soon as for example an FTP upload on the 1:10 chains kicks in, I expect the traffic to be shared in the ration 340:20 (=17:1) between class 1:10 and 1:20. This is the understanding I got form the manpage and the LARTC Howto.
But in reality the ratio is approx. 7:2 when both classes are saturated
The actual output of "tc -s -d class show dev eth0" after a few minutes of full traffic on both classes is:
| Code: | class htb 1:10 parent 1:1 prio 3 quantum 43520 rate 340Kbit ceil 360Kbit burst 2Kb/8 mpu 0b cburst 2Kb/8 mpu 0b level 0
Sent 7263338 bytes 5158 pkts (dropped 0, overlimits 0)
rate 34448bps 24pps backlog 4p
lended: 5147 borrowed: 7 giants: 0
tokens: -46580 ctokens: -14579
class htb 1:20 parent 1:1 prio 4 quantum 2560 rate 20Kbit ceil 180Kbit burst 1624b/8 mpu 0b cburst 1829b/8 mpu 0b level 0
Sent 2070191 bytes 1559 pkts (dropped 0, overlimits 0)
rate 9997bps 7pps backlog 65p
lended: 408 borrowed: 1086 giants: 0
tokens: -931521 ctokens: -39038 |
As you can see class 1:10 gets an average of ~34kb/s and 1:20 gets around 10kb/s which is nowhere near the desired share ratio between both of them.
If I look at the traffic with iftop then I can see that the FTP upload on 1:10 gets around 42kb/s (=340kbit) MOST OF THE TIME but in frequent intervals it drops down to something between 10 and 20 kb/s and then slowly rises again to the normal level. 
I experimented with the burst, cburst, quantum and prio options of HTB and got nowhere, always the same "drops".
Anyone of you has some idea or pointer where those frequent drops come from? I´ll be glad about every hint I can get, as it is imortant for us that as soon as there´s some customer downloading from our FTP server (this is what 1:10 is used for) they gets nearly all of our available bandwidth. Of course I´ve also a class for interactive traffic which doesn´t get blocked by 1:10 but, this is working fine so I didn´t include it here 
The box is running:
grsec-sources 2.4.27 (tried vanilla too)
iproute2-2.4.7.20020116-r1
kernel mode PPPoE with 2048/384 connection.
If you need more details I´ll happy to provide them!
Thank you for ANY ideas
Last edited by fls on Wed Sep 08, 2004 7:52 am; edited 1 time in total |
|
| Back to top |
|
 |
devon
l33t
Joined: 23 Jun 2003
Posts: 943
|
| Posted: Sun Sep 05, 2004 4:31 pm Post subject: |
|
|
| What is the "r2q" in your script? I assume a classless qdisc, but I tried looking at the iproute2/tc docs and did not find that listed. You may also want to check out the HTB User Guide for some additional information. |
|
| Back to top |
|
|
fls
Tux's lil' helper
Joined: 18 Apr 2003
Posts: 111
Location: Germany
|
| Posted: Sun Sep 05, 2004 8:56 pm Post subject: |
|
|
Thanks for your reply devon!
the r2q is NOT a qdisc but it is a (fairly undocumented as you noticed) parameter to the HTB qdisc.
It is used to calculate the quantum. Here is a minimalistic explanation from tc itself:
| Code: | tc qdisc add dev eth0 htb help
Usage: ... qdisc add ... htb [default N] [r2q N]
default minor id of class to which unclassified packets are sent {0}
r2q DRR quantums are computed as rate in Bps/r2q {10}
debug string of 16 numbers each 0-3 {0}
<snip>
|
It´s also explained a bit in the link you gave me near the end of chapter 2 (link sharing).
I have to use it because the rate in my 1:20 class is so low, that HTB computes a quantum smaller than my mtu (1492 for PPPoE) and the kernel prints a warning about this in syslog:
| Code: | | kernsrc@methusalix HTB: quantum of class 10020 is small. Consider r2q change. |
I could circumvent the problem by specifying a quantum for every class, but this is easier. And it doesn´t affect my problem
Thank for the pointer to the user guide. I´ve already read it several times, but can´t seem to find the fault in my setup. It seems like a simple setup and in fact the shaping IS WORKING but HTB doesn´t keep the rate permanently (see first post), thus making it useless. |
|
| Back to top |
|
|
fls
Tux's lil' helper
Joined: 18 Apr 2003
Posts: 111
Location: Germany
|
| Posted: Sat Sep 18, 2004 5:20 pm Post subject: |
|
|
Really no one who´s got an idea? I´d be very happy for even the smallest hint or the wildest guess
_________________
First they ignore you, then they laugh at you, then they fight you, then you win. Mahatma Ghandi |
|
| Back to top |
|
|
tnt
Veteran
Joined: 27 Feb 2004
Posts: 1227
|
| Posted: Fri Jun 10, 2005 3:27 pm Post subject: |
|
|
I've had problems whenever I've used ceil.
Try writting a script without ceil, burst and cburst, but only with priority settings. 
Something like:
| Code: | tc qdisc add dev eth0 root handle 1: htb default 20 r2q 1
tc class add dev eth0 parent 1: classid 1:1 htb rate 360kbit
tc class add dev eth0 parent 1:1 classid 1:10 htb rate 340kbit prio 1
tc class add dev eth0 parent 1:1 classid 1:20 htb rate 180kbit prio 2 |
and see how it works. In theory, prio 1 traffic (ftp upload) should suppress prio 2 traffic, but you should try on your settup to see if that works.
Please, report your results here.
_________________
gentoo user |
|
| Back to top |
|
|
fls
Tux's lil' helper
Joined: 18 Apr 2003
Posts: 111
Location: Germany
|
| Posted: Fri Jun 10, 2005 7:14 pm Post subject: |
|
|
Thanks for the suggestion
I tried changed our production setup to only specify rate and prio.
The code looks like this:
| Code: | $TC qdisc add dev $DEV root handle 1: htb default 20 r2q 1
$TC class add dev $DEV parent 1: classid 1:1 htb rate 360kbit
$TC class add dev $DEV parent 1:1 classid 1:10 htb rate 48kbit prio 1
$TC class add dev $DEV parent 1:1 classid 1:20 htb rate 340kbit prio 2
$TC class add dev $DEV parent 1:1 classid 1:30 htb rate 180kbit prio 3 |
Class 1:10 is for TCP SYN's, ACK's and DNS traffic Class 1:20 is FTP traffic Class 1:30 is Bittorrent uploads
When ceil is not given tc seems to set ceil = rate as you can see in the output of tc -s
| Code: | class htb 1:1 root rate 372000bit ceil 372000bit burst 1645b/8 mpu 0b overhead 0b cburst 1645b/8 mpu 0b overhead 0b level 7
Sent 11556013 bytes 24970 pkt (dropped 0, overlimits 0 requeues 0)
rate 199584bit 80pps backlog 0b 0p requeues 0
lended: 0 borrowed: 0 giants: 0
tokens: -2192 ctokens: -2192
class htb 1:10 parent 1:1 leaf 10: prio 1 quantum 6000 rate 48000bit ceil 48000bit burst 1605b/8 mpu 0b overhead 0b cburst 1605b/8 mpu 0b overhead 0b level 0
Sent 556282 bytes 10800 pkt (dropped 0, overlimits 0 requeues 0)
rate 18360bit 44pps backlog 0b 0p requeues 0
lended: 10800 borrowed: 0 giants: 0
tokens: 403953 ctokens: 403953
class htb 1:20 parent 1:1 leaf 20: prio 2 quantum 42500 rate 340000bit ceil 340000bit burst 1641b/8 mpu 0b overhead 0b cburst 1641b/8 mpu 0b overhead 0b level 0
Sent 992851 bytes 678 pkt (dropped 0, overlimits 0 requeues 0)
rate 96bit 0pps backlog 0b 0p requeues 0
lended: 678 borrowed: 0 giants: 0
tokens: 57468 ctokens: 57468
class htb 1:30 parent 1:1 leaf 30: prio 3 quantum 22500 rate 180000bit ceil 180000bit burst 1621b/8 mpu 0b overhead 0b cburst 1621b/8 mpu 0b overhead 0b level 0
Sent 10070114 bytes 13540 pkt (dropped 0, overlimits 0 requeues 0)
rate 181312bit 35pps backlog 0b 48p requeues 0
lended: 13492 borrowed: 0 giants: 0
tokens: -118218 ctokens: -118218 |
Now the testresults:
When Class 1:30 is fully saturated and traffic on 1:20 kicks in then 1:30 reduces to about 120kbit and 1:20 only gets the remaining ~240kbit
So prio 2 traffic is not really suppressing prio 3 traffic and I have the impression even the prio 1 traffic in class 1:10 gets slowed down by heavy traffic in the other classes.
What where your findings when you experimented with and without ceil??
Thanks for the input
_________________
First they ignore you, then they laugh at you, then they fight you, then you win. Mahatma Ghandi |
|
| Back to top |
|
|
tnt
Veteran
Joined: 27 Feb 2004
Posts: 1227
|
| Posted: Fri Jun 10, 2005 8:58 pm Post subject: |
|
|
Very sad to hear that.
Well, I've started with similar idealistic script as yours, but latter I've found that ceil thing does not work so reliable and just made 3 prio classes.
It worked better in my case, but I use very slow (single channel ISDN) line. I guess ceil couldn't work well with classes as slow as 3, 3.5 and 5KB/s.
I hope I'll get ADSL soon and I will be really disapointed if I could not control traffic the way I want.
And what to do with those prioritys if they don't work?
Don't know what to suggest now... If you have time (and will), try to set some rally agressive setup, something like:
| Code: | $TC qdisc add dev $DEV root handle 1: htb default 20 r2q 1
$TC class add dev $DEV parent 1: classid 1:1 htb rate 360kbit
$TC class add dev $DEV parent 1:1 classid 1:10 htb rate 48kbit prio 1
$TC class add dev $DEV parent 1:1 classid 1:20 htb rate 300kbit prio 2
$TC class add dev $DEV parent 1:1 classid 1:30 htb rate 12kbit ceil 180kbit prio 3 |
to try how it works with two classes without ceil (ceil = rate) and just one class that can borrow bandwidth.
(I've lowered down rates to achieve sum of 360kbps which is rate of your parent class.)
I think I've read somewhere that class that exceeds it's rate (borows some bandwidth) is served last... So, it doesn't cost anything to try out...
I'm very interested in results you will get, so please, post them here.
Any other thoughts about this topic?
_________________
gentoo user |
|
| Back to top |
|
|
fls
Tux's lil' helper
Joined: 18 Apr 2003
Posts: 111
Location: Germany
|
| Posted: Fri Jun 10, 2005 10:45 pm Post subject: |
|
|
Okay, I tried your aggressive setup and it was equally good as the best result I could get (with ceil, burst, cburst, quantum, etc. all specified)
Class 1:30 was filled by constant bittorrent traffic hovering at around 21kb/s (-> ~180kbit/s)
As soon as FTP upload on 1:20 started 1:30 went down to around 7kb/s and the FTP upload got around 35kb/s (-> 280kbit).
Pings (Class 1:10) were around 130ms - not too much above the idle pings of around 60ms
The ratio 7:35 is not too bad but after reading the theory about traffic shaping I had a much more idealistic view - just like you.
But I think with this approach we cannot get much better - at least have I dont LOTS of parameter tuning without success.
| Quote: | | Any other thoughts about this topic? |
Yes, two:
After switching the firewall to another host some month ago I noticed that the traffic shaping was much worse. But setup was same so it had to be hardware. I substituted the cheap RTL8139 card with a old 3com 3c905 and traffic shaping was okay again. Maybe it's the size of the transmit buffer in the card. Have you taken a look at the PRIO qdisc? My idea was to have a PRIO qdisc with 3 bands each having a HTB class in it to enforce maximum bandwidth per class.
That way traffic in band 3 would only be sent if band 1 and 2 were empty, band 2 would transmit if band 1 was empty and band 1 would always have top priority.
I experimented on a testmachine with its ethernet card and it worked PERFECTLY. I could have a 10MB/s filetransfer on band 2 and as soon as I started another upload on band 1 the previous upload didn't even transmit a single kilobyte per minute. Once the new upload stopped band 2 was back at its normal bandwidth.
"Wonderful" I thought and went on to implement the setup on the box with the ADSL connection. But it did not work at all - the PRIO qdisc just didn't seem to do anything. I spent some nights hunting for the error but had no success.
Maybe you want to try it and report your results?!
If so, I wish good luck
_________________
First they ignore you, then they laugh at you, then they fight you, then you win. Mahatma Ghandi |
|
| Back to top |
|
|
tnt
Veteran
Joined: 27 Feb 2004
Posts: 1227
|
| Posted: Fri Jun 10, 2005 11:21 pm Post subject: |
|
|
I'll try PRIO setup but I'm afraid I can't get any results on my 64kbps link - it's just too slow to be shaped correctly. I hope I'll get ADSL next week and do all the testing.
Btw, have you tried to lower MTU to 1000 bytes or even 576 bytes (I've read somewhere about this number).
And, in many traffic shaping scripts I've seen that tx queue on ADSL setups is reduced to 30 (!!!) instead of 100 on older linuxes and 1000 on my current Gentoo.
Maybe one of these or combination (MTU 576 and txqeue length of 30) would help... Just guessing, ofcorse...
| Code: | ippp0 Link encap:Point-to-Point Protocol
inet addr:212.124.166.141 P-t-P:212.124.160.54 Mask:255.255.255.0
UP POINTOPOINT RUNNING NOARP MTU:576 Metric:1
RX packets:1486881 errors:0 dropped:0 overruns:0 frame:0
TX packets:1540685 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:30
RX bytes:885687810 (844.6 Mb) TX bytes:197028274 (187.9 Mb) |
Try setting that by:
| Code: | | ifconfig eth0 txqueuelen 30 mtu 576 |
or whatever eth you use
_________________
gentoo user |
|
| Back to top |
|
|
bigfunkymo
Apprentice
Joined: 23 Jan 2004
Posts: 237
|
| Posted: Sat Jun 11, 2005 2:39 am Post subject: |
|
|
[edit: wrong thread  ]
_________________
[No package... Grabbing a set.]
Last edited by bigfunkymo on Sun Jun 12, 2005 6:08 am; edited 1 time in total |
|
| Back to top |
|
|
fls
Tux's lil' helper
Joined: 18 Apr 2003
Posts: 111
Location: Germany
|
| Posted: Sat Jun 11, 2005 6:58 pm Post subject: |
|
|
| Quote: | | Btw, have you tried to lower MTU to 1000 bytes or even 576 bytes (I've read somewhere about this number). |
Hm, don't have the time to try it at the moment. I also don't see how it could benefit - but I'll try when I've some time.
| tnt wrote: | | And, in many traffic shaping scripts I've seen that tx queue on ADSL setups is reduced to 30 (!!!) instead of 100 on older linuxes and 1000 on my current Gentoo. |
Well, with ADSL which is PPPoE you have two devices involved:
First, the ethernet card connected to the ADSL Modem. Here's mine:
| Code: | eth3 Link encap:Ethernet HWaddr 00:E0:81:2F:E9:3F
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:13559575 errors:0 dropped:0 overruns:0 frame:0
TX packets:12705530 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:7837124390 (7474.0 Mb) TX bytes:7399185092 (7056.4 Mb)
Interrupt:18 |
As you can see this device has the default tx queuelen of 1000
Second, there's the ppp interface:
| Code: | ppp0 Link encap:Point-to-Point Protocol
inet addr:AAA.BBB.CCC.DDD P-t-P:AAA.BBB.CCC.DDD Mask:255.255.255.255
UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1492 Metric:1
RX packets:4499949 errors:0 dropped:0 overruns:0 frame:0
TX packets:4384279 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:3
RX bytes:2284424289 (2178.5 Mb) TX bytes:2219116974 (2116.3 Mb) |
This interface has a transmit queue of only 3, so I increased it in my experiments but sadly I could not notice any difference
I'm also not entirely sure wether the big txqueue of the ethernet card is used or only the queue of the ppp link or both ...
_________________
First they ignore you, then they laugh at you, then they fight you, then you win. Mahatma Ghandi |
|
| Back to top |
|
|
fls
Tux's lil' helper
Joined: 18 Apr 2003
Posts: 111
Location: Germany
|
| Posted: Sat Jun 11, 2005 7:04 pm Post subject: |
|
|
@bigfunkymo:
I read your HOWTO. The egress shaping you're doing is the same as what we have discussed above except that you specify burst = 10k which I also tried in my experiments, but no success.
Then you do ingress shaping and that's the part abour your post I don't understand:
Why could ingress shaping solve the problem that egress shaping doesn't work perfectly?
And sorry for being dumb, but what do you mean with "your internet radio's traffic"?? You mean listening music?
_________________
First they ignore you, then they laugh at you, then they fight you, then you win. Mahatma Ghandi |
|
| Back to top |
|
|
bigfunkymo
Apprentice
Joined: 23 Jan 2004
Posts: 237
|
| Posted: Sun Jun 12, 2005 6:03 am Post subject: |
|
|
I think I posted that reply in this thread by mistake, actually... there was some guy having trouble listening to music because his brother was downloading. I had quite a bit to drink that night and I can see I got a little confused, sorry.
I am not a TC expert, but I think what you are experiencing may be related to this:
| Quote: | | I´ve also a class for interactive traffic which doesn´t get blocked by 1:10 |
if it is not blocked by 1:10, then it may be blocking 1:10. Do you have too much traffic going into this intactive traffic class? Do you have stochastic fairness enabled anywhere?
If you were not on the torrent AND you were not shaping traffic at all, would your FTP uploads work significantly faster? If not, then your problem may be caused by the remote system.
_________________
[No package... Grabbing a set.] |
|
| Back to top |
|
|
fls
Tux's lil' helper
Joined: 18 Apr 2003
Posts: 111
Location: Germany
|
| Posted: Sun Jun 12, 2005 7:18 am Post subject: |
|
|
| bigfunkymo wrote: | | I think I posted that reply in this thread by mistake, |
Okay, no problem
| Quote: | I am not a TC expert, but I think what you are experiencing may be related to this:
| Quote: | | I´ve also a class for interactive traffic which doesn´t get blocked by 1:10 |
if it is not blocked by 1:10, then it may be blocking 1:10. Do you have too much traffic going into this intactive traffic class? Do you have stochastic fairness enabled anywhere? |
This interactive class has an rate + ceil of 48kbit so it cannot really block the others, or am I wrong?
Yes, I have a SFQ qdisc attached to all HTB classes, so stochastic fairness should be given inside each class.
| Quote: | | If you were not on the torrent AND you were not shaping traffic at all, would your FTP uploads work significantly faster? |
Yes, then my upload hovers at ~375kbit/s which is nearly the advertised 384kbit/s of my ISP.
Thanks for the sugggestions!
_________________
First they ignore you, then they laugh at you, then they fight you, then you win. Mahatma Ghandi |
|
| Back to top |
|
|
tnt
Veteran
Joined: 27 Feb 2004
Posts: 1227
|
| Posted: Sun Jun 12, 2005 10:44 pm Post subject: |
|
|
As I already said, I have NO expiriance with ADSL, but now I'm waiting for one and I've read some topics about it, so:
it seems that all filtering and shaping should be done on ppp0 iface and NOT on eth0
Try clearing all tc stuff for your eth0 and set:
| Code: | tc qdisc add dev ppp0 root handle 1: htb default 20 r2q 1
tc class add dev ppp0 parent 1: classid 1:1 htb rate 360kbit ceil 360kbit
tc class add dev ppp0 parent 1:1 classid 1:10 htb rate 340kbit ceil 360kbit prio 1
tc class add dev ppp0 parent 1:1 classid 1:20 htb rate 20kbit ceil 180kbit prio 2 |
_________________
gentoo user |
|
| Back to top |
|
|
fls
Tux's lil' helper
Joined: 18 Apr 2003
Posts: 111
Location: Germany
|
| Posted: Mon Jun 13, 2005 5:14 am Post subject: |
|
|
| tnt wrote: | | it seems that all filtering and shaping should be done on ppp0 iface and NOT on eth0 |
Sorry if that was unclear, but I do all my shaping on ppp0 and nowhere else.
Just wanted to point out that the packets flow through eth0 too so maybe the txqueuelen there also matters.
Good luck with your new connection, BTW
_________________
First they ignore you, then they laugh at you, then they fight you, then you win. Mahatma Ghandi |
|
| Back to top |
|
|
tnt
Veteran
Joined: 27 Feb 2004
Posts: 1227
|
|
| Back to top |
|
|
Nossie
Apprentice
Joined: 19 Apr 2002
Posts: 181
|
|
| Back to top |
|
|
|
Networking & Security
