If you don't know, how to compile kernel with QoS (CBQ) support then check this page.
tc -d qdiscIf you'll get nothing as a result then your kernel support QoS (CBQ).
If you'll get RTNETLINK error then this mean that your kernel don't support QoS (CBQ) (or you forgot to load modules for QoS (CBQ)).
- should to write to /etc/shaper/shaper.X.cfg file a parameters of the local network interfaces - where X is a digit 0-7 meaning number of configuration file for next, working simultaneously shaperd daemons. In case, when we don't want to start some shaperds - you should to use 0 for every of configuration files.
Example of configuration file for server with: 1 local network interface eth0 with 10Mbit bandwidth and 1 internet interface ppp0 with 115Kbit bandwidth (Erricson's HIS) It does not have larger differences in configuration for other types network links (like DSL or Cable).ipchains_path=/sbin/ipchains iptables_path=/sbin/iptables tc_path=/sbin/tc timeout_ipchains_22=3600 timeout_ipchains_24=7200 timeout_iptables_24=432000 timeout_global=300 start_speed=0 even_division=no continuous_control=yes squid_support=no squid_port=8080 divide_upload=yes extended_queue=no nomasq=yes check_sum_of_bandwidth=yes class_type=cbq upload_class_type=cbq check_download_factor_cnt=5 check_upload_factor_cnt=5 remove_dead_class=yes refresh_dead_fwrule=yes burst_type=maximum cburst_type=maximum htb_burst=8 htb_cburst=8 jump_type=5 nop_factor=60 add_factor=80 speed_ext=bit delay_mesg=yes mask_mesg=yes check_firewall_counters=12 debug=0 delay=10 write_delay=1 delay_pools_download=0/100,5000/75,20000/50 delay_pools_upload=0/100,5000/75,20000/50 inter_int=ppp0;8000;92000;1000;60000;auto;1 local_int=eth0;10485760;192.168.0.0/16;192.168.1.0/24;2
Principies:- ipchains_path - ipchains path
- iptables_path - iptables path
- tc_path - tc path
- timeout_ipchains_22 - timeout of masqueraded tcp connections on 2.2.x kernel and ipchains. This value is used to set tcp timeout with ipchains -M -S command.
- timeout_ipchains_24 - timeout of masqueraded tcp connections on 2.4.x kernel and ipchains. Default is 7200 minutes. This value will not affect on real masquerade timeout. It's only used by shaperd for ignoring older connection.
- timeout_iptables_24 - timeout of masqueraded tcp connections on 2.4.x kernel and iptables. Default is 432000 seconds. This value will not affect on real masquerade timeout. It's only used by shaperd for ignoring older connection.
- timeout_global - every connection older than this value (in seconds) will be ignored by shaperd.
- start_speed - method of assigment bandwidth for new users. If 0 then initailly assigned bandwidth is counted through straight division of quantity of hispeed through quantity of downloading users (e.g. for HIS with 5 downloading users = 100000/5 that is 20000 (Kbit)). For speed_start great than 0 assigned bandwidth is always the same as lospeed * start_speed.
- even_division - 1 will turn on even division of bandwidth assigned to every working user.
- continuous_control - yes will turn on continuous bandwidth control.
- squid_support - yes will turn on co-operation with SQUID. (iptables only). You should start SQUID on the same IP number as the first IP with mask in local_int. Also it must use standard squid port 3128. Before you start SQUID you should apply special patch (description is available here) to sources and recompile it. You do not have to create CBQ class with tc command as it was described in Uzycie section. Andrzej Romanowski sent me his version of 2.5STABLE3 squid patch - squid-2.5_hit_miss_mark.patch. Added rules_olny value which force shaperd to create squid firewall rules but for visualisation only. I'm using it on system with 2 ISP interfaces and on 2'nd link I'm using squid with delay_pools without shaperd control.
- squid_port - port number of squid.
- divide_upload - yes will turn on upload bandwidth dividing.
- extended_queue - yes will turn on extended queue separately for every user. (iptables only).
- nomasq - yes will turn off masquerade connection tracking for bandwidth allocation. In this case shaperd is watching firewall counters for connection tracking.
- check_sum_of_bandwidth - yes will turn on control of total sum of assigned bandwidths.
- class_type - type of main download traffic manager class - CBQ or HTB.
- upload_class_type - type of main upload traffic manager class - CBQ or HTB.
- check_download_factor_cnt - counter will down every cycle if any user will download data with speed more than assigned. If it reach 0 - in log will appear error Can't control download bandwidth of IP and if remove_dead_class=yes option is enabled - shaperd will remove forks for this user and create it again in next turn.
- check_upload_factor_cnt - counter will down every cycle if any user will upload data with speed more than assigned. If it reach 0 - in log will appear error Can't control upload bandwidth of IP and if remove_dead_class=yes option is enabled - shaperd will remove forks for this user and create it again in next turn.
- ceil_download - (HTB only) percentage amount of download bandwidth that can be borrow by user. Amount 50 means that user can borrow from other users 50% of his bandwidth assignment.
- ceil_upload - (HTB only) percentage amount of upload bandwidth that can be borrow by user. Amount 50 means that user can borrow from other users 50% of his bandwidth assignment.
- burst_type - (For HTB,EHTB,SHTB only) burst creating mode. For current - is using current user speed. For maximum - is using maximum possible speed for user.
- cburst_type - (For HTB,EHTB,SHTB only) cburst creating mode. For current - is using current user speed. For maximum - is using maximum possible speed for user.
- htb_burst - (For HTB,EHTB,SHTB only) factor for calculation - how many data can be send to user without limits. DataSize = speed / factor. If factor is equal 0 then HTB will automaticaly calculate burst parameter.
- htb_cburst - (For HTB,EHTB,SHTB only) factor for calculation - how many data can be send to user without limits. DataSize = speed / factor. If factor is equal 0 then HTB will automaticaly calculate cburst parameter.
- remove_dead_class - yes will turn on bad class removing if shaperd can't control speed with this class.
- refresh_dead_fwrule - yes will turn on bad firewall rules removing when shaperd will remove dead class if remove_dead_class parameter is turned on.
- check_firewall_counters - value of this parameter means how often (1 - means every shaperd cycle) shaperd will control firewall counters and check if is any counter overfilled. If 0 then firewall counters control is turned off. I added this parameter because I found some servers where was a problem with overfilled firewall. After reaching 4294967295 value counter don't reset to 0 value and after it shaperd throttled all traffic. In case of this now shaperd will check firewall counters and if any will reach 2147483647 (2GB) value then all counters will be reset.
- speed_ext - unit of speed (bit, Kbit or Mbit). Remember about count of all parameters including speed onto new unit.:
- inter_int
- local_int
- also in file iplist.X should to count units if assigned individual limits for individual users
- jump_type - amount of increasing bandwidth. 0 - rest of unused bandwidth, more than is a multipler of lospeed value. So if you set it to 1 it's mean that bandwidth will be lospeed value increased, 2 means that bandwidth will be 2*lospeed value increased etc. I suggest don't use 0 value.
- nop_factor - first link used factor. Value in percent. Default value is 50.
- add_factor - second link used factor. Value in percent. Default value is 75. By manipulating with these two factors you can increase total used bandwidth.
- debug - for debugging (availabl;e debugging levels 0 - none, 1, 2, 3)
- delay - Time of expectation between next cycles (minimum of 10 seconds - at larger quantity of IP numbers can appear arrangements of longer time one needs.
- delay_mesg - yes will turn off some log information about time of creating allocations.
- mask_mesg - no will turn off log messages Can't find mask for IP.
- write_delay - After which of cycle it has to be generated file bitrate_user_sh.X.old which contains list of IP numbers and list at present assigned bandwidths (necessary only to visualization at help of promissory note kto.php - Example)
- inter_int - parameters of internet link interfaces
- ppp0 - name of internet interface.
- 8000 - lospeed - minimum guaraneed download speed.
- 92000 - hispeed - maximum possible download speed.
- 1000 - upload_lospeed - minimum guaranteed upload speed.
- 60000 - upload_hispeed - maximum possible upload speed.
- auto - IP - IP number of internet interface. "auto" will turn on automatic taking IP number of internet interface.
- 1 - CBQ class identifier (For every interface it has to be different - best next after 1).
- local_int - parameters of local network interfaces
- eth0 - name of local network interface. Do not write down here name of internet interface!
- 10485760 - speed of interface (Remember about count onto suitable unit)(10MBit=10*1024*1024 bit=10485760 if unit is bit)
- 192.168.0.0/16 - Number IP of interface with mask (in this of case it's the mask of net of class C what causes, that all transfers from numbers IP fitting to this mask will not have limitations. Thanks to it local transfer from server sent through local interfaces from local people in net was not clipped).
- 192.168.1.0/24 - target ip number with mask (mask of applied subnet on given interface - e.g. for eth0 - 192.168.1.0/24 and for eth1 - 192.168.2.0/24).
- 2 - CBQ class identifier (For every interface it has to be different - best next after 1).
Example of configuration file for server with 2 local interfaces: eth0 and eth1 with 10Mbit and 100Mbit bandwidth and applying unit of speed Kbit. Internet link ppp0 from ISP about speeds 930Kbit.
divide_upload=yes class_type=cbq upload_class_type=cbq jump_type=5 speed_ext=Kbit delay=10 write_delay=0 inter_int=ppp0;16;900;1;300;auto;1 local_int=eth0;10240;192.168.0.0/16;192.168.1.0/24;2 local_int=eth1;102400;192.168.0.0/16;192.168.2.0/24;3
Example of configuration file for server with 3 local interfaces: eth0, eth1 and ppp2 with 10Mbit, 100Mbit and 768Kbit bandwidth (for this of interface we limit also local transfers) and applying unit of speed bit. Internet link ppp0 and ppp1 from ISP about speeds 115Kbit.
divide_upload=yes class_type=cbq upload_class_type=cbq jump_type=5 speed_ext=Kbit debug=0 delay=10 write_delay=0 inter_int=eth0;8;440;1;100;auto;1 inter_int=ppp0;8;92;1;60;auto;2 local_int=eth1;10240;192.168.0.0/16;192.168.1.0/24;3 local_int=eth2;102400;192.168.0.0/16;192.168.2.0/24;4 local_int=ppp2;768;192.168.3.0/24;192.168.3.0/24;5
cd /usr/src/shaperd makeor in older versions:
gcc /usr/src/shaperd/shaperd.c -o /sbin/shaperd
192.168.1.2=eth0 ppp0 192.168.1.3=eth0 ppp0 192.168.1.4=eth0 ppp0 8000 16000 8000 16000 192.168.1.5=eth0 ppp0 192.168.2.2=eth1 ppp0 192.168.2.3=eth1 ppp0 192.168.2.4=eth1 ppp1 192.168.2.5=eth1 ppp1I introduced possibility of placing of individual limits for individual IP numbers additionally. In above been mentioned example for IP number 192.168.1.4. I put following forks: 8000 (1KB of - guaranteed minimum) and 16000 (2KB - maximum assigned speed) and second pair for upload control. Remember, that value this such myself are passed in unit of speed how put unit in file shaper.X.cfg in parameter speed_ext.
192.168.1.2=eth0 ppp0 192.168.1.3=eth0 ppp0 # IP numbers: 192.168.1.4 and 192.168.1.5 have one common allotment of bandwidth 192.168.1.4/31=eth0 ppp0 192.168.2.2=eth1 ppp0 192.168.2.3=eth1 ppp0 192.168.2.4=eth1 ppp1 192.168.2.5=eth1 ppp1Another example:
192.168.1.0/24=eth0 ppp0 192.168.2.0/24=eth1 ppp0In case of lack of registrations shaperd accepts values written in file shaper.X.cfg for every number.
- lospeed - minimum guaranteed speed. We compute it into straight way:
lospeed = hispeed / max_number_of_IPs
where:- hispeed - maximum possible upload speed.
- max_number_of_IPs - maximum quantity of IP numbers, which can work through internet link
I ask not to pass silly value of lospeed. If we have link SDI where hispeed carries out 92000 and it's 10 connected users to this net - this is nonsense given every lospeed even 16000(that is transfer 2KB/sek) because when 10 users will be worked on link and every getting allotment 16000, so after straight multiplying goes out, that link should have speed about 160Kbit! - hispeed - use speed of link no declared by provider'a but real, which it pretends to measure downloading something through longer time through internet link in direction from internet to local network.
If you do not know possibility of one's link, put declared speed onto beginning, install (and configure) programmelstat to making statistician and then overload link by some minutes (download something large - best some large files in the same time) e.g. wget is a good solution. On statisticians you will see real speed of your internet link. - upload_hispeed - similarly how from hispeed - it should write such value with which speed it was been possible to send datas through link in direction from users to internet. In case synchronic links value of this is even hispeed and in case asynchronous links it can be also the same but in case of uploading of datas with full speed it's possible to notice fall of download speed. So you should set this value on level such, on which we do not notice fall of download speed. In case DSL links value upload_speed is much lower than hispeed and you should to put it about some percentage less than declared for given type of upload link speed (e.g. for DSL 512/128 upload_speed it were been possible to try to put on 110Kbit).
tc -s class show dev eth1and for HTB you can see:
class htb 2:5 parent 2:1 prio 3 rate 214Kbit ceil 214Kbit burst 27391b cburst 1872b Sent 27025335 bytes 34960 pkts (dropped 0, overlimits 0) rate 21702bps 28pps lended: 0 borrowed: 0 giants: 0 injects: 0 tokens: 778767 ctokens: 15581and for upload:
tc -s class show dev eth0for CBQ you can see:
class cbq 1:5 parent 1:1 rate 14Kbit (bounded) prio 5 Sent 554775 bytes 4674 pkts (dropped 0, overlimits 0) borrowed 30 overactions 0 avgidle 379301 undertime 0Additional 4 classes for queueing has a numbers based on rule X:Y where X is class identifier and Y = (4*(A+4))+PRIO
A - IP line position in iplist file,
prio - priority - possible 100 (highest), 101, 102 and 103 (lowest - default for every not classified traffic).
Bellow I put example of iptables rules created by default by shaperd after start. For ipchains shaperd don't create any rules because I don't know ipchains and how to do it. Exapmle:
# download # highest priority for download - class 2:120 iptables -t mangle -A FORWARD -d 192.168.1.2 -m tos --tos 0x10 -j MARK --set-mark 2120 # medium priority for download - class 2:121 iptables -t mangle -A FORWARD -d 192.168.1.2 -m tos --tos 0x08 -j MARK --set-mark 2121 # low priority for download - 2:122 iptables -t mangle -A FORWARD -d 192.168.1.2 -m tos --tos 0x04 -j MARK --set-mark 2122 # Rest unclassified traffic will go to 2:123 class.Now example script for creating firewall rules based on /etc/shaper/queue_download_high.0 file: Analogically ypu should do the same for other queue files except queue_upload_server_* files which are still used by shaperd:
#!/bin/sh extqueue=`grep extended_queue=yes /etc/shaper/shaper.0.cfg | wc -l` IFS=';' if [ $extqueue -eq 1 ];then grep -E -v "^#|^$" /etc/shaper/queue_download_high.0 | while read prot src sport dst dport do if [ "$prot" == "*" ];then prot="!icmp" fi if [ "$src" == "*" ];then src="0/0" fi if [ "$dst" == "*" ];then dst="0/0" fi if [ "$sport" == "*" ];then sport="0:65535" fi if [ "$dport" == "*" ];then dport="0:65535" fi if [ "$prot" != "icmp" ];then iptables -t mangle -I FORWARD -p $prot -s $src --sport $sport -d $dst --dport $dport -j TOS --set-tos 0x10 else iptables -t mangle -I FORWARD -p $prot -s $src -d $dst -j TOS --set-tos 0x10 fi done fiMARK we will count from formula
M = (1000 * X) + PRIO + (4 * (A + 4))where M - Mark, X - class identifier, A - ip line position in iplist file, PRIO - 100, 101 or 102
For first user and prio=100 subclass number will be 2120. Let's check if we correctly counted Mark. After converting this value 2120 to hexadecimal system we will get 0x848, now we write:
tc -s filter show dev eth1 | grep 848and we will get as a result:
filter parent 2: protocol ip pref 10 fw handle 0x848 classid 2:120So it's correct. All packets with 0x848 MARK will go to 2:120 class. Now we can observe traffic going through this class:
watch tc -s class show dev eth1And we can see (if something was sent by this class) - for HTB:
class htb 2:120 parent 2:5 leaf a688: prio 0 rate 273Kbit ceil 273Kbit burst 15Kb cburst 1948b Sent 10644 bytes 133 pkts (dropped 0, overlimits 0) rate 11bps lended: 133 borrowed: 0 giants: 0 injects: 0 tokens: 315623 ctokens: 39721Now we create firewall rules for traffic to internet (upload):
# upload # lowest priority for upload - class 1:123 iptables -t mangle -A FORWARD -s 192.168.1.2 -d ! 192.168.0.0/16 -j MARK --set-mark 1123 # highest priority for upload - class 1:120 iptables -t mangle -A FORWARD -s 192.168.1.2 -d ! 192.168.0.0/16 -m tos --tos 0x10 -j MARK --set-mark 1120 # medium priority for upload - class 1:121 iptables -t mangle -A FORWARD -s 192.168.1.2 -d ! 192.168.0.0/16 -m tos --tos 0x08 -j MARK --set-mark 1121 # low priority for upload - class 1:122 iptables -t mangle -A FORWARD -s 192.168.1.2 -d ! 192.168.0.0/16 -m tos --tos 0x10 -j MARK --set-mark 1122The formula for counting MARK is the same as for download. You should know that upload traffic is limited on internet interface - not on local. So MARK for 192.168.1.2 and 100 priority (first IP in iplist file) is 1120 (in Hexadecimal system 0x460):
tc -s filter show dev eth0 | grep 460and we will see:
filter parent 1: protocol ip pref 10 fw handle 0x460 classid 1:120So it's ok. Now we will see traffic in this class:
watch tc -s class show dev eth0and we see:
class cbq 1:120 parent 1:5 leaf a69c: rate 18Kbit prio 3 Sent 12122 bytes 208 pkts (dropped 0, overlimits 0) borrowed 0 overactions 0 avgidle 8.07342e+06 undertime 0
- squid_port must be the same as port in squid config file.
- I tested shaperd with squid also with transparent proxy configuration and it seems tah it's working.
- shaperd for squid_support will create special firewall rules (only for iptables available):
iptables -t mangle -vxL POSTROUTING -n | grep 8080
andiptables -vxL squid0 -n
- If you are not sure is your squid correctly marking TOS field for shaperd - you can check it writing:
iptables -vxL squid0 -n | grep -E -v "destination|squid" | awk '{print $9,"\t",$2}'And if squid correctly mark TOS filed you will see (e.g.):192.168.1.3 135967 0.0.0.0/0 2074683 192.168.1.7 2432944 192.168.1.6 18121074
If all rules for users will have 0 and only 0.0.0.0/0 will count then you have problem - recompile your squid.
modprobe sch_cbq modprobe sch_tbf modprobe cls_u32For HTB you should load following modules:
modprobe sch_htb modprobe sch_sfq modprobe cls_u32
217.96.55.5 411 - IP number and port 411
213.180.130.190 - IP number nad all ports
22$ - all IP numers and port 22 (telnet)
I made this therefore, in order to daemon did not assign link onto connections generating practically no's transfer (e.g. IRC, telnet). You should know that every IP number must end with space character.
Current list taken under attention at division of numbers ip it were been possible to see writing:
/sbin/shaperd shownat
192.168.1.2 40000 192.168.1.9 58000or if speed_ext is set as Kbit:
192.168.1.2 40 Kbit 192.168.1.9 58 Kbit
tc -s qdiscResults of his working can look so:
qdisc tbf d09f: dev eth0 rate 5430bps burst 10Kb lat 1.2s
Sent 108930 bytes 73 pkts (dropped 0, overlimits 0)
qdisc tbf d09e: dev eth0 rate 5430bps burst 10Kb lat 1.2s
Sent 126618 bytes 89 pkts (dropped 0, overlimits 0)
qdisc tbf d09d: dev eth0 rate 1638bps burst 10Kb lat 3.8s
Sent 54699 bytes 65 pkts (dropped 0, overlimits 321)
backlog 6110b 5p
qdisc tbf d09c: dev eth0 rate 10Mbit burst 10Kb lat 4.8ms
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
qdisc tbf d09b: dev eth0 rate 10Mbit burst 10Kb lat 4.8ms
Sent 5690 bytes 53 pkts (dropped 0, overlimits 0)
qdisc cbq 10: dev eth0 rate 10Mbit (bounded,isolated) prio no-transmit
Sent 310829 bytes 302 pkts (dropped 0, overlimits 1555)
backlog 5p
borrowed 0 overactions 0 avgidle 624 undertime 0
Look on line from rate different from 10Mbit (or different in case of local link about different maximum speeds). In this of case of three first class illustrate transfers of three different computers in local net. Check, whether registered size of sent datas Sent are. If so this it means, that CBQ work correctly and cuts transfer.- shaperd version
- description of system (distribution, kernel version, remark)
- url of your server www page (if exists)
<p><a href="https://cbq.trzepak.net/"> <img src="https://cbq.trzepak.net/linux/pics/shaperd.gif" border="0" width="102" height="47" alt="Powered by Shaper CBQ"></a></p>
