03-LVS
1. DR配置原理
/proc虚拟目录，存放在内存。存放内核和启动的进程，linux一切皆文件，会把里面的变量和参数映射成文件。修改文件的值即改参数的值。开机有了内核有了进程，才将子目录挂载进去- 目录里的文件只能覆盖，修改会产生临时文件，这个是不被允许的
1. 隐藏vip
1. 一个网卡多个ip
- 网线两端相匹配的网络号就可以进行通讯
2. lo网卡
- Local Loopback
- 对外隐藏，对内可见（内核里的虚拟网卡）
- 域名：localhost，ip：127.0.0.1
- 跑本地代码时，前端访问后端经过的即为lo网卡
[root@bogon ~]# ifconfig
# 1. 物理对外网卡
enp2s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.*.* netmask 255.255.255.0 broadcast 192.168.90.255
inet6 fe80::506:5cc9:9d4:8a97 prefixlen 64 scopeid 0x20<link>
ether 00:e0:70:8f:2c:03 txqueuelen 1000 (Ethernet)
RX packets 106489256 bytes 93605874727 (87.1 GiB)
RX errors 0 dropped 7 overruns 0 frame 0
TX packets 66646942 bytes 61370728830 (57.1 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
# 2. 本地循环虚拟网卡。本地前端，请求本地后端，走的就是lo网卡
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 488275879 bytes 114235221146 (106.3 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 488275879 bytes 114235221146 (106.3 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
3. arp协议参数
- 默认值都为0
- 隐藏VIP方法：对外隐藏，对内可见
kernel parameter：目标mac地址为全F，交换机触发广播， /proc/sys/net/ipv4/conf/*IF*/
- arp_ignore：定义接收到ARP请求时，响应级别
- 0：只要本地配置有相应地址，就给予响应
- 1：仅在请求的目标（MAC）地址配置请求。到达的接口上的时候，才给予响应
- arp_announce：定义将自己地址向外通告时，通告级别
- 0：将本地任何接口上的任何地址向外通告
- 1：试图仅向目标网络通告与其网络匹配的地址
- 2：仅向与本地接口上地址匹配的网络进行通告
一台主机两张网卡。网卡1为物理网卡
- arp_ignore（响应）：
- 0：pkg（mac1 + ip2 || mac1 + ip1）都进行响应
- 1：pkg（mac1 + ip1）才进行响应
- arp_announce（通知）：
- 只通知网线的另一端在同一网络号上的mac
2. 调度算法
按照何种规则负载
- 四种静态
- rr：轮循
- wrr：加权轮循
- dh
- sh
- 动态调度方法
- lc：最少连接
- wlc：加权最少连接
- sed：最短期望延迟
- nq：never queue
- LBLC：基于本地的最少连接
- DH
- LBLCR：基于本地的带复制功能的最少连接
怎样负载记录（猜测）
- 偷窥，握手包（sync）分手包（fin）并登记，从而得知server上的负载。响应包（ask）没办法偷窥
- 三次握手，记录两次请求pkg
- 四次分手，记录一次请求一次响应
3. ipvs
- linux内核集成了LVS，模块名称为ipvs。需要装一个用户空间程序
# ipvs内核模块
yum install ipvsadm -y
1. 入包规则
- 一个入包规则即为一个系统的负载，可以配置多个
# 管理LVS
ipvsadm -A -t 192.168.9.100:80 -s rr
- 添加：
-A -t|u|f service-address [-s scheduler]- -t：TCP协议的集群
- -u：UDP协议的集群
- -f：FWM防火墙标记
- service-address：IP:PORT
- 修改：-E
- 删除：-D -t|u|f service-address
2. 出包规则
# 管理集群服务中的RS
ipvsadm -a -t 192.168.10.100:80 -r 192.168.10.12 –g
ipvsadm -a -t 192.168.10.100:80 -r 192.168.10.13 -g
- 添加：
-a -t|u|f service-address -r server-address [-g|i|m] [-w weight]-t|u|f service-address：事先定义好的某集群服务-r server-address：某RS的地址，在NAT模型中，可使用IP:PORT实现端口映射；[-g|i|m]： LVS类型- -g：DR
- -i：TUN
- -m：NAT
[-w weight]：定义服务器权重
- 修改：-e
- 删除：
-d -t|u|f service-address -r server-address - 查看：
-L|l- -n：数字格式显示主机地址和端口
- --stats：统计数据
- --rate：速率
- --timeout：显示tcp、tcpfin和udp的会话超时时长
- -c：显示当前的ipvs连接状况
- 删除所有集群服务
- -C：清空ipvs规则
- 保存规则：-S
ipvsadm -S > /path/to/somefile
- 载入此前的规则：-R
ipvsadm -R < /path/form/somefile
4. 架构图
- httpd就是 Web_Server
- node01：192.168.150.11
- node02：192.168.150.12
- node03：192.168.150.13
- node04：192.168.150.14
5. LVS
- 192.168.10.100/24
- 24代表：255.255.255.0
- 16代表：255.255.0.0
1. 配置网卡（物理）
node1
# 1. 网卡添加ip
ifconfig eth0:2 192.168.10.100/24
# 或者
ifconfig eth0:2 192.168.10.100 net mask 255.255.255.0
# 2. 删除网卡ip
ifconfig etho0:2 down
# 1. 网卡添加ip
[root@localhost ~]# ifconfig ens33:8 192.168.10.100/24
[root@localhost ~]# ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.10.11 netmask 255.255.255.0 broadcast 192.168.10.255
inet6 fe80::e9b4:7ddd:b4a9:ecf5 prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:b2:ac:57 txqueuelen 1000 (Ethernet)
RX packets 20853 bytes 30155026 (28.7 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 9213 bytes 597417 (583.4 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
ens33:8: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.10.100 netmask 255.255.255.0 broadcast 192.168.10.255
ether 00:0c:29:b2:ac:57 txqueuelen 1000 (Ethernet)
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 72 bytes 6256 (6.1 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 72 bytes 6256 (6.1 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
# 2. 删除网卡ip
[root@localhost ~]# ifconfig ens33:8 down
[root@localhost ~]# ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.10.11 netmask 255.255.255.0 broadcast 192.168.10.255
inet6 fe80::e9b4:7ddd:b4a9:ecf5 prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:b2:ac:57 txqueuelen 1000 (Ethernet)
RX packets 20961 bytes 30163315 (28.7 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 9261 bytes 603251 (589.1 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 72 bytes 6256 (6.1 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 72 bytes 6256 (6.1 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
2. ipvs客户端配置
# 安装ipvs
yum install ipvsadm
# 入包规则
ipvsadm -A -t 192.168.10.100:80 -s rr
# 出包规则
ipvsadm -a -t 192.168.10.100:80 -r 192.168.10.12 -g -w 1
ipvsadm -a -t 192.168.10.100:80 -r 192.168.10.13 -g -w 1
# 配置列表
ipvsadm -ln
# LVS服务配置
[root@localhost ~]# ipvsadm -A -t 192.168.10.100:80 -s rr
[root@localhost ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.10.100:80 rr
[root@localhost ~]# ipvsadm -a -t 192.168.10.100:80 -r 192.168.10.12 -g -w 1
[root@localhost ~]# ipvsadm -a -t 192.168.10.100:80 -r 192.168.10.13 -g -w 1
[root@localhost ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.10.100:80 rr
-> 192.168.10.12:80 Route 1 0 0
-> 192.168.10.13:80 Route 1 0 0
6. RS
1. 配置arp协议
- 先配置协议，不然网卡信息通知出去了
node2、node3
[root@localhost ~]# cd /proc/sys/net/ipv4/conf
[root@localhost conf]# ll
总用量 0
dr-xr-xr-x. 1 root root 0 6月 6 10:35 all
dr-xr-xr-x. 1 root root 0 6月 6 10:35 default
dr-xr-xr-x. 1 root root 0 6月 6 10:39 ens33
dr-xr-xr-x. 1 root root 0 6月 6 10:39 lo
[root@localhost conf]# cd ens33/
[root@localhost ens33]# ll
总用量 0
-rw-r--r--. 1 root root 0 6月 6 10:39 accept_local
-rw-r--r--. 1 root root 0 6月 6 10:39 accept_redirects
-rw-r--r--. 1 root root 0 6月 6 10:39 accept_source_route
-rw-r--r--. 1 root root 0 6月 6 10:39 arp_accept
-rw-r--r--. 1 root root 0 6月 6 10:39 arp_announce
-rw-r--r--. 1 root root 0 6月 6 10:39 arp_filter
-rw-r--r--. 1 root root 0 6月 6 10:39 arp_ignore
-rw-r--r--. 1 root root 0 6月 6 10:39 arp_notify
# 没办法vi，只能重定向，覆盖
echo 1 > /proc/sys/net/ipv4/conf/ens33/arp_ignore
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/ens33/arp_announce
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
# 恢复默认
echo 0 > /proc/sys/net/ipv4/conf/ens33/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/ens33/arp_announce
echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
2. 配置网卡（lo）
node2、node3
- 虚拟网卡优先于物理网卡，距离内核更近。所以必须为4个255
route -n路由表只能看到物理网卡
ifconfig lo:2 192.168.10.100 netmask 255.255.255.255
[root@localhost ~]# ifconfig lo:2 192.168.10.100 netmask 255.255.255.255
[root@localhost ~]# ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.10.12 netmask 255.255.255.0 broadcast 192.168.10.255
inet6 fe80::e9b4:7ddd:b4a9:ecf5 prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:92:d2:fe txqueuelen 1000 (Ethernet)
RX packets 1246 bytes 95459 (93.2 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 488 bytes 64790 (63.2 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 72 bytes 6256 (6.1 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 72 bytes 6256 (6.1 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo:2: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 192.168.10.100 netmask 255.255.255.255
loop txqueuelen 1000 (Local Loopback)
3. RS中的服务
yum install httpd -y
service httpd start
vi /var/www/html/index.html
# 写到index.html里的内容：from 192.168.10.1x
- 浏览器单独访问应该都没问题的：
192.168.10.12:80
7. 验证
- 浏览器访问：192.168.150.100:80 看到负载，疯狂F5
1. 握手
- node1
netstat -natp结论看不到socket连接
# lvs上没有连接
[root@localhost ~]# netstat -natp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 6803/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 7075/master
tcp 0 0 192.168.10.11:22 192.168.10.1:57045 ESTABLISHED 15317/sshd: root@pt
tcp6 0 0 :::22 :::* LISTEN 6803/sshd
tcp6 0 0 ::1:25 :::* LISTEN 7075/master
- node2、node3
netstat -natp结论看到很多的socket连接
[root@localhost ~]# netstat -natp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 6777/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 7045/master
tcp 0 52 192.168.10.12:22 192.168.10.1:56829 ESTABLISHED 16957/sshd: root@pt
tcp6 0 0 :::80 :::* LISTEN 17046/httpd
tcp6 0 0 :::22 :::* LISTEN 6777/sshd
tcp6 0 0 ::1:25 :::* LISTEN 7045/master
tcp6 0 0 192.168.10.100:80 192.168.10.1:57668 TIME_WAIT - # socket连接
[root@localhost ~]# netstat -natp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 192.168.10.100:80 192.168.10.1:57669 SYN_RECV -
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 6799/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 7071/master
tcp 0 52 192.168.10.13:22 192.168.10.1:57000 ESTABLISHED 16979/sshd: root@pt
tcp6 0 0 :::80 :::* LISTEN 17064/httpd
tcp6 0 0 :::22 :::* LISTEN 6799/sshd
tcp6 0 0 ::1:25 :::* LISTEN 7071/master
tcp6 0 0 192.168.10.100:80 192.168.10.1:57661 TIME_WAIT - # socket连接
2. 偷窥记录
- node1，查看偷窥记录本
- virtual：ipvs规则
# lvs偷窥记录本
[root@localhost ~]# ipvsadm -lnc
IPVS connection entries
pro expire state source virtual destination
TCP 00:50 FIN_WAIT 192.168.10.1:57668 192.168.10.100:80 192.168.10.12:80
TCP 00:29 FIN_WAIT 192.168.10.1:57660 192.168.10.100:80 192.168.10.12:80
TCP 00:50 FIN_WAIT 192.168.10.1:57669 192.168.10.100:80 192.168.10.13:80
TCP 00:29 FIN_WAIT 192.168.10.1:57661 192.168.10.100:80 192.168.10.13:80
- node2，网卡卸载，只能收到LVS的PKG，无法响应
[root@localhost ~]# ifconfig lo:2 down
[root@localhost ~]# ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.10.12 netmask 255.255.255.0 broadcast 192.168.10.255
inet6 fe80::e9b4:7ddd:b4a9:ecf5 prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:92:d2:fe txqueuelen 1000 (Ethernet)
RX packets 4414 bytes 3489539 (3.3 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1881 bytes 224222 (218.9 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 72 bytes 6256 (6.1 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 72 bytes 6256 (6.1 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
- node2，没有办法响应握手
[root@localhost ~]# ipvsadm -lnc
IPVS connection entries
pro expire state source virtual destination
TCP 15:00 ESTABLISHED 192.168.10.1:57730 192.168.10.100:80 192.168.10.13:80
TCP 00:58 SYN_RECV 192.168.10.1:57729 192.168.10.100:80 192.168.10.12:80 # 12无法响应握手
- state
- FIN_WAIT： 连接过，偷窥了包
- SYN_RECV：握手失败。基本上LVS都记录了，证明LVS没事，一定是后边网络层出问题