运维笔记43 使用saltstack配置完整线上服务(haproxy+keepalived,nginx+php+memcache,mysql主从)

r囧r小猫 2022-06-17 06:43 121阅读 0赞

# **概述**: #

之前我们所介绍过了很多实用的服务，有负载均衡类的，web服务类的，数据库类的等等，这些服务有的配置容易，有的配置困难，那我们现在设想一下在生成环境中，有上百台，甚至上千台服务器的情况下，难道要我们去挨个去配置每一台服务器吗，这是无法想象的，所以有了saltstack这类自动化运维工具的出现，帮助我们高效快速的部署服务。

# **线上服务拓扑** #

![这里写图片描述][SouthEast]  
这次的拓扑基本上已经可以胜任一个相当活跃的服务的后端架构了，但是我们一般没有这么多机器，就算是用虚拟机资源也很有可能不够，为了能在一台机器上完成此架构，我们决定做如下分配。  
**haprxoy1:**  
cpu:1  
ram:512MB  
硬盘：20GB  
运行服务:haproxy+keepalived+salt\_master  
ip:172.25.3.101  
**haproxy2：**  
cpu:1  
ram:512MB  
硬盘：20GB  
运行服务:haproxy+keepalived+salt\_minion  
ip:172.25.3.102  
**lnmp1：**  
cpu:1  
ram:1024MB  
硬盘:20GB  
运行服务:nginx+php\_fpm+memchached+mysql\_master+salt\_minion  
ip:172.25.3.103  
**lnmp2：**  
cpu1:  
ram:1024MB  
硬盘:20GB  
运行服务:nginx+php\_fpm+memcached+mysql\_slave+salt\_minion  
ip:172.25.3.104

# 一.saltstack配置 #

我们先简要讲解下salt的安装配置与使用，这部分涉及的机器与上面拓扑图中的无关  
salt1.mo.com : 172.25.3.100  
salt2.mo.com：172.25.3.101  
**1.配置host文件**  
/etc/hosts

172.25.3.100 salt1.mo.com
    172.25.3.101 salt2.mo.com

修改hostname  
salt1:

[root@salt1 ~]# cat /etc/sysconfig/network
    NETWORKING=yes
    HOSTNAME=salt1.mo.com

salt2:

[root@salt2 ~]# cat /etc/sysconfig/network
    NETWORKING=yes
    HOSTNAME=salt2.mo.com

**2.安装**  
salt1相当于salt的服务端，我们需要安装salt-master与salt-minion，而salt2相当于salt客户端，在之后的集群部署中，一般salt服务端只有一个，但是客户端可以有很多，服务端与客户端通信，从完成相关配置文件，软件包的配置与部署。  
由于我们的机器不能与外网连接，所以我将saltstack需要的rpm包全下载了下来制作成一个yum源在172.25.3.250这台机器上。所以我们需要部署salt的机器上的yum的配置如下。

[salt]
    name=saltstack_soft
    baseurl=ftp://172.25.3.250/pub/saltstack/saltstack/rhel6
    enabled=1
    gpgcheck=0

开始安装  
salt1:

[root@salt1 ~]# yum install -y salt-master salt-minion

slat2:

[root@salt2 ~]# yum install -y salt-minion

**3.配置**  
salt的配置也十分简单，master不用动，只需修改minion，使其知道自己的msater是谁即可。

vim /etc/salt/minion
    
    master: 172.25.3.100

**4.启动服务**  
salt1:

[root@salt1 ~]# /etc/init.d/salt-master status
    salt-master (pid  1022) is running...
    [root@salt1 ~]# /etc/init.d/salt-minion status
    salt-minion:root:salt1.mo.com is running: 2332

salt2:

[root@salt2 ~]# /etc/init.d/salt-minion status
    salt-minion:root:salt2.mo.com is running: 1355

**5.配置认证**  
**1）.服务端与客户端认证连接**

[root@salt1 ~]# salt-key 
    Accepted Keys:
    Denied Keys:
    Unaccepted Keys:
    Rejected Keys:

直接输入salt-key查看现在key的情况，只有出现在Accepted Keys中的才是允许的客户端。  
现在我们将自己salt1与salt2添加进去。

[root@salt1 master]# salt-key -a salt1.mo.com
    The following keys are going to be accepted:
    Unaccepted Keys:
    salt1.mo.com
    Proceed? [n/Y] y           
    Key for minion salt1.mo.com accepted.
    
    [root@salt1 master]# salt-key -a salt2.mo.com
    The following keys are going to be accepted:
    Unaccepted Keys:
    salt2.mo.com
    Proceed? [n/Y] y
    Key for minion salt2.mo.com accepted.

当出现如下报错时:

The key glob 'salt2.mo.com' does not match any unaccepted keys.

可能是你之前接收过该节点，却又删除了，请重启相应节点的salt-minion即可。  
现在再查看一遍，key的接收情况

[root@salt1 master]# salt-key 
    Accepted Keys:
    salt1.mo.com
    salt2.mo.com
    Denied Keys:
    Unaccepted Keys:
    Rejected Keys:

说明：-a ：accept ，-A：accept-all，-d：delete，-D：delete-all。可以使用 salt-key 命令查看到已经签名的客户端。此时我们在客户端的 /etc/salt/pki/minion 目录下面会多出一个minion\_master.pub 文件。

**2）测试通讯是否正常**

[root@salt1 master]# salt '*' test.ping salt1.mo.com:
        True
    salt2.mo.com:
        True

如上显示，证明通讯正常  
**3)远程执行命令测试**

[root@salt1 master]# salt '*' cmd.run 'ip addr show'
    salt2.mo.com:
        1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
            link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
            inet 127.0.0.1/8 scope host lo
            inet6 ::1/128 scope host 
               valid_lft forever preferred_lft forever
        2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
            link/ether 52:54:00:f0:10:ce brd ff:ff:ff:ff:ff:ff
            inet 172.25.3.101/24 brd 172.25.3.255 scope global eth0
            inet6 fe80::5054:ff:fef0:10ce/64 scope link 
               valid_lft forever preferred_lft forever
    salt1.mo.com:
        1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
            link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
            inet 127.0.0.1/8 scope host lo
            inet6 ::1/128 scope host 
               valid_lft forever preferred_lft forever
        2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
            link/ether 52:54:00:6c:98:48 brd ff:ff:ff:ff:ff:ff
            inet 172.25.3.100/24 brd 172.25.3.255 scope global eth0
            inet6 fe80::5054:ff:fe6c:9848/64 scope link 
               valid_lft forever preferred_lft foreve

上面执行的命令中\*表示所有已经被接受过的客户端。

# 二.saltstack应用 #

**1.配置双机haproxy**  
haproxy可以直接使用yum安装，所以我们直接使用salt自带的pkg模块进行haproxy的配置。而且为了将配置文件修改成负载均衡模式，我们还要使用文件管理模块。

**1）配置**  
打开如下注释内容，使salt-master知道，相应部署配置文件的定义位置。

[root@salt1 ~]# vim /etc/salt/master
    file_roots:
      base:
        - /srv/salt/

**2)配置top文件，确定执行顺序**  
我们业务的执行顺序是，安装包，同步文件，所以top.sls文件内容如下:

base:
        'salt2.mo.com':
        - install_pkg     - conf

**3)配置包服务最终状态与配置文件的同步**  
install\_pkg.sls

haproxy-service:
        pkg.installed:
            - names:             -haproxy     service.running:
            - name: haproxy         - enable: True

conf.sls:

[root@salt1 salt]# cat conf.sls 
    haproxy-conf:
        file.managed:
            - name: /etc/haproxy/haproxy.cfg         - source: salt://haproxy.cfg         - user: root         - group:root         - mode: 777

**4）执行**

[root@salt1 salt]# salt 'salt2.mo.com' state.highstate
    salt2.mo.com: ----------
     ID: haproxy-service
     Function: pkg.installed
     Name: haproxy
     Result: True
     Comment: The following packages were installed/updated: haproxy
     Started: 17:57:11.655588
     Duration: 6911.527 ms
     Changes: 
     ----------
     haproxy:
     ----------
     new:
     1.4.24-2.el6
     old: ----------
     ID: haproxy-service
     Function: service.running
     Name: haproxy
     Result: True
     Comment: Service haproxy has been enabled, and is running
     Started: 17:57:18.580255
     Duration: 332.323 ms
     Changes: 
     ----------
     haproxy:
     True ----------
     ID: haproxy-conf
     Function: file.managed
     Name: /etc/haproxy/haproxy.cfg
     Result: True
     Comment: File /etc/haproxy/haproxy.cfg updated
     Started: 17:57:18.916540
     Duration: 155.754 ms
     Changes: 
     ----------
     diff:
     --- 
     +++ 
     @@ -84,3 +84,7 @@
     server app3 127.0.0.1:5003 check
     server app4 127.0.0.1:5004 check
    
     +listen mo *:80
     + balance roundrobin 
     + server static 172.25.3.250 check 
     + server static 172.25.3.101 check 
     mode:
     0777
    
    Summary for salt2.mo.com ------------
    Succeeded: 3 (changed=3)
    Failed: 0 ------------
    Total states run:     3
    Total run time:   7.400 s

执行成功，去salt2上看下是否正确执行。

[root@salt2 html]# /etc/init.d/haproxy status
    haproxy (pid  2736) is running...
    [root@salt2 html]# tail -5 /etc/haproxy/haproxy.cfg 
    
    listen mo *:80
        balance     roundrobin  
        server      static 172.25.3.250 check 
        server      static 172.25.3.101 check

**5）.注意事项**  
编写salt配置文件一定要注意缩进与空格

1.haproxy-service:
    2. pkg.installed:
    3. - names:
    4. -haproxy
    5. service.running:
    6. - name:haproxy
    7. - enable: True

比如上面就是一个错误的配置，错误出现在第四行’-‘与’haproxy’一定要有空格，第六行’:’与’haproxy’也要有空格。

**2.配置keepalived双机集群**  
keepalived需要源码安装，所以刚才的使用pkg安装模块不能使用了，需要我们使用命令来安装了。首先把部署逻辑想好，这是十分重要的。  
现在有salt1与salt2两台机器。salt1作为keepalived主,salt2作为keepalived从。  
部署逻辑:  
**安装keepalived：**  
将源码包同步到salt1/salt2  
解压源码包  
解决源码包安装时的依赖条件  
源码包安装  
配置keepalived环境  
**配置keepalived:**  
根据master与salve不同，同步不同的配置文件  
同步监控脚本文件  
根据上述逻辑，写好的salt脚本逻辑如下:

[root@salt1 salt]# tree /srv/salt/
    /srv/salt/
    ├── conf1.sls
    ├── conf-keep-master.sls
    ├── conf-keep-slave.sls
    ├── conf.sls
    ├── haproxy
    │   └── haproxy.cfg
    ├── install-keep.sls
    ├── install_pkg.sls
    ├── keepalived
    │   ├── check_haproxy
    │   ├── keepalived-1.2.24.tar.gz
    │   ├── keepalived-master.conf
    │   └── keepalived-slave.conf
    └── top.sls

top.sls:

base:
        'salt2.mo.com':
            - install-keep         - conf-keep-slave     'salt1.mo.com':
            - install-keep         - conf-keep-master

install-keep.sls:

#install keepalived
    keepalived-source:
        file.managed:
            - name: /root/keepalived-1.2.24.tar.gz         - source: salt://keepalived/keepalived-1.2.24.tar.gz         - user: root         - group: root         - mode: 644 #tar source keepalived
    keepalived-tar:
        cmd.run:
            - cwd: /root         - names:             - tar -xf keepalived-1.2.24.tar.gz         - user: root #install keepalived require
    keepalived-require:
        pkg.installed:
            - pkgs:             - gcc             - openssl-devel #install keepalived in source package
    keepalived-source-install:
        cmd.run:
            - cwd: /root/keepalived-1.2.24         - names:             - ./configure --prefix=/usr/local/keepalived             - make             - make install         - require:             - cmd: keepalived-tar             - pkg: keepalived-require         - unless: test -d /usr/local/keealived #configure keepalived environment
    keepalived-env:
         cmd.run:
             - names:              - ln -s /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/              - chmod a+x /etc/init.d/keepalived              - ln -s /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/              - ln -s /usr/local/keepalived/sbin/keepalived /sbin/              - ln -s /usr/local/keepalived/etc/keepalived /etc/          - require:              - cmd: keepalived-source-install

conf-keep-master.sls:

#keepalived conf master
    keepalived-conf:
        file.managed:
            - name: /etc/keepalived/keepalived.conf         - source: salt://keepalived/keepalived-master.conf         - user: root         - mode: 755 #keepalied healthy check script
    keepalived-check-script:
        file.managed:
            - name: /opt/check_haproxy         - source: salt://keepalived/check_haproxy         - user: root         - mode: 755

conf-keep-slave与master只是修改了不同的keepalived配置文件。

我们整个线上服务的配置无外乎上面两种使用salt的方式，如果能理解上面的脚本，那么使用salt配置该服务业就非常容易了，下面让我们进入正题。

# **三.使用salt配置完整线上服务** #

**1.前端高可用负载均衡业务的部署**  
根据刚才二中所描述的，我们把两步和在一起，相当于把我们的线上系统的两台前端haproxy负载均衡集群自动化部署好。  
salt目录的结构如下:

/srv/salt/
    ├── conf-haproxy.sls            
    ├── conf-keep-master.sls
    ├── conf-keep-slave.sls
    ├── haproxy
    │   └── haproxy.cfg
    ├── install-haproxy.sls
    ├── install-keep.sls
    ├── keepalived
    │   ├── check_haproxy
    │   ├── keepalived-1.2.24.tar.gz
    │   ├── keepalived-master.conf
    │   └── keepalived-slave.conf
    └── top.sls

**top.sls:**

base:
        'front2.mo.com':
            - install-haproxy         - conf-haproxy         - install-keep         - conf-keep-slave     'front1.mo.com':
            - install-haproxy         - conf-haproxy         - install-keep         - conf-keep-master

仍然是先理清主要思路，我们前端的两台负载均衡的host分别是front1.mo.com与front2.mo.con，因为还要使用keepalived作高可用，所以配置文件会有所不同，我们将其分成了两个部分。  
**install-haproxy:**安装haprxoy包并使haproxy开启

haproxy-service:
        pkg.installed:
            - names:             - haproxy     service.running:
            - name: haproxy         - enable: True

**conf-haproxy:**将salt-masters上haproxy配置文件同步至salt-minoin上。并重启haproxy服务

haproxy-conf:
        file.managed:
            - name: /etc/haproxy/haproxy.cfg         - source: salt://haproxy/haproxy.cfg         - user: root         - group: root         - mode: 777 haproxy-restart:
        cmd.run:
            - names:             - /etc/init.d/haproxy restart

**install-keep:**负责分发keepalived源码包，并解压，编译，部署keepalived环境

#install keepalived
    keepalived-source:
        file.managed:
            - name: /root/keepalived-1.2.24.tar.gz         - source: salt://keepalived/keepalived-1.2.24.tar.gz         - user: root         - group: root         - mode: 644 #tar source keepalived
    keepalived-tar:
        cmd.run:
            - cwd: /root         - names:             - tar -xf keepalived-1.2.24.tar.gz         - user: root         - unless: test -d keepalived-1.2.24 #install keepalived require
    keepalived-require:
        pkg.installed:
            - pkgs:             - gcc             - openssl-devel #install keepalived in source package
    keepalived-source-install:
        cmd.run:
            - cwd: /root/keepalived-1.2.24         - names:             - ./configure --prefix=/usr/local/keepalived             - make             - make install         - require:             - cmd: keepalived-tar             - pkg: keepalived-require         - unless: test -d /usr/local/keealived #configure keepalived environment
    keepalived-env:
         cmd.run:
             - names:               - chmod a+x /usr/local/keepalived/etc/rc.d/init.d/keepalived               - ln -s /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/               - ln -s /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/               - ln -s /usr/local/keepalived/sbin/keepalived /sbin/               - ln -s /usr/local/keepalived/etc/keepalived /etc/          - require:              - cmd: keepalived-source-install          - unless: test -e /etc/init.d/keepalived          - user: root

conf-keep-master.sls/conf-keep-slave.sls:两个配置文件基本相似，只是同步的配置文件不同，负责同步keepalived配置文件，同步keepalived脚本，最后开启keepalived，这里配置的时候发现了一个坑，总结起来是因为自己基础问题没有搞清楚，**就是软连接的权限问题，将一个脚本软连接到另一个目录下，如果源文件没有执行权限，就算给了软连接后文件执行权限也是没有用的。**

**2.整个后端服务的搭建**  
后端服务其实就是一个lnmp框架，nginx采用源码安装方式，我们套用二中方法既可，php采用yum安装，mysql采用yum安装并且进行主从的配置，其中在Mysql取随机密码的时候踩了个坑，就是Yaml格式对:的识别问题，因为awk若要以’:’分割可以用如下写法

awk -F: '{print $NF}'

但是这个:会被识别为别的符号，所以我们只有采用下面的写法，才不会报错

awk -F ':' '{print $NF}'
    还有关于mysql5.7你第一次想在命令行运行sql语句(也就是使用-e参数)时候需要添加一个参数就是'--connect-expired-password'才可以。

现在看一下整体的saltsatck脚本树。

/srv/salt/
    ├── conf-haproxy.sls
    ├── conf-keep-master.sls
    ├── conf-keep-slave.sls
    ├── conf-mysql-master.sls
    ├── conf-mysql-slave.sls
    ├── conf-nginx.sls
    ├── conf-php.sls
    ├── haproxy
    │   └── haproxy.cfg
    ├── install-haproxy.sls
    ├── install-keep.sls
    ├── install-memcached.sls
    ├── install-mysql.sls
    ├── install-nginx.sls
    ├── install-php.sls
    ├── keepalived
    │   ├── check_haproxy
    │   ├── keepalived-1.2.24.tar.gz
    │   ├── keepalived-master.conf
    │   └── keepalived-slave.conf
    ├── mysql
    │   ├── my.cnf
    │   ├── my-master.cnf
    │   ├── my-slave.cnf
    │   └── mysql.repo
    ├── nginx
    │   ├── nginx-0.7.69.tar.gz
    │   └── nginx.conf
    ├── php
    │   ├── php.ini
    │   └── php.repo
    ├── test
    ├── top.sls
    ├── top.sls.bak
    └── top.sls.bak2

**top.sls:**整个后端的安装逻辑

[root@salt1 salt]# cat top.sls.bak2 
    base:
         'salt1.mo.com':
            - install-nginx         - conf-nginx         - install-php         - conf-php         - install-memcached         - install-mysql         - conf-mysql-master      'front1.mo.com'
            - install-nginx         - conf-nginx         - install-php         - conf-php         - install-memcached         - install-mysql         - conf-mysql-slave

\*\*install-nginx:\*\*nginx源码文件的同步，源码包的解压，nginx源码安装依赖的解决，编译nginx

#copy nginx source code package 
    copysource-nginx:
        file.managed:
            - name: /root/nginx-0.7.69.tar.gz         - user: root         - mode: 755         - source: salt://nginx/nginx-0.7.69.tar.gz         - unless: test -e /root/nginx-0.7.69.tar.gz #extra nginx package
    tar-nginx:
         cmd.run:
             - cwd: /root          - names:              - tar -xf nginx-0.7.69.tar.gz          - onlyif: test -e /root/nginx-0.7.69.tar.gz #solove nginx source make require
    require-nginx:
         pkg.installed:
             - pkgs:              - gcc              - pcre-devel #configure and make nginx
    compile-nginx:
         cmd.run:
             - cwd: /root/nginx-0.7.69          - names:              - ./configure --prefix=/usr/local/nginx --with-http_stub_status_module              - make && make install          - require:              - cmd: tar-nginx              - pkg: require-nginx

**conf-nginx:**同步nginx的配置文件，部署nginx运行环境

#nginx configure file 
    conf-nginx:
        file.managed:
            - name: /usr/local/nginx/conf/nginx.conf         - user: root         - mode: 755         - source: salt://nginx/nginx.conf #running nginx
    execute-nginx:
        cmd.run:
            - names:             - ln -s /usr/local/nginx/sbin/nginx /usr/sbin && nginx         - user: root

**install-php:**同步php安装yum源，说明需要安装的php包

yumrepo-php:
        file.managed:
           - name: /etc/yum.repos.d/php.repo        - user: root        - mode: 755        - source: salt://php/php.repo        - unless: test -e /etc/yum.repos.d/php.repo install-php:
        pkg.installed:
           - pkgs:            - php-5.3.3-49.el6            - php-fpm-5.3.3-49.el6            - php-cli-5.3.3-49.el6            - php-mysql-5.3.3-49.el6            - php-common-5.3.3-49.el6            - php-pdo-5.3.3-49.el6     service.running:
            - name: php-fpm         - enable: True

conf-php:同步php配置文件

**conf-phpini:**
        file.managed:
            - name: /etc/php.ini         - user: root         - mode: 755         - source: salt://php/php.ini

**install-memcached:**安装memcached，并重启php-fpm

#memcache pkg install
    install-memcached:
        pkg.installed:
            - pkgs:             - memcached             - php-pecl-memcache     service.running:
            - name: memcached         - enable: True #reboot php-fpm
    restart-phpfpm:
        cmd.run:
            - names:             - /etc/init.d/php-fpm restart

**install-mysql:**同步mysql5.7yum源，说明安装哪些mysql包

#copy mysql yum pkg file
    copypkg-mysql:
        file.managed:
            - name: /etc/yum.repos.d/mysql.repo         - user: root         - mode: 755         - source: salt://mysql/mysql.repo #yum pkg install mysql
    
        pkg.installed:
            - pkgs:             - mysql-community-server-5.7.18     service.running:
            - name: mysqld         - enable: True

**conf-mysql-master:**同步mysql-master的配置文件，重启mysql，更新mysql密码，创建repl用户。

#conf mysql masetr
    conf-mysql:
        file.managed:
            - name: /etc/my.cnf         - user: root         - mode: 755         - source: salt://mysql/my-master.cnf #restart mysql
    restart-mysql:
        cmd.run:
            - name: /etc/init.d/mysqld restart         - user: root 
    #modify ini-password
    updatepass-mysql:
        cmd.run:
            - name: pass=`cat /var/log/mysqld.log | grep 'temporary password'| awk -F ':' '{print $NF}'` && mysql -uroot -p`echo $pass` --connect-expired-password -e "alter user root@localhost identified by 'Redhat+001';"         - user: root #create repl host
    repluser-mysql:
        cmd.run:
           - names:            - mysql -uroot -p'Redhat+001' -e "grant replication slave on *.* to 'repl'@'localhost' identified by 'Redhat+001'"

**mysql-slave:**同步mysql-slave配置文件，重启mysql，重置mysql密码，设置mysql-master

#conf mysql masetr
    conf-mysql:
        file.managed:
            - name: /etc/my.cnf         - user: root         - mode: 755         - source: salt://mysql/my-slave.cnf #restart mysql
    restart-mysql:
        cmd.run:
            - name: /etc/init.d/mysqld restart         - user: root 
    #modify ini-password
    updatepass-mysql:
        cmd.run:
            - name: pass=`cat /var/log/mysqld.log | grep 'temporary password'| awk -F ':' '{print $NF}'` && mysql -uroot -p`echo $pass` --connect-expired-password -e "alter user root@localhost identified by 'Redhat+001';"         - user: root #change master to master
    repluser-mysql:
        cmd.run:
            - names:             - mysql -uroot -p'Redhat+001' -e "CHANGE MASTER TO MASTER_HOST='172.25.3.100',MASTER_USER='repl',MASTER_PASSWORD='Redhat+001',MASTER_AUTO_POSITION=1;" && mysql -uroot -p'Redhat+001' -e 'START SLAVE;'

[SouthEast]: /images/20220617/c5383c1c4d5c41d4aad87a462d2fd843.png