MinIO分布式对象存储

朴灿烈づ我的快乐病毒、 2022-11-21 15:17 598阅读 0赞

MinIO 分布式部署需要4块磁盘，可以是4台服务器分别挂载一块磁盘，也可以是2台服务器分别挂载两块磁盘，可参考：[https://docs.min.io/cn/deploy-minio-on-docker-swarm.html][https_docs.min.io_cn_deploy-minio-on-docker-swarm.html]

MinIO 分布式部署支持Docker Compose、Docker Swarm、Kubernetes等几种方式，Docker-Compose是单机多实例，伪分布式，可测试使用，因没有kubernetes环境，这里使用Docker Swarm

有3台服务器，一台docker swarm manger ，两台worker

<table> 
 <thead> 
  <tr> 
   <th>主机IP</th> 
   <th>角色</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>172.16.0.5</td> 
   <td>manager</td> 
  </tr> 
  <tr> 
   <td>172.16.0.2</td> 
   <td>worker</td> 
  </tr> 
  <tr> 
   <td>172.16.0.8</td> 
   <td>woker</td> 
  </tr> 
 </tbody> 
</table>

*  1.在 172.16.0.5 机器上使用：docker swarm init --advertise-addr 172.16.0.5
 *  2.一旦swarm初使化了，你可以看到下面的响应信息，复制这段信息，在两台worker节点上执行

docker swarm join \
      --token  SWMTKN-1-49nj1cmql0jkz5s954yi3oex3nedyz0fb0xx14ie39trti4wxv-8vxv8rssmk743ojnwacrr2e7c \
      172.16.0.5:2377

*  3.在manger节点上  
    查看节点信息： docker node ls  
    添加节点标签： docker node update --label-add minio1=true (ID) ，该ID为docker node ls 查询出来的worker节点ID

docker node update --label-add minio1=true lzqc0e1lojlq64mtzhurkqojc
     docker node update --label-add minio2=true lzqc0e1lojlq64mtzhurkqojc
     docker node update --label-add minio3=true 9kyb2ztxmrk4uwyor3xxjysb6
     docker node update --label-add minio4=true 9kyb2ztxmrk4uwyor3xxjysb6

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3podXl1MTk5MTEwMTY1MjA_size_16_color_FFFFFF_t_70_pic_center]

*  4.为MinIO创建Docker secret

echo "AKIAIOSFODNN7EXAMPLE" | docker secret create access_key -
    echo "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" | docker secret create secret_key -

*  5.下载docker-compose-secrets.yaml 部署文件

wget https://raw.githubusercontent.com/minio/minio/master/docs/orchestration/docker-swarm/docker-compose-secrets.yaml

*  6.修改部署文件里面的内容（基本不用改什么，可以修改volumes，挂载宿主机指定磁盘目录）

version: '3.7'
    
    services:
      minio1:
        image: minio/minio:RELEASE.2020-10-28T08-16-50Z
        hostname: minio1
        volumes:
          - minio1-data:/export
        ports:
          - "9001:9000"
        networks:
          - minio_distributed
        deploy:
          restart_policy:
            delay: 10s
            max_attempts: 10
            window: 60s
          placement:
            constraints:
              - node.labels.minio1==true
        command: server http://minio{ 1...4}/export
        secrets:
          - secret_key
          - access_key
        healthcheck:
          test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
          interval: 30s
          timeout: 20s
          retries: 3
    
      minio2:
        image: minio/minio:RELEASE.2020-10-28T08-16-50Z
        hostname: minio2
        volumes:
          - minio2-data:/export
        ports:
          - "9002:9000"
        networks:
          - minio_distributed
        deploy:
          restart_policy:
            delay: 10s
            max_attempts: 10
            window: 60s
          placement:
            constraints:
              - node.labels.minio2==true
        command: server http://minio{ 1...4}/export
        secrets:
          - secret_key
          - access_key
        healthcheck:
          test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
          interval: 30s
          timeout: 20s
          retries: 3
    
      minio3:
        image: minio/minio:RELEASE.2020-10-28T08-16-50Z
        hostname: minio3
        volumes:
          - minio3-data:/export
        ports:
          - "9003:9000"
        networks:
          - minio_distributed
        deploy:
          restart_policy:
            delay: 10s
            max_attempts: 10
            window: 60s
          placement:
            constraints:
              - node.labels.minio3==true
        command: server http://minio{ 1...4}/export
        secrets:
          - secret_key
          - access_key
        healthcheck:
          test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
          interval: 30s
          timeout: 20s
          retries: 3
    
      minio4:
        image: minio/minio:RELEASE.2020-10-28T08-16-50Z
        hostname: minio4
        volumes:
          - minio4-data:/export
        ports:
          - "9004:9000"
        networks:
          - minio_distributed
        deploy:
          restart_policy:
            delay: 10s
            max_attempts: 10
            window: 60s
          placement:
            constraints:
              - node.labels.minio4==true
        command: server http://minio{ 1...4}/export
        secrets:
          - secret_key
          - access_key
        healthcheck:
          test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
          interval: 30s
          timeout: 20s
          retries: 3
    
    volumes:
      minio1-data:
      minio2-data:
      minio3-data:
      minio4-data:
    
    networks:
      minio_distributed:
        driver: overlay
    
    secrets:
      secret_key:
        external: true
      access_key:
        external: true

*  7.启动集群：docker stack deploy --compose-file=./docker-compose-secrets.yaml minio\_stack  
    使用docker service ls 查看4个minio服务  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3podXl1MTk5MTEwMTY1MjA_size_16_color_FFFFFF_t_70_pic_center 1]  
    在两台worker节点上执行docker ps，可以看到每台worker服务器分别启动了2个minio实例  
    ![在这里插入图片描述][20201102162241515.png_pic_center]  
    在两台worker节点上执行docker volume ls ，查看minio容器把文件存放在什么位置  
    docker volume inspect minio\_stack\_minio1-data  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3podXl1MTk5MTEwMTY1MjA_size_16_color_FFFFFF_t_70_pic_center 2]
 *  8.使用Nginx对4个MinIO服务实例进行代理  
    安装nginx请参考：[https://zhuyu.blog.csdn.net/article/details/103696133][https_zhuyu.blog.csdn.net_article_details_103696133]  
    nginx.conf配置文件内容如下：

upstream minio-cluster { 
           server 172.16.0.2:9001;
           server 172.16.0.2:9002;
           server 172.16.0.8:9003;
           server 172.16.0.8:9004;  
        }
        server { 
            listen       9000;
            server_name  172.16.0.5;
            location / { 
                proxy_pass http://minio-cluster;
                proxy_set_header Host $http_host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_http_version 1.1;
                proxy_set_header Connection "";
                proxy_connect_timeout 90;
                proxy_send_timeout 90;
                proxy_buffer_size 4k;
                proxy_buffers 4 32k;
                client_max_body_size 10m;
                client_body_buffer_size 256k;
                #请注意加上下面2个配置参数
                proxy_buffering off;
                proxy_redirect off;
            }
    
            error_page   500 502 503 504  /50x.html;
            location = /50x.html { 
                root   html;
            }
        }

访问 172.16.0.5:9000 就可以访问 MinIO服务操作文件了，如下创建了名称为test的桶，创建目录，上传了文件，在MinIO worker服务器的数据存放目录查看是否有文件  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3podXl1MTk5MTEwMTY1MjA_size_16_color_FFFFFF_t_70_pic_center 3]

可以看到在4个minio实例的每个volume目录下都能看到如下图的信息，每个实例（每个磁盘）都全量存储数据，这样实现了高可用，但是未实现分布式大数据量存储呀（可能是我配置有误，如有网友知道请留言）  
![在这里插入图片描述][20201102163039677.png_pic_center]

*  9.删除分布式MinIO services

#删除集群
    docker stack rm minio_stack
    
    #删除volume
    docker volume ls
    docker volume rm volume_name

*  10.扩展集群也比较方便，请查看官网：

启动分布式Minio实例，4节点，每节点4块盘，需要在4个节点上都运行下面的命令
    minio server http://192.168.1.11/export1 http://192.168.1.11/export2 \
                   http://192.168.1.11/export3 http://192.168.1.11/export4 \
                   http://192.168.1.12/export1 http://192.168.1.12/export2 \
                   http://192.168.1.12/export3 http://192.168.1.12/export4 \
                   http://192.168.1.13/export1 http://192.168.1.13/export2 \
                   http://192.168.1.13/export3 http://192.168.1.13/export4 \
                   http://192.168.1.14/export1 http://192.168.1.14/export2 \
                   http://192.168.1.14/export3 http://192.168.1.14/export4

MinIO支持通过命令，指定新的集群来扩展现有集群（纠删码模式），命令行如下：
    minio server http://host{ 1...32}/export{ 1...32} http://host{ 33...64}/export{ 1...32}

*  10.指定minio挂载磁盘目录，在两台worker节点上创建目录，如果挂载的是新磁盘请格式化等处理后mount到 根目录  
    worker1节点：mkdir /minio1\_data ，mkdir /minio2\_data  
    worker2节点：mkdir /minio3\_data ，mkdir /minio3\_data

version: '3.7'
    
    services:
      minio1:
        image: minio/minio:RELEASE.2020-10-28T08-16-50Z
        hostname: minio1
        volumes:
          # 指定宿主机根目录下的minio1_data目录挂载映射到容器/export目录
          - /minio1_data:/export
        ports:
          - "9001:9000"
        networks:
          - minio_distributed
        deploy:
          restart_policy:
            delay: 10s
            max_attempts: 10
            window: 60s
          placement:
            constraints:
              - node.labels.minio1==true
        command: server http://minio{ 1...4}/export
        secrets:
          - secret_key
          - access_key
        healthcheck:
          test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
          interval: 30s
          timeout: 20s
          retries: 3
    
      minio2:
        image: minio/minio:RELEASE.2020-10-28T08-16-50Z
        hostname: minio2
        volumes:
          - /minio2_data:/export
        ports:
          - "9002:9000"
        networks:
          - minio_distributed
        deploy:
          restart_policy:
            delay: 10s
            max_attempts: 10
            window: 60s
          placement:
            constraints:
              - node.labels.minio2==true
        command: server http://minio{ 1...4}/export
        secrets:
          - secret_key
          - access_key
        healthcheck:
          test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
          interval: 30s
          timeout: 20s
          retries: 3
    
      minio3:
        image: minio/minio:RELEASE.2020-10-28T08-16-50Z
        hostname: minio3
        volumes:
          - /minio3_data:/export
        ports:
          - "9003:9000"
        networks:
          - minio_distributed
        deploy:
          restart_policy:
            delay: 10s
            max_attempts: 10
            window: 60s
          placement:
            constraints:
              - node.labels.minio3==true
        command: server http://minio{ 1...4}/export
        secrets:
          - secret_key
          - access_key
        healthcheck:
          test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
          interval: 30s
          timeout: 20s
          retries: 3
    
      minio4:
        image: minio/minio:RELEASE.2020-10-28T08-16-50Z
        hostname: minio4
        volumes:
          - /minio4_data:/export
        ports:
          - "9004:9000"
        networks:
          - minio_distributed
        deploy:
          restart_policy:
            delay: 10s
            max_attempts: 10
            window: 60s
          placement:
            constraints:
              - node.labels.minio4==true
        command: server http://minio{ 1...4}/export
        secrets:
          - secret_key
          - access_key
        healthcheck:
          test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
          interval: 30s
          timeout: 20s
          retries: 3
    
    networks:
      minio_distributed:
        driver: overlay
    
    secrets:
      secret_key:
        external: true
      access_key:
        external: true

[https_docs.min.io_cn_deploy-minio-on-docker-swarm.html]: https://docs.min.io/cn/deploy-minio-on-docker-swarm.html
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3podXl1MTk5MTEwMTY1MjA_size_16_color_FFFFFF_t_70_pic_center]: /images/20221120/79d2841cac2743bf8d54353b24c7f354.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3podXl1MTk5MTEwMTY1MjA_size_16_color_FFFFFF_t_70_pic_center 1]: /images/20221120/10d87f07da9341dcb591c74f868165ee.png
[20201102162241515.png_pic_center]: /images/20221120/88e2f96389984a14abcffaf0ee8e9f58.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3podXl1MTk5MTEwMTY1MjA_size_16_color_FFFFFF_t_70_pic_center 2]: /images/20221120/ff75513ad8d04104a46a3809cc3548ae.png
[https_zhuyu.blog.csdn.net_article_details_103696133]: https://zhuyu.blog.csdn.net/article/details/103696133
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3podXl1MTk5MTEwMTY1MjA_size_16_color_FFFFFF_t_70_pic_center 3]: /images/20221120/9fb0d261b69346919b57d7f62d1f5d34.png
[20201102163039677.png_pic_center]: /images/20221120/1e5c502f030b4f44be56b55e6f8cd33b.png