kubernetes学习：1.构建harbor镜像仓库

系统管理员 2022-05-28 00:49 252阅读 0赞

# 构建harbor镜像仓库 #

Harbor是一个用于存储和分发Docker镜像的企业级Registry服务器，可以在测试和生产环境中部署harbor对业务上的镜像进行统一的管理。harbor 的官方网站如下所示：  
[https://vmware.github.io/harbor/cn/][https_vmware.github.io_harbor_cn]

一般来讲我们在生产环境部署harbor的时候需要使用多节点方案。将数据库从配置项中抽取出来，后端的镜像资源存储使用统一的共享资源（ceph分配的磁盘挂载到指定目录中）。前端可以使用nginx或者haproxy对多个harbor服务进行负载均衡。

本文搭建一个单机的harbor镜像仓库，体会对镜像进行上传和下载的过程。上述的复杂架构就是在此基础上的不断拓展。

## 构建harbor ##

--------------------

### 构建环境 ###

具体的构建过程主要分为以下几个步骤：

*  操作系统： centos7.2
 *  docker版本： 1.12.6
 *  docker-compose： 1.9.0

首先安装docker服务：

[root@wecloud-test-harbor ~]# yum install docker -y

启动docker服务：

[root@wecloud-test-harbor ~]# systemctl start docker.service 
    [root@wecloud-test-harbor ~]# systemctl enable docker.service 
    Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.

查看docker服务状态：

[root@wecloud-test-harbor ~]# systemctl status docker.service 
    ● docker.service - Docker Application Container Engine
       Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
       Active: active (running) since 一 2018-03-26 22:33:04 CST; 12s ago
         Docs: http://docs.docker.com
     Main PID: 2311 (dockerd-current)
       CGroup: /system.slice/docker.service
               ├─2311 /usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-run...
               └─2318 /usr/bin/docker-containerd-current -l

--------------------

安装docker-compose：

[root@wecloud-test-harbor ~]# yum install docker-compose -y

下载harbor的离线安装包（离线的安装速度会比较快一些）,下载地址如下：  
[https://storage.googleapis.com/harbor-releases/release-1.4.0/harbor-offline-installer-v1.4.0.tgz][https_storage.googleapis.com_harbor-releases_release-1.4.0_harbor-offline-installer-v1.4.0.tgz]

解压harbor-offline-installer-v1.4.0.tgz：

[root@wecloud-test-harbor ~]# tar xvf harbor-offline-installer-v1.4.0.tgz

跳转到harbor目录，修改harbor.cfg配置文件：

[root@wecloud-test-harbor harbor]# grep "^[^#]" harbor.cfg 
    # harbor的主机地址
    hostname = 192.168.99.180
    # 这里使用http协议，如果需要上传镜像，需要在docker.service文件中添加一个配置，后续会介绍到
    ui_url_protocol = http
    max_job_workers = 3 
    customize_crt = on
    ssl_cert = /data/cert/server.crt
    ssl_cert_key = /data/cert/server.key
    secretkey_path = /data
    admiral_url = NA
    log_rotate_count = 50
    log_rotate_size = 200M
    #××××××××××××××××××邮件的配置信息××××××××××××××××××
    email_identity = 
    email_server = smtp.mydomain.com
    email_server_port = 25
    email_username = sample_admin@mydomain.com
    email_password = abc
    email_from = admin <sample_admin@mydomain.com>
    email_ssl = false
    email_insecure = false
    #××××××××××××××××××邮件的配置信息××××××××××××××××××
    # 管理员密码，管理员帐号为：admin
    harbor_admin_password = Harbor12345
    # 使用数据库存储认证信息
    auth_mode = db_auth
    # 配置ldap信息
    ldap_url = ldaps://ldap.mydomain.com
    ldap_basedn = ou=people,dc=mydomain,dc=com
    ldap_uid = uid 
    ldap_scope = 2 
    ldap_timeout = 5
    ldap_verify_cert = true
    self_registration = on
    token_expiration = 30
    project_creation_restriction = everyone
    # 指定mysql数据库的信息
    db_host = mysql
    db_password = root123
    db_port = 3306
    db_user = root
    # redis地址
    redis_url =
    clair_db_host = postgres
    clair_db_password = password
    clair_db_port = 5432
    clair_db_username = postgres
    clair_db = postgres
    uaa_endpoint = uaa.mydomain.org
    uaa_clientid = id
    uaa_clientsecret = secret
    uaa_verify_cert = true
    uaa_ca_cert = /path/to/ca.pem
    registry_storage_provider_name = filesystem
    registry_storage_provider_config =

配置信息修改结束后，执行install.sh脚本执行具体的安装过程，安装完成后，harbor的容器都会启动：

安装过程主要分为以下几个步骤：

*  检测环境信息，docker的版本和docker-compose的版本信息;
 *  导入harbor的镜像信息，这些镜像内容都已经包含在离线包中了，所以导入速度相对比较快;
 *  检查Harbor的现有实例，如果现有实例存在则会删除;
 *  启动harbor  
    最后会提示我们启动成功，然后可以访问刚才配置的本地ip地址：  
    Now you should be able to visit the admin portal at [http://192.168.99.180][http_192.168.99.180].  
    For more details, please visit [https://github.com/vmware/harbor][https_github.com_vmware_harbor] .

[root@wecloud-test-harbor harbor]# ./install.sh

查看harbor使用的镜像列表：

[root@wecloud-test-harbor harbor]# docker images 
    REPOSITORY                    TAG                 IMAGE ID            CREATED             SIZE
    vmware/clair-photon           v2.0.1-v1.4.0       a1df3526fe43        6 weeks ago         300.3 MB
    vmware/notary-server-photon   v0.5.1-v1.4.0       3edfddb8ece2        6 weeks ago         211.2 MB
    vmware/notary-signer-photon   v0.5.1-v1.4.0       cc70a05cdb6a        6 weeks ago         208.7 MB
    vmware/registry-photon        v2.6.2-v1.4.0       8920f621ddd1        6 weeks ago         197.8 MB
    vmware/nginx-photon           v1.4.0              20c8a01ac6ab        6 weeks ago         134.5 MB
    vmware/harbor-log             v1.4.0              9e818c7a27ab        6 weeks ago         199.7 MB
    vmware/harbor-jobservice      v1.4.0              29c14d91b043        6 weeks ago         190.6 MB
    vmware/harbor-ui              v1.4.0              6cb4318eda6a        6 weeks ago         209.5 MB
    vmware/harbor-adminserver     v1.4.0              8145970fa013        6 weeks ago         182.2 MB
    vmware/harbor-db              v1.4.0              c38da34727f0        6 weeks ago         521 MB
    vmware/mariadb-photon         v1.4.0              8457013cf6e3        6 weeks ago         521 MB
    vmware/postgresql-photon      v1.4.0              59aa61520094        6 weeks ago         220.6 MB
    vmware/harbor-db-migrator     1.4                 7a4d871b612e        9 weeks ago         1.146 GB
    vmware/photon                 1.0                 9b411d78ad9e        10 weeks ago        129.7 MB

查看harbor的启动容器：

[root@wecloud-test-harbor harbor]# docker ps
    CONTAINER ID        IMAGE                                  COMMAND                  CREATED             STATUS                   PORTS                                                              NAMES
    6231d7bb19c3        vmware/nginx-photon:v1.4.0             "nginx -g 'daemon off"   3 minutes ago       Up 2 minutes             0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:4443->4443/tcp   nginx
    23d83a8b15ce        vmware/harbor-jobservice:v1.4.0        "/harbor/start.sh"       3 minutes ago       Up 2 minutes (healthy)                                                                      harbor-jobservice
    ddbd8ea06f59        vmware/harbor-ui:v1.4.0                "/harbor/start.sh"       3 minutes ago       Up 2 minutes (healthy)                                                                      harbor-ui
    d8e70cc126b3        vmware/harbor-db:v1.4.0                "/usr/local/bin/docke"   3 minutes ago       Up 3 minutes (healthy)   3306/tcp                                                           harbor-db
    f533c4ae2e5d        vmware/harbor-adminserver:v1.4.0       "/harbor/start.sh"       3 minutes ago       Up 2 minutes (healthy)                                                                      harbor-adminserver
    68afc0b58219        vmware/registry-photon:v2.6.2-v1.4.0   "/entrypoint.sh serve"   3 minutes ago       Up 3 minutes (healthy)   5000/tcp                                                           registry
    1aa9ed019604        vmware/harbor-log:v1.4.0               "/bin/sh -c /usr/loca"   3 minutes ago       Up 3 minutes (healthy)   127.0.0.1:1514->10514/tcp                                          harbor-log

直接访问harbor的网址：  
输入的用户名为admin，密码为配置文件中harbor\_admin\_password的内容  
![这里写图片描述][70]  
默认有一个项目为library  
![这里写图片描述][70 1]

我们可以点击新建创建额外的项目：  
![这里写图片描述][70 2]

后续我们可以将自己的镜像推送到该项目中。

harbor的单机版已经搭建完成，后边我们需要介绍如何推送和拉取镜像信息。

--------------------

## 推送镜像到harbor仓库 ##

我们首先从其他的仓库获取一个测试镜像，如果是默认配置下，docker指定的仓库地址是docker hub，国内因为一些特殊原因拉取速度很慢，而且登录验证也无法通过。如下图所示：

[root@wecloud-test-harbor ~]# docker search centos
    Error response from daemon: Get https://index.docker.io/v1/search?q=centos&n=25: x509: certificate signed by unknown authority
    [root@wecloud-test-harbor ~]# docker login
    Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
    Username: zhangchiwd371
    Password: 
    Error response from daemon: Get https://index.docker.io/v1/users/: x509: certificate signed by unknown authority

所以可以配置docker的守护进程，修改镜像源为国内源。编辑/etc/docker/daemon.json文件（新版本的docker服务），设置镜像源的地址：

{
      "registry-mirrors": ["https://docker.mirrors.ustc.edu.cn"] }

修改完启动文件后重启docker服务：

[root@wecloud-test-harbor ~]# systemctl restart docker.service

再次搜索查看需要的docker镜像，速度飞快：

[root@wecloud-test-harbor ~]# docker search nginx
    INDEX       NAME                                                             DESCRIPTION                                     STARS     OFFICIAL   AUTOMATED
    docker.io   docker.io/nginx                                                  Official build of Nginx.                        8217      [OK]       
    docker.io   docker.io/jwilder/nginx-proxy                                    Automated Nginx reverse proxy for docker c...   1299                 [OK]
    docker.io   docker.io/richarvey/nginx-php-fpm                                Container running Nginx + PHP-FPM capable ...   536                  [OK]
    docker.io   docker.io/jrcs/letsencrypt-nginx-proxy-companion                 LetsEncrypt container to use with nginx as...   333                  [OK]
    docker.io   docker.io/kong                                                   Open-source Microservice & API Management ...   170       [OK]       
    docker.io   docker.io/webdevops/php-nginx                                    Nginx with PHP-FPM                              97                   [OK]
    docker.io   docker.io/kitematic/hello-world-nginx                            A light-weight nginx container that demons...   95                   
    docker.io   docker.io/zabbix/zabbix-web-nginx-mysql                          Zabbix frontend based on Nginx web-server ...   48                   [OK]
    docker.io   docker.io/bitnami/nginx                                          Bitnami nginx Docker Image                      45                   [OK]
    docker.io   docker.io/linuxserver/nginx                                      An Nginx container, brought to you by Linu...   33                   
    docker.io   docker.io/1and1internet/ubuntu-16-nginx-php-phpmyadmin-mysql-5   ubuntu-16-nginx-php-phpmyadmin-mysql-5          29                   [OK]
    docker.io   docker.io/tobi312/rpi-nginx                                      NGINX on Raspberry Pi / armhf                   19                   [OK]
    docker.io   docker.io/wodby/drupal-nginx                                     Nginx for Drupal container image                9                    [OK]
    docker.io   docker.io/blacklabelops/nginx                                    Dockerized Nginx Reverse Proxy Server.          8                    [OK]
    docker.io   docker.io/nginxdemos/nginx-ingress                               NGINX Ingress Controller for Kubernetes         8                    
    docker.io   docker.io/webdevops/nginx                                        Nginx container                                 8                    [OK]
    docker.io   docker.io/centos/nginx-18-centos7                                Platform for running nginx 1.8 or building...   6                    
    docker.io   docker.io/1science/nginx                                         Nginx Docker images that include Consul Te...   4                    [OK]
    docker.io   docker.io/nginxdemos/hello                                       NGINX webserver that serves a simple page ...   4                    [OK]
    docker.io   docker.io/behance/docker-nginx                                   Provides base OS, patches and stable nginx...   2                    [OK]
    docker.io   docker.io/pebbletech/nginx-proxy                                 nginx-proxy sets up a container running ng...   2                    [OK]
    docker.io   docker.io/toccoag/openshift-nginx                                Nginx reverse proxy for Nice running on sa...   1                    [OK]
    docker.io   docker.io/travix/nginx                                           NGinx reverse proxy                             1                    [OK]
    docker.io   docker.io/goodguide/nginx-application-proxy                      No-configuration Nginx reverse proxy for a...   0                    [OK]
    docker.io   docker.io/mailu/nginx                                            Mailu nginx frontend                            0                    [OK]
    [root@wecloud-test-harbor ~]#

为了测试效果，我们拉取nginx的官方镜像：

[root@wecloud-test-harbor ~]# docker pull docker.io/nginx
    Using default tag: latest
    Trying to pull repository docker.io/library/nginx ... 
    latest: Pulling from docker.io/library/nginx
    2a72cbf407d6: Pull complete 
    fefa2faca81f: Pull complete 
    080aeede8114: Pull complete 
    Digest: sha256:c4ee0ecb376636258447e1d8effb56c09c75fe7acf756bf7c13efadf38aa0aca

可以看到已经镜像列表中已经多出了nginx 镜像：

[root@wecloud-test-harbor ~]# docker images 
    REPOSITORY                              TAG                 IMAGE ID            CREATED             SIZE
    docker.io/nginx                         latest              7f70b30f2cc6        4 days ago          108.7 MB

--------------------

#### 上传镜像 ####

如果需要将这个镜像上传到我们自己的harbor仓库中，需要给该镜像重新生成一个tag标签。

[root@wecloud-test-harbor ~]# docker tag docker.io/nginx:latest 192.168.99.180/k8s/nginx:latest

注意这里的写法，新的标签是一个路径的格式：**192.168.99.180/k8s/nginx:latest**，其中192.168.99.180是我们的harbor仓库地址，k8s是项目名称，nginx是镜像名称，latest是镜像版本号。

查看镜像列表，可以看到我们多了一个新的镜像标签：

[root@wecloud-test-harbor ~]# docker images 
    REPOSITORY                              TAG                 IMAGE ID            CREATED             SIZE
    docker.io/nginx                         latest              7f70b30f2cc6        5 days ago          108.7 MB
    192.168.99.180/k8s/nginx                latest              7f70b30f2cc6        5 days ago          108.7 MB

将该镜像推送到harbor仓库，这是一个写操作，需要事先登录harbor仓库：

[root@wecloud-test-harbor ~]# docker login 192.168.99.180
    Username: admin
    Password: 
    Error response from daemon: Get https://192.168.99.180/v1/users/: dial tcp 192.168.99.180:443: getsockopt: connection refused

直接登录发现并不允许我们登录镜像，其实这是因为默认情况下是需要使用https协议的，而我们在harbor的配置中使用的是http协议。所以需要在docker.service启动脚本中指定非加密的登录方式。编辑/usr/lib/systemd/system/docker.service文件，在启动选项中追加如下配置：–insecure-registry=192.168.99.180：

[Unit]
    Description=Docker Application Container Engine
    Documentation=http://docs.docker.com
    After=network.target
    Wants=docker-storage-setup.service
    Requires=docker-cleanup.timer
    
    [Service]
    Type=notify
    NotifyAccess=all
    EnvironmentFile=-/run/containers/registries.conf
    EnvironmentFile=-/etc/sysconfig/docker
    EnvironmentFile=-/etc/docker/daemon.json
    EnvironmentFile=-/etc/sysconfig/docker-storage
    EnvironmentFile=-/etc/sysconfig/docker-network
    Environment=GOTRACEBACK=crash
    Environment=DOCKER_HTTP_HOST_COMPAT=1
    Environment=PATH=/usr/libexec/docker:/usr/bin:/usr/sbin
    ExecStart=/usr/bin/dockerd-current \
              --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current \
              --default-runtime=docker-runc \
              --exec-opt native.cgroupdriver=systemd \
              --userland-proxy-path=/usr/libexec/docker/docker-proxy-current \
              --insecure-registry=192.168.99.183 \
              $OPTIONS \
              $DOCKER_STORAGE_OPTIONS \
              $DOCKER_NETWORK_OPTIONS \
              $ADD_REGISTRY \
              $BLOCK_REGISTRY \
              $INSECURE_REGISTRY\
              $REGISTRIES
    ExecReload=/bin/kill -s HUP $MAINPID
    LimitNOFILE=1048576
    LimitNPROC=1048576
    LimitCORE=infinity
    TimeoutStartSec=0
    Restart=on-abnormal
    MountFlags=slave
    KillMode=process
    
    [Install]
    WantedBy=multi-user.target

重新加载systemctl服务：

[root@wecloud-test-harbor ~]# systemctl daemon-reload

重启docker服务：

[root@wecloud-test-harbor ~]# systemctl restart docker.service

再次进行登录harbor:

[root@wecloud-test-harbor ~]# docker login 192.168.99.180
    Username: admin
    Password: 
    Login Succeeded

发现harbor登录成功，现在尝试推送镜像到仓库中：

[root@wecloud-test-harbor ~]# docker push 192.168.99.180/k8s/nginx:latest
    The push refers to a repository [192.168.99.180/k8s/nginx]
    8e5e010ce6ad: Pushed 
    974dc1373097: Pushed 
    3358360aedad: Pushed 
    latest: digest: sha256:22650ea37ad4ccf8472330f141de0712c14fbad2b6792b6aba1687d3d2fa9aa5 size: 948

推送成功后可以在harbor的控制台进行查看：  
![这里写图片描述][70 3]

发现镜像已经上传成功

--------------------

#### 拉取镜像 ####

在另外一个机器上可以尝试拉取刚才上传的nginx镜像（同样需要配置/usr/lib/systemd/system/docker.service，启动项添加192.168.99.180的镜像地址），配置完成后进行拉取操作：

[root@wecloud-test-zhangchi-harbor ~]# docker pull 192.168.99.180/k8s/nginx:latest
    Trying to pull repository 192.168.99.180/k8s/nginx ... 
    latest: Pulling from 192.168.99.180/k8s/nginx
    2a72cbf407d6: Pull complete 
    fefa2faca81f: Pull complete 
    080aeede8114: Pull complete 
    Digest: sha256:22650ea37ad4ccf8472330f141de0712c14fbad2b6792b6aba1687d3d2fa9aa5

拉取成功，并且镜像列表中可以看到新的镜像信息：

[root@wecloud-test-zhangchi-harbor ~]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE 192.168.99.180/k8s/nginx latest 7f70b30f2cc6 5 days ago 108.7 MB

--------------------

## 小结 ##

本章介绍了单节点harbor的部署方案，非常的简单易行，如果部署多节点，更加实用的harbor方案。需要考虑更多的因素，后续将会给大家介绍多节点harbor的方案。

[https_vmware.github.io_harbor_cn]: https://vmware.github.io/harbor/cn/
[https_storage.googleapis.com_harbor-releases_release-1.4.0_harbor-offline-installer-v1.4.0.tgz]: https://storage.googleapis.com/harbor-releases/release-1.4.0/harbor-offline-installer-v1.4.0.tgz
[http_192.168.99.180]: http://192.168.99.180
[https_github.com_vmware_harbor]: https://github.com/vmware/harbor
[70]: /images/20220528/2e8941d8cda644fe893a3ed18b4db4a0.png
[70 1]: /images/20220528/f32019eada8e4cdf9a4ddc2edd8010b7.png
[70 2]: /images/20220528/f69cad1ff0f54b9ba6dbbc5739cd56be.png
[70 3]: /images/20220528/89a8fee783114823b9d9c8efeedfba85.png