Gitlab之间数据迁移的5种方式

傷城~ 2023-10-01 23:33 8阅读 0赞

#### 目录 ####

*  方式一：Gitlab project export / import
 *   *  1. 限制
     *   *  版本限制
         *  速率限制
         *  导出限制
     *  2.导入导出过程
     *   *  导出
         *  导入
 *  方式二：Gitlab backup & restore
 *   *  1.backup
     *   *  命令
         *  额外需要备份文件
         *  Backup options
     *  2.Restore
     *   *  Prerequisites
         *  Command
 *  方式三： Git clone、pull & push
 *  方式四：Gitlab import bare repostries
 *  方式五：Gitlab官方 \[Congregate\]工具

--------------------

## 方式一：Gitlab project export / import ##

任意一个Gitlab实例中的项目都能通过导出成文件然后导入到新Gitlab的方式来完成迁移。

**优点**：

1.  支持Gitlab之间的数据迁移，故而可以从多个Gitlab迁移数据到一个Gitlab；
2.  操作简单，流程不是很复杂，可以通过调用API实现导入导出；

**缺点**：

1.  不支持增量数据同步；
2.  受到 rate limit的影响，速率相对较慢；
3.  有版本限制，不同版本之间并不通用；

### 1. 限制 ###

#### 版本限制 ####

Gitlab 的v13.0之前，只能在兼容版本的Gitlab之间实现导入导出，例如Gitlab v12.10导出的项目只能导入到版本号为v11.70到v12.10的Gitlab，具体参考：  
![请添加图片描述][db4c57d328a34c6e8974c80b0a73c56c.png]

从GitLab 13.0开始，GitLab可以导入从不同的GitLab部署中导出的包，仅限于之前的两个GitLab小版本，例如：

![请添加图片描述][39f6703783a1448d954e0826d9b71757.png]

#### 速率限制 ####

![请添加图片描述][a5b50ab20bf2418d89d465f45abf841d.png]

#### 导出限制 ####

在项目导出时，不是所有的数据都会被导出，有一部分数据并不会导出，具体参考：

**The following items are exported**:

*  Project and wiki repositories
 *  Project uploads
 *  Project configuration, excluding integrations
 *  Issues with comments, merge requests with diffs and comments, labels, milestones, snippets, time tracking, and other project entities
 *  Design Management files and data
 *  LFS objects
 *  Issue boards
 *  Pipelines history
 *  Push Rules
 *  Awards
 *  Group members are exported as project members, as long as the user has the Maintainer role in the exported project’s group, or is an administrator

**The following items are not exported**:

*  Child pipeline history
 *  Build traces and artifacts
 *  Package and container registry images
 *  CI/CD variables
 *  Pipeline triggers
 *  Webhooks
 *  Any encrypted tokens
 *  Merge Request Approvers and the number of required approvals
 *  Repository size limits
 *  Deploy keys allowed to push to protected branches
 *  Secure Files

These content rules also apply to creating projects from templates on the group or instance levels, because the same export and import mechanisms are used.

### 2.导入导出过程 ###

#### 导出 ####

1.  On the top bar, select Menu > Projects and find your project.
2.  On the left sidebar, select Settings > General.
3.  Expand Advanced.
4.  Select Export project.
5.  After the export is generated, you should receive an email with a link to download the file.
6.  Alternatively, you can come back to the project settings and download the file from there or generate a new export. After the file is available, the page should show the Download export button.

#### 导入 ####

1.  When creating a new project, select Import project.
2.  In Import project from, select GitLab export.
3.  Enter your project name and URL. Then select the file you exported previously.
4.  Select Import project to begin importing. Your newly imported project page appears shortly.

--------------------

## 方式二：Gitlab backup & restore ##

根据官方文档说，单个gitlab之间数据的迁移，这是最推荐的方式。

**优点**：

1.  流程简单，依赖于Gitlab自带工具；
2.  从Gitlab v14.9 开始支持增量数据备份与恢复;
3.  可以根据需要，选择备份与恢复的内容；

**缺点**：

1.  不支持多个Gitlab到一个Gitlab的数据同步；
2.  恢复时可能会产生数据覆盖，也就是恢复的Gitlab最好是一个空的Gitlab；
3.  同样受限于版本，只支持相同版本Gitlab之间的backup & restore ，You can only restore a backup to exactly the same version and type (CE/EE) of GitLab that you created it on；
4.  不能直接识别使用对象存储保存的项目，需要工具转换为文件存储；

### 1.backup ###

**GitLab provides a command line interface to back up your entire instance, including**:

*  Database
 *  Attachments
 *  Git repositories data
 *  CI/CD job output logs
 *  CI/CD job artifacts
 *  LFS objects
 *  Terraform states ([introduced][] in GitLab 14.7)
 *  Container Registry images
 *  GitLab Pages content
 *  Packages ([introduced][introduced 1] in GitLab 14.7)
 *  Snippets
 *  [Group wikis][]

**Backups do not include**:

*  [Mattermost data][]
 *  Redis (and thus Sidekiq jobs)

#### 命令 ####

# Install gitlab by using the Omnibus package
    ## Gitlab 12.2 or later
    sudo gitlab-backup create
    ## Gitlab 12.1 and earlier
    sudo gitlab-rake gitlab:backup:create
    
    
    # Install gitlab from source
    cd /home/git/gitlab
    sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production

#### 额外需要备份文件 ####

The [backup Rake task][] GitLab provides does *not* store your configuration files. The primary reason for this is that your database contains items including encrypted information for two-factor authentication and the CI/CD *secure variables*. Storing encrypted information in the same location as its key defeats the purpose of using encryption in the first place.

For Omnibus:

*  /etc/gitlab/gitlab-secrets.json
 *  /etc/gitlab/gitlab.rb

For installation from source:

*  /home/git/gitlab/config/secrets.yml
 *  /home/git/gitlab/config/gitlab.yml

#### Backup options ####

具体参考

The command line tool GitLab provides to backup your instance can accept more options.

*  backup strategy option **STRATEGY=copy**

## For Omnibus GitLab packages
    sudo gitlab-backup create STRATEGY=copy
    
    ## For installations from source
    sudo -u git -H bundle exec rake gitlab:backup:create STRATEGY=copy RAILS_ENV=production

*  bakcup filename **BACKUP=**
 *  confirm archive can be transferred **GZIP\_RSYNCABLE=yes**
 *  Excluding specific directories from the backup **SKIP=**

You can exclude specific directories from the backup by adding the environment variable SKIP, whose values are a comma-separated list of the following options:

*   *   *  db (database)
         *  uploads (attachments)
         *  builds (CI job output logs)
         *  artifacts (CI job artifacts)
         *  lfs (LFS objects)
         *  terraform\_state (Terraform states)
         *  registry (Container Registry images)
         *  pages (Pages content)
         *  repositories (Git repositories data)
         *  packages (Packages)

# skip db and uploads
    ## For Omnibus GitLab packages
    sudo gitlab-backup create SKIP=db,uploads
    
    
    ## For installations from source
    sudo -u git -H bundle exec rake gitlab:backup:create SKIP=db,uploads RAILS_ENV=production

*  Disabling prompts during restore **GITLAB\_ASSUME\_YES=1**
 *  Back up Git repositories concurrently
    
     *  **Introduced in GitLab 13.3**
     *  When using [multiple repository storages][], repositories can be backed up or restored concurrently to help fully use CPU time.
        
         *  GITLAB\_BACKUP\_MAX\_CONCURRENCY: The maximum number of projects to back up at the same time. Defaults to the number of logical CPUs (in GitLab 14.1 and earlier, defaults to 1).
         *  GITLAB\_BACKUP\_MAX\_STORAGE\_CONCURRENCY: The maximum number of projects to back up at the same time on each storage. This allows the repository backups to be spread across storages. Defaults to 2 (in GitLab 14.1 and earlier, defaults to 1).
 *  Incremental repository backups **INCREMENTAL=yes PREVIOUS\_BACKUP=<timestamp\_of\_backup>**
    
     *  **Introduced in GitLab 14.9 with a flag named incremental\_repository\_backup. Disabled by default**
     *  **PREVIOUS\_BACKUP option introduced in GitLab 15.0**
 *  Back up specific repository storages **REPOSITORIES\_STORAGES=<storage1，…，storageN>**
    
     *  **ntroduced in GitLab 15.0**
 *  Uploading backups to a remote (cloud) storage

### 2.Restore ###

#### Prerequisites ####

1.  You need to have a working GitLab installation before you can perform a restore. This is because the system user performing the restore actions (git) is usually not allowed to create or delete the SQL database needed to import data into (gitlabhq\_production). All existing data is either erased (SQL) or moved to a separate directory (such as repositories and uploads). 数据库相关
2.  To restore a backup, you must restore /etc/gitlab/gitlab-secrets.json (for Omnibus packages) or /home/git/gitlab/.secret (for installations from source). This file contains the database encryption key, CI/CD variables, and variables used for two-factor authentication. If you fail to restore this encryption key file along with the application data backup, users with two-factor authentication enabled and GitLab Runner loses access to your GitLab server. 密钥
3.  You may also want to restore your previous /etc/gitlab/gitlab.rb (for Omnibus packages) or /home/git/gitlab/config/gitlab.yml (for installations from source) and any TLS keys, certificates (/etc/gitlab/ssl, /etc/gitlab/trusted-certs), or SSH host keys. 配置文件
4.  Starting with GitLab 12.9, if an untarred backup (like the ones made with SKIP=tar) is found, and no backup is chosen with BACKUP=, the untarred backup is used.

#### Command ####

# Restore for installation from source
    ## Stop processes that are connected to the database
    sudo service gitlab stop
    ## restore
    sudo -u git -H bundle exec rake gitlab:backup:restore RAILS_ENV=production

参考：

[Back up and restore GitLab | GitLab][backup Rake task]

--------------------

## 方式三： Git clone、pull & push ##

通过git的方式来实现数据的迁移，是一种非常简单且机械的方式，不用考虑多到单，单到单、全量迁移、增量迁移等种种问题，通过git clone Gitlab-A / Gitlab-B 等到本地，然后git push到Gitlab-new

有增量的数据，可以pull拉取最新的代码，然后push到新Gitlab，以此来实现增量的更新；

**缺点**：

1.  需要在迁移的Gitlab中提前建好git库，这样才能push到新Gitlab；（不过可以通过写代码，获取API token，调用API来做）；
2.  只能迁移git中存储的信息，那些存储在gitlab中的并不能实现迁移，比如merge记录、CI/CD记录等等信息；

--------------------

## 方式四：Gitlab import bare repostries ##

Rake tasks are available to import bare repositories into a GitLab instance. When migrating from an existing GitLab instance, and to preserve ownership by users and their namespaces, please use our project-based import/export. 也就是使用方式一。

注意事项：

*  The owner of the project is the first administrator.
 *  The groups are created as needed, including subgroups.
 *  The owner of the group is the first administrator.
 *  Existing projects are skipped.
 *  Projects in hashed storage may be skipped. For more information, see Importing bare repositories from hashed storage. 从Gitlab 10.0开始引入hashed storage，12.0之后作为默认存储方式，不过迁移方式类似
 *  The existing Git repositories ware moved from disk (removed from the original path).

--------------------

## 方式五：Gitlab官方 \[Congregate\]工具 ##

\[Congregate\]地址：[https://gitlab.com/gitlab-org/professional-services-automation/tools/migration/congregate/-/tree/master][https_gitlab.com_gitlab-org_professional-services-automation_tools_migration_congregate_-_tree_master]

Congregate is a migration tool that can move data from many different Source Code Management (SCM) and Continuous Integration (CI) systems into a GitLab Self Managed (SM) instance hosted in the customer data center/cloud or GitLab.com. It is what GitLab Professional Services uses for large scale migration engagements. It is being shared in a source-available way for collaboration with Customers and Partners.

![777c7e9cdd0fa88e169dd5921cec3dc6.png][]

快速启动参考：

[docs/using-congregate.md · master · GitLab.org / Professional Services Automation / Tools / Migration / Congregate · GitLab][docs_using-congregate.md _ master _ GitLab.org _ Professional Services Automation _ Tools _ Migration _ Congregate _ GitLab]

[db4c57d328a34c6e8974c80b0a73c56c.png]: https://img-blog.csdnimg.cn/db4c57d328a34c6e8974c80b0a73c56c.png
[39f6703783a1448d954e0826d9b71757.png]: https://img-blog.csdnimg.cn/39f6703783a1448d954e0826d9b71757.png
[a5b50ab20bf2418d89d465f45abf841d.png]: https://img-blog.csdnimg.cn/a5b50ab20bf2418d89d465f45abf841d.png
[introduced]: https://gitlab.com/gitlab-org/gitlab/-/issues/331806
[introduced 1]: https://gitlab.com/gitlab-org/gitlab/-/issues/332006
[Group wikis]: https://docs.gitlab.com/ee/user/project/wiki/group.html
[Mattermost data]: https://docs.mattermost.com/administration/config-settings.html#file-storage
[backup Rake task]: https://docs.gitlab.com/ee/raketasks/backup_restore.html#back-up-gitlab
[multiple repository storages]: https://docs.gitlab.com/ee/administration/repository_storage_paths.html
[https_gitlab.com_gitlab-org_professional-services-automation_tools_migration_congregate_-_tree_master]: https://gitlab.com/gitlab-org/professional-services-automation/tools/migration/congregate/-/tree/master
[777c7e9cdd0fa88e169dd5921cec3dc6.png]: https://img-blog.csdnimg.cn/img_convert/777c7e9cdd0fa88e169dd5921cec3dc6.png
[docs_using-congregate.md _ master _ GitLab.org _ Professional Services Automation _ Tools _ Migration _ Congregate _ GitLab]: https://gitlab.com/gitlab-org/professional-services-automation/tools/migration/congregate/-/blob/master/docs/using-congregate.md#quick-start