kubernetes的编译、打包

港控/mmm° 2022-06-04 04:50 900阅读 0赞

## **使用本地环境编译** ##

kubernetes对golang的版本有要求，具体情况见k8s development Guide:

kubernetes requires Go  
1.0 - 1.2 1.4.2  
1.3, 1.4 1.6  
1.5, 1.6 1.7 - 1.7.5  
1.7+ 1.8.1  
如果是在MAC上操作，因为MAC的shell命令是BSD风格的，因此需要安装GNU command tools。

brew install coreutils  
brew install gnu-tar  
获取代码，用本地go进行编译:

go get -d k8s.io/kubernetes  
cd $GOPATH/src/k8s.io/kubernetes  
KUBE\_BUILD\_PLATFORMS=linux/amd64 make all  
使用官方容器编译  
要在容器中编译，因为墙的缘故，需要先在本地准备好docker镜像:

gcr.io/google\_containers/kube-cross:KUBE\_BUILD\_IMAGE\_CROSS\_TAG  
TAG在文件build-image/cross/VERSION中:

$ cat build-image/cross/VERSION  
v1.7.5-2  
因为gcr.io镜像需要翻墙获取，可以使用docker.io中他人上传的cross镜像，例如:

docker pull tacylee/kube-cross:v1.7.5-2  
docker tag tacylee/kube-cross:v1.7.5-2 gcr.io/google\_containers/kube-cross:v1.7.5-2  
开始编译目标:

KUBE\_BUILD\_PLATFORMS=linux/amd64 build/run.sh make all  
将编译得到的文件从容器中拷贝出来：

build/copy-output.sh  
进入编译容器:

build/shell.sh  
一步完成编译、打镜像

因为墙的缘故，先准备好build/common.sh中指定的base镜像:

kube::build::get\_docker\_wrapped\_binaries() \{  
debian\_iptables\_version=v7  
…  
kube-proxy,gcr.io/google-containers/debian-iptables-amd64:$\{debian\_iptables\_version\}  
可以从docker.io上获取他人上传的镜像:

docker pull googlecontainer/debian-iptables-amd64:v7  
docker tag googlecontainer/debian-iptables-amd64:v7 gcr.io/google-containers/debian-iptables-amd64:v7

docker pull googlecontainer/debian-iptables-arm:v7  
docker tag googlecontainer/debian-iptables-arm:v7 gcr.io/google-containers/debian-iptables-arm:v7

docker pull googlecontainer/debian-iptables-arm64:v7  
docker tag googlecontainer/debian-iptables-arm64:v7 gcr.io/google-containers/debian-iptables-arm64:v7

docker pull googlecontainer/debian-iptables-ppc64le:v7  
docker tag googlecontainer/debian-iptables-ppc64le:v7 gcr.io/google-containers/debian-iptables-ppc64le:v7

docker pull googlecontainer/debian-iptables-s390x:v7  
docker tag googlecontainer/debian-iptables-s390x:v7 gcr.io/google-containers/debian-iptables-s390x:v7  
并且把build/lib/release.sh中的–pull去掉，避免构建镜像继续拉取镜像：

“DOCKER\[@\]"build−−pull−q−t"\{docker\_image\_tag\}” $\{docker\_build\_path\} >/dev/null

修改为:

“DOCKER\[@\]"build−q−t"\{docker\_image\_tag\}” $\{docker\_build\_path\} >/dev/null  
编译所有的目标平台，编译过程会很久，特别是test的时间会很长:

make release  
只编译linux/amd64，且略过test:

make quick-release  
如果在mac上执行还会编译mac平台，可以通过设置环境变量OSTYPE忽略mac平台:

OSTYPE=notdetected make quick-release  
默认是在容器中完成编译的，如果想打包在本地编译得到的程序，可以注释掉build/release.sh中代码:

#kube::build::verify_prereqs
    #kube::build::build_image
    #kube::build::run_build_command make cross
    
    #if [[ $KUBE_RELEASE_RUN_TESTS =~ ^[yY]$ ]]; then
    #  kube::build::run_build_command make test
    #  kube::build::run_build_command make test-integration
    #fi
    
    #kube::build::copy_output

**将编译的代码都注释，只保留打包用的代码**  
kube::release::package\_tarballs  
kube::release::package\_hyperkube  
服务端以镜像的形式发布在\_output/release-stage目录中:

$cd ./_output/
    $find ./release-stage/ -name "*.tar"
    ./release-stage/server/linux-amd64/kubernetes/server/bin/kube-aggregator.tar
    ./release-stage/server/linux-amd64/kubernetes/server/bin/kube-apiserver.tar
    ./release-stage/server/linux-amd64/kubernetes/server/bin/kube-controller-manager.tar
    ./release-stage/server/linux-amd64/kubernetes/server/bin/kube-proxy.tar
    ./release-stage/server/linux-amd64/kubernetes/server/bin/kube-scheduler.tar
    ...

客户端以压缩包的形式发布在\_output/release-tars目录中:

$ls _output/release-tars/
    kubernetes-client-darwin-386.tar.gz    kubernetes-client-windows-386.tar.gz
    kubernetes-client-darwin-amd64.tar.gz  kubernetes-client-windows-amd64.tar.gz
    kubernetes-client-linux-386.tar.gz     kubernetes-manifests.tar.gz
    kubernetes-client-linux-amd64.tar.gz   kubernetes-node-linux-amd64.tar.gz
    kubernetes-client-linux-arm.tar.gz     kubernetes-node-linux-arm.tar.gz
    kubernetes-client-linux-arm64.tar.gz   kubernetes-salt.tar.gz
    kubernetes-client-linux-ppc64le.tar.gz kubernetes-server-linux-amd64.tar.gz
    kubernetes-client-linux-s390x.tar.gz   kubernetes-src.tar.gz

开始分析  
kubernetes编译有两种方式，直接编译和在docker中编译。

如果是在MAC上操作，因为MAC的shell命令是BSD风格的，因此需要安装GNU command tools。

brew install coreutils
    brew install gnu-tar

目标平台与目标组件  
KUBE\_BUILD\_PLATFORMS指定目标平台，WHAT指定编译的组件，通过GOFLAGS和GOGCFLAGS传入编译时参数:

KUBE_BUILD_PLATFORMS=linux/amd64 make all WHAT=cmd/kubelet GOFLAGS=-v GOGCFLAGS="-N -l"

如果不指定WHAT，则编译全部。

make all是在本地环境中进行编译的。

make release和make quick-release在容器中完成编译、打包成docker镜像。

支持的目标平台  
通过环境变量KUBE\_BUILD\_PLATFORMS指定目标平台，格式为GOOS/GOARCH:

KUBE_BUILD_PLATFORMS=linux/amd64

GOOS选项:

linux, darwin, windows, netbsd  
GOARCH选项:

amd64, 386, arm, ppc64  
支持的目标组件  
目标组件在src/k8s.io/kubernetes/hack/lib/golang.sh中定义：

readonly KUBE_ALL_TARGETS=(
      "${KUBE_SERVER_TARGETS[@]}"
      "${KUBE_CLIENT_TARGETS[@]}"
      "${KUBE_TEST_TARGETS[@]}"
      "${KUBE_TEST_SERVER_TARGETS[@]}"
      cmd/gke-certificates-controller
    )
    ...
    if [[ ${#targets[@]} -eq 0 ]]; then
      targets=("${KUBE_ALL_TARGETS[@]}")
    fi

相关变量也在hack/lib/golang.sh中定义：

KUBE_SERVER_TARGETS:
        cmd/kube-proxy
        cmd/kube-apiserver
        cmd/kube-controller-manager
        cmd/cloud-controller-manager
        cmd/kubelet
        cmd/kubeadm
        cmd/hyperkube
        vendor/k8s.io/kube-aggregator
        vendor/k8s.io/kube-apiextensions-server
        plugin/cmd/kube-scheduler
    
    KUBE_CLIENT_TARGETS
        cmd/kubectl
        federation/cmd/kubefed
    
    KUBE_TEST_TARGETS
        cmd/gendocs
        cmd/genkubedocs
        cmd/genman
        cmd/genyaml
        cmd/mungedocs
        cmd/genswaggertypedocs
        cmd/linkcheck
        federation/cmd/genfeddocs
        vendor/github.com/onsi/ginkgo/ginkgo
        test/e2e/e2e.test
    
    KUBE_TEST_SERVER_TARGETS
        cmd/kubemark
        vendor/github.com/onsi/ginkgo/ginkgo
    
    cmd/gke-certificates-controller

在容器中编译  
Building kubernetes中给出了在容器中编译的方法。

build/run.sh、build/copy-out.sh、build/make-clean.sh和build/shell.sh是在容器中编译时直接使用的脚本。

build/run.sh: 在编译容器中执行命令，后面跟随的所有指令都是在容器中运行的。  
build/copy-output.sh: 将容器中编译得到的文件拷贝到本地。  
build/make-clean.sh: 清理编译过程中产生的容器和文件。  
build/shell.sh: 进入编译容器  
如下所示，make all命令将在容器中运行。

build/run.sh make all  
需要特别注意，make release以及make quick-release本身就是在容器中完成编译过程，不需要通过build/run.sh。

在容器中编译的过程与直接在本地编译的过程相同，都支持下面的make目标:

make       
    make cross
    make test
    make test-integration
    make test-cmd

build/run.sh运行时会使用目录build/build-image中的文件构建编译用的镜像，然后开始编译：

...
    kube::build::verify_prereqs
    kube::build::build_image
    kube::build::run_build_command "$@"
    ...
    build/build-image中的文件:

▾ build/
      ▾ build-image/
        ▸ cross/
          Dockerfile  <-- 用于构建镜像的Dockerfile
          rsyncd.sh*
          VERSION

build/copy-output.sh运行时，将编译后得到的文件拷贝出来：

kube::build::verify\_prereqs  
kube::build::copy\_output  
容器中编译的整个过程中会有data、rsyncd、build、rsyncd四个容器参与。

这四个容器都来自run.sh中构建的镜像，只是执行了不同的命令：

data: 创建volume，编译过程中容器退出后不会被删除，用于存放代码和编译后的文件  
rsnycd: 挂载data容器的volumes，启动rsyncd服务，接收代码上传，完成上传后被删除  
build: 挂载data容器的voluems，开始编译，编译完成后被删除  
rsyncd: 挂载data容器的volumes，启动rsyncd服务，将编译后的容器中的内容同步到本地  
构建镜像

kube::build::build_image在build/common.sh中实现:
    
    function kube::build::build_image() { 
      ...
      cp /etc/localtime "${LOCAL_OUTPUT_BUILD_CONTEXT}/"
      cp build/build-image/Dockerfile "${LOCAL_OUTPUT_BUILD_CONTEXT}/Dockerfile"
      cp build/build-image/rsyncd.sh "${LOCAL_OUTPUT_BUILD_CONTEXT}/"
      dd if=/dev/urandom bs=512 count=1 2>/dev/null | LC_ALL=C tr -dc 'A-Za-z0-9' | dd bs=32 count=1 2>/dev/null > "${LOCAL_OUTPUT_BUILD_CONTEXT}/rsyncd.password"
      chmod go= "${LOCAL_OUTPUT_BUILD_CONTEXT}/rsyncd.password"
      kube::build::update_dockerfile
      kube::build::docker_build "${KUBE_BUILD_IMAGE}" "${LOCAL_OUTPUT_BUILD_CONTEXT}" 'false'
      ...

构建的镜像名称为:

${KUBE_BUILD_IMAGE}
    => ${KUBE_BUILD_IMAGE_REPO}:${KUBE_BUILD_IMAGE_TAG}
    => kube-build:${KUBE_BUILD_IMAGE_TAG_BASE}-${KUBE_BUILD_IMAGE_VERSION}
    => kube-build:build-${KUBE_ROOT_HASH}-${KUBE_BUILD_IMAGE_VERSION_BASE}-${KUBE_BUILD_IMAGE_CROSS_TAG}
    KUBE_ROOT_HASH是根据HOSTNAME和KUBE_ROOT生成的。
    
    KUBE_BUILD_IMAGE_VERSION_BASE在build/build-image/VERSION中定义。
    
    KUBE_BUILD_IMAGE_CROSS_TAG在build/build-image/cross/VERSION中定义。
    
    Dockerfile在build/build-image中:
    
    FROM gcr.io/google_containers/kube-cross:KUBE_BUILD_IMAGE_CROSS_TAG
    ...
    可以看到是以gcr.io/google_containers/kube-cross:KUBE_BUILD_IMAGE_CROSS_TAG为基础镜像。

编译时需要翻墙获取的就是这个镜像。

创建data容器，准备volumes

镜像构建完成后，会调用kube::build::ensure\_data\_container，创建一个data容器。

function kube::build::build_image() { 
      ...
      kube::build::ensure_data_container
      kube::build::sync_to_container
      ...

一个名为$\{KUBE\_DATA\_CONTAINER\_NAME\}的data容器:

src/k8s.io/kubernetes/build/common.sh
    
    function kube::build::ensure_data_container() { 
    ...
    local -ra docker_cmd=(
      "${DOCKER[@]}" run
      --volume "${REMOTE_ROOT}"   # white-out the whole output dir
      --volume /usr/local/go/pkg/linux_386_cgo
      --volume /usr/local/go/pkg/linux_amd64_cgo
      --volume /usr/local/go/pkg/linux_arm_cgo
      --volume /usr/local/go/pkg/linux_arm64_cgo
      --volume /usr/local/go/pkg/linux_ppc64le_cgo
      --volume /usr/local/go/pkg/darwin_amd64_cgo
      --volume /usr/local/go/pkg/darwin_386_cgo
      --volume /usr/local/go/pkg/windows_amd64_cgo
      --volume /usr/local/go/pkg/windows_386_cgo
      --name "${KUBE_DATA_CONTAINER_NAME}"
      --hostname "${HOSTNAME}"
      "${KUBE_BUILD_IMAGE}"
      chown -R ${USER_ID}:${GROUP_ID}
        "${REMOTE_ROOT}"
        /usr/local/go/pkg/
    )
    "${docker_cmd[@]}"

其中：

REMOTE_ROOT="/go/src/${KUBE_GO_PACKAGE}"
    
    KUBE_DATA_CONTAINER_NAME
    =>${ KUBE_DATA_CONTAINER_NAME_BASE}-${ KUBE_BUILD_IMAGE_VERSION} =>kube-build-data-${ KUBE_ROOT_HASH}-${ KUBE_BUILD_IMAGE_VERSION}

data容器中准备好了volume，将来的rsync和build容器会通过–volume-from使用data容器的volume。

启动rsyncd容器，完成代码上传

启动运行rsyncd服务的容器，将代码通过rsync命令传输到容器中:

src/k8s.io/kubernetes/build/common.sh
    
    function kube::build::sync_to_container() { 
      kube::log::status "Syncing sources to container"
      ...
      kube::build::start_rsyncd_container
      kube::build::rsync \
        --delete \
        --filter='+ /staging/**' \
        --filter='- /.git/' \
        --filter='- /.make/' \
        --filter='- /_tmp/' \
        --filter='- /_output/' \
        --filter='- /' \
        --filter='- zz_generated.*' \
        --filter='- generated.proto' \
        "${KUBE_ROOT}/" "rsync://k8s@${KUBE_RSYNC_ADDR}/k8s/"

rsync容器启动的时候会挂载data容器的volume:

function kube::build::start_rsyncd_container() { 
        ...
      kube::build::run_build_command_ex \
        "${KUBE_RSYNC_CONTAINER_NAME}" -p 127.0.0.1:${KUBE_RSYNC_PORT}:${KUBE_CONTAINER_RSYNC_PORT} -d -- /rsyncd.sh >/dev/null
        ...
    
    function kube::build::run_build_command_ex() { 
        ...
        local -a docker_run_opts=(
            "--name=${container_name}"
            "--user=$(id -u):$(id -g)"
            "--hostname=${HOSTNAME}"
            "${DOCKER_MOUNT_ARGS[@]}"
        )
        ...

变量DOCKER\_MOUNT\_ARGS是:

DOCKER_MOUNT_ARGS=(--volumes-from "${KUBE_DATA_CONTAINER_NAME}")

所以代码其实是上传到了最开始创建的data容器的volume上。

function kube::build::rsync { 
      local -a rsync_opts=(
        --archive
        --prune-empty-dirs
        --password-file="${LOCAL_OUTPUT_BUILD_CONTEXT}/rsyncd.password"
      )
      if (( ${KUBE_VERBOSE} >= 6 )); then
        rsync_opts+=("-iv")
      fi
      if (( ${KUBE_RSYNC_COMPRESS} > 0 )); then
         rsync_opts+=("--compress-level=${KUBE_RSYNC_COMPRESS}")
      fi
      V=3 kube::log::status "Running rsync"
      rsync "${rsync_opts[@]}" "$@"
    }

启动build容器，开始编译

代码上传到data容器的volumes中后，启动build容器，在buid容器的HOME目录下执行编译命令。 执行完成后，再将buid容器中的文件同步到本地。

function kube::build::run_build_command() { 
      kube::log::status "Running build command..."
      kube::build::run_build_command_ex "${KUBE_BUILD_CONTAINER_NAME}" -- "$@"
    }

function kube::build::run_build_command_ex() {
     ...
      local -a docker_run_opts=(
        "--name=${container_name}"
        "--user=$(id -u):$(id -g)"
        "--hostname=${HOSTNAME}"
        "${DOCKER_MOUNT_ARGS[@]}"
      )
    ...
        docker_run_opts+=("$1")    //附加$@参数
    ...
      docker_run_opts+=(
        --env "KUBE_FASTBUILD=${KUBE_FASTBUILD:-false}"
        --env "KUBE_BUILDER_OS=${OSTYPE:-notdetected}"
        --env "KUBE_VERBOSE=${KUBE_VERBOSE}"
      )
    ...
      local -ra docker_cmd=(
        "${DOCKER[@]}" run "${docker_run_opts[@]}" "${KUBE_BUILD_IMAGE}")

编译时，就是直接在容器中执行传入的make命令，例如make all、make cross等。

启动rsyncd容器，将编译后文件拷贝到本地

function kube::build::copy_output() { 
      ...
      kube::build::start_rsyncd_container
      ...

常用变量  
以KUBE\_开头的变量经常在后面的脚本中使用到，这些变量一般都是通过hack/lib/init.sh引入的。

KUBE_CLIENT_TARGETS 与 KUBE_CLIENT_BINARIES
    在hack/lib/golang.sh中定义，client程序:
    
    readonly KUBE_CLIENT_TARGETS=(
      cmd/kubectl
      federation/cmd/kubefed
    )

readonly KUBE\_CLIENT\_BINARIES=(“$\{KUBE\_CLIENT\_TARGETS\[@\]\#\#\*/\}”)  
release的时候，被打包到client包里的程序。

KUBE\_NODE\_TARGETS 与 KUBE\_NODE\_BINARIES  
在hack/lib/golang.sh中定义，node中的程序:

kube::golang::node_targets() {
      local targets=(
        cmd/kube-proxy
        cmd/kubelet
      )
      echo "${targets[@]}"
    }
    
    readonly KUBE_NODE_TARGETS=($(kube::golang::node_targets))
    readonly KUBE_NODE_BINARIES=("${KUBE_NODE_TARGETS[@]##*/}")

本地编译与容器内的编译过程  
Development Guide中给出了本地编译的方法，构建过程，用make管理:

make all
    make cross
    make test
    make test-integration
    make test-cmd
    kubernetes对golang的版本有要求，编译时需要注意:
    
    kubernetes   requires Go 1.0 - 1.2     1.4.2 1.3, 1.4      1.6 1.5, 1.6      1.7 - 1.7.5 1.7+          1.8.1

顶层Makefile

# Build code.
    #
    # Args:
    #   WHAT: Directory names to build.  If any of these directories has a 'main'
    #     package, the build will produce executable files under $(OUT_DIR)/go/bin.
    #     If not specified, "everything" will be built.
    #   GOFLAGS: Extra flags to pass to 'go' when building.
    #   GOLDFLAGS: Extra linking flags passed to 'go' when building.
    #   GOGCFLAGS: Additional go compile flags passed to 'go' when building.
    #
    # Example:
    #   make
    #   make all
    #   make all WHAT=cmd/kubelet GOFLAGS=-v
    #   make all GOGCFLAGS="-N -l"
    #     Note: Use the -N -l options to disable compiler optimizations an inlining.
    #           Using these build options allows you to subsequently use source
    #           debugging tools like delve.
    make all
    src/k8s.io/kubernetes/Makefile:
    
    .PHONY: all
    ifeq ($(PRINT_HELP),y)
    all:
        @echo "$$ALL_HELP_INFO"
    else
    all: generated_files
        hack/make-rules/build.sh $(WHAT)
    endif
    hack/make-rules/build.sh开始构建, $(WHAT)是要构建的目标。
    
    generated_files
    .PHONY: generated_files
    ifeq ($(PRINT_HELP),y)
    generated_files:
        @echo "$$GENERATED_FILES_HELP_INFO"
    else
    generated_files:
        $(MAKE) -f Makefile.generated_files $@ CALLED_FROM_MAIN_MAKEFILE=1
    endif
    generated_files是在另一个Makefile中完成的：src/k8s.io/kubernetes/Makefile.generated_files
    
    Makefile.generated_files
    Makefile.generated_files正如其名，定义了用于进行代码的自动生成的Target。
    
    .PHONY: generated_files
    generated_files: gen_deepcopy gen_defaulter gen_conversion gen_openapi
    generated_files依赖的四项的工作模式相同，都是用同名的程序，对在代码注释中打了标记的文件就行处理，自动生成对应的方法。生成器代码位于名为gengo的repo中。
    
    gen_deepcopy
    
    .PHONY: gen_deepcopy
    gen_deepcopy: $(DEEPCOPY_FILES) $(DEEPCOPY_GEN)
        $(RUN_GEN_DEEPCOPY)
    执行的命令:
    
    RUN_GEN_DEEPCOPY =                                                          \
        function run_gen_deepcopy() {                                           \
            if [[ -f $(META_DIR)/$(DEEPCOPY_GEN).todo ]]; then                  \
                ./hack/run-in-gopath.sh $(DEEPCOPY_GEN)                         \
                    --v $(KUBE_VERBOSE)                                         \
                    --logtostderr                                               \
                    -i $$(cat $(META_DIR)/$(DEEPCOPY_GEN).todo | paste -sd, -)  \
                    --bounding-dirs $(PRJ_SRC_PATH)                             \
                    -O $(DEEPCOPY_BASENAME)                                     \
                    "$$@";                                                      \
            fi                                                                  \
        };                                                                      \
        run_gen_deepcopy
    DEEPCOPY_GEN是一个可执行程序:
    
    # The tool used to generate deep copies.
    DEEPCOPY_GEN := $(BIN_DIR)/deepcopy-gen
    run-in-gopaht.sh的作用是设置好GOPATH等变量，并在这个环境下运行传入的命令：
    
    set -o errexit
    set -o nounset
    set -o pipefail
    
    KUBE_ROOT=$(dirname "${BASH_SOURCE}")/..
    source "${KUBE_ROOT}/hack/lib/init.sh"
    
    # This sets up a clean GOPATH and makes sure we are currently in it.
    kube::golang::setup_env
    
    # Run the user-provided command.
    "${@}"
    问题就是deepcopy-gen做了什么事情了，到依赖项中看一下deepcopy-gen是怎样生成的。
    
    $(DEEPCOPY_FILES): $(DEEPCOPY_GEN)
        mkdir -p $$(dirname $(META_DIR)/$(DEEPCOPY_GEN))
        echo $(PRJ_SRC_PATH)/$(@D) >> $(META_DIR)/$(DEEPCOPY_GEN).todo
    
    $(DEEPCOPY_GEN):
        hack/make-rules/build.sh cmd/libs/go2idl/deepcopy-gen
        touch $@
    k8s.io/kubernetes/cmd/libs/go2idl/deepcopy-gen/main.go:
    
    // deepcopy-gen is a tool for auto-generating DeepCopy functions.
    //
    // Given a list of input directories, it will generate functions that
    // efficiently perform a full deep-copy of each type. For any type that
    // offers a `.DeepCopy()` method, it will simply call that. Otherwise it will
    // use standard value assignment whenever possible. If that is not possible it
    // will try to call its own generated copy function for the type, if the type is
    // within the allowed root packages. Failing that, it will fall back on
    // `conversion.Cloner.DeepCopy(val)` to make the copy. The resulting file will
    // be stored in the same directory as the processed source package.

标记为需要deep-copy

$(DEEPCOPY\_FILES)生成了输入参数中的.todo文件，在其中写入需要deep-copy处理的目录。

如果文件中包含注释// +k8s:deepcopy-gen，表示这个文件需要进行deep-copy处理。

k8s.io/kubernetes/Makefile.generated\_files:

# Deep-copy generation
    #
    # Any package that wants deep-copy functions generated must include a
    # comment-tag in column 0 of one file of the form:
    # // +k8s:deepcopy-gen=<VALUE>
    #
    # The <VALUE> may be one of:
    # generate: generate deep-copy functions into the package
    # register: generate deep-copy functions and register them with a
    # scheme
    gen_defaulter
    
    # Defaulter generation
    #
    # Any package that wants defaulter functions generated must include a
    # comment-tag in column 0 of one file of the form:
    # // +k8s:defaulter-gen=<VALUE>
    #
    # The <VALUE> depends on context:
    # on types:
    # true: always generate a defaulter for this type
    # false: never generate a defaulter for this type
    # on functions:
    # covers: if the function name matches SetDefault_NAME, instructs
    # the generator not to recurse
    # on packages:
    # FIELDNAME: any object with a field of this name is a candidate
    # for having a defaulter generated
    gen_conversion
    
    # Conversion generation
    #
    # Any package that wants conversion functions generated must include one or
    # more comment-tags in any .go file, in column 0, of the form:
    # // +k8s:conversion-gen=<CONVERSION_TARGET_DIR>
    #
    # The CONVERSION_TARGET_DIR is a project-local path to another directory which
    # should be considered when evaluating peer types for conversions. Types which
    # are found in the source package (where conversions are being generated)
    # but do not have a peer in one of the target directories will not have
    # conversions generated.
    gen_openapi
    
    # Open-api generation
    #
    # Any package that wants open-api functions generated must include a
    # comment-tag in column 0 of one file of the form:
    # // +k8s:openapi-gen=true
    ## hack/make-rules/build.sh
    hack/make-rules/build.sh
    build.sh用来编译具体的目标。
    
    构建目标
    
    set -o errexit
    set -o nounset
    set -o pipefail
    
    KUBE_ROOT=$(dirname "${BASH_SOURCE}")/../..
    KUBE_VERBOSE="${KUBE_VERBOSE:-1}"
    source "${KUBE_ROOT}/hack/lib/init.sh"
    
    kube::golang::build_binaries "$@"
    kube::golang::place_bins

如果没有传入构建目标，默认构建所有目标。

函数kube::golang::build\_binaries()，接收构建目标，进行构建。

设置编译环境

设置环境变量: kube::golang::setup\_env()

编译时的GOPATH为: \_output/local/go

KUBE_OUTPUT_SUBPATH="${KUBE_OUTPUT_SUBPATH:-_output/local}"
    KUBE_OUTPUT="${KUBE_ROOT}/${KUBE_OUTPUT_SUBPATH}"
    KUBE_GOPATH="${KUBE_OUTPUT}/go"
    GOPATH=${KUBE_GOPATH}
    GOPATH="${GOPATH}:${KUBE_EXTRA_GOPATH}"

可以通过设置环境变量KUBE\_EXTRA\_GOPATH，增加GOPATH中的路径  
编译时源码路径: \_out/local/go/src/k8s.io/kuberne

KUBE_GO_PACKAGE=k8s.io/kubernetes
    ${KUBE_GOPATH}/src/${KUBE_GO_PACKAGE}

选项:

goflags=(${KUBE_GOFLAGS:-})
    gogcflags="${KUBE_GOGCFLAGS:-}"
    goldflags="${KUBE_GOLDFLAGS:-} $(kube::version::ldflags)"

链接时选项，就是通过-X，修改pkg/version/和vendor/k8s.io/client-go/pkg/version中的变量：

kube::version::ldflag() {
      local key=${
       1}
      local val=${
       2}
    
      echo "-X ${KUBE_GO_PACKAGE}/pkg/version.${key}=${val}"
      echo "-X ${KUBE_GO_PACKAGE}/vendor/k8s.io/client-go/pkg/version.${key}=${val}"
    }
    
    # Prints the value that needs to be passed to the -ldflags parameter of go build
    # in order to set the kubernetes based on the git tree status.
    kube::version::ldflags() {
      kube::version::get_version_vars
    
      local -a ldflags=($(kube::version::ldflag "buildDate" "$(date -u +'%Y-%m-%dT%H:%M:%SZ')"))
      if [[ -n ${KUBE_GIT_COMMIT-} ]]; then
        ldflags+=($(kube::version::ldflag "gitCommit" "${KUBE_GIT_COMMIT}"))
        ldflags+=($(kube::version::ldflag "gitTreeState" "${KUBE_GIT_TREE_STATE}"))
      fi
    
      if [[ -n ${KUBE_GIT_VERSION-} ]]; then
        ldflags+=($(kube::version::ldflag "gitVersion" "${KUBE_GIT_VERSION}"))
      fi
    
      if [[ -n ${KUBE_GIT_MAJOR-} && -n ${KUBE_GIT_MINOR-} ]]; then
        ldflags+=(
          $(kube::version::ldflag "gitMajor" "${KUBE_GIT_MAJOR}")
          $(kube::version::ldflag "gitMinor" "${KUBE_GIT_MINOR}")
        )
      fi
    
      # The -ldflags parameter takes a single string, so join the output.
      echo "${ldflags[*]-}"
    }
    运行时，传入的以-开始的参数，被认为是新增的goflags:
    
    for arg; do
      if [[ "${arg}" == "--use_go_build" ]]; then
        use_go_build=true
      elif [[ "${arg}" == -* ]]; then
        # Assume arguments starting with a dash are flags to pass to go.
        goflags+=("${arg}")
      else
        targets+=("${arg}")
      fi
    done

准备工具链

准备编译时工具链: kube::golang::build\_kube\_toolchain()

kube::golang::build_kube_toolchain() {
      local targets=(
        hack/cmd/teststale
        vendor/github.com/jteeuwen/go-bindata/go-bindata
      )
    
      local binaries
      binaries=($(kube::golang::binaries_from_targets "${targets[@]}"))
    
      kube::log::status "Building the toolchain targets:" "${binaries[@]}"
      go install "${goflags[@]:+${goflags[@]}}" \
            -gcflags "${gogcflags}" \
            -ldflags "${goldflags}" \
            "${binaries[@]:+${binaries[@]}}"
    }

go-bindata用于将任意文件编译到go源码文件中。

源码预处理

go generate生成bindata

readonly KUBE_BINDATAS=(
      test/e2e/generated/gobindata_util.go
    )
    ...
    for bindata in ${KUBE_BINDATAS[@]}; do
      if [[ -f "${KUBE_ROOT}/${bindata}" ]]; then
        go generate "${goflags[@]:+${goflags[@]}}" "${KUBE_ROOT}/${bindata}"
      fi
    done
    go generate会运行目标.go文件中以//go:generate开始的注释行中的指令。
    
    在test/e2e/generated/gobindata_util.go中，运行generate-bindata.sh:
    
    //go:generate ../../../hack/generate-bindata.sh

generate-bindata.sh将一下二进制文件打包到对应的源码中:

# These are files for e2e tests.
    BINDATA_OUTPUT="test/e2e/generated/bindata.go"
    go-bindata -nometadata -o "${BINDATA_OUTPUT}.tmp" -pkg generated \
        -ignore .jpg -ignore .png -ignore .md \
        "examples/..." \
        "test/e2e/testing-manifests/..." \
        "test/images/..." \
        "test/fixtures/..."
    
    BINDATA_OUTPUT="pkg/generated/bindata.go"
    go-bindata -nometadata -nocompress -o "${BINDATA_OUTPUT}.tmp" -pkg generated \
        -ignore .jpg -ignore .png -ignore .md \
        "translations/..."

设置目标平台

设置目标平台，kube::golang::set\_platform\_envs()

export GOOS=${ platform%/*} export GOARCH=${platform##*/}

通过环境变量KUBE\_BUILD\_PLATFORMS指定目标平台，格式为OS/ARCH

local -a platforms=(${KUBE_BUILD_PLATFORMS:-})
    if [[ ${#platforms[@]} -eq 0 ]]; then
      platforms=("${host_platform}")
    fi

例如：

darwin/amd64
    GOARCH:
    
    目标CPU结构，amd64, 386, arm, ppc64
    GOOS:

目标操作系统，linux, darwin, windows, netbsd  
开始编译

开始编译，kube::golang::build\_binaries\_for\_platform()。

编译时将目标分为静态链接、动态链接、测试三组。

//binary就是上面列出的构建目标

for binary in "${binaries[@]}"; do
        if [[ "${binary}" =~ ".test"$ ]]; then
          tests+=($binary)
        elif kube::golang::is_statically_linked_library "${binary}"; then
          statics+=($binary)
        else
          nonstatics+=($binary)
        fi
    done

以.test结尾的为测试用，除了下面指定为静态链接的，其它为动态链接：

readonly KUBE_STATIC_LIBRARIES=(
      cloud-controller-manager
      kube-apiserver
      kube-controller-manager
      kube-scheduler
      kube-proxy
      kube-aggregator
      kubeadm
      kubectl
    )

静态链接目标的编译：

CGO_ENABLED=0 go build -o "${outfile}" \
    "${goflags[@]:+${goflags[@]}}" \
    -gcflags "${gogcflags}" \
    -ldflags "${goldflags}" \
    "${binary}"

非静态目标的编译：

go build -o "${outfile}" \
    "${goflags[@]:+${goflags[@]}}" \
    -gcflags "${gogcflags}" \
    -ldflags "${goldflags}" \
    "${binary}"
    make update
    .PHONY: update
    ifeq ($(PRINT_HELP),y)
    update:
        @echo "$$UPDATE_HELP_INFO"
    else
    update:
        hack/update-all.sh
    endif
    hack/update-all.sh
    update-all.sh会依次执行hack/XXX.sh，完成更新操作。

BASH_TARGETS="
        update-generated-protobuf update-codegen update-codecgen update-generated-docs update-generated-swagger-docs update-swagger-spec update-openapi-spec update-api-reference-docs update-federation-openapi-spec update-staging-client-go update-staging-godeps update-bazel" hack/update-staging-client-go.sh

update-staging-client-go.sh目的是更新staging/src/k8s.io/client-go/中的文件。

首先会检查是否已经执行go restore，确保kubernetes的依赖包已经安装在$GOPATH中。

之后运行staging/copy.sh，copy.sh的作用是更新staging/src/k8s.io/client-go，具体过程见：k8s的第三方包的使用

make release

make release直接执行build/releash.sh脚本：

release:
        build/release.sh

如果变量KUBE\_FASTBUILD为“true”，只发布linux/amd64，否则发布所有平台。

hack/lib/golang.sh:
    
    if [[ "${KUBE_FASTBUILD:-}" == "true" ]]; then
      readonly KUBE_SERVER_PLATFORMS=(linux/amd64)
      readonly KUBE_NODE_PLATFORMS=(linux/amd64)
      if [[ "${KUBE_BUILDER_OS:-}" == "darwin"* ]]; then
        readonly KUBE_TEST_PLATFORMS=(
          darwin/amd64
          linux/amd64
        )
        readonly KUBE_CLIENT_PLATFORMS=(
          darwin/amd64
          linux/amd64
        )
    ...
    build/release.sh

release.sh首先在容器中进行编译，然后进行打包：

src/k8s.io/kubernetes/build/release.sh:
    
    kube::build::verify_prereqs
    kube::build::build_image
    kube::build::run_build_command make cross
    
    if [[ $KUBE_RELEASE_RUN_TESTS =~ ^[yY]$ ]]; then
      kube::build::run_build_command make test
      kube::build::run_build_command make test-integration
    fi
    
    kube::build::copy_output
    
    kube::release::package_tarballs
    kube::release::package_hyperkube
    kube::release::package_tarballs
    
    function kube::release::package_tarballs() {
      # Clean out any old releases
      rm -rf "${RELEASE_DIR}"
      mkdir -p "${RELEASE_DIR}"
      #源代码打包
      kube::release::package_src_tarball &
      #client端程序打包
      kube::release::package_client_tarballs &
      kube::release::package_salt_tarball &
      kube::release::package_kube_manifests_tarball &
      kube::util::wait-for-jobs || { kube::log::error "previous tarball phase failed"; return 1; }
    
      # _node and _server tarballs depend on _src tarball
      kube::release::package_node_tarballs &
      kube::release::package_server_tarballs &
      kube::util::wait-for-jobs || { kube::log::error "previous tarball phase failed"; return 1; }
    
      kube::release::package_final_tarball & # _final depends on some of the previous phases
      kube::release::package_test_tarball & # _test doesn't depend on anything
      kube::util::wait-for-jobs || { kube::log::error "previous tarball phase failed"; return 1; }
    }
    kube::release::package_src_tarball()
    
    # Package the source code we built, for compliance/licensing/audit/yadda.
    function kube::release::package_src_tarball() {
      kube::log::status "Building tarball: src"
      local source_files=(
        $(cd "${KUBE_ROOT}" && find . -mindepth 1 -maxdepth 1 \
          -not $ \
            \( -path ./_\*        -o \
               -path ./.git\*     -o \
               -path ./.config\* -o \
               -path ./.gsutil\*    \
            $ -prune \
          \))
      )
      "${TAR}" czf "${RELEASE_DIR}/kubernetes-src.tar.gz" -C "${KUBE_ROOT}" "${source_files[@]}"
    }
    kube::release::package_client_tarballs

将变量KUBE\_CLIENT\_BINARIES中的程序打包:

.:
    kubernetes
    
    ./kubernetes:
    client
    
    ./kubernetes/client:
    bin
    
    ./kubernetes/client/bin:
    kubectl  kubefed
    kube::release::package_salt_tarball

将cluster/saltbase/目录打包, 这是一套用saltstack部署k8s集群的脚本。

kube::release::package_kube_manifests_tarball
    
    # This will pack kube-system manifests files for distros without using salt
    # such as GCI and Ubuntu Trusty. We directly copy manifests from
    # cluster/addons and cluster/saltbase/salt. The script of cluster initialization
    # will remove the salt configuration and evaluate the variables in the manifests.
    kube::release::package_node_tarballs

将变量KUBE\_NODE\_BINARIES中列出的程序打包:

.:
    kubernetes
    
    ./kubernetes:
    kubernetes-src.tar.gz  LICENSES  node
    
    ./kubernetes/node:
    bin
    
    ./kubernetes/node/bin:
    kubectl  kubefed  kubelet  kube-proxy
    kube::release::package_server_tarballs

将变量KUBE\_NODE\_BINARIES和变量KUBE\_SERVER\_BINARIES中列出的程序打包:

.:
    kubernetes
    
    ./kubernetes:
    addons  kubernetes-src.tar.gz  LICENSES  server
    
    ./kubernetes/addons:
    
    ./kubernetes/server:
    bin
    
    ./kubernetes/server/bin:
    cloud-controller-manager  kube-aggregator          kubectl  kube-proxy
    hyperkube                 kube-apiserver           kubefed  kube-scheduler
    kubeadm                   kube-controller-manager  kubelet

并为server端的程序创建docker image:

kube::release::create_docker_images_for_server "${release_stage}/server/bin" "${arch}"
    kube::release::create_docker_images_for_server

在lib/release.sh中定义，分别将这些程序打包到各自的docker image中:

local binaries=($(kube::build::get_docker_wrapped_binaries ${arch}))
    get_docker_wrapped_binaries在build/common.sh中定义:
    
    kube::build::get_docker_wrapped_binaries() {
      debian_iptables_version=v7
      case $1 in
        "amd64")
            local targets=(
              kube-apiserver,busybox
              kube-controller-manager,busybox
              kube-scheduler,busybox
              kube-aggregator,busybox
              kube-proxy,gcr.io/google-containers/debian-iptables-amd64:${debian_iptables_version}
            );;
        "arm")
            local targets=(
              kube-apiserver,armel/busybox
              kube-controller-manager,armel/busybox
              kube-scheduler,armel/busybox
              kube-aggregator,armel/busybox
              kube-proxy,gcr.io/google-containers/debian-iptables-arm:${debian_iptables_version}
            );;
        ......
      esac
    
    echo "${targets[@]}"
    }

每个target的,之前是要打包进容器的程序名binary\_name，,之后是base\_image。

Dockerfile:
    
    printf " FROM ${base_image} \n ADD ${binary_name} /usr/local/bin/${binary_name}\n" > ${docker_file_path}

最终镜像保存在:

"${DOCKER[@]}" save ${docker_image_tag} > ${binary_dir}/${binary_name}.tar
    
    ./_output/elease-stage/server/linux-amd64/kubernetes/server/bin/kube-controller-manager.tar
    ./_output/elease-stage/server/linux-amd64/kubernetes/server/bin/kube-scheduler.tar
    ./_output/elease-stage/server/linux-amd64/kubernetes/server/bin/kube-proxy.tar
    ./_output/elease-stage/server/linux-amd64/kubernetes/server/bin/kube-apiserver.tar
    ./_output/elease-stage/server/linux-amd64/kubernetes/server/bin/kube-aggregator.tar
    kube::release::package_final_tarball
    
    # This is all the platform-independent stuff you need to run/install kubernetes.
    # Arch-specific binaries will need to be downloaded separately (possibly by
    # using the bundled cluster/get-kube-binaries.sh script).
    # Included in this tarball:
    #   - Cluster spin up/down scripts and configs for various cloud providers
    #   - Tarballs for salt configs that are ready to be uploaded
    #     to master by whatever means appropriate.
    #   - Examples (which may or may not still work)
    #   - The remnants of the docs/ directory
    kube::release::package_test_tarball() {
    
    # This is the stuff you need to run tests from the binary distribution.
    release

release已经被作为一个单独项目发布k8s release发布。

制作

cd rpm
    ./docker-build.sh

make all的输出：

+++ [0518 15:19:14] Building the toolchain targets:
        k8s.io/kubernetes/hack/cmd/teststale
        k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata
    +++ [0518 15:19:14] Generating bindata:
        test/e2e/generated/gobindata_util.go
    ~/Work/Docker/GOPATH/src/k8s.io/kubernetes ~/Work/Docker/GOPATH/src/k8s.io/kubernetes/test/e2e/generated
    ~/Work/Docker/GOPATH/src/k8s.io/kubernetes/test/e2e/generated
    +++ [0518 15:19:14] Building go targets for darwin/amd64:
        cmd/libs/go2idl/deepcopy-gen
    +++ [0518 15:19:18] Building the toolchain targets:
        k8s.io/kubernetes/hack/cmd/teststale
        k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata
    +++ [0518 15:19:18] Generating bindata:
        test/e2e/generated/gobindata_util.go
    ~/Work/Docker/GOPATH/src/k8s.io/kubernetes ~/Work/Docker/GOPATH/src/k8s.io/kubernetes/test/e2e/generated
    ~/Work/Docker/GOPATH/src/k8s.io/kubernetes/test/e2e/generated
    +++ [0518 15:19:19] Building go targets for darwin/amd64:
        cmd/libs/go2idl/defaulter-gen
    +++ [0518 15:19:23] Building the toolchain targets:
        k8s.io/kubernetes/hack/cmd/teststale
        k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata
    +++ [0518 15:19:23] Generating bindata:
        test/e2e/generated/gobindata_util.go
    ~/Work/Docker/GOPATH/src/k8s.io/kubernetes ~/Work/Docker/GOPATH/src/k8s.io/kubernetes/test/e2e/generated
    ~/Work/Docker/GOPATH/src/k8s.io/kubernetes/test/e2e/generated
    +++ [0518 15:19:23] Building go targets for darwin/amd64:
        cmd/libs/go2idl/conversion-gen
    +++ [0518 15:19:27] Building the toolchain targets:
        k8s.io/kubernetes/hack/cmd/teststale
        k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata
    +++ [0518 15:19:27] Generating bindata:
        test/e2e/generated/gobindata_util.go
    ~/Work/Docker/GOPATH/src/k8s.io/kubernetes ~/Work/Docker/GOPATH/src/k8s.io/kubernetes/test/e2e/generated
    ~/Work/Docker/GOPATH/src/k8s.io/kubernetes/test/e2e/generated
    +++ [0518 15:19:28] Building go targets for darwin/amd64:
        cmd/libs/go2idl/openapi-gen
    +++ [0518 15:19:33] Building the toolchain targets:
        k8s.io/kubernetes/hack/cmd/teststale
        k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata
    +++ [0518 15:19:33] Generating bindata:
        test/e2e/generated/gobindata_util.go
    ~/Work/Docker/GOPATH/src/k8s.io/kubernetes ~/Work/Docker/GOPATH/src/k8s.io/kubernetes/test/e2e/generated
    ~/Work/Docker/GOPATH/src/k8s.io/kubernetes/test/e2e/generated
    +++ [0518 15:19:34] Building go targets for darwin/amd64:
        cmd/kube-proxy
        cmd/kube-apiserver
        cmd/kube-controller-manager
        cmd/cloud-controller-manager
        cmd/kubelet
        cmd/kubeadm
        cmd/hyperkube
        vendor/k8s.io/kube-aggregator
        plugin/cmd/kube-scheduler
        cmd/kubectl
        federation/cmd/kubefed
        cmd/gendocs
        cmd/genkubedocs
        cmd/genman
        cmd/genyaml
        cmd/mungedocs
        cmd/genswaggertypedocs
        cmd/linkcheck
        examples/k8petstore/web-server/src
        federation/cmd/genfeddocs
        vendor/github.com/onsi/ginkgo/ginkgo
        test/e2e/e2e.test
        cmd/kubemark
        vendor/github.com/onsi/ginkgo/ginkgo
        test/e2e_node/e2e_node.test
        cmd/gke-certificates-controller

**参考**  
[http://www.lijiaocn.com/%E9%A1%B9%E7%9B%AE/2017/05/15/Kubernetes-compile.html\#][http_www.lijiaocn.com_E9_A1_B9_E7_9B_AE_2017_05_15_Kubernetes-compile.html]使用本地环境编译

[http_www.lijiaocn.com_E9_A1_B9_E7_9B_AE_2017_05_15_Kubernetes-compile.html]: http://www.lijiaocn.com/%E9%A1%B9%E7%9B%AE/2017/05/15/Kubernetes-compile.html#