运维调试记录：Ubuntu14.04下部署Opendaylight Nitrogen集群

待我称王封你为后i 2022-05-29 09:24 132阅读 0赞

--------------------

**建议参考官网教程：**[Setting Up Clustering][]

--------------------

# 一、实验环境 #

*  主机节点系统版本： Ubuntu 14.04 (64bit)

odl@mpodl:~$ uname -a
    Linux mpodl 4.2.0-27-generic #32~14.04.1-Ubuntu SMP Fri Jan 22 15:32:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

*  主机节点硬件配置：单核CPU+4GB内存+50GB硬盘

odl@mpodl:~$ sudo lshw        
        description: Computer
        product: Standard PC (i440FX + PIIX, 1996) ()
        vendor: QEMU
        version: pc-i440fx-trusty
        width: 64 bits
        capabilities: smbios-2.4 dmi-2.4 vsyscall32
        configuration: boot=normal uuid=053D43B6-2E3C-CEA4-4C52-833DDD1749BE
      *-core
           description: Motherboard
           physical id: 0
         # CPU信息 
         *-cpu
              description: CPU
              product: QEMU Virtual CPU version 2.0.0
              vendor: Intel Corp.
              physical id: 401
              bus info: cpu@0
              slot: CPU 1
              size: 2GHz
              capacity: 2GHz
              width: 64 bits
         # 内存信息
         *-memory
              description: System Memory
              physical id: 1000
              size: 4GiB
         # 硬盘信息
         *-disk
               description: ATA Disk
               product: QEMU HARDDISK
               physical id: 0.0.0
               bus info: scsi@0:0.0.0
               logical name: /dev/sda
               version: 0
               serial: QM00001
               size: 50GiB (53GB)
               *-volume:0
                    description: EXT4 volume
                    vendor: Linux
                    physical id: 1
                    bus info: scsi@0:0.0.0,1
                    logical name: /dev/sda1
                    logical name: /
                    version: 1.0
                    serial: 65d80188-ddfb-4f08-8018-a0c2e1da8af3
                    size: 46GiB
                    capacity: 46GiB
               *-volume:1
                    description: Extended partition
                    physical id: 2
                    bus info: scsi@0:0.0.0,2
                    logical name: /dev/sda2
                    size: 4093MiB
                    capacity: 4093MiB

*  集群环境：3台主机节点

Cluster_Node1: Ubuntu 14.04 -- [IP_Addr]=192.168.1.124
    Cluster_Node2: Ubuntu 14.04 -- [IP_Addr]=192.168.1.125
    Cluster_Node3: Ubuntu 14.04 -- [IP_Addr]=192.168.1.104

*  所有主机节点都连接到同一台OpenvSwitch交换机上，且能互相PING通！！！
 *  每台主机节点都安装配置有JDK 1.8，安装方法见：[Ubuntu下通过PPA方式安装Java 8并自动配置环境变量][Ubuntu_PPA_Java 8]

# 二、部署方法 #

## 1. 下载Opendaylight Nitrogen ##

在每台主机节点上执行如下命令，下载tar.gz格式的压缩包到用户目录下的ODL\_N子目录：

odl@mpodl:~/ODL_N$ wget -P . https://nexus.opendaylight.org/content/repositories/public/org/opendaylight/integration/karaf/0.7.2/karaf-0.7.2.tar.gz

或在官网下载，然后通过xftp软件从本地上传到服务器的Opendaylight运行环境中。

官网地址：[https://www.opendaylight.org/technical-community/getting-started-for-developers/downloads-and-documentation][https_www.opendaylight.org_technical-community_getting-started-for-developers_downloads-and-documentation]

## 2. 安装Opendaylight Nitrogen ##

在每台主机节点上执行如下命令，解压压缩包到ODL\_N子目录：

odl@mpodl:~/ODL_N$ tar -zxvf karaf-0.7.2.tar.gz

解压完成，即认为Opendaylight Nitrogen安装成功。

## 3. 配置Opendaylight Nitrogen集群 ##

在每台主机节点下执行如下命令，完成集群脚本配置工作：

命令格式：`sudo bash ./karaf-0.7.2/bin/configure_cluster.sh [index] [seed_node_list]`

其中，`[index]`为正整数，表示在`[seed_node_list]`中对应哪个主机节点上配置集群脚本。另外，`[seed_node_list]`是组成集群的各个主机节点对应的IP地址列表，以空格或逗号隔开。

因此，具体执行命令如下：

# Cluster_Node1: IP=192.168.1.124
    odl@mpodl:~/ODL_N$ sudo bash ./karaf-0.7.2/bin/configure_cluster.sh 1 192.168.1.124 192.168.1.125 192.168.1.104
    
    # Cluster_Node2: IP=192.168.1.125
    odl@mpodl:~/ODL_N$ sudo bash ./karaf-0.7.2/bin/configure_cluster.sh 2 192.168.1.124 192.168.1.125 192.168.1.104
    
    # Cluster_Node3: IP=192.168.1.104
    odl@mpodl:~/ODL_N$ sudo bash ./karaf-0.7.2/bin/configure_cluster.sh 3 192.168.1.124 192.168.1.125 192.168.1.104

当执行完之后，结果如下所示：

# Cluster_Node1: IP=192.168.1.124
    odl@mpodl:~/ODL_N$ sudo bash ./karaf-0.7.2/bin/configure_cluster.sh 1 192.168.1.124 192.168.1.125 192.168.1.104
    ################################################
    ## Configure Cluster ##
    ################################################
    Configuring unique name in akka.conf
    Configuring hostname in akka.conf
    Configuring data and rpc seed nodes in akka.conf
    modules = [
    
        {
            name = "inventory"
            namespace = "urn:opendaylight:inventory"
            shard-strategy = "module"
        },
        {
            name = "topology"
            namespace = "urn:TBD:params:xml:ns:yang:network-topology"
            shard-strategy = "module"
        },
        {
            name = "toaster"
            namespace = "http://netconfcentral.org/ns/toaster"
            shard-strategy = "module"
        }
    ]
    Configuring replication type in module-shards.conf
    ################################################
    ## NOTE: Manually restart controller to ##
    ## apply configuration. ##
    ################################################

备注：

（1）执行如上命令后，会在`karaf-0.7.2/configuration`目录下生成`initial`子目录，结果如下所示：

odl@mpodl:~/ODL_N$ ls ./karaf-0.7.2/configuration/
    context.xml  factory  initial  logback.xml  tomcat-logging.properties  tomcat-server.xml
    
    odl@mpodl:~/ODL_N$ ls ./karaf-0.7.2/configuration/initial/
    akka.conf  modules.conf  module-shards.conf

可见，集群配置脚本在`initial`子目录下生成了`akka.conf`、`modules.conf`和`module-shards.conf`三个配置文件。

（2）查看第一台Ubuntu主机节点的`akka.conf`文件内容，具体如下所示：

odl@mpodl:~/ODL_N$ cat ./karaf-0.7.2/configuration/initial/akka.conf 
    
    odl-cluster-data {
      akka {
        remote {
          artery {
            enabled = off
            canonical.hostname = "192.168.1.124"   # 本机IP地址
            canonical.port = 2550
          }
          netty.tcp {
            hostname = "192.168.1.124"             # 本机IP地址
            port = 2550
          }
          # when under load we might trip a false positive on the failure detector # transport-failure-detector { # heartbeat-interval = 4 s # acceptable-heartbeat-pause = 16s # } } cluster { # Remove ".tcp" when using artery. # 集群节点列表 seed-nodes = ["akka.tcp://opendaylight-cluster-data@192.168.1.124:2550", "akka.tcp://opendaylight-cluster-data@192.168.1.125:2550", "akka.tcp://opendaylight-cluster-data@192.168.1.104:2550"] roles = ["member-1"] } persistence { # By default the snapshots/journal directories live in KARAF_HOME. You can choose to put it somewhere else by # modifying the following two properties. The directory location specified may be a relative or absolute path. # The relative path is always relative to KARAF_HOME. # snapshot-store.local.dir = "target/snapshots" # journal.leveldb.dir = "target/journal" journal { leveldb { # Set native = off to use a Java-only implementation of leveldb. # Note that the Java-only version is not currently considered by Akka to be production quality. # native = off } } } } }

同样地，第二台和第三台Ubuntu主机节点的`akka.conf`文件内容相似，只是对应的IP地址不同。

（3）查看第一台Ubuntu主机节点的`module-shards.conf`文件内容，具体如下所示：

odl@mpodl:~/ODL_N$ cat ./karaf-0.7.2/configuration/initial/module-shards.conf 
    module-shards = [
        {
            name = "default"
            shards = [
                {
                    name = "default"
                    replicas = ["member-1",
                    "member-2",
                    "member-3"]
                }
            ]
        },
        {
            name = "inventory"
            shards = [
                {
                    name="inventory"
                    replicas = ["member-1",
                    "member-2",
                    "member-3"]
                }
            ]
        },
        {
            name = "topology"
            shards = [
                {
                    name="topology"
                    replicas = ["member-1",
                    "member-2",
                    "member-3"]
                }
            ]
        },
        {
            name = "toaster"
            shards = [
                {
                    name="toaster"
                    replicas = ["member-1",
                    "member-2",
                    "member-3"]
                }
            ]
        }
    ]

同样地，第二台和第三台Ubuntu主机节点的`module-shards.conf`文件内容完全相同。

## 4. 启动Opendaylight Nitrogen集群 ##

在每台主机节点下执行如下命令，完成Opendaylight节点启动工作：

odl@mpodl:~/ODL_N$ ./karaf-0.7.2/bin/karaf karaf: JAVA_HOME not set; results may vary Apache Karaf starting up. Press Enter to open the shell now... 100% [========================================================================] Karaf started in 8s. Bundle stats: 208 active, 209 total ________ ________ .__ .__ .__ __ \_____ \ ______ ____ ____ \______ \ _____ ___.__.| | |__| ____ | |___/ |_ / | \\____ \_/ __ \ / \ | | \\__ \< | || | | |/ ___\| | \ __\ / | \ |_> > ___/| | \| ` \/ __ \\___ || |_| / /_/ > Y \ | \_______ / __/ \___ >___| /_______ (____ / ____||____/__\___ /|___| /__| \/|__| \/ \/ \/ \/\/ /_____/ \/ Hit '<tab>' for a list of available commands and '[cmd] --help' for help on a specific command. Hit '<ctrl-d>' or type 'system:shutdown' or 'logout' to shutdown OpenDaylight. opendaylight-user@root>

然后，执行命令`feature:list -i`检查`odl-mdsal-clustering`是否处于已安装状态。如果没有安装，则执行命令`feature:install odl-mdsal-clustering`完成对应Feature的安装。

## 5. 检查Opendaylight Nitrogen集群是否建立 ##

在每台主机节点下执行如下命令，获取主机节点的角色信息（Leader/Follower）：

opendaylight-user@root> ld | grep clustering

于是，在主机节点一，输出如下：

2018-03-21 13:22:45,688 | INFO  | d-dispatcher-125 | Shard                            | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | member-1-shard-prefix-configuration-shard-config (Candidate): Starting new election term 21
    2018-03-21 13:22:45,741 | INFO  | d-dispatcher-125 | Shard                            | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | member-1-shard-prefix-configuration-shard-config (Candidate) :- Switching from behavior Candidate to Leader, election term: 21
    2018-03-21 13:22:45,742 | INFO  | ult-dispatcher-5 | RoleChangeNotifier               | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | RoleChangeNotifier for member-1-shard-prefix-configuration-shard-config , received role change from Candidate to Leader
    2018-03-21 13:22:45,821 | INFO  | d-dispatcher-121 | Shard                            | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | member-1-shard-prefix-configuration-shard-operational (Candidate): Starting new election term 21
    2018-03-21 13:22:45,858 | INFO  | d-dispatcher-125 | Shard                            | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | member-1-shard-prefix-configuration-shard-operational (Candidate) :- Switching from behavior Candidate to Leader, election term: 21
    2018-03-21 13:22:45,858 | INFO  | ult-dispatcher-5 | RoleChangeNotifier               | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | RoleChangeNotifier for member-1-shard-prefix-configuration-shard-operational , received role change from Candidate to Leader
    2018-03-21 13:22:45,872 | INFO  | d-dispatcher-125 | EntityOwnershipShard             | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | member-1-shard-entity-ownership-operational (Candidate): Starting new election term 21
    2018-03-21 13:22:45,912 | INFO  | d-dispatcher-145 | EntityOwnershipShard             | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | member-1-shard-entity-ownership-operational (Candidate) :- Switching from behavior Candidate to Leader, election term: 21
    2018-03-21 13:22:45,921 | INFO  | lt-dispatcher-21 | RoleChangeNotifier               | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | RoleChangeNotifier for member-1-shard-entity-ownership-operational , received role change from Candidate to Leader

在主机节点二，输出如下：

2018-03-21 13:22:36,286 | INFO  | ult-dispatcher-4 | RoleChangeNotifier               | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | RoleChangeNotifier for member-2-shard-prefix-configuration-shard-config , received role change from null to Follower
    2018-03-21 13:22:36,287 | INFO  | ult-dispatcher-4 | RoleChangeNotifier               | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | RoleChangeNotifier for member-2-shard-prefix-configuration-shard-operational , received role change from null to Follower
    2018-03-21 13:22:36,287 | INFO  | ult-dispatcher-4 | RoleChangeNotifier               | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | RoleChangeNotifier for member-2-shard-prefix-configuration-shard-config , registered listener akka://opendaylight-cluster-data/user/shardmanager-config
    2018-03-21 13:22:36,287 | INFO  | ult-dispatcher-4 | RoleChangeNotifier               | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | RoleChangeNotifier for member-2-shard-prefix-configuration-shard-operational , registered listener akka://opendaylight-cluster-data/user/shardmanager-operational
    2018-03-21 13:22:36,305 | INFO  | rd-dispatcher-32 | EntityOwnershipShard             | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | Starting recovery for member-2-shard-entity-ownership-operational with journal batch size 1
    2018-03-21 13:22:36,313 | INFO  | rd-dispatcher-38 | EntityOwnershipShard             | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | Recovery completed - Switching actor to Follower - Persistence Id =  member-2-shard-entity-ownership-operational Last index in log = -1, snapshotIndex = -1, snapshotTerm = -1, journal-size = 0
    2018-03-21 13:22:36,317 | INFO  | ult-dispatcher-2 | RoleChangeNotifier               | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | RoleChangeNotifier for member-2-shard-entity-ownership-operational , received role change from null to Follower

在主机节点三，输出如下：

2018-03-21 13:22:39,418 | INFO  | ult-dispatcher-6 | RoleChangeNotifier               | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | RoleChangeNotifier for member-3-shard-prefix-configuration-shard-operational , received role change from null to Follower
    2018-03-21 13:22:39,418 | INFO  | ult-dispatcher-6 | RoleChangeNotifier               | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | RoleChangeNotifier for member-3-shard-prefix-configuration-shard-config , received role change from null to Follower
    2018-03-21 13:22:39,418 | INFO  | ult-dispatcher-6 | RoleChangeNotifier               | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | RoleChangeNotifier for member-3-shard-prefix-configuration-shard-operational , registered listener akka://opendaylight-cluster-data/user/shardmanager-operational
    2018-03-21 13:22:39,418 | INFO  | ult-dispatcher-6 | RoleChangeNotifier               | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | RoleChangeNotifier for member-3-shard-prefix-configuration-shard-config , registered listener akka://opendaylight-cluster-data/user/shardmanager-config
    2018-03-21 13:22:39,466 | INFO  | rd-dispatcher-23 | EntityOwnershipShard             | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | Starting recovery for member-3-shard-entity-ownership-operational with journal batch size 1
    2018-03-21 13:22:39,470 | INFO  | rd-dispatcher-23 | EntityOwnershipShard             | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | Recovery completed - Switching actor to Follower - Persistence Id =  member-3-shard-entity-ownership-operational Last index in log = -1, snapshotIndex = -1, snapshotTerm = -1, journal-size = 0
    2018-03-21 13:22:39,473 | INFO  | lt-dispatcher-31 | RoleChangeNotifier               | 120 - org.opendaylight.controller.sal-clustering-commons - 1.6.2 | RoleChangeNotifier for member-3-shard-entity-ownership-operational , received role change from null to Follower

可以看出，第一台Ubuntu主机节点成为Leader，其它两台Ubuntu主机节点成为 Follower ，集群配置成功。但是，从日志也可以看出，对于不同的Shard，存在不同的集群关系。

# 三、小结 #

本文详细介绍了Opendaylight Nitrogen集群的搭建指南，后续碰到问题将继续补充。

[Setting Up Clustering]: http://docs.opendaylight.org/en/latest/getting-started-guide/common-features/clustering.html
[Ubuntu_PPA_Java 8]: http://blog.csdn.net/zhaobryant/article/details/51125246
[https_www.opendaylight.org_technical-community_getting-started-for-developers_downloads-and-documentation]: https://www.opendaylight.org/technical-community/getting-started-for-developers/downloads-and-documentation