openstack cinder snapshot快照源码分析

客官°小女子只卖身不卖艺 2024-04-19 10:04 58阅读 0赞

#### cinder snapshot快照源码分析 ####

源码下载地址：  
https://releases.openstack.org/index.html

--------------------

##### 1. 介绍 #####

Cinder是OpenStack Block Storage服务，用于为Nova虚拟机，Ironic裸机主机，容器等提供卷。Cinder的一些目标是：

基于组件的体系结构：快速添加新行为

高度可用：扩展到非常严重的工作负载

容错：隔离的进程可以避免级联故障

--------------------

##### 2.命令行 #####

命令行查看功能：

[root@cinder ~]# cinder -h | grep snap
        cgsnapshot-create   Creates a cgsnapshot.
        cgsnapshot-delete   Removes one or more cgsnapshots.
        cgsnapshot-list     Lists all cgsnapshots.
        cgsnapshot-show     Shows cgsnapshot details.
                            Creates a consistency group from a cgsnapshot or a
        snapshot-create     Creates a snapshot.
        snapshot-delete     Removes one or more snapshots.
        snapshot-list       Lists all snapshots.
        snapshot-manage     Manage an existing snapshot.
        snapshot-manageable-list
                            Lists all manageable snapshots.
        snapshot-metadata   Sets or deletes snapshot metadata.
        snapshot-metadata-show
                            Shows snapshot metadata.
        snapshot-metadata-update-all
                            Updates snapshot metadata.
        snapshot-rename     Renames a snapshot.
        snapshot-reset-state
                            Explicitly updates the snapshot state.
        snapshot-show       Shows snapshot details.
        snapshot-unmanage   Stop managing a snapshot.
    [root@cinder ~]# 
    [root@cinder ~]#

查看命令详情帮助

[root@cinder ~]# cinder help snapshot-create
    usage: cinder snapshot-create [--force [<True|False>]] [--name <name>]
                                  [--description <description>]
                                  [--metadata [<key=value> [<key=value> ...]]]
                                  <volume>
    
    Creates a snapshot.
    
    Positional arguments:
      <volume>              Name or ID of volume to snapshot.
    
    Optional arguments:
      --force [<True|False>]
                            Allows or disallows snapshot of a volume when the
                            volume is attached to an instance. If set to True,
                            ignores the current status of the volume when
                            attempting to snapshot it rather than forcing it to be
                            available. Default=False.
      --name <name>         Snapshot name. Default=None.
      --description <description>
                            Snapshot description. Default=None.
      --metadata [<key=value> [<key=value> ...]]
                            Snapshot metadata key and value pairs. Default=None.
    [root@cinder ~]#

--------------------

##### 3. 源码分析 #####

从最新版本开始查找，目前为V3版本：

cinder snapshot快照功能的API入口在文件cinder/api/v3/snapshots.py

快照功能定义类：

class SnapshotsController(snapshots_v2.SnapshotsController):
        """The Snapshots API controller for the OpenStack API."""
    
        _view_builder_class = snapshot_views.ViewBuilder

实际使用v2版本， 源码文件在cinder/api/v2/snapshots.py

class SnapshotsController(wsgi.Controller):
        """The Snapshots API controller for the OpenStack API."""
    
        _view_builder_class = snapshot_views.ViewBuilder
        
        def __init__(self, ext_mgr=None):
            self.volume_api = volume.API()
            self.ext_mgr = ext_mgr
            super(SnapshotsController, self).__init__()

###### 一. show 详情功能： ######

def show(self, req, id):
            """Return data about the given snapshot."""
            context = req.environ['cinder.context']
        
            # Not found exception will be handled at the wsgi level
            snapshot = self.volume_api.get_snapshot(context, id)
            req.cache_db_snapshot(snapshot)
        
            return self._view_builder.detail(req, snapshot)

获取信息self.volume\_api.get\_snapshot(context, id)， 具体实现在API中，  
定义在文件： cinder/volume/api.py

class API(base.Base):
        """API for interacting with the volume manager."""
    
    		.....
    		
    	def get_snapshot(self, context, snapshot_id):
    	    check_policy(context, 'get_snapshot')
    	    snapshot = objects.Snapshot.get_by_id(context, snapshot_id)
    	
    	    # FIXME(jdg): The objects don't have the db name entries
    	    # so build the resource tag manually for now.
    	    LOG.info(_LI("Snapshot retrieved successfully."),
    	             resource={'type': 'snapshot',
    	                       'id': snapshot.id})
    	    return snapshot

objects.Snapshot.get\_by\_id(context, snapshot\_id)查询db，获取快照信息

可以看到该功能基本是直接获取db的操作， 比较简单

###### 二. list， update与show差不多， 基本都是操作db信息， 这里不做分析 ######

--------------------

###### 三. 创建create创建快照功能： ######

@wsgi.response(202)
        def create(self, req, body):
            """Creates a new snapshot."""
            kwargs = {}
            
    #获取并校验参数信息        
            context = req.environ['cinder.context']
    
            self.assert_valid_body(body, 'snapshot')
        
            snapshot = body['snapshot']
            kwargs['metadata'] = snapshot.get('metadata', None)
        
            try:
                volume_id = snapshot['volume_id']
            except KeyError:
                msg = _("'volume_id' must be specified")
                raise exc.HTTPBadRequest(explanation=msg)
    
    
    #获取并检查volume卷
            volume = self.volume_api.get(context, volume_id)
            force = snapshot.get('force', False)
            msg = _LI("Create snapshot from volume %s")
            LOG.info(msg, volume_id)
            self.validate_name_and_description(snapshot)
    
            # NOTE(thingee): v2 API allows name instead of display_name
            if 'name' in snapshot:
                snapshot['display_name'] = snapshot.pop('name')
        
            try:
                force = strutils.bool_from_string(force, strict=True)
            except ValueError as error:
                err_msg = encodeutils.exception_to_unicode(error)
                msg = _("Invalid value for 'force': '%s'") % err_msg
                raise exception.InvalidParameterValue(err=msg)
        
            if force:
                new_snapshot = self.volume_api.create_snapshot_force(
                    context,
                    volume,
                    snapshot.get('display_name'),
                    snapshot.get('description'),
                    **kwargs)
            else:
                new_snapshot = self.volume_api.create_snapshot(
                    context,
                    volume,
                    snapshot.get('display_name'),
                    snapshot.get('description'),
                    **kwargs)
            req.cache_db_snapshot(new_snapshot)
        
            return self._view_builder.detail(req, new_snapshot)

创建快照：

new_snapshot = self.volume_api.create_snapshot(
          context,
          volume,
          snapshot.get('display_name'),
          snapshot.get('description'),
          **kwargs)

功能定义在API的create\_snapshot ， 位置在： cinder/volume/api.py

def create_snapshot(self, context,
                            volume, name, description,
                            metadata=None, cgsnapshot_id=None,
                            group_snapshot_id=None):
            result = self._create_snapshot(context, volume, name, description,
                                           False, metadata, cgsnapshot_id,
                                           group_snapshot_id)
            LOG.info(_LI("Snapshot create request issued successfully."),
                     resource=result)
            return result

其中实际调用`_create_snapshot`实现：

def _create_snapshot(self, context,
                             volume, name, description,
                             force=False, metadata=None,
                             cgsnapshot_id=None,
                             group_snapshot_id=None):
    # 创建db信息
            snapshot = self.create_snapshot_in_db(
                context, volume, name,
                description, force, metadata, cgsnapshot_id,
                True, group_snapshot_id)
            
            调用驱动创建快照
            self.volume_rpcapi.create_snapshot(context, volume, snapshot)
        
            return snapshot

`self.volume_rpcapi.create_snapshot(context, volume, snapshot)`进行远程rpc调用  
最后实际的处理方法create\_snapshot， 定义在nova/volume/manager.py中：

搜索create\_snapshot

定义：

def create_snapshot(self, context, volume_id, snapshot):
            """Creates and exports the snapshot."""
            context = context.elevated()
        
            self._notify_about_snapshot_usage(
                context, snapshot, "create.start")
        
            try:
                # NOTE(flaper87): Verify the driver is enabled
                # before going forward. The exception will be caught
                # and the snapshot status updated.
                utils.require_driver_initialized(self.driver)
        
                # Pass context so that drivers that want to use it, can,
                # but it is not a requirement for all drivers.
                snapshot.context = context
    
    # 调用driver创建快照
                model_update = self.driver.create_snapshot(snapshot)
    
    
    # 更新快照信息
                if model_update:
                    snapshot.update(model_update)
                    snapshot.save()
        
            except Exception:
                with excutils.save_and_reraise_exception():
                    snapshot.status = fields.SnapshotStatus.ERROR
                    snapshot.save()
        
            vol_ref = self.db.volume_get(context, snapshot.volume_id)
            if vol_ref.bootable:
                try:
                    self.db.volume_glance_metadata_copy_to_snapshot(
                        context, snapshot.id, snapshot.volume_id)
                except exception.GlanceMetadataNotFound:
                    # If volume is not created from image, No glance metadata
                    # would be available for that volume in
                    # volume glance metadata table
                    pass
                except exception.CinderException as ex:
                    LOG.exception(_LE("Failed updating snapshot"
                                      " metadata using the provided volumes"
                                      " %(volume_id)s metadata"),
                                  {'volume_id': snapshot.volume_id},
                                  resource=snapshot)
                    snapshot.status = fields.SnapshotStatus.ERROR
                    snapshot.save()
                    raise exception.MetadataCopyFailure(reason=six.text_type(ex))
    
    
    #更新snapshot信息
    
            snapshot.status = fields.SnapshotStatus.AVAILABLE
            snapshot.progress = '100%'
            snapshot.save()
        
            self._notify_about_snapshot_usage(context, snapshot, "create.end")
            LOG.info(_LI("Create snapshot completed successfully"),
                     resource=snapshot)
            return snapshot.id

其中最关键的步骤是调用driver信息创建snapshot：

`model_update = self.driver.create_snapshot(snapshot)`

在cinder/volume/drivers/ 下寻找对应的驱动文件

比如：  
使用本地LVM， 则对应的驱动文件为cinder/volume/drivers/lvm.py

定义如下：

def create_snapshot(self, snapshot):
            """Creates a snapshot."""
    
            self.vg.create_lv_snapshot(self._escape_snapshot(snapshot['name']),
                                       snapshot['volume_name'],
                                       self.configuration.lvm_type)

create\_lv\_snapshot实际使用的是操作系统的vg管理命令

定义如下：

@utils.retry(putils.ProcessExecutionError)
     def create_lv_snapshot(self, name, source_lv_name, lv_type='default'):
         """Creates a snapshot of a logical volume.
    
         :param name: Name to assign to new snapshot
         :param source_lv_name: Name of Logical Volume to snapshot
         :param lv_type: Type of LV (default or thin)
        
         """
         source_lvref = self.get_volume(source_lv_name)
         if source_lvref is None:
             LOG.error(_LE("Trying to create snapshot by non-existent LV: %s"),
                       source_lv_name)
             raise exception.VolumeDeviceNotFound(device=source_lv_name)
         cmd = LVM.LVM_CMD_PREFIX + ['lvcreate', '--name', name, '--snapshot',
                                     '%s/%s' % (self.vg_name, source_lv_name)]
         if lv_type != 'thin':
             size = source_lvref['size']
             cmd.extend(['-L', '%sg' % (size)])
        
         try:
             self._execute(*cmd,
                           root_helper=self._root_helper,
                           run_as_root=True)
         except putils.ProcessExecutionError as err:
             LOG.exception(_LE('Error creating snapshot'))
             LOG.error(_LE('Cmd     :%s'), err.cmd)
             LOG.error(_LE('StdOut  :%s'), err.stdout)
             LOG.error(_LE('StdErr  :%s'), err.stderr)
             raise

`_execute`以root权限执行命令

cmd = LVM.LVM_CMD_PREFIX + ['lvcreate', '--name', name, '--snapshot',
                                '%s/%s' % (self.vg_name, source_lv_name)]

命令实际上使用的是lvcreate

lvcreate 创建, 可以直接在操作系统上查看功能

如果使用ceph存储， 则对接的驱动是RBD  
对应驱动文件在cinder/volume/drivers/rbd.py

定义如下：

def create_snapshot(self, snapshot):
            """Creates an rbd snapshot."""
            with RBDVolumeProxy(self, snapshot.volume_name) as volume:
                snap = utils.convert_str(snapshot.name)
                volume.create_snap(snap)
                volume.protect_snap(snap)

实际就是连接rbd， 调用对应的方法：  
volume.create\_snap(snap)  
volume.protect\_snap(snap)

对应调用create\_snap 和protect\_snap 方法

定义：

class RBDVolumeProxy(object):
        """Context manager for dealing with an existing rbd volume.
    
        This handles connecting to rados and opening an ioctx automatically, and
        otherwise acts like a librbd Image object.
        
        The underlying librados client and ioctx can be accessed as the attributes
        'client' and 'ioctx'.
        """
        def __init__(self, driver, name, pool=None, snapshot=None,
                     read_only=False):
            client, ioctx = driver._connect_to_rados(pool)
            if snapshot is not None:
                snapshot = utils.convert_str(snapshot)
        
            try:
                self.volume = driver.rbd.Image(ioctx,
                                               utils.convert_str(name),
                                               snapshot=snapshot,
                                               read_only=read_only)
            except driver.rbd.Error:
                LOG.exception(_LE("error opening rbd image %s"), name)
                driver._disconnect_from_rados(client, ioctx)
                raise
            self.driver = driver
            self.client = client
            self.ioctx = ioctx

--------------------

##### 四. 删除功能： #####

api入口： cinder/api/v2/snapshot.py

def delete(self, req, id):
            """Delete a snapshot."""
            context = req.environ['cinder.context']
        
            LOG.info(_LI("Delete snapshot with id: %s"), id)
        
            # Not found exception will be handled at the wsgi level
    
    # 获取快照信息， 并校验快照是否存在
            snapshot = self.volume_api.get_snapshot(context, id)
    
    #删除快照
            self.volume_api.delete_snapshot(context, snapshot)
    
            return webob.Response(status_int=202)

API 在cinder/volume/api.py

@wrap_check_policy
        def delete_snapshot(self, context, snapshot, force=False,
                            unmanage_only=False):
            # Build required conditions for conditional update
            expected = {'cgsnapshot_id': None,
                        'group_snapshot_id': None}
            # If not force deleting we have status conditions
    
    # 如果不是进行强制删除， 则检察状态， 值允许available/error
            if not force:
                expected['status'] = (fields.SnapshotStatus.AVAILABLE,
                                      fields.SnapshotStatus.ERROR)
        
            result = snapshot.conditional_update(
                {'status': fields.SnapshotStatus.DELETING}, expected)
            if not result:
                status = utils.build_or_str(expected.get('status'),
                                            _('status must be %s and'))
                msg = (_('Snapshot %s must not be part of a group.') %
                       status)
                LOG.error(msg)
                raise exception.InvalidSnapshot(reason=msg)
        
            # Make RPC call to the right host
    #获取volume卷信息
            volume = objects.Volume.get_by_id(context, snapshot.volume_id)
            
            
    #volume.host  调用volume  host 主机的volume服务进行删除
            self.volume_rpcapi.delete_snapshot(context, snapshot, volume.host,
                                               unmanage_only=unmanage_only)
            LOG.info(_LI("Snapshot delete request issued successfully."),
                     resource=snapshot)

主机volume删除通过rpc进行调用

volume远程服务定义在cinder/volume/manager.py

@coordination.synchronized('{snapshot.id}-{f_name}')
        def delete_snapshot(self, context, snapshot, unmanage_only=False):
            """Deletes and unexports snapshot."""
            context = context.elevated()
            snapshot._context = context
            project_id = snapshot.project_id
    
            self._notify_about_snapshot_usage(
                context, snapshot, "delete.start")
        
            try:
                # NOTE(flaper87): Verify the driver is enabled
                # before going forward. The exception will be caught
                # and the snapshot status updated.
                utils.require_driver_initialized(self.driver)
        
                # Pass context so that drivers that want to use it, can,
                # but it is not a requirement for all drivers.
                snapshot.context = context
                snapshot.save()
        
                if unmanage_only:
    # unmanage 不进行管理， 进行信息剔除
                    self.driver.unmanage_snapshot(snapshot)
                else:
    # 删除快照信息
                    self.driver.delete_snapshot(snapshot)
            except exception.SnapshotIsBusy:
    
    #异常回滚，将snapshot的状态设置为available
    
                LOG.error(_LE("Delete snapshot failed, due to snapshot busy."),
                          resource=snapshot)
                snapshot.status = fields.SnapshotStatus.AVAILABLE
                snapshot.save()
                return
            except Exception:
                with excutils.save_and_reraise_exception():
                    snapshot.status = fields.SnapshotStatus.ERROR_DELETING
                    snapshot.save()
        
            # Get reservations
            try:
    
    # 删除snapshot快照信息， 并调整quota信息
    
                if CONF.no_snapshot_gb_quota:
                    reserve_opts = {'snapshots': -1}
                else:
                    reserve_opts = {
                        'snapshots': -1,
                        'gigabytes': -snapshot.volume_size,
                    }
                volume_ref = self.db.volume_get(context, snapshot.volume_id)
                QUOTAS.add_volume_type_opts(context,
                                            reserve_opts,
                                            volume_ref.get('volume_type_id'))
                reservations = QUOTAS.reserve(context,
                                              project_id=project_id,
                                              **reserve_opts)
            except Exception:
                reservations = None
                LOG.exception(_LE("Update snapshot usages failed."),
                              resource=snapshot)
            self.db.volume_glance_metadata_delete_by_snapshot(context, snapshot.id)
            snapshot.destroy()
            self._notify_about_snapshot_usage(context, snapshot, "delete.end")
        
            # Commit the reservations
            if reservations:
                QUOTAS.commit(context, reservations, project_id=project_id)
            LOG.info(_LI("Delete snapshot completed successfully"),
                     resource=snapshot)

self.driver.delete\_snapshot(snapshot)调用驱动删除快照

驱动拥有多种， 驱动文件统一放在cinder/volume/dirvers/目录下

例如： 如果使用lvm 的vg管理功能则对应驱动文件为cinder/volume/dirvers/lvm.py

定义如下：

def delete_snapshot(self, snapshot):
            """Deletes a snapshot."""
            
    检察卷， 如果已经没有卷信息， 则直接返回true成功即可
            if self._volume_not_present(self._escape_snapshot(snapshot['name'])):
                # If the snapshot isn't present, then don't attempt to delete
                LOG.warning(_LW("snapshot: %s not found, "
                                "skipping delete operations"), snapshot['name'])
                LOG.info(_LI('Successfully deleted snapshot: %s'), snapshot['id'])
                return True
    
            # TODO(yamahata): zeroing out the whole snapshot triggers COW.
            # it's quite slow.
    # 删除volume snapshot
            self._delete_volume(snapshot, is_snapshot=True)

\_delete\_volume 定义：

def _delete_volume(self, volume, is_snapshot=False):
            """Deletes a logical volume."""
            if self.configuration.volume_clear != 'none' and \
                    self.configuration.lvm_type != 'thin':
                self._clear_volume(volume, is_snapshot)
        
            name = volume['name']
            if is_snapshot:
                name = self._escape_snapshot(volume['name'])
            self.vg.delete(name)

vg delete 定义在cinder/brick/loacl\_dev/lvm.py

@utils.retry(putils.ProcessExecutionError)
        def delete(self, name):
            """Delete logical volume or snapshot.
        
            :param name: Name of LV to delete
        
            """
        
            def run_udevadm_settle():
                self._execute('udevadm', 'settle',
                              root_helper=self._root_helper, run_as_root=True,
                              check_exit_code=False)
        
            # LV removal seems to be a race with other writers or udev in
            # some cases (see LP #1270192), so we enable retry deactivation
            LVM_CONFIG = 'activation { retry_deactivation = 1} '
        
            try:
                self._execute(
                    'lvremove',
                    '--config', LVM_CONFIG,
                    '-f',
                    '%s/%s' % (self.vg_name, name),
                    root_helper=self._root_helper, run_as_root=True)
            except putils.ProcessExecutionError as err:
                LOG.debug('Error reported running lvremove: CMD: %(command)s, '
                          'RESPONSE: %(response)s',
                          {'command': err.cmd, 'response': err.stderr})
        
                LOG.debug('Attempting udev settle and retry of lvremove...')
                run_udevadm_settle()
        
                # The previous failing lvremove -f might leave behind
                # suspended devices; when lvmetad is not available, any
                # further lvm command will block forever.
                # Therefore we need to skip suspended devices on retry.
                LVM_CONFIG += 'devices { ignore_suspended_devices = 1}'
        
                self._execute(
                    'lvremove',
                    '--config', LVM_CONFIG,
                    '-f',
                    '%s/%s' % (self.vg_name, name),
                    root_helper=self._root_helper, run_as_root=True)
                LOG.debug('Successfully deleted volume: %s after '
                          'udev settle.', name)

lvremove 删除快照, 可以查看具体命令， 这里不做介绍

如果是ceph  
则对应的驱动定义在cinder/volume/dirvers/rbd.py

delete\_snapshot 删除snapshot定义

def delete_snapshot(self, snapshot):
            """Deletes an rbd snapshot."""
            # NOTE(dosaboy): this was broken by commit cbe1d5f. Ensure names are
            #                utf-8 otherwise librbd will barf.
            volume_name = utils.convert_str(snapshot.volume_name)
            snap_name = utils.convert_str(snapshot.name)
        
            with RBDVolumeProxy(self, volume_name) as volume:
                try:
                    volume.unprotect_snap(snap_name)
                except self.rbd.InvalidArgument:
                    LOG.info(
                        _LI("InvalidArgument: Unable to unprotect snapshot %s."),
                        snap_name)
                except self.rbd.ImageNotFound:
                    LOG.info(
                        _LI("ImageNotFound: Unable to unprotect snapshot %s."),
                        snap_name)
                except self.rbd.ImageBusy:
                    children_list = self._get_children_info(volume, snap_name)
        
                    if children_list:
                        for (pool, image) in children_list:
                            LOG.info(_LI('Image %(pool)s/%(image)s is dependent '
                                         'on the snapshot %(snap)s.'),
                                     {'pool': pool,
                                      'image': image,
                                      'snap': snap_name})
        
                    raise exception.SnapshotIsBusy(snapshot_name=snap_name)
                try:
                    volume.remove_snap(snap_name)
                except self.rbd.ImageNotFound:
                    LOG.info(_LI("Snapshot %s does not exist in backend."),
                             snap_name)

RBDVolumeProxy 调用rbd

删除volume.remove\_snap(snap\_name)