Caffe | 自定义字段和层

绝地灬酷狼 2022-03-25 10:24 274阅读 0赞

#### 1.自定义字段 ####

最近在老版本的caffe上跑resnext网络的时候出现如下所示的bug，正如我们上一篇文章[Caffe | 核心积木Layer层类详解][Caffe _ _Layer]中说到的，在caffe.proto文件的PoolingParameter中没有ceil\_mode这个field字段。因此只有在源码中添加这个参数以及相关实现代码，并重新编译caffe。

Message type “caffe.PoolingParameter” has no field named “ceil_mode”.

（1）修改pooling\_layer.hpp文件  
首先在头文件中添加参数的定义，因为参数在使用之前需要先定义（C/C++语言的特性）。

int height_, width_;
        int pooled_height_, pooled_width_;
        bool global_pooling_;
        bool ceil_mode_;   //添加bug中指出的缺少的属性
        Blob<Dtype> rand_idx_;
        Blob<int> max_idx_;

（2）修改pooling\_layer.cpp文件中对应参数  
也是在上一篇文章[Caffe | 核心积木Layer层类详解][Caffe _ _Layer]中说到的，每一个层（包括当前存在问题的PoolingLayer）都继承自基类LayerParameter。而基类参数中有两个重要的函数如下：

*  Layersetup：读取指定层类的layer param（层参数），为后续reshape做准备。
 *  reshape：根据输入该层的bottom blob的形状，和改成定制化的计算策略（也就是当前层的逻辑）计算得到对应的top blob的形状，并预先分配好内存空间。

所以在这里这两个函数跟参数紧密相关，因此我们主要修改的就是pooling\_layer.cpp中的这两个函数。  
Layersetup函数：

|| (!pool_param.has_stride_h() && !pool_param.has_stride_w()))
            << "Stride is stride OR stride_h and stride_w are required.";
        global_pooling_ = pool_param.global_pooling();
    
     // 添加的代码-----------------------------------
             ceil_mode_ = pool_param.ceil_mode();      //添加的代码，
                                 //主要作用是从参数文件中获取ceil_mode_的参数数值。
    // ------------------------------------------------------
    
        if (global_pooling_) {
          kernel_h_ = bottom[0]->height();
          kernel_w_ = bottom[0]->width();
       if (pad_h_ != 0 || pad_w_ != 0) {
         CHECK(this->layer_param_.pooling_param().pool()
             == PoolingParameter_PoolMethod_AVE
             || this->layer_param_.pooling_param().pool()
             == PoolingParameter_PoolMethod_MAX)
             << "Padding implemented only for average and max pooling.";
         CHECK_LT(pad_h_, kernel_h_);
         CHECK_LT(pad_w_, kernel_w_);

Reshape函数：

void PoolingLayer<Dtype>::Reshape(const vector<Blob<Dtype>*>& bottom,
           const vector<Blob<Dtype>*>& top) {
       CHECK_EQ(4, bottom[0]->num_axes()) << "Input must have 4 axes, "
           << "corresponding to (num, channels, height, width)";
       channels_ = bottom[0]->channels();
       height_ = bottom[0]->height();
       width_ = bottom[0]->width();
       if (global_pooling_) {
          kernel_h_ = bottom[0]->height();
          kernel_w_ = bottom[0]->width();
        }
     -  pooled_height_ = static_cast<int>(ceil(static_cast<float>(
     -      height_ + 2 * pad_h_ - kernel_h_) / stride_h_)) + 1;
     -  pooled_width_ = static_cast<int>(ceil(static_cast<float>(
     -      width_ + 2 * pad_w_ - kernel_w_) / stride_w_)) + 1;
     +  // Specify the structure by ceil or floor mode
     + 
     + // 添加的代码-----------------------------------
     +  if (ceil_mode_) {
     +    pooled_height_ = static_cast<int>(ceil(static_cast<float>(
     +        height_ + 2 * pad_h_ - kernel_h_) / stride_h_)) + 1;
     +    pooled_width_ = static_cast<int>(ceil(static_cast<float>(
     +        width_ + 2 * pad_w_ - kernel_w_) / stride_w_)) + 1;
     +  } else {
     +    pooled_height_ = static_cast<int>(floor(static_cast<float>(
     +        height_ + 2 * pad_h_ - kernel_h_) / stride_h_)) + 1;
     +    pooled_width_ = static_cast<int>(floor(static_cast<float>(
     +        width_ + 2 * pad_w_ - kernel_w_) / stride_w_)) + 1;
     +  }
     + // ------------------------------------------------------
     + 
        if (pad_h_ || pad_w_) {
          // If we have padding, ensure that the last pooling starts strictly
          // inside the image (instead of at the padding); otherwise clip the last.

（3）修改caffe.proto文件中PoolingParameter  
因为所有层的参数定义都存放在caffe.proto文件中，因此修改参数后需要将新的参数添加到该文件对应的层参数中，本例中就将参数添加到PoolingParameter中。

/ If global_pooling then it will pool over the size of the bottom by doing
        // kernel_h = bottom->height and kernel_w = bottom->width
        optional bool global_pooling = 12 [default = false];
    
    
     // 添加的代码-----------------------------------
     +  // Specify floor/ceil mode
    // 为pooling层添加参数，这样可以在net.prototxt文件中为pooling层设置该参数，
    // 注意后面需要给其设置一个ID，同时设置一个默认值。（下面是我之前文章中提到过的ID的作用）
    //Message的tag：
    //每个message里面的每个field都对应一个tag，
    //分别是1~15或者以上，比如required string number=1；
    //这个数字就是用来在生成的二进制文件中搜索查询的标签（怪不得会快）。
    //关于这个数字，1到15会花费1byte的编码空间，16到2047花费2byte。
    //所以一般建议把那些频繁使用的名字的标签设为1到15之间的值～
             +  optional bool ceil_mode = 13 [default = true];
     // ------------------------------------------------------
      }

（4）重新编译caffe

返回到caffe的根目录，使用make指令(下面几个都可以，任选一个)，即可。

make
     make -j
     make -j16
     make -j32    // 这里j后面的数字与电脑配置有关系，可以加速编译

#### 2.自定义新层 ####

这里以新增一个简单的Loss层（OHEM）举例说明自定义新层的流程。简单介绍下OHEM，难例挖掘（OHEM）是指,针对模型训练过程中导致损失值很大的一些样本(即使模型很大概率分类错误的样本),重新训练它们.维护一个错误分类样本池, 把每个batch训练数据中的出错率很大的样本放入该样本池中,当积累到一个batch以后,将这些样本放回网络重新训练。这是RGB大神的策略，用于Faster Rcnn。主要就是新增3个文件，hpp/cpp/cu。主要就是修改forward\_gpu(cpu)，backward\_gpu(cpu)这几个函数。

（1）hpp文件

#ifndef CAFFE_SOFTMAX_WITH_LOSS_OHEM_LAYER_HPP_
    #define CAFFE_SOFTMAX_WITH_LOSS_OHEM_LAYER_HPP_
    
    #include <vector>
    
    #include "caffe/blob.hpp"
    #include "caffe/layer.hpp"
    #include "caffe/proto/caffe.pb.h"
    
    #include "caffe/layers/loss_layer.hpp"
    #include "caffe/layers/softmax_layer.hpp"
    
    namespace caffe {
    
    /**
     * @brief Computes the multinomial logistic loss for a one-of-many
     *        classification task, passing real-valued predictions through a
     *        softmax to get a probability distribution over classes.
     *
     * This layer should be preferred over separate
     * SoftmaxLayer + MultinomialLogisticLossLayer
     * as its gradient computation is more numerically stable.
     * At test time, this layer can be replaced simply by a SoftmaxLayer.
     *
     * @param bottom input Blob vector (length 2)
     *   -# @f$ (N \times C \times H \times W) @f$
     *      the predictions @f$ x @f$, a Blob with values in
     *      @f$ [-\infty, +\infty] @f$ indicating the predicted score for each of
     *      the @f$ K = CHW @f$ classes. This layer maps these scores to a
     *      probability distribution over classes using the softmax function
     *      @f$ \hat{p}_{nk} = \exp(x_{nk}) /
     *      \left[\sum_{k'} \exp(x_{nk'})\right] @f$ (see SoftmaxLayer).
     *   -# @f$ (N \times 1 \times 1 \times 1) @f$
     *      the labels @f$ l @f$, an integer-valued Blob with values
     *      @f$ l_n \in [0, 1, 2, ..., K - 1] @f$
     *      indicating the correct class label among the @f$ K @f$ classes
     * @param top output Blob vector (length 1)
     *   -# @f$ (1 \times 1 \times 1 \times 1) @f$
     *      the computed cross-entropy classification loss: @f$ E =
     *        \frac{-1}{N} \sum\limits_{n=1}^N \log(\hat{p}_{n,l_n})
     *      @f$, for softmax output class probabilites @f$ \hat{p} @f$
     */
    template <typename Dtype>
    class SoftmaxWithLossOHEMLayer : public LossLayer<Dtype> {
     public:
       /**
        * @param param provides LossParameter loss_param, with options:
        *  - ignore_label (optional)
        *    Specify a label value that should be ignored when computing the loss.
        *  - normalize (optional, default true)
        *    If true, the loss is normalized by the number of (nonignored) labels
        *    present; otherwise the loss is simply summed over spatial locations.
        */
      explicit SoftmaxWithLossOHEMLayer(const LayerParameter& param)
          : LossLayer<Dtype>(param) {}
      virtual void LayerSetUp(const vector<Blob<Dtype>*>& bottom,
          const vector<Blob<Dtype>*>& top);
      virtual void Reshape(const vector<Blob<Dtype>*>& bottom,
          const vector<Blob<Dtype>*>& top);
    
      virtual inline const char* type() const { return "SoftmaxWithLossOHEM"; }
      virtual inline int ExactNumTopBlobs() const { return -1; }
      virtual inline int MinTopBlobs() const { return 1; }
      virtual inline int MaxTopBlobs() const { return 3; }
    
     protected:
      virtual void Forward_cpu(const vector<Blob<Dtype>*>& bottom,
          const vector<Blob<Dtype>*>& top);
      virtual void Forward_gpu(const vector<Blob<Dtype>*>& bottom,
          const vector<Blob<Dtype>*>& top);
      /**
       * @brief Computes the softmax loss error gradient w.r.t. the predictions.
       *
       * Gradients cannot be computed with respect to the label inputs (bottom[1]),
       * so this method ignores bottom[1] and requires !propagate_down[1], crashing
       * if propagate_down[1] is set.
       *
       * @param top output Blob vector (length 1), providing the error gradient with
       *      respect to the outputs
       *   -# @f$ (1 \times 1 \times 1 \times 1) @f$
       *      This Blob's diff will simply contain the loss_weight* @f$ \lambda @f$,
       *      as @f$ \lambda @f$ is the coefficient of this layer's output
       *      @f$\ell_i@f$ in the overall Net loss
       *      @f$ E = \lambda_i \ell_i + \mbox{other loss terms}@f$; hence
       *      @f$ \frac{\partial E}{\partial \ell_i} = \lambda_i @f$.
       *      (*Assuming that this top Blob is not used as a bottom (input) by any
       *      other layer of the Net.)
       * @param propagate_down see Layer::Backward.
       *      propagate_down[1] must be false as we can't compute gradients with
       *      respect to the labels.
       * @param bottom input Blob vector (length 2)
       *   -# @f$ (N \times C \times H \times W) @f$
       *      the predictions @f$ x @f$; Backward computes diff
       *      @f$ \frac{\partial E}{\partial x} @f$
       *   -# @f$ (N \times 1 \times 1 \times 1) @f$
       *      the labels -- ignored as we can't compute their error gradients
       */
      virtual void Backward_cpu(const vector<Blob<Dtype>*>& top,
          const vector<bool>& propagate_down, const vector<Blob<Dtype>*>& bottom);
      virtual void Backward_gpu(const vector<Blob<Dtype>*>& top,
          const vector<bool>& propagate_down, const vector<Blob<Dtype>*>& bottom);
    
      /// Read the normalization mode parameter and compute the normalizer based
      /// on the blob size.  If normalization_mode is VALID, the count of valid
      /// outputs will be read from valid_count, unless it is -1 in which case
      /// all outputs are assumed to be valid.
      virtual Dtype get_normalizer(
          LossParameter_NormalizationMode normalization_mode, int valid_count);
    
      /// The internal SoftmaxLayer used to map predictions to a distribution.
      shared_ptr<Layer<Dtype> > softmax_layer_;
      /// prob stores the output probability predictions from the SoftmaxLayer.
      Blob<Dtype> prob_;
      /// bottom vector holder used in call to the underlying SoftmaxLayer::Forward
      vector<Blob<Dtype>*> softmax_bottom_vec_;
      /// top vector holder used in call to the underlying SoftmaxLayer::Forward
      vector<Blob<Dtype>*> softmax_top_vec_;
      /// Whether to ignore instances with a certain label.
      bool has_ignore_label_;
      /// The label indicating that an instance should be ignored.
      int ignore_label_;
      /// How to normalize the output loss.
      LossParameter_NormalizationMode normalization_;
    
      int softmax_axis_, outer_num_, inner_num_;
    };
    
    }  // namespace caffe
    
    #endif  // CAFFE_SOFTMAX_WITH_LOSS_OHEMLAYER_HPP_

（2）cpp文件

#include <algorithm>
    #include <cfloat>
    #include <vector>
    
    #include "caffe/layers/softmax_loss_ohem_layer.hpp"
    #include "caffe/util/math_functions.hpp"
    
    namespace caffe {
    
    template <typename Dtype>
    void SoftmaxWithLossOHEMLayer<Dtype>::LayerSetUp(
        const vector<Blob<Dtype>*>& bottom, const vector<Blob<Dtype>*>& top) {
      LossLayer<Dtype>::LayerSetUp(bottom, top);
      LayerParameter softmax_param(this->layer_param_);
      softmax_param.clear_loss_weight();
      softmax_param.set_type("Softmax");
      softmax_layer_ = LayerRegistry<Dtype>::CreateLayer(softmax_param);
      softmax_bottom_vec_.clear();
      softmax_bottom_vec_.push_back(bottom[0]);
      softmax_top_vec_.clear();
      softmax_top_vec_.push_back(&prob_);
      softmax_layer_->SetUp(softmax_bottom_vec_, softmax_top_vec_);
    
      has_ignore_label_ =
        this->layer_param_.loss_param().has_ignore_label();
      if (has_ignore_label_) {
        ignore_label_ = this->layer_param_.loss_param().ignore_label();
      }
      if (!this->layer_param_.loss_param().has_normalization() &&
          this->layer_param_.loss_param().has_normalize()) {
        normalization_ = this->layer_param_.loss_param().normalize() ?
                         LossParameter_NormalizationMode_VALID :
                         LossParameter_NormalizationMode_BATCH_SIZE;
      } else {
        normalization_ = this->layer_param_.loss_param().normalization();
      }
    }
    
    template <typename Dtype>
    void SoftmaxWithLossOHEMLayer<Dtype>::Reshape(
        const vector<Blob<Dtype>*>& bottom, const vector<Blob<Dtype>*>& top) {
      LossLayer<Dtype>::Reshape(bottom, top);
      softmax_layer_->Reshape(softmax_bottom_vec_, softmax_top_vec_);
      softmax_axis_ =
          bottom[0]->CanonicalAxisIndex(this->layer_param_.softmax_param().axis());
      outer_num_ = bottom[0]->count(0, softmax_axis_);
      inner_num_ = bottom[0]->count(softmax_axis_ + 1);
      CHECK_EQ(outer_num_ * inner_num_, bottom[1]->count())
          << "Number of labels must match number of predictions; "
          << "e.g., if softmax axis == 1 and prediction shape is (N, C, H, W), "
          << "label count (number of labels) must be N*H*W, "
          << "with integer values in {0, 1, ..., C-1}.";
      if (top.size() >= 2) {
        // softmax output
        top[1]->ReshapeLike(*bottom[0]);
      }
    }
    
    template <typename Dtype>
    Dtype SoftmaxWithLossOHEMLayer<Dtype>::get_normalizer(
        LossParameter_NormalizationMode normalization_mode, int valid_count) {
      Dtype normalizer;
      switch (normalization_mode) {
        case LossParameter_NormalizationMode_FULL:
          normalizer = Dtype(outer_num_ * inner_num_);
          break;
        case LossParameter_NormalizationMode_VALID:
          if (valid_count == -1) {
            normalizer = Dtype(outer_num_ * inner_num_);
          } else {
            normalizer = Dtype(valid_count);
          }
          break;
        case LossParameter_NormalizationMode_BATCH_SIZE:
          normalizer = Dtype(outer_num_);
          break;
        case LossParameter_NormalizationMode_NONE:
          normalizer = Dtype(1);
          break;
        default:
          LOG(FATAL) << "Unknown normalization mode: "
              << LossParameter_NormalizationMode_Name(normalization_mode);
      }
      // Some users will have no labels for some examples in order to 'turn off' a
      // particular loss in a multi-task setup. The max prevents NaNs in that case.
      return std::max(Dtype(1.0), normalizer);
    }
    
    template <typename Dtype>
    void SoftmaxWithLossOHEMLayer<Dtype>::Forward_cpu(
        const vector<Blob<Dtype>*>& bottom, const vector<Blob<Dtype>*>& top) {
      NOT_IMPLEMENTED;
      
    }
    
    template <typename Dtype>
    void SoftmaxWithLossOHEMLayer<Dtype>::Backward_cpu(const vector<Blob<Dtype>*>& top,
        const vector<bool>& propagate_down, const vector<Blob<Dtype>*>& bottom) {
      NOT_IMPLEMENTED;
    }
    
    #ifdef CPU_ONLY
    STUB_GPU(SoftmaxWithLossOHEMLayer);
    #endif
    
    INSTANTIATE_CLASS(SoftmaxWithLossOHEMLayer);
    REGISTER_LAYER_CLASS(SoftmaxWithLossOHEM);
    
    }  // namespace caffe

（3）cu文件

#include <algorithm>
    #include <cfloat>
    #include <vector>
    
    #include "caffe/layers/softmax_loss_ohem_layer.hpp"
    #include "caffe/util/math_functions.hpp"
    
    namespace caffe {
    
    template <typename Dtype>
    __global__ void SoftmaxLossForwardGPU(const int nthreads,
              const Dtype* prob_data, const Dtype* label, Dtype* loss,
              const int num, const int dim, const int spatial_dim,
              const bool has_ignore_label_, const int ignore_label_,
              Dtype* counts) {
      CUDA_KERNEL_LOOP(index, nthreads) {
        const int n = index / spatial_dim;
        const int s = index % spatial_dim;
        const int label_value = static_cast<int>(label[n * spatial_dim + s]);
        if (has_ignore_label_ && label_value == ignore_label_) {
          loss[index] = 0;
          counts[index] = 0;
        } else {
          loss[index] = -log(max(prob_data[n * dim + label_value * spatial_dim + s],
                          Dtype(FLT_MIN)));
          counts[index] = 1;
        }
      }
    }
    
    template <typename Dtype>
    void SoftmaxWithLossOHEMLayer<Dtype>::Forward_gpu(
        const vector<Blob<Dtype>*>& bottom, const vector<Blob<Dtype>*>& top) {
      softmax_layer_->Forward(softmax_bottom_vec_, softmax_top_vec_);
      const Dtype* prob_data = prob_.gpu_data();
      const Dtype* label = bottom[1]->gpu_data();
      const int dim = prob_.count() / outer_num_;
      const int nthreads = outer_num_ * inner_num_;
      // Since this memory is not used for anything, we use it here to avoid having
      // to allocate new GPU memory to accumulate intermediate results.
      Dtype* loss_data = bottom[0]->mutable_gpu_diff();
      // Similarly, this memory is never used elsewhere, and thus we can use it
      // to avoid having to allocate additional GPU memory.
      Dtype* counts = prob_.mutable_gpu_diff();
      // NOLINT_NEXT_LINE(whitespace/operators)
      SoftmaxLossForwardGPU<Dtype><<<CAFFE_GET_BLOCKS(nthreads),
          CAFFE_CUDA_NUM_THREADS>>>(nthreads, prob_data, label, loss_data,
          outer_num_, dim, inner_num_, has_ignore_label_, ignore_label_, counts);
      Dtype loss;
      caffe_gpu_asum(nthreads, loss_data, &loss);
      Dtype valid_count = -1;
      // Only launch another CUDA kernel if we actually need the count of valid
      // outputs.
      if (normalization_ == LossParameter_NormalizationMode_VALID &&
          has_ignore_label_) {
        caffe_gpu_asum(nthreads, counts, &valid_count);
      }
      top[0]->mutable_cpu_data()[0] = loss / get_normalizer(normalization_,
                                                            valid_count);
      if (top.size() >= 2) {
        top[1]->ShareData(prob_);
      }
    
      if (top.size() >= 3){
        //output per-instance loss
        caffe_gpu_memcpy(top[2]->count() * sizeof(Dtype), loss_data,
           top[2]->mutable_gpu_data());
     }
    
      // Clear scratch memory to prevent interfering with backward (see #6202).
      caffe_gpu_set(bottom[0]->count(), Dtype(0), bottom[0]->mutable_gpu_diff());
    }
    
    template <typename Dtype>
    __global__ void SoftmaxLossBackwardGPU(const int nthreads, const Dtype* top,
              const Dtype* label, Dtype* bottom_diff, const int num, const int dim,
              const int spatial_dim, const bool has_ignore_label_,
              const int ignore_label_, Dtype* counts) {
      const int channels = dim / spatial_dim;
    
      CUDA_KERNEL_LOOP(index, nthreads) {
        const int n = index / spatial_dim;
        const int s = index % spatial_dim;
        const int label_value = static_cast<int>(label[n * spatial_dim + s]);
    
        if (has_ignore_label_ && label_value == ignore_label_) {
          for (int c = 0; c < channels; ++c) {
            bottom_diff[n * dim + c * spatial_dim + s] = 0;
          }
          counts[index] = 0;
        } else {
          bottom_diff[n * dim + label_value * spatial_dim + s] -= 1;
          counts[index] = 1;
        }
      }
    }
    
    template <typename Dtype>
    void SoftmaxWithLossOHEMLayer<Dtype>::Backward_gpu(const vector<Blob<Dtype>*>& top,
        const vector<bool>& propagate_down, const vector<Blob<Dtype>*>& bottom) {
      if (propagate_down[1]) {
        LOG(FATAL) << this->type()
                   << " Layer cannot backpropagate to label inputs.";
      }
      if (propagate_down[0]) {
        Dtype* bottom_diff = bottom[0]->mutable_gpu_diff();
        const Dtype* prob_data = prob_.gpu_data();
        const Dtype* top_data = top[0]->gpu_data();
        caffe_gpu_memcpy(prob_.count() * sizeof(Dtype), prob_data, bottom_diff);
        const Dtype* label = bottom[1]->gpu_data();
        const int dim = prob_.count() / outer_num_;
        const int nthreads = outer_num_ * inner_num_;
        // Since this memory is never used for anything else,
        // we use to to avoid allocating new GPU memory.
        Dtype* counts = prob_.mutable_gpu_diff();
        // NOLINT_NEXT_LINE(whitespace/operators)
        SoftmaxLossBackwardGPU<Dtype><<<CAFFE_GET_BLOCKS(nthreads),
            CAFFE_CUDA_NUM_THREADS>>>(nthreads, top_data, label, bottom_diff,
            outer_num_, dim, inner_num_, has_ignore_label_, ignore_label_, counts);
    
        Dtype valid_count = -1;
        // Only launch another CUDA kernel if we actually need the count of valid
        // outputs.
        if (normalization_ == LossParameter_NormalizationMode_VALID &&
            has_ignore_label_) {
          caffe_gpu_asum(nthreads, counts, &valid_count);
        }
        const Dtype loss_weight = top[0]->cpu_diff()[0] /
                                  get_normalizer(normalization_, valid_count);
        caffe_gpu_scal(prob_.count(), loss_weight , bottom_diff);
      }
    }
    
    INSTANTIATE_LAYER_GPU_FUNCS(SoftmaxWithLossOHEMLayer);
    
    }  // namespace caffe

（4）重新编译caffe

返回到caffe的根目录，使用make指令(下面几个都可以，任选一个)，即可。

make
     make -j
     make -j16
     make -j32    // 这里j后面的数字与电脑配置有关系，可以加速编译

layer {
      name: "loss"
      type: "SoftmaxWithLossOHEM"
      bottom: "fc_end"
      bottom: "label"
      top: "loss"
    
    }

[Caffe _ _Layer]: https://www.jianshu.com/p/b6ec5eaf737f