Dl4j-fit(DataSetIterator iterator)源码阅读（六）反向传播部分

青旅半醒 2022-06-04 06:42 118阅读 0赞

*  backprop
    
     *  1 calcBackpropGradients
        
         *  11 initGradientsView
            
             *  111 numParamsNeuralNetConfiguration conf
         *  11 initGradientsView
            
             *  112 flattenedGradientsget
             *  113 pointint point
             *  114 intervalint begin int end
     *  1 calcBackpropGradients
        
         *  21 outputLayerbackpropGradientINDArray epsilon
            
             *  211 preOutput2dtrue
             *  212 getGradientsAndDeltaINDArray preOut
                
                 *  2121 lossFunctioncomputeGradient
                 *  2121 lossFunctioncomputeGradient
             *  212 getGradientsAndDeltaINDArray preOut
         *  21 outputLayerbackpropGradientINDArray epsilon
     *  1 calcBackpropGradients
 *  backprop
 *  Pair数据结构源码解读
 *  Gradient
 *  DefaultGradient
 *  Nd4jgemm

# 1 backprop() #

/** Calculate and set gradients for MultiLayerNetwork, based on OutputLayer and labels*/
    protected void backprop() {
        Pair<Gradient, INDArray> pair = calcBackpropGradients(null, true);
        this.gradient = (pair == null ? null : pair.getFirst());
        this.epsilon = (pair == null ? null : pair.getSecond());
    }

根据注释可以看出是基于输出层和标签计算多层网络的梯度。之后跳进另外一个函数内。

## 1.1 calcBackpropGradients ##

这段代码提供的注释很多，这里直接翻译原有注释

/** 计算梯度和偏差. 在一下的两个地方使用: * (a) backprop (用于标准的网络结构) * (b) backpropGradient (layer类方法, 用于当MultiLayerNetwork类被用作layer的时候) * @param epsilon 偏差 (technically errors .* activations). 当withOutputLayer = true时，不被使用 * @param withOutputLayer 如果为true: 认为最后一层为输出层, 并且根据标签计算偏差. 在这种情况下输入的epsilon将不会被使用（可能为null） * 如果为false: 计算反向传播的梯度 * @return 输入的梯度和偏差 (epsilon) */
    protected Pair<Gradient, INDArray> calcBackpropGradients(INDArray epsilon, boolean withOutputLayer) {
        if (flattenedGradients == null)
            initGradientsView();

在刚开始运行的时候`flattenedGradients`字段为null。之后进入`initGradientsView()`方法。

### 1.1.1 initGradientsView() ###

/** * This method: 用于初始化展平的梯度数据 (用于反向传播)并且对于所有的网络层设置梯度子集 * 作为一般规则，在通过fit（DataSet）或Fit（DataSetIterator）进行训练时，不需要手动调用它， */
    public void initGradientsView() {
        if (layers == null)
            init();
    
        //获取网络层的层数
        int nLayers = layers.length;
    
        //首先: 计算（反向传播）参数的总长度
        int backpropParamLength = 0;
        int[] nParamsPerLayer = new int[nLayers];
        for (int i = 0; i < nLayers; i++) {
            NeuralNetConfiguration conf = layerWiseConfigurations.getConf(i);
            nParamsPerLayer[i] = layers[i].conf().getLayer().initializer().numParams(conf);
            backpropParamLength += nParamsPerLayer[i];
        }

#### 1.1.1.1 numParams(NeuralNetConfiguration conf) ####

@Override
    public int numParams(NeuralNetConfiguration conf) {
        org.deeplearning4j.nn.conf.layers.FeedForwardLayer layerConf =
                        (org.deeplearning4j.nn.conf.layers.FeedForwardLayer) conf.getLayer();
        int nIn = layerConf.getNIn();
        int nOut = layerConf.getNOut();
        return nIn * nOut + nOut; //weights + bias
    }

因为需要计算每一层的参数个数，所以需要调用如上的函数，每一个层参数的计算个数方法为`nIn * nOut + nOut`。  
然后继续返回到上层函数继续执行。

### 1.1.1 initGradientsView() ###

//以上计算得出参数的总个数之后，创建ndarray。'f'代表数组的存储顺序，有兴趣可以查阅nd4j官网
    flattenedGradients = Nd4j.zeros(new int[] {
       1, backpropParamLength}, 'f');
    
    int backpropParamsSoFar = 0;
    for (int i = 0; i < layers.length; i++) {
        //如果该层参数个数为0则跳过当前层
        if (nParamsPerLayer[i] == 0)
            continue; //This layer doesn't have any parameters...
    
        //NDArrayIndex.point(0)用于指定是第几行，这里就是指定第0行
        //NDArrayIndex.interval(backpropParamsSoFar, backpropParamsSoFar + nParamsPerLayer[i]) 用于获取列坐标索引
        INDArray thisLayerGradView = flattenedGradients.get(NDArrayIndex.point(0),
                        NDArrayIndex.interval(backpropParamsSoFar, backpropParamsSoFar + nParamsPerLayer[i]));
        layers[i].setBackpropGradientsViewArray(thisLayerGradView);
        backpropParamsSoFar += nParamsPerLayer[i];
    }

刚开始看这段代码的时候，不懂很明白这是要做什么，这时候需要回看到这个函数的目的是什么。这个函数的目的是`initializes the flattened gradients array (used in backprop) and sets the appropriate subset in all layers.`。这里的for loop主要是设置各个层的梯度数组。

#### 1.1.1.2 flattenedGradients.get() ####

/**
     * Returns a subset of this array based on the specified
     * indexes
     *
     * @param indexes the indexes in to the array
     * @return a view of the array with the specified indices
     */
    INDArray get(INDArrayIndex... indexes);

这个方法主要是根据给定的数组索引获取数组的子集。

#### 1.1.1.3 point(int point) ####

/** * Returns a point index * @param point the point index * @return the point index based * on the specified point */
    public static INDArrayIndex point(int point) {
        return new PointIndex(point);
    }

用户返回指定点的索引，在这里使用主要是指定行索引。

#### 1.1.1.4 interval(int begin, int end) ####

/** * Generates an interval from begin (inclusive) to end (exclusive) * * @param begin the begin * @param end the end index * @return the interval */
    public static INDArrayIndex interval(int begin, int end) {
        return interval(begin, 1, end, false);
    }

生成区间\[begin, end)区间内数据的索引，用于获取列索引。

## 1.1 calcBackpropGradients ##

String multiGradientKey;
    //使用初始化之后的flattenedGradients构造梯度类
    Gradient gradient = new DefaultGradient(flattenedGradients);
    Layer currLayer;
    
    //计算并应用每个图层的后向梯度
    /** * 跳过索引的输出层，只是向后循环更新每个层的系数。 * (当 withOutputLayer == true) * * 激活为每个图层应用激活函数，并将其设置为下一图层的输入。 * * Typical literature contains most trivial case for the error calculation: wT * weights * This interpretation transpose a few things to get mini batch because ND4J is rows vs columns organization for params */
    int numLayers = getnLayers();
    //将梯度存储为列表; used to ensure iteration order in DefaultGradient linked hash map. i.e., layer 0 first instead of output layer
    LinkedList<Triple<String, INDArray, Character>> gradientList = new LinkedList<>();

在构造梯度类，梯度列表以及获取网络结构的层数之后，继续执行以下语句：

int layerFrom;
    Pair<Gradient, INDArray> currPair;
    //判断是否使用输出层
    if (withOutputLayer) {
        //对输出层做类型检查
        if (!(getOutputLayer() instanceof IOutputLayer)) {
            log.warn("Warning: final layer isn't output layer. You cannot use backprop without an output layer.");
            return null;
        }
    
        //获取输出层
        IOutputLayer outputLayer = (IOutputLayer) getOutputLayer();
        //对标签进行检查
        if (labels == null)
            throw new IllegalStateException("No labels found");
        //设置输出层的标签，用于计算偏差
        outputLayer.setLabels(labels);
        //首先获取输出层的梯度
        currPair = outputLayer.backpropGradient(null);

接下来单步进入输出层反向传播梯度的的函数

### 1.2.1 outputLayer.backpropGradient(INDArray epsilon) ###

@Override
    public Pair<Gradient, INDArray> backpropGradient(INDArray epsilon) {
        Pair<Gradient, INDArray> pair = getGradientsAndDelta(preOutput2d(true)); //Returns Gradient and delta^(this), not Gradient and epsilon^(this-1)
        INDArray delta = pair.getSecond();
    
        INDArray epsilonNext = params.get(DefaultParamInitializer.WEIGHT_KEY).mmul(delta.transpose()).transpose();
        return new Pair<>(pair.getFirst(), epsilonNext);
    }

#### 1.2.1.1 preOutput2d(true) ####

调用链如下

protected INDArray preOutput2d(boolean training) {
        return preOutput(training);
    }
    
    public INDArray preOutput(boolean training) {
        applyDropOutIfNecessary(training);
        INDArray b = getParam(DefaultParamInitializer.BIAS_KEY);
        INDArray W = getParam(DefaultParamInitializer.WEIGHT_KEY);
    
        //Input validation:
        if (input.rank() != 2 || input.columns() != W.rows()) {
            if (input.rank() != 2) {
                throw new DL4JInvalidInputException("Input that is not a matrix; expected matrix (rank 2), got rank "
                                + input.rank() + " array with shape " + Arrays.toString(input.shape()));
            }
            throw new DL4JInvalidInputException("Input size (" + input.columns() + " columns; shape = "
                            + Arrays.toString(input.shape())
                            + ") is invalid: does not match layer input size (layer # inputs = " + W.size(0) + ")");
        }
    
        if (conf.isUseDropConnect() && training && conf.getLayer().getDropOut() > 0) {
            W = Dropout.applyDropConnect(this, DefaultParamInitializer.WEIGHT_KEY);
        }
    
        INDArray ret = input.mmul(W).addiRowVector(b);
    
        if (maskArray != null) {
            applyMask(ret);
        }
    
        return ret;
    }

因为这里是OutputLayer在调用，这里相当于使用`y = xw + b`计算并得出输出层还未经过激活函数变换的输出。并将计算得出的结果传入到`getGradientsAndDelta(INDArray preOut)`方法中

#### 1.2.1.2 getGradientsAndDelta(INDArray preOut) ####

/** Returns tuple: {Gradient,Delta,Output} given preOut */
    private Pair<Gradient, INDArray> getGradientsAndDelta(INDArray preOut) {
        //首先获取当前层的损失函数
        ILossFunction lossFunction = layerConf().getLossFn();
        //获取2维的列表。 （主要是针对RNN CNN这种网络，因为他们的数据组成方式是3d或者4d，需要转化为2d之后才能残油矩阵运算）
        INDArray labels2d = getLabels2d();
        //判断两个矩阵的形状，进行一个检验
        if (labels2d.size(1) != preOut.size(1)) {
            throw new DL4JInvalidInputException("Labels array numColumns (size(1) = " + labels2d.size(1)
                            + ") does not match output layer" + " number of outputs (nOut = " + preOut.size(1) + ")");
        }
        //传入标签，输出层的输出，激活函数和掩码，利用损失函数来计算偏差
        INDArray delta = lossFunction.computeGradient(labels2d, preOut, layerConf().getActivationFn(), maskArray);

##### 1.2.1.2.1 lossFunction.computeGradient #####

@Override
    public INDArray computeGradient(INDArray labels, INDArray preOutput, IActivation activationFn, INDArray mask) {
        INDArray gradients = super.computeGradient(labels, preOutput, activationFn, mask);
        return gradients.divi(labels.size(1));
    }

之后调用父类的`computeGradient()`方法来计算梯度

@Override
    public INDArray computeGradient(INDArray labels, INDArray preOutput, IActivation activationFn, INDArray mask) {
        //因为前面获取的preOutput只是 y = xw + b部分，尚未经过激活函数的变化
        //所以这里先 复制一份preOutput，然后经过激活函数的变换获得output
        INDArray output = activationFn.getActivation(preOutput.dup(), true);
    
        //这里先计算两个矩阵之间的差距， 然后再*2
        //这里是因为当前的损失函数类型为LossMSE()， 所以需要对
        INDArray dLda = output.subi(labels).muli(2);
    
    
        //损失函数的权重为null，为此不进行改步操作
        if (weights != null) {
            dLda.muliRowVector(weights);
        }
    
        //如若使用掩码，则与掩码进行计算
        f(mask != null && LossUtil.isPerOutputMasking(dLda, mask)){
            //For *most* activation functions: we don't actually need to mask dL/da in addition to masking dL/dz later
            //but: some, like softmax, require both (due to dL/dz_i being a function of dL/da_j, for i != j)
            //We could add a special case for softmax (activationFn instanceof ActivationSoftmax) but that would be
            // error prone - but buy us a tiny bit of performance
            LossUtil.applyMask(dLda, mask);
        }
    
        //根据激活函数计算梯度
        INDArray gradients = activationFn.backprop(preOutput, dLda).getFirst();

这里会调用激活函数的backprop，主要是用于求其在激活函数之后的梯度。  
因为我这里最后一层的函数为IDENTITY，本身对输入不会做任何变换，所以直接返回本身。

@Override
    public Pair<INDArray, INDArray> backprop(INDArray in, INDArray epsilon) {
        return new Pair<>(epsilon, null);
    }

之后调用Pair.getFirst()，就是为了获取激活函数求导之后的epsilon

//Loss function with masking
        if (mask != null) {
            LossUtil.applyMask(gradients, mask);
        }
    
        return gradients;
    }

在这个函数最后调用掩码进行计算，然后这个函数到这里执行完毕，返回上层函数。

##### 1.2.1.2.1 lossFunction.computeGradient #####

INDArray gradients = super.computeGradient(labels, preOutput, activationFn, mask);
        return gradients.divi(labels.size(1));
    }

调用`gradients.divi(labels.size(1))`，梯度除以labels的第二个维度。一般情况下为最后一层的神经元个数。然后返回到`getGradientsAndDelta(INDArray preOut)`方法中。

#### 1.2.1.2 getGradientsAndDelta(INDArray preOut) ####

//初始化新的梯度类
        Gradient gradient = new DefaultGradient();
    
        //获取权重梯度视图
        INDArray weightGradView = gradientViews.get(DefaultParamInitializer.WEIGHT_KEY);
        //获取偏重梯度视图
        INDArray biasGradView = gradientViews.get(DefaultParamInitializer.BIAS_KEY);
    
        //Equivalent to:  weightGradView.assign(input.transpose().mmul(delta));
        //相当于更新权重的梯度
        Nd4j.gemm(input, delta, weightGradView, true, false, 1.0, 0.0);
    
        //对权重梯度进行赋值，初始值为 delta的第0行之和
        biasGradView.assign(delta.sum(0));
    
        //将权重的梯度放入到初始化之后的梯度类中
        gradient.gradientForVariable().put(DefaultParamInitializer.WEIGHT_KEY, weightGradView);
        gradient.gradientForVariable().put(DefaultParamInitializer.BIAS_KEY, biasGradView);
    
        //返回梯度和delta
        return new Pair<>(gradient, delta);
    }

返回到上层函数`backpropGradient(INDArray epsilon)`中

### 1.2.1 outputLayer.backpropGradient(INDArray epsilon) ###

//获取delta
        INDArray delta = pair.getSecond();
        //误差 = (w * delta^T) ^ T
        INDArray epsilonNext = params.get(DefaultParamInitializer.WEIGHT_KEY).mmul(delta.transpose()).transpose();
        return new Pair<>(pair.getFirst(), epsilonNext);
    }

## 1.1 calcBackpropGradients ##

然后返回上层函数继续执行

//获取到了当前层的Pair<Gradient, INDArray>
        currPair = outputLayer.backpropGradient(null);
    
        //遍历梯度map里面的 权重梯度以及偏置梯度
        for (Map.Entry<String, INDArray> entry : currPair.getFirst().gradientForVariable().entrySet()) {
            //获取原始的名称，基本为"W", "b"
            String origName = entry.getKey();
            //然后根据当前所在的层数进行拼装，比如变成"1_W", "1_b"
            multiGradientKey = String.valueOf(numLayers - 1) + "_" + origName;
    
            //然后构建三元组， 字符串新名称， 梯度INDArray，以及展平之后的梯度INDArray
            //添加到链表的最后
            gradientList.addLast(new Triple<>(multiGradientKey, entry.getValue(),
                            currPair.getFirst().flatteningOrderForVariable(origName)));
        }
    
        //判断是否有输入预处理操作
        if (getLayerWiseConfigurations().getInputPreProcess(numLayers - 1) != null)
            currPair = new Pair<>(currPair.getFirst(),
                            this.layerWiseConfigurations.getInputPreProcess(numLayers - 1)
                                            .backprop(currPair.getSecond(), getInputMiniBatchSize()));
    
        //numLayers - 1为输出层，且输出层的梯度和误差以及计算好了
        //所以layerFrom为 numLayers - 2
        layerFrom = numLayers - 2;
    } else {
    
        //如果无输出层，则从numLayers - 1开始
        currPair = new Pair<>(null, epsilon);
        layerFrom = numLayers - 1;
    }
    
    
    //根据前面计算的梯度来进行反向传播
    // Calculate gradients for previous layers & drops output layer in count
    for (int j = layerFrom; j >= 0; j--) {
        //获取当前网络层
        currLayer = getLayer(j);
        //如果当前层是FrozenLayer，终止反向传播
        if (currLayer instanceof FrozenLayer)
            break;
    
        //根据上一层的误差来重新计算梯度和误差便于继续反向传播
        currPair = currLayer.backpropGradient(currPair.getSecond());
    
        //新建三元组子列表
        LinkedList<Triple<String, INDArray, Character>> tempList = new LinkedList<>();
    
        //遍历梯度和误差
        for (Map.Entry<String, INDArray> entry : currPair.getFirst().gradientForVariable().entrySet()) {
            String origName = entry.getKey();
            multiGradientKey = String.valueOf(j) + "_" + origName;
            tempList.addFirst(new Triple<>(multiGradientKey, entry.getValue(),
                            currPair.getFirst().flatteningOrderForVariable(origName)));
        }
    
        //加入到gradientList的前面
        for (Triple<String, INDArray, Character> triple : tempList)
            gradientList.addFirst(triple);
    
        //Pass epsilon through input processor before passing to next layer (if applicable)
        if (getLayerWiseConfigurations().getInputPreProcess(j) != null)
            currPair = new Pair<>(currPair.getFirst(), getLayerWiseConfigurations().getInputPreProcess(j)
                            .backprop(currPair.getSecond(), getInputMiniBatchSize()));
    }
    
    //Add gradients to Gradients (map), in correct order
    //把所有梯度以正确的顺序加入到Gradients (map)中
    for (Triple<String, INDArray, Character> triple : gradientList) {
        gradient.setGradientFor(triple.getFirst(), triple.getSecond(), triple.getThird());
    }
    
    //返回当前的梯度，和误差
    return new Pair<>(gradient, currPair.getSecond());

# 1 backprop() #

对于当前的`MultiLayerNetwork`类设置成员变量的值。

# Pair数据结构源码解读 #

package org.deeplearning4j.berkeley;
    
    /** * A generic-typed pair of objects. * @author Dan Klein */
    public class Pair<F, S> implements Serializable, Comparable<Pair<F, S>> { 
        static final long serialVersionUID = 42;
    
        F first;
        S second;
    
        public F getFirst() {
            return first;
        }
    
        public S getSecond() {
            return second;
        }
    
        public void setFirst(F pFirst) {
            first = pFirst;
        }
    
        public void setSecond(S pSecond) {
            second = pSecond;
        }
    
        public Pair<S, F> reverse() {
            return new Pair<>(second, first);
        }
    }

# Gradient #

package org.deeplearning4j.nn.gradient;
    
    /** * Generic gradient * * @author Adam Gibson */
    public interface Gradient extends Serializable { 
    
        /** * Gradient look up table * * @return the gradient look up table */
        Map<String, INDArray> gradientForVariable();
    
        /** * The full gradient as one flat vector * * @return */
        INDArray gradient(List<String> order);
    
        /** * The full gradient as one flat vector * * @return */
        INDArray gradient();
    
        /** * Clear residual parameters (useful for returning a gradient and then clearing old objects) */
        void clear();
    
        /** * The gradient for the given variable * * @param variable the variable to get the gradient for * @return the gradient for the given variable or null */
        INDArray getGradientFor(String variable);
    
        /** * Update gradient for the given variable * * @param variable the variable to get the gradient for * @param gradient the gradient values * @return the gradient for the given variable or null */
        INDArray setGradientFor(String variable, INDArray gradient);
    
        /** * Update gradient for the given variable; also (optionally) specify the order in which the array should be flattened * to a row vector * * @param variable the variable to get the gradient for * @param gradient the gradient values * @param flatteningOrder the order in which gradients should be flattened (null ok - default) * @return the gradient for the given variable or null */
        INDArray setGradientFor(String variable, INDArray gradient, Character flatteningOrder);
    
        /** * Return the gradient flattening order for the specified variable, or null if it is not explicitly set * @param variable Variable to return the gradient flattening order for * @return Order in which the specified variable's gradient should be flattened */
        Character flatteningOrderForVariable(String variable);
    
    }

# DefaultGradient #

package org.deeplearning4j.nn.gradient;
    
    /** * Default gradient implementation. Basically lookup table * for ndarrays * * @author Adam Gibson */
    
    public class DefaultGradient implements Gradient { 
        public static final char DEFAULT_FLATTENING_ORDER = 'f';
        private Map<String, INDArray> gradients = new LinkedHashMap<>();
        private Map<String, Character> flatteningOrders;
        private INDArray flattenedGradient;
    
        public DefaultGradient() {}
    
        public DefaultGradient(INDArray flattenedGradient) {
            this.flattenedGradient = flattenedGradient;
        }
    
        @Override
        public Map<String, INDArray> gradientForVariable() {
            return gradients;
        }
    
        @Override
        public INDArray gradient(List<String> order) {
            List<INDArray> toFlatten = new ArrayList<>();
            if (flatteningOrders == null) {
                for (String s : order) {
                    if (!gradients.containsKey(s))
                        continue;
                    toFlatten.add(gradients.get(s));
                }
            } else {
                for (String s : order) {
                    if (!gradients.containsKey(s))
                        continue;
                    if (flatteningOrders.containsKey(s) && flatteningOrders.get(s) != DEFAULT_FLATTENING_ORDER) {
                        //Arrays with non-default order get flattened to row vector first, then everything is flattened to f order
                        //TODO revisit this, and make more efficient
                        toFlatten.add(Nd4j.toFlattened(flatteningOrders.get(s), gradients.get(s)));
                    } else {
                        toFlatten.add(gradients.get(s));
                    }
                }
            }
            return Nd4j.toFlattened(DEFAULT_FLATTENING_ORDER, toFlatten);
        }
    
        private void flattenGradient() {
            if (flatteningOrders != null) {
                //Arrays with non-default order get flattened to row vector first, then everything is flattened to f order
                //TODO revisit this, and make more efficient
                List<INDArray> toFlatten = new ArrayList<>();
                for (Map.Entry<String, INDArray> entry : gradients.entrySet()) {
                    if (flatteningOrders.containsKey(entry.getKey())
                                    && flatteningOrders.get(entry.getKey()) != DEFAULT_FLATTENING_ORDER) {
                        //Specific flattening order for this array, that isn't the default
                        toFlatten.add(Nd4j.toFlattened(flatteningOrders.get(entry.getKey()), entry.getValue()));
                    } else {
                        //default flattening order for this array
                        toFlatten.add(entry.getValue());
                    }
                }
                flattenedGradient = Nd4j.toFlattened(DEFAULT_FLATTENING_ORDER, toFlatten);
            } else {
                //Standard case: flatten all to f order
                flattenedGradient = Nd4j.toFlattened(DEFAULT_FLATTENING_ORDER, gradients.values());
            }
        }
    
        @Override
        public INDArray gradient() {
            if (flattenedGradient != null)
                return flattenedGradient;
            flattenGradient();
            return flattenedGradient;
        }
    
        @Override
        public void clear() {
            gradients.clear();
        }
    
        @Override
        public INDArray getGradientFor(String variable) {
            return gradients.get(variable);
        }
    
        @Override
        public INDArray setGradientFor(String variable, INDArray newGradient) {
            INDArray last = gradients.put(variable, newGradient);
            // TODO revisit whether setGradientFor should update the gradient that can be pulled from this object in any form - currently does not update flattened
            // use of unitialized var for flattengradient in backprop is generating an error in gradient calc if bellow is used
            // flattenGradient();
            return last;
        }
    
        @Override
        public INDArray setGradientFor(String variable, INDArray gradient, Character flatteningOrder) {
            INDArray last = setGradientFor(variable, gradient);
    
            if (flatteningOrder != null) {
                if (flatteningOrders == null)
                    flatteningOrders = new LinkedHashMap<>();
                flatteningOrders.put(variable, flatteningOrder);
            }
            return last;
        }
    
        @Override
        public Character flatteningOrderForVariable(String variable) {
            if (flatteningOrders == null)
                return null;
            return flatteningOrders.get(variable);
        }
    
    
        @Override
        public String toString() {
            return "DefaultGradient{" + "gradients=" + gradients + (flatteningOrders != null ? flatteningOrders : "") + '}';
        }
    }

# Nd4j.gemm #

/** Matrix multiply: Implements c = alpha*op(a)*op(b) + beta*c where op(X) means transpose X (or not)
     * depending on setting of arguments transposeA and transposeB.<br>
     * Note that matrix c MUST be fortran order, have zero offset and have c.data().length == c.length().
     * An exception will be thrown otherwise.<br>
     * Don't use this unless you know about level 3 blas and NDArray storage orders.
     * @param a First matrix
     * @param b Second matrix
     * @param c result matrix. Used in calculation (assuming beta != 0) and result is stored in this. f order,
     *          zero offset and length == data.length only
     * @param transposeA if true: transpose matrix a before mmul
     * @param transposeB if true: transpose matrix b before mmul
     * @return result, i.e., matrix c is returned for convenience
     */
    public static INDArray gemm(INDArray a, INDArray b, INDArray c, boolean transposeA, boolean transposeB,
                    double alpha, double beta) {
        getBlasWrapper().level3().gemm(a, b, c, transposeA, transposeB, alpha, beta);
     return c;
    }