tensorflow教程:tf.contrib.rnn.DropoutWrapper

素颜马尾好姑娘i 2022-04-14 05:54 172阅读 0赞

tf.contrib.rnn.DropoutWrapper   
Defined in tensorflow/python/ops/rnn\_cell\_impl.py.

__init__(
        cell,
        input_keep_prob=1.0,
        output_keep_prob=1.0,
        state_keep_prob=1.0,
        variational_recurrent=False,
        input_size=None,
        dtype=None,
        seed=None,
        dropout_state_filter_visitor=None
    )

Args:
    
    cell:  an RNNCell, a projection to output_size is added to it.
    input_keep_prob:  unit Tensor or float between 0 and 1, input keep probability; if it is constant and 1, no input dropout will be added.
    output_keep_prob:  unit Tensor or float between 0 and 1, output keep probability; if it is constant and 1, no output dropout will be added.
    state_keep_prob:  unit Tensor or float between 0 and 1, output keep probability; if it is constant and 1, no output dropout will be added. State dropout is performed on the outgoing states of the cell. Note the state components to which dropout is applied when state_keep_prob is in (0, 1) are also determined by the argument dropout_state_filter_visitor (e.g. by default dropout is never applied to the c component of an LSTMStateTuple).
    variational_recurrent:  Python bool. If True, then the same dropout pattern is applied across all time steps per run call. If this parameter is set, input_size must be provided.
    input_size:  (optional) (possibly nested tuple of) TensorShape objects containing the depth(s) of the input tensors expected to be passed in to the DropoutWrapper. Required and used iff variational_recurrent = True and input_keep_prob < 1.
    dtype:  (optional) The dtype of the input, state, and output tensors. Required and used iff variational_recurrent = True.
    seed:  (optional) integer, the randomness seed.
    dropout_state_filter_visitor:  (optional), default: (see below). Function that takes any hierarchical level of the state and returns a scalar or depth=1 structure of Python booleans describing which terms in the state should be dropped out. In addition, if the function returns True, dropout is applied across this sublevel. If the function returns False, dropout is not applied across this entire sublevel. Default behavior: perform dropout on all terms except the memory (c) state of LSTMCellState objects, and don't try to apply dropout to TensorArray objects: def dropout_state_filter_visitor(s):  if isinstance(s, LSTMCellState): # Never perform dropout on the c state. return LSTMCellState(c=False, h=True) elif isinstance(s, TensorArray): return False return True

所谓dropout,就是指网络中每个单元在每次有数据流入时以一定的概率(keep prob)正常工作，否则输出0值。这是是一种有效的正则化方法，可以有效防止过拟合。在rnn中使用dropout的方法和cnn不同，推荐大家去把recurrent neural network regularization看一遍。   
在rnn中进行dropout时，对于rnn的部分不进行dropout，也就是说从t-1时候的状态传递到t时刻进行计算时，这个中间不进行memory的dropout；仅在同一个t时刻中，多层cell之间传递信息的时候进行dropout，如下图所示   
   
上图中，t-2时刻的输入xt−2首先传入第一层cell，这个过程有dropout，但是从t−2时刻的第一层cell传到t−1,t,t+1的第一层cell这个中间都不进行dropout。再从t+1时候的第一层cell向同一时刻内后续的cell传递时，这之间又有dropout了。

在使用tf.nn.rnn\_cell.DropoutWrapper时，同样有一些参数，例如input\_keep\_prob,output\_keep\_prob等，分别控制输入和输出的dropout概率，很好理解。   
可以从官方文档中看到，它有input\_keep\_prob和output\_keep\_prob，也就是说裹上这个DropoutWrapper之后，如果我希望是input传入这个cell时dropout掉一部分input信息的话，就设置input\_keep\_prob，那么传入到cell的就是部分input；如果我希望这个cell的output只部分作为下一层cell的input的话，就定义output\_keep\_prob。   
备注：Dropout只能是层与层之间（输入层与LSTM1层、LSTM1层与LSTM2层）的Dropout；同一个层里面，T时刻与T+1时刻是不会Dropout的。