一种结合门控循环单元 (GRU) 和支持向量机 (SVM) 的神经网络架构(代码实现）

ゞ浴缸里的玫瑰 2022-10-04 15:49 73阅读 0赞

门控循环单元 网络结构(GRU) 是基于长短期记忆 (LSTM) 单元的变体，两者都是循环神经网络 (RNN) 的类型。通过经验证据，两种模型都已被证明在各种机器学习任务中有效，例如自然语言处理、语音识别和文本分类等。传统上与大多数神经网络一样，上述两种 RNN 变体都使用 Softmax 函数作为最终的输出层，并使用交叉熵函数来计算其损失。在本文中，我们通过在 GRU 模型的最终输出层中引入支持向量机 (SVM) 作为 Softmax 的替代品来对该模式进行修正。此外，交叉熵函数应替换为 margin-based function loss。该模型命名为GRU-SVM。

模型结构如下:![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MDY1MTUxNQ_size_16_color_FFFFFF_t_70][]

其实现代码：

class GruSvm:
        def __init__(
            self,
            alpha,
            batch_size,
            cell_size,
            dropout_rate,
            num_classes,
            sequence_length,
            svm_c,
        ):
            self.alpha = alpha
            self.batch_size = batch_size
            self.cell_size = cell_size
            self.dropout_rate = dropout_rate
            self.num_classes = num_classes
            self.sequence_length = sequence_length
            self.svm_c = svm_c
    
            def __graph__():
                """Build the inference graph"""
                with tf.name_scope("input"):
                    # [BATCH_SIZE, SEQUENCE_LENGTH]
                    x_input = tf.placeholder(
                        dtype=tf.uint8, shape=[None, self.sequence_length], name="x_input"
                    )
    
                    # [BATCH_SIZE, SEQUENCE_LENGTH, 10]
                    x_onehot = tf.one_hot(
                        indices=x_input,
                        depth=10,
                        on_value=1.0,
                        off_value=0.0,
                        name="x_onehot",
                    )
    
                    # [BATCH_SIZE]
                    y_input = tf.placeholder(dtype=tf.uint8, shape=[None], name="y_input")
    
                    # [BATCH_SIZE, N_CLASSES]
                    y_onehot = tf.one_hot(
                        indices=y_input,
                        depth=self.num_classes,
                        on_value=1.0,
                        off_value=-1.0,
                        name="y_onehot",
                    )
    
                state = tf.placeholder(
                    dtype=tf.float32, shape=[None, self.cell_size], name="initial_state"
                )
    
                p_keep = tf.placeholder(dtype=tf.float32, name="p_keep")
                learning_rate = tf.placeholder(dtype=tf.float32, name="learning_rate")
    
                cell = tf.contrib.rnn.GRUCell(self.cell_size)
                drop_cell = tf.contrib.rnn.DropoutWrapper(cell, input_keep_prob=p_keep)
    
                # outputs: [BATCH_SIZE, SEQUENCE_LENGTH, CELL_SIZE]
                # states: [BATCH_SIZE, CELL_SIZE]
                outputs, states = tf.nn.dynamic_rnn(
                    drop_cell, x_onehot, initial_state=state, dtype=tf.float32
                )
    
                states = tf.identity(states, name="H")
    
                with tf.name_scope("final_training_ops"):
                    with tf.name_scope("weights"):
                        weight = tf.get_variable(
                            "weights",
                            initializer=tf.random_normal(
                                [self.cell_size, self.num_classes], stddev=0.01
                            ),
                        )
                        self.variable_summaries(weight)
                    with tf.name_scope("biases"):
                        bias = tf.get_variable(
                            "biases", initializer=tf.constant(0.1, shape=[self.num_classes])
                        )
                        self.variable_summaries(bias)
                    hf = tf.transpose(outputs, [1, 0, 2])
                    last = tf.gather(hf, int(hf.get_shape()[0]) - 1)
                    with tf.name_scope("Wx_plus_b"):
                        output = tf.matmul(last, weight) + bias
                        tf.summary.histogram("pre-activations", output)
    
                # L2-SVM
                with tf.name_scope("svm"):
                    regularization_loss = 0.5 * tf.reduce_sum(tf.square(weight))
                    hinge_loss = tf.reduce_sum(
                        tf.square(
                            tf.maximum(
                                tf.zeros([self.batch_size, self.num_classes]),
                                1 - y_onehot * output,
                            )
                        )
                    )
                    with tf.name_scope("loss"):
                        loss = regularization_loss + self.svm_c * hinge_loss
                tf.summary.scalar("loss", loss)
    
                optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(
                    loss
                )
    
                with tf.name_scope("accuracy"):
                    predicted_class = tf.sign(output)
                    predicted_class = tf.identity(predicted_class, name="prediction")
                    with tf.name_scope("correct_prediction"):
                        correct = tf.equal(
                            tf.argmax(predicted_class, 1), tf.argmax(y_onehot, 1)
                        )
                    with tf.name_scope("accuracy"):
                        accuracy = tf.reduce_mean(tf.cast(correct, "float"))
                tf.summary.scalar("accuracy", accuracy)
    
                # merge all the summaries collected from the TF graph
                merged = tf.summary.merge_all()
    
                # set class properties
                self.x_input = x_input
                self.y_input = y_input
                self.y_onehot = y_onehot
                self.p_keep = p_keep
                self.loss = loss
                self.optimizer = optimizer
                self.state = state
                self.states = states
                self.learning_rate = learning_rate
                self.predicted_class = predicted_class
                self.accuracy = accuracy
                self.merged = merged
    
            sys.stdout.write("\n<log> Building Graph...")
            __graph__()
            sys.stdout.write("</log>\n")

其中该模型进行网络流量入侵检测实验后和GRU-softmax、SVM进行对比得出了不错的实验结果。

其完整代码下载连接：[https://download.csdn.net/download/weixin\_40651515/20204836][https_download.csdn.net_download_weixin_40651515_20204836]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MDY1MTUxNQ_size_16_color_FFFFFF_t_70 1][]

实验环境：python3.7.6 matplotlib==2.1.0 numpy==1.13.3 tensorflow==1.15.4 pandas==0.20.3  
scikit\_learn==0.19.1等。

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MDY1MTUxNQ_size_16_color_FFFFFF_t_70]: /images/20221004/5e76a50b158b4d3c89db7ea04ff5a317.png
[https_download.csdn.net_download_weixin_40651515_20204836]: https://download.csdn.net/download/weixin_40651515/20204836
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MDY1MTUxNQ_size_16_color_FFFFFF_t_70 1]: /images/20221004/8b1dea238f6d4eaf9234b8060d2c426e.png