Pytorch 不同的网络参数初始化对梯度消失的影响

亦凉 2021-11-04 16:24 398阅读 0赞

尽管ReLU相比其它激活函数已经大大避免了发生梯度消失或者为0的现象，但不好的参数初始化方法也会导致这种情况的出现。

1. 使用标准正态分布初始化

这里用`torch.randn()`来做实际上就是对参数w和b进行了N~(0,1)的标准正态分布初始化，这个初始化值都很小，比较接近0。

import torch
    from torch.nn import functional as F
    from torch import nn
    from torch import optim
    from torchvision import datasets, transforms
    
    """可以将超参数都放到一起写在最前面,方便调参"""
    batch_size = 200  # 每批的样本数量
    learning_rate = 0.01  # 学习率
    epochs = 10  # 跑多少次样本集
    
    """
    读取MNIST手写数字数据集,初次运行下载到../data/目录下
    
    要注意所有样本进行标准化的参数要保持一致(这里是样本和方差)
    这个标准化的参数是数据提供方计算好的
    所以就不用自己计算了,在网上查好然后标准化时候写进去就可以了
    """
    # 训练集
    train_loader = torch.utils.data.DataLoader(
    	datasets.MNIST('../data', train=True, download=True,  # train=True则得到的是训练集
    				   transform=transforms.Compose([  # 进行数据预处理
    					   transforms.ToTensor(),  # 这表示转成Tensor类型的数据
    					   transforms.Normalize((0.1307,), (0.3081,))  # 这里是进行数据标准化(减去均值除以方差)
    				   ])),
    	batch_size=batch_size, shuffle=True)  # 按batch_size分出一个batch维度在最前面,shuffle=True打乱顺序
    # 测试集
    test_loader = torch.utils.data.DataLoader(
    	datasets.MNIST('../data', train=False, transform=transforms.Compose([
    		transforms.ToTensor(),
    		transforms.Normalize((0.1307,), (0.3081,))
    	])),
    	batch_size=batch_size, shuffle=True)
    
    """建立三个线性层"""
    # 这里表示输入是784个分量,输出是200个分量的,注意这个顺序
    w1 = torch.randn(200, 784, requires_grad=True)
    b1 = torch.randn(200, requires_grad=True)
    w2 = torch.randn(200, 200, requires_grad=True)
    b2 = torch.randn(200, requires_grad=True)
    w3 = torch.randn(10, 200, requires_grad=True)
    b3 = torch.randn(10, requires_grad=True)
    
    
    def forward(x):
    	"""对输入的样本矩阵x的前向计算过程,x的shape是[样本数,784]"""
    	x = x @ w1.t() + b1
    	x = F.relu(x)  # 非线性激活
    	x = x @ w2.t() + b2
    	x = F.relu(x)
    	x = x @ w3.t() + b3
    	x = F.relu(x)
    	return x
    
    
    # 建立优化器,指明优化目标和学习率
    optimizer = optim.SGD([w1, b1, w2, b2, w3, b3], lr=1e-3)
    # 计算CEL的函数
    CEL = nn.CrossEntropyLoss()
    
    """训练+测试过程"""
    # 每个epoch是遍历一次样本集
    for epoch in range(epochs):
    	"""训练"""
    	# 对训练集中每个batch的样本,标签
    	for batch_idx, (data, target) in enumerate(train_loader):
    		# 摊平成shape=[样本数,784]的形状
    		data = data.reshape(-1, 28 * 28)
    		# 前向计算出logits
    		logits = forward(data)
    		# 计算Loss,这里不需要再Softmax一次,PyTorch计算CEL时已经做了Softmax了
    		loss = CEL(logits, target)
    		# 清空梯度
    		optimizer.zero_grad()
    		# 反向传播计算各个参数(优化目标)相对于Loss的梯度信息
    		loss.backward()
    		# 执行优化器
    		optimizer.step()
    		# 每100个batch输出一次信息
    		if batch_idx % 100 == 0:
    			print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
    				epoch, batch_idx * len(data), len(train_loader.dataset),
    					   100. * batch_idx / len(train_loader), loss.item()))
    	"""测试"""
    	test_loss = 0  # 在测试集上的Loss,反映了模型的表现
    	correct = 0  # 记录正确分类的样本数
    	# 对测试集中每个batch的样本,标签
    	for data, target in test_loader:
    		# 摊平成shape=[样本数,784]的形状
    		data = data.reshape(-1, 28 * 28)
    		logits = forward(data)
    		test_loss += CEL(logits, target).item()
    		# 得到的预测值输出是一个10个分量的概率,在第2个维度上取max
    		# logits.data是一个shape=[batch_size,10]的Tensor
    		# 注意Tensor.max(dim=1)是在这个Tensor的1号维度上求最大值
    		# 得到一个含有两个元素的元组,这两个元素都是shape=[batch_size]的Tensor
    		# 第一个Tensor里面存的都是最大值的值,第二个Tensor里面存的是对应的索引
    		# 这里要取索引,所以取了这个tuple的第二个元素
    		# print(type(logits.data), logits.data.shape,type(logits.data.max(dim=1)))
    		pred = logits.data.max(dim=1)[1]
    		# 对应位置相等则对应位置为True,这里用sum()即记录了True的数量
    		correct += pred.eq(target.data).sum()
    	test_loss /= len(test_loader.dataset)
    	print('\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n'.format(
    		test_loss, correct, len(test_loader.dataset),
    		100. * correct / len(test_loader.dataset)))

从运行结果中可以看出，测试数据上的Loss变化很小，在几个epoch之后仍然没有提高acc，发生了梯度消失或者梯度太小。

2. 使用何恺明的初始化方法

在第43行后面加入：

torch.nn.init.kaiming_normal_(w1)
    torch.nn.init.kaiming_normal_(w2)
    torch.nn.init.kaiming_normal_(w3)

运行结果：

Train Epoch: 0 [0/60000 (0%)]	Loss: 3.478527
    Train Epoch: 0 [20000/60000 (33%)]	Loss: 2.086296
    Train Epoch: 0 [40000/60000 (67%)]	Loss: 1.689571
    
    Test set: Average loss: 0.0067, Accuracy: 5949/10000 (59%)
    
    Train Epoch: 1 [0/60000 (0%)]	Loss: 1.394030
    Train Epoch: 1 [20000/60000 (33%)]	Loss: 1.099167
    Train Epoch: 1 [40000/60000 (67%)]	Loss: 1.029864
    
    Test set: Average loss: 0.0040, Accuracy: 7760/10000 (77%)
    
    Train Epoch: 2 [0/60000 (0%)]	Loss: 0.878572
    Train Epoch: 2 [20000/60000 (33%)]	Loss: 0.749158
    Train Epoch: 2 [40000/60000 (67%)]	Loss: 0.696058
    
    Test set: Average loss: 0.0031, Accuracy: 8268/10000 (82%)
    
    Train Epoch: 3 [0/60000 (0%)]	Loss: 0.594997
    Train Epoch: 3 [20000/60000 (33%)]	Loss: 0.583601
    Train Epoch: 3 [40000/60000 (67%)]	Loss: 0.575014
    
    Test set: Average loss: 0.0026, Accuracy: 8532/10000 (85%)
    
    Train Epoch: 4 [0/60000 (0%)]	Loss: 0.514690
    Train Epoch: 4 [20000/60000 (33%)]	Loss: 0.558908
    Train Epoch: 4 [40000/60000 (67%)]	Loss: 0.475922
    
    Test set: Average loss: 0.0024, Accuracy: 8665/10000 (86%)
    
    Train Epoch: 5 [0/60000 (0%)]	Loss: 0.509288
    Train Epoch: 5 [20000/60000 (33%)]	Loss: 0.482504
    Train Epoch: 5 [40000/60000 (67%)]	Loss: 0.624165
    
    Test set: Average loss: 0.0022, Accuracy: 8774/10000 (87%)
    
    Train Epoch: 6 [0/60000 (0%)]	Loss: 0.497107
    Train Epoch: 6 [20000/60000 (33%)]	Loss: 0.458236
    Train Epoch: 6 [40000/60000 (67%)]	Loss: 0.394187
    
    Test set: Average loss: 0.0020, Accuracy: 8870/10000 (88%)
    
    Train Epoch: 7 [0/60000 (0%)]	Loss: 0.368239
    Train Epoch: 7 [20000/60000 (33%)]	Loss: 0.375526
    Train Epoch: 7 [40000/60000 (67%)]	Loss: 0.390056
    
    Test set: Average loss: 0.0019, Accuracy: 8927/10000 (89%)
    
    Train Epoch: 8 [0/60000 (0%)]	Loss: 0.315364
    Train Epoch: 8 [20000/60000 (33%)]	Loss: 0.405996
    Train Epoch: 8 [40000/60000 (67%)]	Loss: 0.369839
    
    Test set: Average loss: 0.0018, Accuracy: 8973/10000 (89%)
    
    Train Epoch: 9 [0/60000 (0%)]	Loss: 0.381817
    Train Epoch: 9 [20000/60000 (33%)]	Loss: 0.399142
    Train Epoch: 9 [40000/60000 (67%)]	Loss: 0.376821
    
    Test set: Average loss: 0.0017, Accuracy: 9017/10000 (90%)

可见网络中参数的初始化方法对网络的运行影响是很大的。