tensorflow estimator 训练完的模型与验证时载入的模型是否一致 hook保存模型

傷城~ 2021-06-24 16:12 484阅读 0赞

estimator有个问题就是验证时是从文件中载入模型的，这样存在问题是无法保证从保存到载入期间的完全正确性。对于这种问题，我们一般采用少量数据，然后在训练集上进行验证。确认预测数据是否一致。主要是使用cond的控制最后几轮不进行训练，并且把数据打印出来。

def train_func():
    
                    # 构建训练节点
                    train_op = create_optimizer(
                        total_loss, lr, optimizer_params, 1., variables_to_train, use_fp16=FLAGS.use_fp16)
    
                    update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
                    update_ops.append(train_op)
                    update_op = tf.group(*update_ops)
                    with tf.control_dependencies([update_op]):
                        train_tensor = tf.identity(total_loss)
                    return train_tensor
    
    
                train_tensor = tf.cond(global_step_tensor>5, lambda : tf.Print(total_loss,
                                [tf.shape(per_example_loss), per_example_loss,
                                 tf.reduce_mean(per_example_loss),
                                 tf.reduce_mean(tf.to_float(per_example_loss<FLAGS.margin))], summarize=32), lambda : train_func() )

[tf.cond and tf.case execute all branches ][tf.cond and tf.case execute all branches]  
[tensorflow: Initializer for variable… is from inside a control-flow construct, a loop or conditional][tensorflow_ Initializer for variable_ is from inside a control-flow construct_ a loop or conditional]

另外可以使用hook把训练完或者验证完的模型参数保存下来，进行比较。

class LogSessionEvalHook(tf.train.SessionRunHook):
        def __init__(self):
            pass
    
        def after_create_session(self, session, coord):
            pass
    
        def before_run(self, run_context):
            pass
    
        def after_run(self, run_context, run_values):
            pass
    
        def end(self, sess):
            pass
            # sess.graph._unsafe_unfinalize()
            # saver = tf.train.Saver()
            # save_path = saver.save(sess, "/data00/huangqingkang/repos/bert/exps/bertrel/tmp/eval")
            # print("******* eval checkpoint: ", save_path, "********")

[tf.cond and tf.case execute all branches]: https://github.com/tensorflow/tensorflow/issues/3287
[tensorflow_ Initializer for variable_ is from inside a control-flow construct_ a loop or conditional]: https://stackoverflow.com/questions/48515034/keras-tensorflow-initializer-for-variable-is-from-inside-a-control-flow-con/51509387