tensorflow之训练数据制作

墨蓝 2022-06-16 08:26 182阅读 0赞

[TensorFlow高效读取数据的方法][TensorFlow]

[TensorFlow 学习（二） 制作自己的TFRecord数据集，读取，显示及代码详解][TensorFlow _ _TFRecord]

convert\_to\_records.py

"""Converts MNIST data to TFRecords file format with Example protos."""
    from __future__ import absolute_import
    from __future__ import division
    from __future__ import print_function
    
    import argparse
    import os
    import sys
    
    import tensorflow as tf
    
    from tensorflow.contrib.learn.python.learn.datasets import mnist
    
    FLAGS = None
    
    
    def _int64_feature(value):
      return tf.train.Feature(int64_list=tf.train.Int64List(value=[value]))
    
    
    def _bytes_feature(value):
      return tf.train.Feature(bytes_list=tf.train.BytesList(value=[value]))
    
    
    def convert_to(data_set, name):
      """Converts a dataset to tfrecords."""
      images = data_set.images
      labels = data_set.labels
      num_examples = data_set.num_examples
    
      if images.shape[0] != num_examples:
        raise ValueError('Images size %d does not match label size %d.' %
                         (images.shape[0], num_examples))
      rows = images.shape[1]
      cols = images.shape[2]
      depth = images.shape[3]
    
      filename = os.path.join(FLAGS.directory, name + '.tfrecords')
      print('Writing', filename)
      writer = tf.python_io.TFRecordWriter(filename)
      for index in range(num_examples):
        image_raw = images[index].tostring()
        example = tf.train.Example(features=tf.train.Features(feature={
            'height': _int64_feature(rows),
            'width': _int64_feature(cols),
            'depth': _int64_feature(depth),
            'label': _int64_feature(int(labels[index])),
            'image_raw': _bytes_feature(image_raw)}))
        writer.write(example.SerializeToString())
      writer.close()
    
    
    def main(unused_argv):
      # Get the data.
      data_sets = mnist.read_data_sets(FLAGS.directory,
                                       dtype=tf.uint8,
                                       reshape=False,
                                       validation_size=FLAGS.validation_size)
    
      # Convert to Examples and write the result to TFRecords.
      convert_to(data_sets.train, 'train')
      convert_to(data_sets.validation, 'validation')
      convert_to(data_sets.test, 'test')
    
    
    if __name__ == '__main__':
      parser = argparse.ArgumentParser()
      parser.add_argument(
          '--directory',
          type=str,
          default='/tmp/data',
          help='Directory to download data files and write the converted result'
      )
      parser.add_argument(
          '--validation_size',
          type=int,
          default=5000,
          help="""\
          Number of examples to separate from the training data for the validation
          set.\
          """
      )
      FLAGS, unparsed = parser.parse_known_args()
      tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)

fully\_connected\_reader.py

#coding:utf-8
    
    """Train and Eval the MNIST network.
    This version is like fully_connected_feed.py but uses data converted
    to a TFRecords file containing tf.train.Example protocol buffers.
    See:
    https://www.tensorflow.org/programmers_guide/reading_data#reading_from_files
    for context.
    YOU MUST run convert_to_records before running this (but you only need to
    run it once).
    """
    from __future__ import absolute_import
    from __future__ import division
    from __future__ import print_function
    
    import argparse
    import os.path
    import sys
    import time
    
    import tensorflow as tf
    
    from tensorflow.examples.tutorials.mnist import mnist
    
    # Basic model parameters as external flags.
    FLAGS = None
    
    # Constants used for dealing with the files, matches convert_to_records.
    TRAIN_FILE = 'train.tfrecords'
    VALIDATION_FILE = 'validation.tfrecords'
    
    
    def read_and_decode(filename_queue):
      reader = tf.TFRecordReader()
      _, serialized_example = reader.read(filename_queue)
      features = tf.parse_single_example(
          serialized_example,
          # Defaults are not specified since both keys are required.
          features={
              'image_raw': tf.FixedLenFeature([], tf.string),
              'label': tf.FixedLenFeature([], tf.int64),
          })
    
      # Convert from a scalar string tensor (whose single string has
      # length mnist.IMAGE_PIXELS) to a uint8 tensor with shape
      # [mnist.IMAGE_PIXELS].
      image = tf.decode_raw(features['image_raw'], tf.uint8)
      image.set_shape([mnist.IMAGE_PIXELS])
    
      # OPTIONAL: Could reshape into a 28x28 image and apply distortions
      # here.  Since we are not applying any distortions in this
      # example, and the next step expects the image to be flattened
      # into a vector, we don't bother.
    
      # Convert from [0, 255] -> [-0.5, 0.5] floats.
      image = tf.cast(image, tf.float32) * (1. / 255) - 0.5
    
      # Convert label from a scalar uint8 tensor to an int32 scalar.
      label = tf.cast(features['label'], tf.int32)
    
      return image, label
    
    
    def inputs(train, batch_size, num_epochs):
      """Reads input data num_epochs times.
      Args:
        train: Selects between the training (True) and validation (False) data.
        batch_size: Number of examples per returned batch.
        num_epochs: Number of times to read the input data, or 0/None to
           train forever.
      Returns:
        A tuple (images, labels), where:
        * images is a float tensor with shape [batch_size, mnist.IMAGE_PIXELS]
          in the range [-0.5, 0.5].
        * labels is an int32 tensor with shape [batch_size] with the true label,
          a number in the range [0, mnist.NUM_CLASSES).
        Note that an tf.train.QueueRunner is added to the graph, which
        must be run using e.g. tf.train.start_queue_runners().
      """
      if not num_epochs: num_epochs = None
      filename = os.path.join(FLAGS.train_dir,
                              TRAIN_FILE if train else VALIDATION_FILE)
    
      with tf.name_scope('input'):
        filename_queue = tf.train.string_input_producer(
            [filename], num_epochs=num_epochs)
    
        # Even when reading in multiple threads, share the filename
        # queue.
        image, label = read_and_decode(filename_queue)
    
        # Shuffle the examples and collect them into batch_size batches.
        # (Internally uses a RandomShuffleQueue.)
        # We run this in two threads to avoid being a bottleneck.
        images, sparse_labels = tf.train.shuffle_batch(
            [image, label], batch_size=batch_size, num_threads=2,
            capacity=1000 + 3 * batch_size,
            # Ensures a minimum amount of shuffling of examples.
            min_after_dequeue=1000)
    
        return images, sparse_labels
    
    
    def run_training():
      """Train MNIST for a number of steps."""
    
      # Tell TensorFlow that the model will be built into the default Graph.
      with tf.Graph().as_default():
        # Input images and labels.
        images, labels = inputs(train=True, batch_size=FLAGS.batch_size,
                                num_epochs=FLAGS.num_epochs)
    
        # Build a Graph that computes predictions from the inference model.
        logits = mnist.inference(images,
                                 FLAGS.hidden1,
                                 FLAGS.hidden2)
    
        # Add to the Graph the loss calculation.
        loss = mnist.loss(logits, labels)
    
        # Add to the Graph operations that train the model.
        train_op = mnist.training(loss, FLAGS.learning_rate)
    
        # The op for initializing the variables.
        init_op = tf.group(tf.global_variables_initializer(),
                           tf.local_variables_initializer())
    
        # Create a session for running operations in the Graph.
        sess = tf.Session()
    
        # Initialize the variables (the trained variables and the
        # epoch counter).
        sess.run(init_op)
    
        # Start input enqueue threads.
        coord = tf.train.Coordinator()
        threads = tf.train.start_queue_runners(sess=sess, coord=coord)
    
        try:
          step = 0
          while not coord.should_stop():
            start_time = time.time()
    
            # Run one step of the model.  The return values are
            # the activations from the `train_op` (which is
            # discarded) and the `loss` op.  To inspect the values
            # of your ops or variables, you may include them in
            # the list passed to sess.run() and the value tensors
            # will be returned in the tuple from the call.
            _, loss_value = sess.run([train_op, loss])
    
            duration = time.time() - start_time
    
            # Print an overview fairly often.
            if step % 100 == 0:
              print('Step %d: loss = %.2f (%.3f sec)' % (step, loss_value,
                                                         duration))
            step += 1
        except tf.errors.OutOfRangeError:
          print('Done training for %d epochs, %d steps.' % (FLAGS.num_epochs, step))
        finally:
          # When done, ask the threads to stop.
          coord.request_stop()
    
        # Wait for threads to finish.
        coord.join(threads)
        sess.close()
    
    
    def main(_):
      run_training()
    
    
    if __name__ == '__main__':
      parser = argparse.ArgumentParser()
      parser.add_argument(
          '--learning_rate',
          type=float,
          default=0.01,
          help='Initial learning rate.'
      )
      parser.add_argument(
          '--num_epochs',
          type=int,
          default=2,
          help='Number of epochs to run trainer.'
      )
      parser.add_argument(
          '--hidden1',
          type=int,
          default=128,
          help='Number of units in hidden layer 1.'
      )
      parser.add_argument(
          '--hidden2',
          type=int,
          default=32,
          help='Number of units in hidden layer 2.'
      )
      parser.add_argument(
          '--batch_size',
          type=int,
          default=100,
          help='Batch size.'
      )
      parser.add_argument(
          '--train_dir',
          type=str,
          default='/tmp/data',
          help='Directory with the training data.'
      )
      FLAGS, unparsed = parser.parse_known_args()
      tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)

TensorFlow支持从csv文件和TFRecords文件读取数据，如果从二进制的TFRecords文件读取，可以采用QueueRunner和Coordinator的方式进行多线程读取，通过设置epoch参数控制训练数据文件迭代训练的次数，通过设置batch\_size的大小来控制一次训练中从训练数据中取得的样本数量，还可以设置随机选取，有利于加快训练速度。

def read_and_decode(filename_queue):#从TFRecords中读取数据
        reader = tf.TFRecordReader()
        _, serialized_example = reader.read(filename_queue)
        features = tf.parse_single_example(serialized_example,
        features={
        "label": tf.FixedLenFeature([], tf.float32),
        "features": tf.FixedLenFeature([FEATURE_SIZE], tf.float32),
        })
        label = features["label"]
        features = features["features"]
    return label, features
    
    
    filename_queue = tf.train.string_input_producer(tf.train.match_filenames_once(trainFile), num_epochs=epoch_number)
    label, features = read_and_decode(filename_queue)
    batch_labels, batch_features = tf.train.shuffle_batch([label, features], batch_size=batch_size, num_threads=thread_number, capacity=capacity, min_after_dequeue=min_after_dequeue)

这里的trainFile可以是一个文件名的列表：  
  
  
trainFile = \['./data/train\_1.tfrecords','./data/train\_2.tfrecords'\]  
还可以是一个正则表达式：  
  
  
trainFile = './data/\*.tfrecords'  
使用Coordinator来管理队列：

coord = tf.train.Coordinator()
    threads = tf.train.start_queue_runners(coord=coord, sess=sess)
    try:
        while not coord.should_stop():
        _, loss_value, step = sess.run([train_op, loss, global_step])
        saver.save(sess, "./checkpoint/checkpoint.ckpt",global_step=step)
    except tf.errors.OutOfRangeError:
        print("Done training after reading all data")
    finally:
        coord.request_stop()

这里经常会碰到的一个问题是在没有训练之前队列就关闭了，类似“get ‘OutOfRange’, the queue will be closed”的问题，这是因为epoch设置过小，在开始训练前就把数据读完退出了，可以把epoch设置的大一些，如果设置成Nnoe，程序会无限制地一直跑下去，当然你可以在结果足够好的时候手动中断程序的运行。这里就是我的问题啦，有没有什么好的方法来设置epoch参数？

[Notes on tensorflow（七）将数据集转换为TFRecord][Notes on tensorflow_TFRecord]

[Notes on tensorflow（八）read tfrecords with slim][Notes on tensorflow_read tfrecords with slim]

[tensorflow学习笔记（四十二）：输入流水线][tensorflow]

[TensorFlow数据读取方法][TensorFlow 1]

[Tensorflow中使用tfrecord方式读取数据][Tensorflow_tfrecord]

[tensorflow中next\_batch][tensorflow_next_batch]

[Tensorflow之构建自己的图片数据集TFrecords][Tensorflow_TFrecords]

[【TensorFlow动手玩】数据导入2][TensorFlow_2]

[自己用到的TensorFlow的几种输入][TensorFlow 2]

[深度学习之图片转成pkl格式的训练集][pkl]

[学习TensorFlow，生成tensorflow输入输出的图像格式][TensorFlow_tensorflow]

[Tenforflow之构建自己的cpkt训练模型][Tenforflow_cpkt]

[tensorflow载入数据的三种方式][tensorflow 1]

https://github.com/kevin28520/My-TensorFlow-tutorials/tree/master/03%20TFRecord

[tf.train.shuffle\_batch][tf.train.shuffle_batch]

https://github.com/markdtw/tensorflow-queue-example

https://github.com/ppplinday/TFrecord-Lenet-Vgg19-tensorflow

[TensorFlow]: http://blog.csdn.net/u012759136/article/details/52232266
[TensorFlow _ _TFRecord]: http://blog.csdn.net/miaomiaoyuan/article/details/56865361
[Notes on tensorflow_TFRecord]: http://blog.csdn.net/weixin_35653315/article/details/71015845
[Notes on tensorflow_read tfrecords with slim]: http://blog.csdn.net/weixin_35653315/article/details/71023596
[tensorflow]: http://blog.csdn.net/u012436149/article/details/72353313
[TensorFlow 1]: http://honggang.io/2016/08/19/tensorflow-data-reading/
[Tensorflow_tfrecord]: http://blog.csdn.net/u010358677/article/details/70544241
[tensorflow_next_batch]: http://blog.csdn.net/appleml/article/details/57413615
[Tensorflow_TFrecords]: http://blog.csdn.net/csuzhaoqinghui/article/details/51377941
[TensorFlow_2]: http://blog.csdn.net/shenxiaolu1984/article/details/52857437
[TensorFlow 2]: http://blog.csdn.net/jyshee/article/details/52566152
[pkl]: http://blog.csdn.net/csuzhaoqinghui/article/details/54346637
[TensorFlow_tensorflow]: http://blog.csdn.net/helei001/article/details/51354404
[Tenforflow_cpkt]: http://blog.csdn.net/csuzhaoqinghui/article/details/51839169
[tensorflow 1]: http://blog.csdn.net/lujiandong1/article/details/53376802
[tf.train.shuffle_batch]: https://www.tensorflow.org/api_docs/python/tf/train/shuffle_batch