tensorflow serving warmup

浅浅的花香味﹌ 2021-11-17 03:44 302阅读 0赞

*  warmup是通过模型启动时加载$\{model\}/$\{version\}/assets.extra/tf\_serving\_warmup\_requests达到热启动的目的。
 *  如下为基于inception模型的warmup生成程序。

import tensorflow as tf
    from tensorflow_serving.apis import model_pb2
    from tensorflow_serving.apis import predict_pb2
    from tensorflow_serving.apis import prediction_log_pb2
    
    def main():
        data = (open('0.jpg','rb').read())
        with tf.python_io.TFRecordWriter("tf_serving_warmup_requests") as writer:
            request = predict_pb2.PredictRequest(
                model_spec=model_pb2.ModelSpec(name="inception", signature_name = 'predict_images'),
                inputs={"images": tf.make_tensor_proto(data, shape=[1])}
            )
            log = prediction_log_pb2.PredictionLog(
            predict_log=prediction_log_pb2.PredictLog(request=request))
            writer.write(log.SerializeToString())
    
    if __name__ == "__main__":
        main()

*  warmup在终端的输出

2019-07-29 12:53:34.948183: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:182] Restoring SavedModel bundle.
    2019-07-29 12:53:37.081668: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:132] Running initialization op on SavedModel bundle.
    2019-07-29 12:53:37.369278: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:285] SavedModel load for tags { serve }; Status: success. Took 4122859 microseconds.
    2019-07-29 12:53:37.651905: I external/org_tensorflow/tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally
    2019-07-29 12:53:40.158449: I tensorflow_serving/servables/tensorflow/saved_model_warmup.cc:143] Finished reading warmup data for model at /tensorflow-serving/model/1/assets.extra/tf_serving_warmup_requests. Number of warmup records read: 1.
    2019-07-29 12:53:40.158715: I tensorflow_serving/core/loader_harness.cc:86] Successfully loaded servable version {name: inception version: 1}
    2019-07-29 12:53:40.190985: I tensorflow_serving/model_servers/server.cc:313] Running gRPC ModelServer at 0.0.0.0:9000 ...

*  效果

gpu:V100
    model:v4
    tm: 2.98701691628  //没有使用warmup时的首次访问时间
    tm: 0.468713998795  //使用warmup时的首次访问时间

参考文献：

1.  [SavedModel Warmup][]
2.  [How to prepare warmup request file for tensorflow serving?][How to prepare warmup request file for tensorflow serving]

[SavedModel Warmup]: https://www.tensorflow.org/tfx/serving/saved_model_warmup
[How to prepare warmup request file for tensorflow serving]: https://stackoverflow.com/questions/51980407/how-to-prepare-warmup-request-file-for-tensorflow-serving