python动作识别库_用于动作识别的3D ResNets

怼烎@ 2022-10-25 14:27 154阅读 0赞

3D ResNets for Action Recognition

Update (2020/4/13)

We published a paper on arXiv.

We uploaded the pretrained models described in this paper including ResNet-50 pretrained on the combined dataset with Kinetics-700 and Moments in Time.

Update (2020/4/10)

We significantly updated our scripts. If you want to use older versions to reproduce our CVPR2018 paper, you should use the scripts in the CVPR2018 branch.

This update includes as follows:

Refactoring whole project

Supporting the newer PyTorch versions

Supporting distributed training

Supporting training and testing on the Moments in Time dataset.

Adding R(2+1)D models

Uploading 3D ResNet models trained on the Kinetics-700, Moments in Time, and STAIR-Actions datasets

Summary

This is the PyTorch code for the following papers:

This code includes training, fine-tuning and testing on Kinetics, Moments in Time, ActivityNet, UCF-101, and HMDB-51.

Citation

If you use this code or pre-trained models, please cite the following:

@inproceedings\{hara3dcnns,

author=\{Kensho Hara and Hirokatsu Kataoka and Yutaka Satoh\},

title=\{Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?\},

booktitle=\{Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)\},

pages=\{6546--6555\},

year=\{2018\},

Pre-trained models

Pre-trained models are available here.

All models are trained on Kinetics-700 (K), Moments in Time (M), STAIR-Actions (S), or merged datasets of them (KM, KS, MS, KMS).

If you want to finetune the models on your dataset, you should specify the following options.

r3d18\_K\_200ep.pth: --model resnet --model\_depth 18 --n\_pretrain\_classes 700

r3d18\_KM\_200ep.pth: --model resnet --model\_depth 18 --n\_pretrain\_classes 1039

r3d34\_K\_200ep.pth: --model resnet --model\_depth 34 --n\_pretrain\_classes 700

r3d34\_KM\_200ep.pth: --model resnet --model\_depth 34 --n\_pretrain\_classes 1039

r3d50\_K\_200ep.pth: --model resnet --model\_depth 50 --n\_pretrain\_classes 700

r3d50\_KM\_200ep.pth: --model resnet --model\_depth 50 --n\_pretrain\_classes 1039

r3d50\_KMS\_200ep.pth: --model resnet --model\_depth 50 --n\_pretrain\_classes 1139

r3d50\_KS\_200ep.pth: --model resnet --model\_depth 50 --n\_pretrain\_classes 800

r3d50\_M\_200ep.pth: --model resnet --model\_depth 50 --n\_pretrain\_classes 339

r3d50\_MS\_200ep.pth: --model resnet --model\_depth 50 --n\_pretrain\_classes 439

r3d50\_S\_200ep.pth: --model resnet --model\_depth 50 --n\_pretrain\_classes 100

r3d101\_K\_200ep.pth: --model resnet --model\_depth 101 --n\_pretrain\_classes 700

r3d101\_KM\_200ep.pth: --model resnet --model\_depth 101 --n\_pretrain\_classes 1039

r3d152\_K\_200ep.pth: --model resnet --model\_depth 152 --n\_pretrain\_classes 700

r3d152\_KM\_200ep.pth: --model resnet --model\_depth 152 --n\_pretrain\_classes 1039

r3d200\_K\_200ep.pth: --model resnet --model\_depth 200 --n\_pretrain\_classes 700

r3d200\_KM\_200ep.pth: --model resnet --model\_depth 200 --n\_pretrain\_classes 1039

Old pretrained models are still available here.

However, some modifications are required to use the old pretrained models in the current scripts.

Requirements

PyTorch (ver. 0.4+ required)

conda install pytorch torchvision cudatoolkit=10.1 -c soumith

FFmpeg, FFprobe

Python 3

Preparation

ActivityNet

Download videos using the official crawler.

Convert from avi to jpg files using util\_scripts/generate\_video\_jpgs.py

python -m util\_scripts.generate\_video\_jpgs mp4\_video\_dir\_path jpg\_video\_dir\_path activitynet

Add fps infomartion into the json file util\_scripts/add\_fps\_into\_activitynet\_json.py

python -m util\_scripts.add\_fps\_into\_activitynet\_json mp4\_video\_dir\_path json\_file\_path

Kinetics

Download videos using the official crawler.

Locate test set in video\_directory/test.

Convert from avi to jpg files using util\_scripts/generate\_video\_jpgs.py

python -m util\_scripts.generate\_video\_jpgs mp4\_video\_dir\_path jpg\_video\_dir\_path kinetics

Generate annotation file in json format similar to ActivityNet using util\_scripts/kinetics\_json.py

The CSV files (kinetics\_\{train, val, test\}.csv) are included in the crawler.

python -m util\_scripts.kinetics\_json csv\_dir\_path 700 jpg\_video\_dir\_path jpg dst\_json\_path

UCF-101

Download videos and train/test splits here.

Convert from avi to jpg files using util\_scripts/generate\_video\_jpgs.py

python -m util\_scripts.generate\_video\_jpgs avi\_video\_dir\_path jpg\_video\_dir\_path ucf101

Generate annotation file in json format similar to ActivityNet using util\_scripts/ucf101\_json.py

annotation\_dir\_path includes classInd.txt, trainlist0\{1, 2, 3\}.txt, testlist0\{1, 2, 3\}.txt

python -m util\_scripts.ucf101\_json annotation\_dir\_path jpg\_video\_dir\_path dst\_json\_path

HMDB-51

Download videos and train/test splits here.

Convert from avi to jpg files using util\_scripts/generate\_video\_jpgs.py

python -m util\_scripts.generate\_video\_jpgs avi\_video\_dir\_path jpg\_video\_dir\_path hmdb51

Generate annotation file in json format similar to ActivityNet using util\_scripts/hmdb51\_json.py

annotation\_dir\_path includes brush\_hair\_test\_split1.txt, ...

python -m util\_scripts.hmdb51\_json annotation\_dir\_path jpg\_video\_dir\_path dst\_json\_path

Running the code

Assume the structure of data directories is the following:

data/

kinetics\_videos/

jpg/

.../ (directories of class names)

.../ (directories of video names)

... (jpg files)

results/

save\_100.pth

kinetics.json

Confirm all options.

python main.py -h

Train ResNets-50 on the Kinetics-700 dataset (700 classes) with 4 CPU threads (for data loading).

Batch size is 128.

Save models at every 5 epochs. All GPUs is used for the training. If you want a part of GPUs, use CUDA\_VISIBLE\_DEVICES=....

python main.py --root\_path ~/data --video\_path kinetics\_videos/jpg --annotation\_path kinetics.json \\

\--result\_path results --dataset kinetics --model resnet \\

\--model\_depth 50 --n\_classes 700 --batch\_size 128 --n\_threads 4 --checkpoint 5

Continue Training from epoch 101. (~/data/results/save\_100.pth is loaded.)

python main.py --root\_path ~/data --video\_path kinetics\_videos/jpg --annotation\_path kinetics.json \\

\--result\_path results --dataset kinetics --resume\_path results/save\_100.pth \\

\--model\_depth 50 --n\_classes 700 --batch\_size 128 --n\_threads 4 --checkpoint 5

Calculate top-5 class probabilities of each video using a trained model (~/data/results/save\_200.pth.)

Note that inference\_batch\_size should be small because actual batch size is calculated by inference\_batch\_size \* (n\_video\_frames / inference\_stride).

python main.py --root\_path ~/data --video\_path kinetics\_videos/jpg --annotation\_path kinetics.json \\

\--result\_path results --dataset kinetics --resume\_path results/save\_200.pth \\

\--model\_depth 50 --n\_classes 700 --n\_threads 4 --no\_train --no\_val --inference --output\_topk 5 --inference\_batch\_size 1

Evaluate top-1 video accuracy of a recognition result (~/data/results/val.json).

python -m util\_scripts.eval\_accuracy ~/data/kinetics.json ~/data/results/val.json --subset val -k 1 --ignore

Fine-tune fc layers of a pretrained model (~/data/models/resnet-50-kinetics.pth) on UCF-101.

python main.py --root\_path ~/data --video\_path ucf101\_videos/jpg --annotation\_path ucf101\_01.json \\

\--result\_path results --dataset ucf101 --n\_classes 101 --n\_pretrain\_classes 700 \\

\--pretrain\_path models/resnet-50-kinetics.pth --ft\_begin\_module fc \\

\--model resnet --model\_depth 50 --batch\_size 128 --n\_threads 4 --checkpoint 5