Tips: You can run step by step of all the steps below to verify your environment is ready. After that, you can directly execute the bash run.sh ${VIDEO_ID} to finish all the steps in one run.

中文文档

Step0. Crop videos to 512 * 512 and 25 FPS, ensure there is face in every frame.

ffmpeg -i data/raw/videos/${VIDEO_ID}.mp4 -vf fps=25,scale=w=512:h=512 -qmin 1 -q:v 1 data/raw/videos/${VIDEO_ID}_512.mp4
mv data/raw/videos/${VIDEO_ID}.mp4 data/raw/videos/${VIDEO_ID}_to_rm.mp4
mv data/raw/videos/${VIDEO_ID}_512.mp4 data/raw/videos/${VIDEO_ID}.mp4

step1: Extract audio features, such as mel, f0, hubuert and esperanto

export CUDA_VISIBLE_DEVICES=0
export VIDEO_ID=May
export PYTHONPATH=./
mkdir -p data/processed/videos/${VIDEO_ID}
ffmpeg -i data/raw/videos/${VIDEO_ID}.mp4 -f wav -ar 16000 data/processed/videos/${VIDEO_ID}/aud.wav
python data_gen/utils/process_audio/extract_hubert.py --video_id=${VIDEO_ID}
python data_gen/utils/process_audio/extract_mel_f0.py --video_id=${VIDEO_ID}

Step2. Extract images

TIPS: add --force_single_process if you discover that multiprocessing does not work well with Mediapipe

export PYTHONPATH=./
export VIDEO_ID=May
export CUDA_VISIBLE_DEVICES=0
mkdir -p data/processed/videos/${VIDEO_ID}/gt_imgs
ffmpeg -i data/raw/videos/${VIDEO_ID}.mp4 -vf fps=25,scale=w=512:h=512 -qmin 1 -q:v 1 -start_number 0 data/processed/videos/${VIDEO_ID}/gt_imgs/%08d.jpg
python data_gen/utils/process_video/extract_segment_imgs.py --ds_name=nerf --vid_dir=data/raw/videos/${VIDEO_ID}.mp4 # extract image, segmap, and background

Step3. Extract lm2d_mediapipe

Extract 2D landmark for Fit 3DMM later

num_workers: number of CPU workers；total_process: number of machines to be used；process_id: machine ID

export PYTHONPATH=./
export VIDEO_ID=May
python data_gen/utils/process_video/extract_lm2d.py --ds_name=nerf --vid_dir=data/raw/videos/${VIDEO_ID}.mp4

Step3. Fit 3DMM

export VIDEO_ID=May
export PYTHONPATH=./
export CUDA_VISIBLE_DEVICES=0
python data_gen/utils/process_video/fit_3dmm_landmark.py --ds_name=nerf --vid_dir=data/raw/videos/${VIDEO_ID}.mp4 --reset  --debug --id_mode=global

Step4. Binarize

export PYTHONPATH=./
export VIDEO_ID=May
python data_gen/runs/binarizer_nerf.py --video_id=${VIDEO_ID}

You can check data/binary/videos/May for generated dataset.