From 248cbedbce784bfca82e5afbeb822cc40cac00b4 Mon Sep 17 00:00:00 2001 From: Xintao Date: Sun, 25 Jul 2021 13:00:29 +0800 Subject: [PATCH 1/3] add readme for training --- README.md | 3 +- Training.md | 97 +++++++++++++++++++++++++ experiments/pretrained_models/README.md | 1 + 3 files changed, 100 insertions(+), 1 deletion(-) create mode 100644 Training.md create mode 100644 experiments/pretrained_models/README.md diff --git a/README.md b/README.md index 6104d7b..e919106 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ Real-ESRGAN aims at developing **Practical Algorithms for General Image Restoration**.
We extend the powerful ESRGAN to a practical restoration application (namely, Real-ESRGAN), which is trained with pure synthetic data. -:triangular_flag_on_post: The training codes have been released. A detailed guide will be provided later (on July 25th). Note that the codes have a lot of refactoring. So there may be some bugs/performance drops. Welcome to report issues and I wil also retrain the models. +:triangular_flag_on_post: The training codes have been released. A detailed guide will be provided later (on July 25th). ### :book: Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data @@ -54,6 +54,7 @@ You can download **Windows executable files** from https://github.com/xinntao/Re This executable file is **portable** and includes all the binaries and models required. No CUDA or PyTorch environment is needed.
You can simply run the following command: + ```bash ./realesrgan-ncnn-vulkan.exe -i input.jpg -o output.png ``` diff --git a/Training.md b/Training.md new file mode 100644 index 0000000..9e54f02 --- /dev/null +++ b/Training.md @@ -0,0 +1,97 @@ +# :computer: How to Train Real-ESRGAN + +The training codes have been released.
+Note that the codes have a lot of refactoring. So there may be some bugs/performance drops. Welcome to report issues and I will also retrain the models. + +## Overview + +The training has been divided into two stages. These two stages have the same data synthesis process and training pipeline, except for the loss functions. Specifically, + +1. We first train Real-ESRNet with L1 loss from the pre-trained model ESRGAN. +1. We then use the trained Real-ESRNet model as an initialization of the generator, and train the Real-ESRGAN with a combination ofL1 loss, perceptual loss and GAN loss. + +## Dataset Preparation + +We use DF2K (DIV2K and Flickr2K) + OST datasets for our training. Only HR images are required.
+You can download from : + +1. DIV2K: http://data.vision.ee.ethz.ch/cvl/DIV2K/DIV2K_train_HR.zip +2. Flickr2K: https://cv.snu.ac.kr/research/EDSR/Flickr2K.tar +3. OST: https://openmmlab.oss-cn-hangzhou.aliyuncs.com/datasets/OST_dataset.zip + +For the DF2K dataset, we use a multi-scale strategy, *i.e.*, we downsample HR images to obtain several Ground-Truth images with different scales. + +We then crop DF2K images into sub-images for faster IO and processing. + +You need to prepare a txt file containing the image paths. Examples in `meta_info_DF2Kmultiscale+OST_sub.txt` (As different users may have different sub-images partition, this file is not suitable for your purpose and you need to prepare your own txt file): + +```txt +DF2K_HR_sub/000001_s001.png +DF2K_HR_sub/000001_s002.png +DF2K_HR_sub/000001_s003.png +... +``` + +## Train Real-ESRNet + +1. Download pre-trained model [ESRGAN](https://drive.google.com/file/d/1b3_bWZTjNO3iL2js1yWkJfjZykcQgvzT/view?usp=sharing) into `experiments/pretrained_models`. +1. Modify the content in the option file `options/train_realesrnet_x4plus.yml` accordingly: + ```yml + train: + name: DF2K+OST + type: RealESRGANDataset + dataroot_gt: datasets/DF2K # modify to the root path of your folder + meta_info: data/meta_info/meta_info_DF2Kmultiscale+OST_sub.txt # modify to your own generate meta info + io_backend: + type: disk + ``` +1. If you want to perform validation during training, uncomment those lines and modify accordingly: + ```yml + # Uncomment these for validation + # val: + # name: validation + # type: PairedImageDataset + # dataroot_gt: path_to_gt + # dataroot_lq: path_to_lq + # io_backend: + # type: disk + + ... + + # Uncomment these for validation + # validation settings + # val: + # val_freq: !!float 5e3 + # save_img: True + + # metrics: + # psnr: # metric name, can be arbitrary + # type: calculate_psnr + # crop_border: 4 + # test_y_channel: false + ``` +1. Before the formal training, you may run in the `--debug` mode to see whether everything is OK. We use four GPUs for training: + ```bash + CUDA_VISIBLE_DEVICES=0,1,2,3 \ + python -m torch.distributed.launch --nproc_per_node=4 --master_port=4321 train.py -opt options/train_realesrnet_x4plus.yml --launcher pytorch --debug + ``` +1. The formal training. We use four GPUs for training. We pass `--auto_resume` to resume the training if necessary automatically. + ```bash + CUDA_VISIBLE_DEVICES=0,1,2,3 \ + python -m torch.distributed.launch --nproc_per_node=4 --master_port=4321 train.py -opt options/train_realesrnet_x4plus.yml --launcher pytorch --auto_resume + ``` + +## Train Real-ESRGAN + +1. After you train Real-ESRNet, you now have the file `experiments/train_RealESRNetx4plus_1000k_B12G4_fromESRGAN/model/net_g_1000000.pth`. If you need to specify the pre-trained path of other files. Modify the `pretrain_network_g` value in the option file `train_realesrgan_x4plus.yml`. +1. Modify the option file `train_realesrgan_x4plus.yml` accordingly. Most modifications are similar to those listed above. +1. Before the formal training, you may run in the `--debug` mode to see whether everything is OK. We use four GPUs for training: + ```bash + CUDA_VISIBLE_DEVICES=0,1,2,3 \ + python -m torch.distributed.launch --nproc_per_node=4 --master_port=4321 train.py -opt options/train_realesrgan_x4plus.yml --launcher pytorch --debug + ``` +1. The formal training. We use four GPUs for training. We pass `--auto_resume` to resume the training if necessary automatically. + ```bash + CUDA_VISIBLE_DEVICES=0,1,2,3 \ + python -m torch.distributed.launch --nproc_per_node=4 --master_port=4321 train.py -opt options/train_realesrgan_x4plus.yml --launcher pytorch --auto_resume + ``` diff --git a/experiments/pretrained_models/README.md b/experiments/pretrained_models/README.md new file mode 100644 index 0000000..d0cc4af --- /dev/null +++ b/experiments/pretrained_models/README.md @@ -0,0 +1 @@ +# Put downloaded pre-trained models here From ee820df2e2da778fd0aca6a2e9e6f7d887b99c80 Mon Sep 17 00:00:00 2001 From: Xintao Date: Sun, 25 Jul 2021 13:08:09 +0800 Subject: [PATCH 2/3] update --- README.md | 2 +- Training.md | 12 ++++++------ 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index e919106..346e89b 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ Real-ESRGAN aims at developing **Practical Algorithms for General Image Restoration**.
We extend the powerful ESRGAN to a practical restoration application (namely, Real-ESRGAN), which is trained with pure synthetic data. -:triangular_flag_on_post: The training codes have been released. A detailed guide will be provided later (on July 25th). +:triangular_flag_on_post: The training codes have been released. A detailed guide can be found [here](Training.md) ### :book: Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data diff --git a/Training.md b/Training.md index 9e54f02..64229d3 100644 --- a/Training.md +++ b/Training.md @@ -8,7 +8,7 @@ Note that the codes have a lot of refactoring. So there may be some bugs/perform The training has been divided into two stages. These two stages have the same data synthesis process and training pipeline, except for the loss functions. Specifically, 1. We first train Real-ESRNet with L1 loss from the pre-trained model ESRGAN. -1. We then use the trained Real-ESRNet model as an initialization of the generator, and train the Real-ESRGAN with a combination ofL1 loss, perceptual loss and GAN loss. +1. We then use the trained Real-ESRNet model as an initialization of the generator, and train the Real-ESRGAN with a combination of L1 loss, perceptual loss and GAN loss. ## Dataset Preparation @@ -23,7 +23,7 @@ For the DF2K dataset, we use a multi-scale strategy, *i.e.*, we downsample HR im We then crop DF2K images into sub-images for faster IO and processing. -You need to prepare a txt file containing the image paths. Examples in `meta_info_DF2Kmultiscale+OST_sub.txt` (As different users may have different sub-images partition, this file is not suitable for your purpose and you need to prepare your own txt file): +You need to prepare a txt file containing the image paths. The following are some examples in `meta_info_DF2Kmultiscale+OST_sub.txt` (As different users may have different sub-images partitions, this file is not suitable for your purpose and you need to prepare your own txt file): ```txt DF2K_HR_sub/000001_s001.png @@ -41,7 +41,7 @@ DF2K_HR_sub/000001_s003.png name: DF2K+OST type: RealESRGANDataset dataroot_gt: datasets/DF2K # modify to the root path of your folder - meta_info: data/meta_info/meta_info_DF2Kmultiscale+OST_sub.txt # modify to your own generate meta info + meta_info: data/meta_info/meta_info_DF2Kmultiscale+OST_sub.txt # modify to your own generate meta info txt io_backend: type: disk ``` @@ -75,7 +75,7 @@ DF2K_HR_sub/000001_s003.png CUDA_VISIBLE_DEVICES=0,1,2,3 \ python -m torch.distributed.launch --nproc_per_node=4 --master_port=4321 train.py -opt options/train_realesrnet_x4plus.yml --launcher pytorch --debug ``` -1. The formal training. We use four GPUs for training. We pass `--auto_resume` to resume the training if necessary automatically. +1. The formal training. We use four GPUs for training. We use the `--auto_resume` argument to automatically resume the training if necessary. ```bash CUDA_VISIBLE_DEVICES=0,1,2,3 \ python -m torch.distributed.launch --nproc_per_node=4 --master_port=4321 train.py -opt options/train_realesrnet_x4plus.yml --launcher pytorch --auto_resume @@ -83,14 +83,14 @@ DF2K_HR_sub/000001_s003.png ## Train Real-ESRGAN -1. After you train Real-ESRNet, you now have the file `experiments/train_RealESRNetx4plus_1000k_B12G4_fromESRGAN/model/net_g_1000000.pth`. If you need to specify the pre-trained path of other files. Modify the `pretrain_network_g` value in the option file `train_realesrgan_x4plus.yml`. +1. After the training of Real-ESRNet, you now have the file `experiments/train_RealESRNetx4plus_1000k_B12G4_fromESRGAN/model/net_g_1000000.pth`. If you need to specify the pre-trained path to other files, modify the `pretrain_network_g` value in the option file `train_realesrgan_x4plus.yml`. 1. Modify the option file `train_realesrgan_x4plus.yml` accordingly. Most modifications are similar to those listed above. 1. Before the formal training, you may run in the `--debug` mode to see whether everything is OK. We use four GPUs for training: ```bash CUDA_VISIBLE_DEVICES=0,1,2,3 \ python -m torch.distributed.launch --nproc_per_node=4 --master_port=4321 train.py -opt options/train_realesrgan_x4plus.yml --launcher pytorch --debug ``` -1. The formal training. We use four GPUs for training. We pass `--auto_resume` to resume the training if necessary automatically. +1. The formal training. We use four GPUs for training. We use the `--auto_resume` argument to automatically resume the training if necessary. ```bash CUDA_VISIBLE_DEVICES=0,1,2,3 \ python -m torch.distributed.launch --nproc_per_node=4 --master_port=4321 train.py -opt options/train_realesrgan_x4plus.yml --launcher pytorch --auto_resume From 8d982a21730448a805121934c011f199de3aab6b Mon Sep 17 00:00:00 2001 From: Xintao Date: Sun, 25 Jul 2021 13:09:02 +0800 Subject: [PATCH 3/3] update --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 346e89b..7160202 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ Real-ESRGAN aims at developing **Practical Algorithms for General Image Restoration**.
We extend the powerful ESRGAN to a practical restoration application (namely, Real-ESRGAN), which is trained with pure synthetic data. -:triangular_flag_on_post: The training codes have been released. A detailed guide can be found [here](Training.md) +:triangular_flag_on_post: The training codes have been released. A detailed guide can be found in [Training.md](Training.md). ### :book: Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data