update setup.py, V0.2.2.5

* Improve performance

* !fixup Fix typo discovered by codespell

* fixup! Fix typo discovered by codespell

* fixup! Add codespell to lint process

.github/workflows/pylint.yml

@@ -20,11 +20,12 @@ jobs:
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip
-        pip install flake8 yapf isort
+        pip install codespell flake8 isort yapf
 
     # modify the folders accordingly
     - name: Lint
       run: |
+        codespell
         flake8 .
         isort --check-only --diff realesrgan/ scripts/ inference_realesrgan.py setup.py
         yapf -r -d realesrgan/ scripts/ inference_realesrgan.py setup.py

.pre-commit-config.yaml

@@ -24,6 +24,12 @@ repos:
     hooks:
       - id: yapf
 
+  # codespell
+  - repo: https://github.com/codespell-project/codespell
+    rev: v2.1.0
+    hooks:
+      - id: codespell
+
   # pre-commit-hooks
   - repo: https://github.com/pre-commit/pre-commit-hooks
     rev: v3.2.0

README.md

@@ -9,7 +9,9 @@
 [![Publish-pip](https://github.com/xinntao/Real-ESRGAN/actions/workflows/publish-pip.yml/badge.svg)](https://github.com/xinntao/Real-ESRGAN/blob/master/.github/workflows/publish-pip.yml)
 
 1. [Colab Demo](https://colab.research.google.com/drive/1k2Zod6kSHEvraybHl50Lys0LerhyTMCo?usp=sharing) for Real-ESRGAN <a href="https://colab.research.google.com/drive/1k2Zod6kSHEvraybHl50Lys0LerhyTMCo?usp=sharing"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="google colab logo"></a>.
-2. Portable [Windows](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.4/realesrgan-ncnn-vulkan-20210901-windows.zip) / [Linux](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.4/realesrgan-ncnn-vulkan-20210901-ubuntu.zip) / [MacOS](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.4/realesrgan-ncnn-vulkan-20210901-macos.zip) **executable files for Intel/AMD/Nvidia GPU**. You can find more information [here](#Portable-executable-files).
+2. Portable [Windows](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.4/realesrgan-ncnn-vulkan-20210901-windows.zip) / [Linux](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.4/realesrgan-ncnn-vulkan-20210901-ubuntu.zip) / [MacOS](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.4/realesrgan-ncnn-vulkan-20210901-macos.zip) **executable files for Intel/AMD/Nvidia GPU**. You can find more information [here](#Portable-executable-files). The ncnn implementation is in [Real-ESRGAN-ncnn-vulkan](https://github.com/xinntao/Real-ESRGAN-ncnn-vulkan).
+
+感谢大家的关注和使用:-) 关于动漫插画的模型，目前还有很多问题，主要有: 1. 视频处理不了; 2. 景深虚化有问题; 3. 不可调节, 效果过了; 4. 改变原来的风格。大家提供了很好的反馈。我会逐步整理这些反馈，更新在 [这个文档](feedback.md)。希望不久之后，有新模型可以使用
 
 Real-ESRGAN aims at developing **Practical Algorithms for General Image Restoration**.<br>
 We extend the powerful ESRGAN to a practical restoration application (namely, Real-ESRGAN), which is trained with pure synthetic data.
@@ -19,6 +21,7 @@ We extend the powerful ESRGAN to a practical restoration application (namely, Re
 :question: Frequently Asked Questions can be found in [FAQ.md](FAQ.md).
 
 :triangular_flag_on_post: **Updates**
+- :white_check_mark: Add the ncnn implementation [Real-ESRGAN-ncnn-vulkan](https://github.com/xinntao/Real-ESRGAN-ncnn-vulkan).
 - :white_check_mark: Add [*RealESRGAN_x4plus_anime_6B.pth*](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.4/RealESRGAN_x4plus_anime_6B.pth), which is optimized for **anime** images with much smaller model size. More details and comparisons with [waifu2x](https://github.com/nihui/waifu2x-ncnn-vulkan) are in [**anime_model.md**](docs/anime_model.md)
 - :white_check_mark: Support finetuning on your own data or paired data (*i.e.*, finetuning ESRGAN). See [here](Training.md#Finetune-Real-ESRGAN-on-your-own-dataset)
 - :white_check_mark: Integrate [GFPGAN](https://github.com/TencentARC/GFPGAN) to support **face enhancement**.
@@ -32,7 +35,7 @@ We extend the powerful ESRGAN to a practical restoration application (namely, Re
 If Real-ESRGAN is helpful in your photos/projects, please help to :star: this repo or recommend it to your friends. Thanks:blush: <br>
 Other recommended projects:<br>
 :arrow_forward: [GFPGAN](https://github.com/TencentARC/GFPGAN): A practical algorithm for real-world face restoration <br>
-:arrow_forward: [BasicSR](https://github.com/xinntao/BasicSR): An ppen-source image and video restoration toolbox<br>
+:arrow_forward: [BasicSR](https://github.com/xinntao/BasicSR): An open-source image and video restoration toolbox<br>
 :arrow_forward: [facexlib](https://github.com/xinntao/facexlib): A collection that provides useful face-relation functions.<br>
 :arrow_forward: [HandyView](https://github.com/xinntao/HandyView): A PyQt5-based image viewer that is handy for view and comparison. <br>
 
@@ -40,10 +43,9 @@ Other recommended projects:<br>
 
 ### :book: Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data
 
-> [[Paper](https://arxiv.org/abs/2107.10833)] &emsp; [Project Page] &emsp; [Demo] <br>
+> [[Paper](https://arxiv.org/abs/2107.10833)] &emsp; [Project Page] &emsp; [[YouTube Video](https://www.youtube.com/watch?v=fxHWoDSSvSc)] &emsp; [[B站讲解](https://www.bilibili.com/video/BV1H34y1m7sS/)] &emsp; [[Poster](https://xinntao.github.io/projects/RealESRGAN_src/RealESRGAN_poster.pdf)] &emsp; [[PPT slides](https://docs.google.com/presentation/d/1QtW6Iy8rm8rGLsJ0Ldti6kP-7Qyzy6XL/edit?usp=sharing&ouid=109799856763657548160&rtpof=true&sd=true)]<br>
 > [Xintao Wang](https://xinntao.github.io/), Liangbin Xie, [Chao Dong](https://scholar.google.com.hk/citations?user=OSDCB0UAAAAJ), [Ying Shan](https://scholar.google.com/citations?user=4oXBp9UAAAAJ&hl=en) <br>
-> Applied Research Center (ARC), Tencent PCG<br>
+> Tencent ARC Lab; Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences
-> Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences
 
 <p align="center">
   <img src="assets/teaser.jpg">
@@ -92,6 +94,28 @@ We have provided three models:
 
 You can use the `-n` argument for other models, for example, `./realesrgan-ncnn-vulkan.exe -i input.jpg -o output.png -n realesrnet-x4plus`
 
+### Usage of executable files
+
+1. Please refer to [Real-ESRGAN-ncnn-vulkan](https://github.com/xinntao/Real-ESRGAN-ncnn-vulkan#computer-usages) for more details.
+1. Note that it does not support all the functions (such as `outscale`) as the python script `inference_realesrgan.py`.
+
+```console
+Usage: realesrgan-ncnn-vulkan.exe -i infile -o outfile [options]...
+
+  -h                   show this help
+  -v                   verbose output
+  -i input-path        input image path (jpg/png/webp) or directory
+  -o output-path       output image path (jpg/png/webp) or directory
+  -s scale             upscale ratio (4, default=4)
+  -t tile-size         tile size (>=32/0=auto, default=0) can be 0,0,0 for multi-gpu
+  -m model-path        folder path to pre-trained models(default=models)
+  -n model-name        model name (default=realesrgan-x4plus, can be realesrgan-x4plus | realesrgan-x4plus-anime | realesrnet-x4plus)
+  -g gpu-id            gpu device to use (default=0) can be 0,1,2 for multi-gpu
+  -j load:proc:save    thread count for load/proc/save (default=1:2:2) can be 1:2,2,2:2 for multi-gpu
+  -x                   enable tta mode
+  -f format            output image format (jpg/png/webp, default=ext/png)
+```
+
 Note that it may introduce block inconsistency (and also generate slightly different results from the PyTorch implementation), because this executable file first crops the input image into several tiles, and then processes them separately, finally stitches together.
 
 This executable file is based on the wonderful [Tencent/ncnn](https://github.com/Tencent/ncnn) and [realsr-ncnn-vulkan](https://github.com/nihui/realsr-ncnn-vulkan) by [nihui](https://github.com/nihui).
@@ -161,16 +185,42 @@ python inference_realesrgan.py --model_path experiments/pretrained_models/RealES
 
 Results are in the `results` folder
 
+### Usage of python script
+
+1. You can use X4 model for **arbitrary output size** with the argument `outscale`. The program will further perform cheap resize operation after the Real-ESRGAN output.
+
+```console
+Usage: python inference_realesrgan.py --model_path experiments/pretrained_models/RealESRGAN_x4plus.pth --input infile --output outfile [options]...
+
+A common command: python inference_realesrgan.py --model_path experiments/pretrained_models/RealESRGAN_x4plus.pth --input infile --netscale 4 --outscale 3.5 --half --face_enhance
+
+  -h                   show this help
+  --input              Input image or folder. Default: inputs
+  --output             Output folder. Default: results
+  --model_path         Path to the pre-trained model. Default: experiments/pretrained_models/RealESRGAN_x4plus.pth
+  --netscale           Upsample scale factor of the network. Default: 4
+  --outscale           The final upsampling scale of the image. Default: 4
+  --suffix             Suffix of the restored image. Default: out
+  --tile               Tile size, 0 for no tile during testing. Default: 0
+  --face_enhance       Whether to use GFPGAN to enhance face. Default: False
+  --half               Whether to use half precision during inference. Default: False
+  --ext                Image extension. Options: auto | jpg | png, auto means using the same extension as inputs. Default: auto
+```
+
 ## :european_castle: Model Zoo
 
-- [RealESRGAN_x4plus](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.1.0/RealESRGAN_x4plus.pth)
+- [RealESRGAN_x4plus](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.1.0/RealESRGAN_x4plus.pth): X4 model for general images
+- [RealESRGAN_x4plus_anime_6B](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.4/RealESRGAN_x4plus_anime_6B.pth): Optimized for anime images; 6 RRDB blocks (slightly smaller network)
+- [RealESRGAN_x2plus](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.1/RealESRGAN_x2plus.pth): X2 model for general images
+- [RealESRNet_x4plus](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.1.1/RealESRNet_x4plus.pth): X4 model with MSE loss (over-smooth effects)
+
+- [official ESRGAN_x4](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.1.1/ESRGAN_SRx4_DF2KOST_official-ff704c30.pth): official ESRGAN model (X4)
+
+The following models are **discriminators**, which are usually used for fine-tuning.
+
 - [RealESRGAN_x4plus_netD](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.3/RealESRGAN_x4plus_netD.pth)
-- [RealESRGAN_x4plus_anime_6B](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.4/RealESRGAN_x4plus_anime_6B.pth)
-- [RealESRGAN_x4plus_anime_6B_netD](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.4/RealESRGAN_x4plus_anime_6B_netD.pth)
-- [RealESRNet_x4plus](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.1.1/RealESRNet_x4plus.pth)
-- [RealESRGAN_x2plus](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.1/RealESRGAN_x2plus.pth)
 - [RealESRGAN_x2plus_netD](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.3/RealESRGAN_x2plus_netD.pth)
-- [official ESRGAN_x4](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.1.1/ESRGAN_SRx4_DF2KOST_official-ff704c30.pth)
+- [RealESRGAN_x4plus_anime_6B_netD](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.4/RealESRGAN_x4plus_anime_6B_netD.pth)
 
 ## :computer: Training and Finetuning on your own dataset
 
@@ -178,11 +228,11 @@ A detailed guide can be found in [Training.md](Training.md).
 
 ## BibTeX
 
-    @Article{wang2021realesrgan,
+    @InProceedings{wang2021realesrgan,
-        title={Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data},
+        author    = {Xintao Wang and Liangbin Xie and Chao Dong and Ying Shan},
-        author={Xintao Wang and Liangbin Xie and Chao Dong and Ying Shan},
+        title     = {Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data},
-        journal={arXiv:2107.10833},
+        booktitle = {International Conference on Computer Vision Workshops (ICCVW)},
-        year={2021}
+        date      = {2021}
     }
 
 ## :e-mail: Contact

Training.md

@@ -7,7 +7,7 @@
   - [Train Real-ESRGAN](#Train-Real-ESRGAN)
 - [Finetune Real-ESRGAN on your own dataset](#Finetune-Real-ESRGAN-on-your-own-dataset)
   - [Generate degraded images on the fly](#Generate-degraded-images-on-the-fly)
-  - [Use paired training data](#Use-paired-training-data)
+  - [Use paired training data](#use-your-own-paired-data)
 
 ## Train Real-ESRGAN
 
@@ -32,7 +32,7 @@ Here are steps for data preparation.
 #### Step 1: [Optional] Generate multi-scale images
 
 For the DF2K dataset, we use a multi-scale strategy, *i.e.*, we downsample HR images to obtain several Ground-Truth images with different scales. <br>
-You can use the [scripts/generate_multiscale_DF2K.py](scripts/generate_multiscale_DF2K.py) script to geneate multi-scale images. <br>
+You can use the [scripts/generate_multiscale_DF2K.py](scripts/generate_multiscale_DF2K.py) script to generate multi-scale images. <br>
 Note that this step can be omitted if you just want to have a fast try.
 
 ```bash
@@ -114,12 +114,22 @@ You can merge several folders into one meta_info txt. Here is the example:
     CUDA_VISIBLE_DEVICES=0,1,2,3 \
     python -m torch.distributed.launch --nproc_per_node=4 --master_port=4321 realesrgan/train.py -opt options/train_realesrnet_x4plus.yml --launcher pytorch --debug
     ```
+
+    Train with **a single GPU** in the *debug* mode:
+    ```bash
+    python realesrgan/train.py -opt options/train_realesrnet_x4plus.yml --debug
+    ```
 1. The formal training. We use four GPUs for training. We use the `--auto_resume` argument to automatically resume the training if necessary.
     ```bash
     CUDA_VISIBLE_DEVICES=0,1,2,3 \
     python -m torch.distributed.launch --nproc_per_node=4 --master_port=4321 realesrgan/train.py -opt options/train_realesrnet_x4plus.yml --launcher pytorch --auto_resume
     ```
 
+    Train with **a single GPU**:
+    ```bash
+    python realesrgan/train.py -opt options/train_realesrnet_x4plus.yml --auto_resume
+    ```
+
 ### Train Real-ESRGAN
 
 1. After the training of Real-ESRNet, you now have the file `experiments/train_RealESRNetx4plus_1000k_B12G4_fromESRGAN/model/net_g_1000000.pth`. If you need to specify the pre-trained path to other files, modify the `pretrain_network_g` value in the option file `train_realesrgan_x4plus.yml`.
@@ -129,12 +139,22 @@ You can merge several folders into one meta_info txt. Here is the example:
     CUDA_VISIBLE_DEVICES=0,1,2,3 \
     python -m torch.distributed.launch --nproc_per_node=4 --master_port=4321 realesrgan/train.py -opt options/train_realesrgan_x4plus.yml --launcher pytorch --debug
     ```
+
+    Train with **a single GPU** in the *debug* mode:
+    ```bash
+    python realesrgan/train.py -opt options/train_realesrgan_x4plus.yml --debug
+    ```
 1. The formal training. We use four GPUs for training. We use the `--auto_resume` argument to automatically resume the training if necessary.
     ```bash
     CUDA_VISIBLE_DEVICES=0,1,2,3 \
     python -m torch.distributed.launch --nproc_per_node=4 --master_port=4321 realesrgan/train.py -opt options/train_realesrgan_x4plus.yml --launcher pytorch --auto_resume
     ```
 
+    Train with **a single GPU**:
+    ```bash
+    python realesrgan/train.py -opt options/train_realesrgan_x4plus.yml --auto_resume
+    ```
+
 ## Finetune Real-ESRGAN on your own dataset
 
 You can finetune Real-ESRGAN on your own dataset. Typically, the fine-tuning process can be divided into two cases:
@@ -185,6 +205,11 @@ CUDA_VISIBLE_DEVICES=0,1,2,3 \
 python -m torch.distributed.launch --nproc_per_node=4 --master_port=4321 realesrgan/train.py -opt options/finetune_realesrgan_x4plus.yml --launcher pytorch --auto_resume
 ```
 
+Finetune with **a single GPU**:
+```bash
+python realesrgan/train.py -opt options/finetune_realesrgan_x4plus.yml --auto_resume
+```
+
 ### Use your own paired data
 
 You can also finetune RealESRGAN with your own paired data. It is more similar to fine-tuning ESRGAN.
@@ -237,3 +262,8 @@ We use four GPUs for training. We use the `--auto_resume` argument to automatica
 CUDA_VISIBLE_DEVICES=0,1,2,3 \
 python -m torch.distributed.launch --nproc_per_node=4 --master_port=4321 realesrgan/train.py -opt options/finetune_realesrgan_x4plus_pairdata.yml --launcher pytorch --auto_resume
 ```
+
+Finetune with **a single GPU**:
+```bash
+python realesrgan/train.py -opt options/finetune_realesrgan_x4plus_pairdata.yml --auto_resume
+```

VERSION

@@ -1 +1 @@
-0.2.2.4
+0.2.2.5

docs/anime_model.md

@@ -6,7 +6,7 @@
   - [PyTorch Inference](#PyTorch-Inference)
   - [ncnn Executable File](#ncnn-Executable-File)
 - [Comparisons with waifu2x](#Comparisons-with-waifu2x)
-- [Comparions with Sliding Bars](#Comparions-with-Sliding-Bars)
+- [Comparisons with Sliding Bars](#Comparions-with-Sliding-Bars)
 
 <p align="center">
   <img src="https://raw.githubusercontent.com/xinntao/public-figures/master/Real-ESRGAN/cmp_realesrgan_anime_1.png">

feedback.md

@@ -0,0 +1,9 @@
+# Feedback 反馈
+
+## 动漫插画模型
+
+1. 视频处理不了: 目前的模型，不是针对视频的，所以视频效果很很不好。我们在探究针对视频的模型了
+1. 景深虚化有问题: 现在的模型把一些景深 和 特意的虚化 都复原了，感觉不好。这个后面我们会考虑把这个信息结合进入。一个简单的做法是识别景深和虚化，然后作为条件告诉神经网络，哪些地方复原强一些，哪些地方复原要弱一些
+1. 不可以调节: 像 Waifu2X 可以调节。可以根据自己的喜好，做调整，但是 Real-ESRGAN-anime 并不可以。导致有些恢复效果过了
+1. 把原来的风格改变了: 不同的动漫插画都有自己的风格，现在的 Real-ESRGAN-anime 倾向于恢复成一种风格（这是受到训练数据集影响的）。风格是动漫很重要的一个要素，所以要尽可能保持
+1. 模型太大: 目前的模型处理太慢，能够更快。这个我们有相关的工作在探究，希望能够尽快有结果，并应用到 Real-ESRGAN 这一系列的模型上

inference_realesrgan.py

@@ -81,10 +81,10 @@ def main():
         h, w = img.shape[0:2]
         if max(h, w) > 1000 and args.netscale == 4:
             import warnings
-            warnings.warn('The input image is large, try X2 model for better performace.')
+            warnings.warn('The input image is large, try X2 model for better performance.')
         if max(h, w) < 500 and args.netscale == 2:
             import warnings
-            warnings.warn('The input image is small, try X4 model for better performace.')
+            warnings.warn('The input image is small, try X4 model for better performance.')
 
         try:
             if args.face_enhance:

options/train_realesrgan_x2plus.yml

@@ -2,7 +2,7 @@
 name: train_RealESRGANx2plus_400k_B12G4
 model_type: RealESRGANModel
 scale: 2
-num_gpu: 4
+num_gpu: auto  # auto: can infer from your visible devices automatically. official: 4 GPUs
 manual_seed: 0
 
 # ----------------- options for synthesizing training data in RealESRGANModel ----------------- #

options/train_realesrgan_x4plus.yml

@@ -2,7 +2,7 @@
 name: train_RealESRGANx4plus_400k_B12G4
 model_type: RealESRGANModel
 scale: 4
-num_gpu: 4
+num_gpu: auto  # auto: can infer from your visible devices automatically. official: 4 GPUs
 manual_seed: 0
 
 # ----------------- options for synthesizing training data in RealESRGANModel ----------------- #

options/train_realesrnet_x2plus.yml

@@ -2,7 +2,7 @@
 name: train_RealESRNetx2plus_1000k_B12G4
 model_type: RealESRNetModel
 scale: 2
-num_gpu: 4
+num_gpu: auto  # auto: can infer from your visible devices automatically. official: 4 GPUs
 manual_seed: 0
 
 # ----------------- options for synthesizing training data in RealESRNetModel ----------------- #

options/train_realesrnet_x4plus.yml

@@ -2,7 +2,7 @@
 name: train_RealESRNetx4plus_1000k_B12G4
 model_type: RealESRNetModel
 scale: 4
-num_gpu: 4
+num_gpu: auto  # auto: can infer from your visible devices automatically. official: 4 GPUs
 manual_seed: 0
 
 # ----------------- options for synthesizing training data in RealESRNetModel ----------------- #

scripts/extract_subimages.py

@@ -106,13 +106,7 @@ def worker(path, opt):
 
     img = cv2.imread(path, cv2.IMREAD_UNCHANGED)
 
-    if img.ndim == 2:
+    h, w = img.shape[0:2]
-        h, w = img.shape
-    elif img.ndim == 3:
-        h, w, c = img.shape
-    else:
-        raise ValueError(f'Image ndim should be 2 or 3, but got {img.ndim}')
-
     h_space = np.arange(0, h - crop_size + 1, step)
     if h - (h_space[-1] + crop_size) > thresh_size:
         h_space = np.append(h_space, h - crop_size)

setup.cfg

@@ -20,3 +20,8 @@ known_first_party = realesrgan
 known_third_party = PIL,basicsr,cv2,numpy,torch,torchvision,tqdm
 no_lines_before = STDLIB,LOCALFOLDER
 default_section = THIRDPARTY
+
+[codespell]
+skip = .git,./docs/build
+count =
+quiet-level = 3

setup.py

@@ -43,12 +43,6 @@ def get_git_hash():
 def get_hash():
     if os.path.exists('.git'):
         sha = get_git_hash()[:7]
-    elif os.path.exists(version_file):
-        try:
-            from realesrgan.version import __version__
-            sha = __version__.split('+')[-1]
-        except ImportError:
-            raise ImportError('Unable to get git version')
     else:
         sha = 'unknown'