Compare commits
28 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
fa4c8a03ae | ||
|
|
37a7c5726d | ||
|
|
382d5be582 | ||
|
|
61e81d3108 | ||
|
|
8f5744bc51 | ||
|
|
d9c2e77853 | ||
|
|
0ac8d66d39 | ||
|
|
89aa45c72d | ||
|
|
f18f613acc | ||
|
|
870a997099 | ||
|
|
576aaddfaf | ||
|
|
b827be13a1 | ||
|
|
e5e79fbde3 | ||
|
|
e5763af574 | ||
|
|
e2db576020 | ||
|
|
6b15fc6936 | ||
|
|
bc77ca5666 | ||
|
|
23d180fd8d | ||
|
|
38c913f1af | ||
|
|
8cb9bd403e | ||
|
|
8041099021 | ||
|
|
40d964c08d | ||
|
|
cdc14b74a5 | ||
|
|
827fae3bdc | ||
|
|
c8346f8823 | ||
|
|
abe36dfd04 | ||
|
|
89d897650c | ||
|
|
5adeaa2588 |
41
.github/workflows/release.yml
vendored
Normal file
41
.github/workflows/release.yml
vendored
Normal file
@@ -0,0 +1,41 @@
|
||||
name: release
|
||||
on:
|
||||
push:
|
||||
tags:
|
||||
- '*'
|
||||
|
||||
jobs:
|
||||
build:
|
||||
permissions: write-all
|
||||
name: Create Release
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v2
|
||||
- name: Create Release
|
||||
id: create_release
|
||||
uses: actions/create-release@v1
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
with:
|
||||
tag_name: ${{ github.ref }}
|
||||
release_name: Real-ESRGAN ${{ github.ref }} Release Note
|
||||
body: |
|
||||
🚀 See you again 😸
|
||||
🚀Have a nice day 😸 and happy everyday 😃
|
||||
🚀 Long time no see ☄️
|
||||
|
||||
✨ **Highlights**
|
||||
✅ [Features] Support ...
|
||||
|
||||
🐛 **Bug Fixes**
|
||||
|
||||
🌴 **Improvements**
|
||||
|
||||
📢📢📢
|
||||
|
||||
<p align="center">
|
||||
<img src="https://raw.githubusercontent.com/xinntao/Real-ESRGAN/master/assets/realesrgan_logo.png" height=150>
|
||||
</p>
|
||||
draft: true
|
||||
prerelease: false
|
||||
2
.gitignore
vendored
2
.gitignore
vendored
@@ -5,7 +5,7 @@ results/*
|
||||
tb_logger/*
|
||||
wandb/*
|
||||
tmp/*
|
||||
realesrgan/weights/*
|
||||
weights/*
|
||||
|
||||
version.py
|
||||
|
||||
|
||||
7
FAQ.md
7
FAQ.md
@@ -1,7 +0,0 @@
|
||||
# FAQ
|
||||
|
||||
1. **How to select models?**<br>
|
||||
A: Please refer to [docs/model_zoo.md](docs/model_zoo.md)
|
||||
|
||||
1. **Can `face_enhance` be used for anime images/animation videos?**<br>
|
||||
A: No, it can only be used for real faces. It is recommended not to use this option for anime images/animation videos to save GPU memory.
|
||||
@@ -5,4 +5,4 @@ include inference_realesrgan.py
|
||||
include VERSION
|
||||
include LICENSE
|
||||
include requirements.txt
|
||||
include realesrgan/weights/README.md
|
||||
include weights/README.md
|
||||
|
||||
306
README.md
306
README.md
@@ -4,6 +4,10 @@
|
||||
|
||||
## <div align="center"><b><a href="README.md">English</a> | <a href="README_CN.md">简体中文</a></b></div>
|
||||
|
||||
<div align="center">
|
||||
|
||||
👀[**Demos**](#-demos-videos) **|** 🚩[**Updates**](#-updates) **|** ⚡[**Usage**](#-quick-inference) **|** 🏰[**Model Zoo**](docs/model_zoo.md) **|** 🔧[Install](#-dependencies-and-installation) **|** 💻[Train](docs/Training.md) **|** ❓[FAQ](docs/FAQ.md) **|** 🎨[Contribution](docs/CONTRIBUTING.md)
|
||||
|
||||
[](https://github.com/xinntao/Real-ESRGAN/releases)
|
||||
[](https://pypi.org/project/realesrgan/)
|
||||
[](https://github.com/xinntao/Real-ESRGAN/issues)
|
||||
@@ -12,83 +16,39 @@
|
||||
[](https://github.com/xinntao/Real-ESRGAN/blob/master/.github/workflows/pylint.yml)
|
||||
[](https://github.com/xinntao/Real-ESRGAN/blob/master/.github/workflows/publish-pip.yml)
|
||||
|
||||
:fire: Update the **RealESRGAN AnimeVideo-v3** model **更新动漫视频的小模型**. Please see [anime video models](docs/anime_video_model.md) and [comparisons](docs/anime_comparisons.md) for more details.
|
||||
</div>
|
||||
|
||||
1. [Colab Demo](https://colab.research.google.com/drive/1k2Zod6kSHEvraybHl50Lys0LerhyTMCo?usp=sharing) for Real-ESRGAN <a href="https://colab.research.google.com/drive/1k2Zod6kSHEvraybHl50Lys0LerhyTMCo?usp=sharing"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="google colab logo"></a>.
|
||||
2. [Colab Demo](https://colab.research.google.com/drive/1yNl9ORUxxlL4N0keJa2SEPB61imPQd1B?usp=sharing) for Real-ESRGAN (**anime videos**) <a href="https://colab.research.google.com/drive/1yNl9ORUxxlL4N0keJa2SEPB61imPQd1B?usp=sharing"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="google colab logo"></a>.
|
||||
3. Portable [Windows](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/realesrgan-ncnn-vulkan-20220424-windows.zip) / [Linux](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/realesrgan-ncnn-vulkan-20220424-ubuntu.zip) / [MacOS](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/realesrgan-ncnn-vulkan-20220424-macos.zip) **executable files for Intel/AMD/Nvidia GPU**. You can find more information [here](#Portable-executable-files). The ncnn implementation is in [Real-ESRGAN-ncnn-vulkan](https://github.com/xinntao/Real-ESRGAN-ncnn-vulkan).
|
||||
🔥 **AnimeVideo-v3 model (动漫视频小模型)**. Please see [[*anime video models*](docs/anime_video_model.md)] and [[*comparisons*](docs/anime_comparisons.md)]<br>
|
||||
🔥 **RealESRGAN_x4plus_anime_6B** for anime images **(动漫插图模型)**. Please see [[*anime_model*](docs/anime_model.md)]
|
||||
|
||||
<!-- 1. You can try in our website: [ARC Demo](https://arc.tencent.com/en/ai-demos/imgRestore) (now only support RealESRGAN_x4plus_anime_6B) -->
|
||||
1. :boom: **Update** online Replicate demo: [](https://replicate.com/xinntao/realesrgan)
|
||||
1. Online Colab demo for Real-ESRGAN: [](https://colab.research.google.com/drive/1k2Zod6kSHEvraybHl50Lys0LerhyTMCo?usp=sharing) **|** Online Colab demo for for Real-ESRGAN (**anime videos**): [](https://colab.research.google.com/drive/1yNl9ORUxxlL4N0keJa2SEPB61imPQd1B?usp=sharing)
|
||||
1. Portable [Windows](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/realesrgan-ncnn-vulkan-20220424-windows.zip) / [Linux](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/realesrgan-ncnn-vulkan-20220424-ubuntu.zip) / [MacOS](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/realesrgan-ncnn-vulkan-20220424-macos.zip) **executable files for Intel/AMD/Nvidia GPU**. You can find more information [here](#portable-executable-files-ncnn). The ncnn implementation is in [Real-ESRGAN-ncnn-vulkan](https://github.com/xinntao/Real-ESRGAN-ncnn-vulkan)
|
||||
<!-- 1. You can watch enhanced animations in [Tencent Video](https://v.qq.com/s/topic/v_child/render/fC4iyCAM.html). 欢迎观看[腾讯视频动漫修复](https://v.qq.com/s/topic/v_child/render/fC4iyCAM.html) -->
|
||||
|
||||
Real-ESRGAN aims at developing **Practical Algorithms for General Image/Video Restoration**.<br>
|
||||
We extend the powerful ESRGAN to a practical restoration application (namely, Real-ESRGAN), which is trained with pure synthetic data.
|
||||
|
||||
:art: Real-ESRGAN needs your contributions. Any contributions are welcome, such as new features/models/typo fixes/suggestions/maintenance, *etc*. See [CONTRIBUTING.md](CONTRIBUTING.md). All contributors are list [here](README.md#hugs-acknowledgement).
|
||||
|
||||
:question: Frequently Asked Questions can be found in [FAQ.md](FAQ.md).
|
||||
|
||||
:milky_way: Thanks for your valuable feedbacks/suggestions. All the feedbacks are updated in [feedback.md](feedback.md).
|
||||
🌌 Thanks for your valuable feedbacks/suggestions. All the feedbacks are updated in [feedback.md](docs/feedback.md).
|
||||
|
||||
---
|
||||
|
||||
If Real-ESRGAN is helpful in your photos/projects, please help to :star: this repo or recommend it to your friends. Thanks:blush: <br>
|
||||
If Real-ESRGAN is helpful, please help to ⭐ this repo or recommend it to your friends 😊 <br>
|
||||
Other recommended projects:<br>
|
||||
:arrow_forward: [GFPGAN](https://github.com/TencentARC/GFPGAN): A practical algorithm for real-world face restoration <br>
|
||||
:arrow_forward: [BasicSR](https://github.com/xinntao/BasicSR): An open-source image and video restoration toolbox<br>
|
||||
:arrow_forward: [facexlib](https://github.com/xinntao/facexlib): A collection that provides useful face-relation functions.<br>
|
||||
:arrow_forward: [HandyView](https://github.com/xinntao/HandyView): A PyQt5-based image viewer that is handy for view and comparison. <br>
|
||||
▶️ [GFPGAN](https://github.com/TencentARC/GFPGAN): A practical algorithm for real-world face restoration <br>
|
||||
▶️ [BasicSR](https://github.com/xinntao/BasicSR): An open-source image and video restoration toolbox<br>
|
||||
▶️ [facexlib](https://github.com/xinntao/facexlib): A collection that provides useful face-relation functions.<br>
|
||||
▶️ [HandyView](https://github.com/xinntao/HandyView): A PyQt5-based image viewer that is handy for view and comparison <br>
|
||||
▶️ [HandyFigure](https://github.com/xinntao/HandyFigure): Open source of paper figures <br>
|
||||
|
||||
---
|
||||
|
||||
<!---------------------------------- Updates --------------------------->
|
||||
<details>
|
||||
<summary>🚩<b>Updates</b></summary>
|
||||
### 📖 Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data
|
||||
|
||||
- ✅ Update the **RealESRGAN AnimeVideo-v3** model. Please see [anime video models](docs/anime_video_model.md) and [comparisons](docs/anime_comparisons.md) for more details.
|
||||
- ✅ Add small models for anime videos. More details are in [anime video models](docs/anime_video_model.md).
|
||||
- ✅ Add the ncnn implementation [Real-ESRGAN-ncnn-vulkan](https://github.com/xinntao/Real-ESRGAN-ncnn-vulkan).
|
||||
- ✅ Add [*RealESRGAN_x4plus_anime_6B.pth*](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.4/RealESRGAN_x4plus_anime_6B.pth), which is optimized for **anime** images with much smaller model size. More details and comparisons with [waifu2x](https://github.com/nihui/waifu2x-ncnn-vulkan) are in [**anime_model.md**](docs/anime_model.md)
|
||||
- ✅ Support finetuning on your own data or paired data (*i.e.*, finetuning ESRGAN). See [here](Training.md#Finetune-Real-ESRGAN-on-your-own-dataset)
|
||||
- ✅ Integrate [GFPGAN](https://github.com/TencentARC/GFPGAN) to support **face enhancement**.
|
||||
- ✅ Integrated to [Huggingface Spaces](https://huggingface.co/spaces) with [Gradio](https://github.com/gradio-app/gradio). See [Gradio Web Demo](https://huggingface.co/spaces/akhaliq/Real-ESRGAN). Thanks [@AK391](https://github.com/AK391)
|
||||
- ✅ Support arbitrary scale with `--outscale` (It actually further resizes outputs with `LANCZOS4`). Add *RealESRGAN_x2plus.pth* model.
|
||||
- ✅ [The inference code](inference_realesrgan.py) supports: 1) **tile** options; 2) images with **alpha channel**; 3) **gray** images; 4) **16-bit** images.
|
||||
- ✅ The training codes have been released. A detailed guide can be found in [Training.md](Training.md).
|
||||
|
||||
</details>
|
||||
|
||||
<!---------------------------------- Projects that use Real-ESRGAN --------------------------->
|
||||
<details>
|
||||
<summary>🧩<b>Projects that use Real-ESRGAN</b></summary>
|
||||
|
||||
👋 If you develop/use Real-ESRGAN in your projects, welcome to let me know.
|
||||
|
||||
- NCNN-Android: [RealSR-NCNN-Android](https://github.com/tumuyan/RealSR-NCNN-Android) by [tumuyan](https://github.com/tumuyan)
|
||||
- VapourSynth: [vs-realesrgan](https://github.com/HolyWu/vs-realesrgan) by [HolyWu](https://github.com/HolyWu)
|
||||
- NCNN: [Real-ESRGAN-ncnn-vulkan](https://github.com/xinntao/Real-ESRGAN-ncnn-vulkan)
|
||||
|
||||
**GUI**
|
||||
|
||||
- [Waifu2x-Extension-GUI](https://github.com/AaronFeng753/Waifu2x-Extension-GUI) by [AaronFeng753](https://github.com/AaronFeng753)
|
||||
- [Squirrel-RIFE](https://github.com/Justin62628/Squirrel-RIFE) by [Justin62628](https://github.com/Justin62628)
|
||||
- [Real-GUI](https://github.com/scifx/Real-GUI) by [scifx](https://github.com/scifx)
|
||||
- [Real-ESRGAN_GUI](https://github.com/net2cn/Real-ESRGAN_GUI) by [net2cn](https://github.com/net2cn)
|
||||
- [Real-ESRGAN-EGUI](https://github.com/WGzeyu/Real-ESRGAN-EGUI) by [WGzeyu](https://github.com/WGzeyu)
|
||||
- [anime_upscaler](https://github.com/shangar21/anime_upscaler) by [shangar21](https://github.com/shangar21)
|
||||
|
||||
</details>
|
||||
|
||||
<!---------------------------------- Demo videos --------------------------->
|
||||
<details open>
|
||||
<summary>👀<b>Demo videos</b></summary>
|
||||
|
||||
- [大闹天宫片段](https://www.bilibili.com/video/BV1ja41117zb)
|
||||
|
||||
</details>
|
||||
|
||||
### :book: Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data
|
||||
|
||||
> [[Paper](https://arxiv.org/abs/2107.10833)]   [Project Page]   [[YouTube Video](https://www.youtube.com/watch?v=fxHWoDSSvSc)]   [[B站讲解](https://www.bilibili.com/video/BV1H34y1m7sS/)]   [[Poster](https://xinntao.github.io/projects/RealESRGAN_src/RealESRGAN_poster.pdf)]   [[PPT slides](https://docs.google.com/presentation/d/1QtW6Iy8rm8rGLsJ0Ldti6kP-7Qyzy6XL/edit?usp=sharing&ouid=109799856763657548160&rtpof=true&sd=true)]<br>
|
||||
> [[Paper](https://arxiv.org/abs/2107.10833)]   [[YouTube Video](https://www.youtube.com/watch?v=fxHWoDSSvSc)]   [[B站讲解](https://www.bilibili.com/video/BV1H34y1m7sS/)]   [[Poster](https://xinntao.github.io/projects/RealESRGAN_src/RealESRGAN_poster.pdf)]   [[PPT slides](https://docs.google.com/presentation/d/1QtW6Iy8rm8rGLsJ0Ldti6kP-7Qyzy6XL/edit?usp=sharing&ouid=109799856763657548160&rtpof=true&sd=true)]<br>
|
||||
> [Xintao Wang](https://xinntao.github.io/), Liangbin Xie, [Chao Dong](https://scholar.google.com.hk/citations?user=OSDCB0UAAAAJ), [Ying Shan](https://scholar.google.com/citations?user=4oXBp9UAAAAJ&hl=en) <br>
|
||||
> Tencent ARC Lab; Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences
|
||||
> [Tencent ARC Lab](https://arc.tencent.com/en/ai-demos/imgRestore); Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences
|
||||
|
||||
<p align="center">
|
||||
<img src="assets/teaser.jpg">
|
||||
@@ -96,75 +56,35 @@ Other recommended projects:<br>
|
||||
|
||||
---
|
||||
|
||||
We have provided a pretrained model (*RealESRGAN_x4plus.pth*) with upsampling X4.<br>
|
||||
**Note that RealESRGAN may still fail in some cases as the real-world degradations are really too complex.**<br>
|
||||
Moreover, it **may not** perform well on **human faces, text**, *etc*, which will be optimized later.
|
||||
<br>
|
||||
<!---------------------------------- Updates --------------------------->
|
||||
## 🚩 Updates
|
||||
|
||||
Real-ESRGAN will be a long-term supported project (in my current plan :smiley:). It will be continuously updated
|
||||
in my spare time.
|
||||
|
||||
Here is a TODO list in the near future:
|
||||
|
||||
- [ ] optimize for human faces
|
||||
- [ ] optimize for texts
|
||||
- [x] optimize for anime images
|
||||
- [ ] support more scales
|
||||
- [ ] support controllable restoration strength
|
||||
|
||||
If you have any good ideas or demands, please open an issue/discussion to let me know. <br>
|
||||
If you have some images that Real-ESRGAN could not well restored, please also open an issue/discussion. I will record it (but I cannot guarantee to resolve it:stuck_out_tongue:). If necessary, I will open a page to specially record these real-world cases that need to be solved, but the current technology is difficult to handle well.
|
||||
- ✅ Add the **realesr-general-x4v3** model - a tiny small model for general scenes. It also supports the **--dn** option to balance the noise (avoiding over-smooth results). **--dn** is short for denoising strength.
|
||||
- ✅ Update the **RealESRGAN AnimeVideo-v3** model. Please see [anime video models](docs/anime_video_model.md) and [comparisons](docs/anime_comparisons.md) for more details.
|
||||
- ✅ Add small models for anime videos. More details are in [anime video models](docs/anime_video_model.md).
|
||||
- ✅ Add the ncnn implementation [Real-ESRGAN-ncnn-vulkan](https://github.com/xinntao/Real-ESRGAN-ncnn-vulkan).
|
||||
- ✅ Add [*RealESRGAN_x4plus_anime_6B.pth*](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.4/RealESRGAN_x4plus_anime_6B.pth), which is optimized for **anime** images with much smaller model size. More details and comparisons with [waifu2x](https://github.com/nihui/waifu2x-ncnn-vulkan) are in [**anime_model.md**](docs/anime_model.md)
|
||||
- ✅ Support finetuning on your own data or paired data (*i.e.*, finetuning ESRGAN). See [here](docs/Training.md#Finetune-Real-ESRGAN-on-your-own-dataset)
|
||||
- ✅ Integrate [GFPGAN](https://github.com/TencentARC/GFPGAN) to support **face enhancement**.
|
||||
- ✅ Integrated to [Huggingface Spaces](https://huggingface.co/spaces) with [Gradio](https://github.com/gradio-app/gradio). See [Gradio Web Demo](https://huggingface.co/spaces/akhaliq/Real-ESRGAN). Thanks [@AK391](https://github.com/AK391)
|
||||
- ✅ Support arbitrary scale with `--outscale` (It actually further resizes outputs with `LANCZOS4`). Add *RealESRGAN_x2plus.pth* model.
|
||||
- ✅ [The inference code](inference_realesrgan.py) supports: 1) **tile** options; 2) images with **alpha channel**; 3) **gray** images; 4) **16-bit** images.
|
||||
- ✅ The training codes have been released. A detailed guide can be found in [Training.md](docs/Training.md).
|
||||
|
||||
---
|
||||
|
||||
### Portable executable files
|
||||
<!---------------------------------- Demo videos --------------------------->
|
||||
## 👀 Demos Videos
|
||||
|
||||
You can download [Windows](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/realesrgan-ncnn-vulkan-20220424-windows.zip) / [Linux](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/realesrgan-ncnn-vulkan-20220424-ubuntu.zip) / [MacOS](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/realesrgan-ncnn-vulkan-20220424-macos.zip) **executable files for Intel/AMD/Nvidia GPU**.
|
||||
#### Bilibili
|
||||
|
||||
This executable file is **portable** and includes all the binaries and models required. No CUDA or PyTorch environment is needed.<br>
|
||||
- [大闹天宫片段](https://www.bilibili.com/video/BV1ja41117zb)
|
||||
- [Anime dance cut 动漫魔性舞蹈](https://www.bilibili.com/video/BV1wY4y1L7hT/)
|
||||
- [海贼王片段](https://www.bilibili.com/video/BV1i3411L7Gy/)
|
||||
|
||||
You can simply run the following command (the Windows example, more information is in the README.md of each executable files):
|
||||
#### YouTube
|
||||
|
||||
```bash
|
||||
./realesrgan-ncnn-vulkan.exe -i input.jpg -o output.png -n model_name
|
||||
```
|
||||
|
||||
We have provided five models:
|
||||
|
||||
1. realesrgan-x4plus (default)
|
||||
2. realesrnet-x4plus
|
||||
3. realesrgan-x4plus-anime (optimized for anime images, small model size)
|
||||
4. realesr-animevideov3 (animation video)
|
||||
|
||||
You can use the `-n` argument for other models, for example, `./realesrgan-ncnn-vulkan.exe -i input.jpg -o output.png -n realesrnet-x4plus`
|
||||
|
||||
### Usage of executable files
|
||||
|
||||
1. Please refer to [Real-ESRGAN-ncnn-vulkan](https://github.com/xinntao/Real-ESRGAN-ncnn-vulkan#computer-usages) for more details.
|
||||
1. Note that it does not support all the functions (such as `outscale`) as the python script `inference_realesrgan.py`.
|
||||
|
||||
```console
|
||||
Usage: realesrgan-ncnn-vulkan.exe -i infile -o outfile [options]...
|
||||
|
||||
-h show this help
|
||||
-i input-path input image path (jpg/png/webp) or directory
|
||||
-o output-path output image path (jpg/png/webp) or directory
|
||||
-s scale upscale ratio (can be 2, 3, 4. default=4)
|
||||
-t tile-size tile size (>=32/0=auto, default=0) can be 0,0,0 for multi-gpu
|
||||
-m model-path folder path to the pre-trained models. default=models
|
||||
-n model-name model name (default=realesr-animevideov3, can be realesr-animevideov3 | realesrgan-x4plus | realesrgan-x4plus-anime | realesrnet-x4plus)
|
||||
-g gpu-id gpu device to use (default=auto) can be 0,1,2 for multi-gpu
|
||||
-j load:proc:save thread count for load/proc/save (default=1:2:2) can be 1:2,2,2:2 for multi-gpu
|
||||
-x enable tta mode"
|
||||
-f format output image format (jpg/png/webp, default=ext/png)
|
||||
-v verbose output
|
||||
```
|
||||
|
||||
Note that it may introduce block inconsistency (and also generate slightly different results from the PyTorch implementation), because this executable file first crops the input image into several tiles, and then processes them separately, finally stitches together.
|
||||
|
||||
---
|
||||
|
||||
## :wrench: Dependencies and Installation
|
||||
## 🔧 Dependencies and Installation
|
||||
|
||||
- Python >= 3.7 (Recommend to use [Anaconda](https://www.anaconda.com/download/#linux) or [Miniconda](https://docs.conda.io/en/latest/miniconda.html))
|
||||
- [PyTorch >= 1.7](https://pytorch.org/)
|
||||
@@ -191,43 +111,69 @@ Note that it may introduce block inconsistency (and also generate slightly diffe
|
||||
python setup.py develop
|
||||
```
|
||||
|
||||
## :zap: Quick Inference
|
||||
---
|
||||
|
||||
### Inference general images
|
||||
## ⚡ Quick Inference
|
||||
|
||||
Download pre-trained models: [RealESRGAN_x4plus.pth](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.1.0/RealESRGAN_x4plus.pth)
|
||||
There are usually three ways to inference Real-ESRGAN.
|
||||
|
||||
1. [Online inference](#online-inference)
|
||||
1. [Portable executable files (NCNN)](#portable-executable-files-ncnn)
|
||||
1. [Python script](#python-script)
|
||||
|
||||
### Online inference
|
||||
|
||||
1. You can try in our website: [ARC Demo](https://arc.tencent.com/en/ai-demos/imgRestore) (now only support RealESRGAN_x4plus_anime_6B)
|
||||
1. [Colab Demo](https://colab.research.google.com/drive/1k2Zod6kSHEvraybHl50Lys0LerhyTMCo?usp=sharing) for Real-ESRGAN **|** [Colab Demo](https://colab.research.google.com/drive/1yNl9ORUxxlL4N0keJa2SEPB61imPQd1B?usp=sharing) for Real-ESRGAN (**anime videos**).
|
||||
|
||||
### Portable executable files (NCNN)
|
||||
|
||||
You can download [Windows](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/realesrgan-ncnn-vulkan-20220424-windows.zip) / [Linux](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/realesrgan-ncnn-vulkan-20220424-ubuntu.zip) / [MacOS](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/realesrgan-ncnn-vulkan-20220424-macos.zip) **executable files for Intel/AMD/Nvidia GPU**.
|
||||
|
||||
This executable file is **portable** and includes all the binaries and models required. No CUDA or PyTorch environment is needed.<br>
|
||||
|
||||
You can simply run the following command (the Windows example, more information is in the README.md of each executable files):
|
||||
|
||||
```bash
|
||||
wget https://github.com/xinntao/Real-ESRGAN/releases/download/v0.1.0/RealESRGAN_x4plus.pth -P experiments/pretrained_models
|
||||
./realesrgan-ncnn-vulkan.exe -i input.jpg -o output.png -n model_name
|
||||
```
|
||||
|
||||
Inference!
|
||||
We have provided five models:
|
||||
|
||||
```bash
|
||||
python inference_realesrgan.py -n RealESRGAN_x4plus -i inputs --face_enhance
|
||||
1. realesrgan-x4plus (default)
|
||||
2. realesrnet-x4plus
|
||||
3. realesrgan-x4plus-anime (optimized for anime images, small model size)
|
||||
4. realesr-animevideov3 (animation video)
|
||||
|
||||
You can use the `-n` argument for other models, for example, `./realesrgan-ncnn-vulkan.exe -i input.jpg -o output.png -n realesrnet-x4plus`
|
||||
|
||||
#### Usage of portable executable files
|
||||
|
||||
1. Please refer to [Real-ESRGAN-ncnn-vulkan](https://github.com/xinntao/Real-ESRGAN-ncnn-vulkan#computer-usages) for more details.
|
||||
1. Note that it does not support all the functions (such as `outscale`) as the python script `inference_realesrgan.py`.
|
||||
|
||||
```console
|
||||
Usage: realesrgan-ncnn-vulkan.exe -i infile -o outfile [options]...
|
||||
|
||||
-h show this help
|
||||
-i input-path input image path (jpg/png/webp) or directory
|
||||
-o output-path output image path (jpg/png/webp) or directory
|
||||
-s scale upscale ratio (can be 2, 3, 4. default=4)
|
||||
-t tile-size tile size (>=32/0=auto, default=0) can be 0,0,0 for multi-gpu
|
||||
-m model-path folder path to the pre-trained models. default=models
|
||||
-n model-name model name (default=realesr-animevideov3, can be realesr-animevideov3 | realesrgan-x4plus | realesrgan-x4plus-anime | realesrnet-x4plus)
|
||||
-g gpu-id gpu device to use (default=auto) can be 0,1,2 for multi-gpu
|
||||
-j load:proc:save thread count for load/proc/save (default=1:2:2) can be 1:2,2,2:2 for multi-gpu
|
||||
-x enable tta mode"
|
||||
-f format output image format (jpg/png/webp, default=ext/png)
|
||||
-v verbose output
|
||||
```
|
||||
|
||||
Results are in the `results` folder
|
||||
Note that it may introduce block inconsistency (and also generate slightly different results from the PyTorch implementation), because this executable file first crops the input image into several tiles, and then processes them separately, finally stitches together.
|
||||
|
||||
### Inference anime images
|
||||
### Python script
|
||||
|
||||
<p align="center">
|
||||
<img src="https://raw.githubusercontent.com/xinntao/public-figures/master/Real-ESRGAN/cmp_realesrgan_anime_1.png">
|
||||
</p>
|
||||
|
||||
Pre-trained models: [RealESRGAN_x4plus_anime_6B](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.4/RealESRGAN_x4plus_anime_6B.pth)<br>
|
||||
More details and comparisons with [waifu2x](https://github.com/nihui/waifu2x-ncnn-vulkan) are in [**anime_model.md**](docs/anime_model.md)
|
||||
|
||||
```bash
|
||||
# download model
|
||||
wget https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.4/RealESRGAN_x4plus_anime_6B.pth -P experiments/pretrained_models
|
||||
# inference
|
||||
python inference_realesrgan.py -n RealESRGAN_x4plus_anime_6B -i inputs
|
||||
```
|
||||
|
||||
Results are in the `results` folder
|
||||
|
||||
### Usage of python script
|
||||
#### Usage of python script
|
||||
|
||||
1. You can use X4 model for **arbitrary output size** with the argument `outscale`. The program will further perform cheap resize operation after the Real-ESRGAN output.
|
||||
|
||||
@@ -248,13 +194,41 @@ A common command: python inference_realesrgan.py -n RealESRGAN_x4plus -i infile
|
||||
--ext Image extension. Options: auto | jpg | png, auto means using the same extension as inputs. Default: auto
|
||||
```
|
||||
|
||||
## :european_castle: Model Zoo
|
||||
#### Inference general images
|
||||
|
||||
Please see [docs/model_zoo.md](docs/model_zoo.md)
|
||||
Download pre-trained models: [RealESRGAN_x4plus.pth](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.1.0/RealESRGAN_x4plus.pth)
|
||||
|
||||
## :computer: Training and Finetuning on your own dataset
|
||||
```bash
|
||||
wget https://github.com/xinntao/Real-ESRGAN/releases/download/v0.1.0/RealESRGAN_x4plus.pth -P weights
|
||||
```
|
||||
|
||||
A detailed guide can be found in [Training.md](Training.md).
|
||||
Inference!
|
||||
|
||||
```bash
|
||||
python inference_realesrgan.py -n RealESRGAN_x4plus -i inputs --face_enhance
|
||||
```
|
||||
|
||||
Results are in the `results` folder
|
||||
|
||||
#### Inference anime images
|
||||
|
||||
<p align="center">
|
||||
<img src="https://raw.githubusercontent.com/xinntao/public-figures/master/Real-ESRGAN/cmp_realesrgan_anime_1.png">
|
||||
</p>
|
||||
|
||||
Pre-trained models: [RealESRGAN_x4plus_anime_6B](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.4/RealESRGAN_x4plus_anime_6B.pth)<br>
|
||||
More details and comparisons with [waifu2x](https://github.com/nihui/waifu2x-ncnn-vulkan) are in [**anime_model.md**](docs/anime_model.md)
|
||||
|
||||
```bash
|
||||
# download model
|
||||
wget https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.4/RealESRGAN_x4plus_anime_6B.pth -P weights
|
||||
# inference
|
||||
python inference_realesrgan.py -n RealESRGAN_x4plus_anime_6B -i inputs
|
||||
```
|
||||
|
||||
Results are in the `results` folder
|
||||
|
||||
---
|
||||
|
||||
## BibTeX
|
||||
|
||||
@@ -265,14 +239,34 @@ A detailed guide can be found in [Training.md](Training.md).
|
||||
date = {2021}
|
||||
}
|
||||
|
||||
## :e-mail: Contact
|
||||
## 📧 Contact
|
||||
|
||||
If you have any question, please email `xintao.wang@outlook.com` or `xintaowang@tencent.com`.
|
||||
|
||||
## :hugs: Acknowledgement
|
||||
<!---------------------------------- Projects that use Real-ESRGAN --------------------------->
|
||||
## 🧩 Projects that use Real-ESRGAN
|
||||
|
||||
If you develop/use Real-ESRGAN in your projects, welcome to let me know.
|
||||
|
||||
- NCNN-Android: [RealSR-NCNN-Android](https://github.com/tumuyan/RealSR-NCNN-Android) by [tumuyan](https://github.com/tumuyan)
|
||||
- VapourSynth: [vs-realesrgan](https://github.com/HolyWu/vs-realesrgan) by [HolyWu](https://github.com/HolyWu)
|
||||
- NCNN: [Real-ESRGAN-ncnn-vulkan](https://github.com/xinntao/Real-ESRGAN-ncnn-vulkan)
|
||||
|
||||
**GUI**
|
||||
|
||||
- [Waifu2x-Extension-GUI](https://github.com/AaronFeng753/Waifu2x-Extension-GUI) by [AaronFeng753](https://github.com/AaronFeng753)
|
||||
- [Squirrel-RIFE](https://github.com/Justin62628/Squirrel-RIFE) by [Justin62628](https://github.com/Justin62628)
|
||||
- [Real-GUI](https://github.com/scifx/Real-GUI) by [scifx](https://github.com/scifx)
|
||||
- [Real-ESRGAN_GUI](https://github.com/net2cn/Real-ESRGAN_GUI) by [net2cn](https://github.com/net2cn)
|
||||
- [Real-ESRGAN-EGUI](https://github.com/WGzeyu/Real-ESRGAN-EGUI) by [WGzeyu](https://github.com/WGzeyu)
|
||||
- [anime_upscaler](https://github.com/shangar21/anime_upscaler) by [shangar21](https://github.com/shangar21)
|
||||
- [Upscayl](https://github.com/upscayl/upscayl) by [Nayam Amarshe](https://github.com/NayamAmarshe) and [TGS963](https://github.com/TGS963)
|
||||
|
||||
## 🤗 Acknowledgement
|
||||
|
||||
Thanks for all the contributors.
|
||||
|
||||
- [AK391](https://github.com/AK391): Integrate RealESRGAN to [Huggingface Spaces](https://huggingface.co/spaces) with [Gradio](https://github.com/gradio-app/gradio). See [Gradio Web Demo](https://huggingface.co/spaces/akhaliq/Real-ESRGAN).
|
||||
- [Asiimoviet](https://github.com/Asiimoviet): Translate the README.md to Chinese (中文).
|
||||
- [2ji3150](https://github.com/2ji3150): Thanks for the [detailed and valuable feedbacks/suggestions](https://github.com/xinntao/Real-ESRGAN/issues/131).
|
||||
- [Jared-02](https://github.com/Jared-02): Translate the Training.md to Chinese (中文).
|
||||
|
||||
25
README_CN.md
25
README_CN.md
@@ -12,20 +12,19 @@
|
||||
[](https://github.com/xinntao/Real-ESRGAN/blob/master/.github/workflows/pylint.yml)
|
||||
[](https://github.com/xinntao/Real-ESRGAN/blob/master/.github/workflows/publish-pip.yml)
|
||||
|
||||
:fire: 更新动漫视频的小模型 **RealESRGAN AnimeVideo-v3**. 更多信息在 [anime video models](docs/anime_video_model.md) 和 [comparisons](docs/anime_comparisons.md)中.
|
||||
:fire: 更新动漫视频的小模型 **RealESRGAN AnimeVideo-v3**. 更多信息在 [[动漫视频模型介绍](docs/anime_video_model.md)] 和 [[比较](docs/anime_comparisons_CN.md)] 中.
|
||||
|
||||
1. Real-ESRGAN的[Colab Demo](https://colab.research.google.com/drive/1k2Zod6kSHEvraybHl50Lys0LerhyTMCo?usp=sharing) <a href="https://colab.research.google.com/drive/1k2Zod6kSHEvraybHl50Lys0LerhyTMCo?usp=sharing"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="google colab logo"></a>.
|
||||
2. Real-ESRGAN的 **动漫视频** 的[Colab Demo](https://colab.research.google.com/drive/1yNl9ORUxxlL4N0keJa2SEPB61imPQd1B?usp=sharing) <a href="https://colab.research.google.com/drive/1yNl9ORUxxlL4N0keJa2SEPB61imPQd1B?usp=sharing"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="google colab logo"></a>.
|
||||
3. **支持Intel/AMD/Nvidia显卡**的绿色版exe文件： [Windows版](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/realesrgan-ncnn-vulkan-20220424-windows.zip) / [Linux版](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/realesrgan-ncnn-vulkan-20220424-ubuntu.zip) / [macOS版](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/realesrgan-ncnn-vulkan-20220424-macos.zip)，详情请移步[这里](#便携版（绿色版）可执行文件)。NCNN的实现在 [Real-ESRGAN-ncnn-vulkan](https://github.com/xinntao/Real-ESRGAN-ncnn-vulkan)。
|
||||
1. Real-ESRGAN的[Colab Demo](https://colab.research.google.com/drive/1k2Zod6kSHEvraybHl50Lys0LerhyTMCo?usp=sharing) | Real-ESRGAN**动漫视频** 的[Colab Demo](https://colab.research.google.com/drive/1yNl9ORUxxlL4N0keJa2SEPB61imPQd1B?usp=sharing)
|
||||
2. **支持Intel/AMD/Nvidia显卡**的绿色版exe文件： [Windows版](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/realesrgan-ncnn-vulkan-20220424-windows.zip) / [Linux版](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/realesrgan-ncnn-vulkan-20220424-ubuntu.zip) / [macOS版](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/realesrgan-ncnn-vulkan-20220424-macos.zip)，详情请移步[这里](#便携版（绿色版）可执行文件)。NCNN的实现在 [Real-ESRGAN-ncnn-vulkan](https://github.com/xinntao/Real-ESRGAN-ncnn-vulkan)。
|
||||
|
||||
Real-ESRGAN 的目标是开发出**实用的图像/视频修复算法**。<br>
|
||||
我们在 ESRGAN 的基础上使用纯合成的数据来进行训练，以使其能被应用于实际的图片修复的场景（顾名思义：Real-ESRGAN）。
|
||||
|
||||
:art: Real-ESRGAN 需要，也很欢迎你的贡献，如新功能、模型、bug修复、建议、维护等等。详情可以查看[CONTRIBUTING.md](CONTRIBUTING.md)，所有的贡献者都会被列在[此处](README_CN.md#hugs-感谢)。
|
||||
:art: Real-ESRGAN 需要，也很欢迎你的贡献，如新功能、模型、bug修复、建议、维护等等。详情可以查看[CONTRIBUTING.md](docs/CONTRIBUTING.md)，所有的贡献者都会被列在[此处](README_CN.md#hugs-感谢)。
|
||||
|
||||
:milky_way: 感谢大家提供了很好的反馈。这些反馈会逐步更新在 [这个文档](feedback.md)。
|
||||
:milky_way: 感谢大家提供了很好的反馈。这些反馈会逐步更新在 [这个文档](docs/feedback.md)。
|
||||
|
||||
:question: 常见的问题可以在[FAQ.md](FAQ.md)中找到答案。（好吧，现在还是空白的=-=||）
|
||||
:question: 常见的问题可以在[FAQ.md](docs/FAQ.md)中找到答案。（好吧，现在还是空白的=-=||）
|
||||
|
||||
---
|
||||
|
||||
@@ -46,12 +45,12 @@ Real-ESRGAN 的目标是开发出**实用的图像/视频修复算法**。<br>
|
||||
- ✅ 添加了针对动漫视频的小模型, 更多信息在 [anime video models](docs/anime_video_model.md) 中.
|
||||
- ✅ 添加了ncnn 实现：[Real-ESRGAN-ncnn-vulkan](https://github.com/xinntao/Real-ESRGAN-ncnn-vulkan).
|
||||
- ✅ 添加了 [*RealESRGAN_x4plus_anime_6B.pth*](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.4/RealESRGAN_x4plus_anime_6B.pth)，对二次元图片进行了优化，并减少了model的大小。详情 以及 与[waifu2x](https://github.com/nihui/waifu2x-ncnn-vulkan)的对比请查看[**anime_model.md**](docs/anime_model.md)
|
||||
- ✅支持用户在自己的数据上进行微调 (finetune)：[详情](Training.md#Finetune-Real-ESRGAN-on-your-own-dataset)
|
||||
- ✅支持用户在自己的数据上进行微调 (finetune)：[详情](docs/Training.md#Finetune-Real-ESRGAN-on-your-own-dataset)
|
||||
- ✅ 支持使用[GFPGAN](https://github.com/TencentARC/GFPGAN)**增强人脸**
|
||||
- ✅ 通过[Gradio](https://github.com/gradio-app/gradio)添加到了[Huggingface Spaces](https://huggingface.co/spaces)（一个机器学习应用的在线平台）：[Gradio在线版](https://huggingface.co/spaces/akhaliq/Real-ESRGAN)。感谢[@AK391](https://github.com/AK391)
|
||||
- ✅ 支持任意比例的缩放：`--outscale`（实际上使用`LANCZOS4`来更进一步调整输出图像的尺寸）。添加了*RealESRGAN_x2plus.pth*模型
|
||||
- ✅ [推断脚本](inference_realesrgan.py)支持: 1) 分块处理**tile**; 2) 带**alpha通道**的图像; 3) **灰色**图像; 4) **16-bit**图像.
|
||||
- ✅ 训练代码已经发布，具体做法可查看：[Training.md](Training.md)。
|
||||
- ✅ 训练代码已经发布，具体做法可查看：[Training.md](docs/Training.md)。
|
||||
|
||||
</details>
|
||||
|
||||
@@ -73,6 +72,7 @@ Real-ESRGAN 的目标是开发出**实用的图像/视频修复算法**。<br>
|
||||
- [Real-ESRGAN_GUI](https://github.com/net2cn/Real-ESRGAN_GUI) by [net2cn](https://github.com/net2cn)
|
||||
- [Real-ESRGAN-EGUI](https://github.com/WGzeyu/Real-ESRGAN-EGUI) by [WGzeyu](https://github.com/WGzeyu)
|
||||
- [anime_upscaler](https://github.com/shangar21/anime_upscaler) by [shangar21](https://github.com/shangar21)
|
||||
- [RealESRGAN-GUI](https://github.com/Baiyuetribe/paper2gui/blob/main/Video%20Super%20Resolution/RealESRGAN-GUI.md) by [Baiyuetribe](https://github.com/Baiyuetribe)
|
||||
|
||||
</details>
|
||||
|
||||
@@ -195,7 +195,7 @@ Usage: realesrgan-ncnn-vulkan.exe -i infile -o outfile [options]...
|
||||
下载我们训练好的模型: [RealESRGAN_x4plus.pth](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.1.0/RealESRGAN_x4plus.pth)
|
||||
|
||||
```bash
|
||||
wget https://github.com/xinntao/Real-ESRGAN/releases/download/v0.1.0/RealESRGAN_x4plus.pth -P experiments/pretrained_models
|
||||
wget https://github.com/xinntao/Real-ESRGAN/releases/download/v0.1.0/RealESRGAN_x4plus.pth -P weights
|
||||
```
|
||||
|
||||
推断!
|
||||
@@ -217,7 +217,7 @@ python inference_realesrgan.py -n RealESRGAN_x4plus -i inputs --face_enhance
|
||||
|
||||
```bash
|
||||
# 下载模型
|
||||
wget https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.4/RealESRGAN_x4plus_anime_6B.pth -P experiments/pretrained_models
|
||||
wget https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.4/RealESRGAN_x4plus_anime_6B.pth -P weights
|
||||
# 推断
|
||||
python inference_realesrgan.py -n RealESRGAN_x4plus_anime_6B -i inputs
|
||||
```
|
||||
@@ -251,7 +251,7 @@ A common command: python inference_realesrgan.py -n RealESRGAN_x4plus -i infile
|
||||
|
||||
## :computer: 训练，在你的数据上微调（Fine-tune）
|
||||
|
||||
这里有一份详细的指南：[Training.md](Training.md).
|
||||
这里有一份详细的指南：[Training.md](docs/Training.md).
|
||||
|
||||
## BibTeX 引用
|
||||
|
||||
@@ -273,3 +273,4 @@ A common command: python inference_realesrgan.py -n RealESRGAN_x4plus -i infile
|
||||
- [AK391](https://github.com/AK391): 通过[Gradio](https://github.com/gradio-app/gradio)添加到了[Huggingface Spaces](https://huggingface.co/spaces)（一个机器学习应用的在线平台）：[Gradio在线版](https://huggingface.co/spaces/akhaliq/Real-ESRGAN)。
|
||||
- [Asiimoviet](https://github.com/Asiimoviet): 把 README.md 文档 翻译成了中文。
|
||||
- [2ji3150](https://github.com/2ji3150): 感谢详尽并且富有价值的[反馈、建议](https://github.com/xinntao/Real-ESRGAN/issues/131).
|
||||
- [Jared-02](https://github.com/Jared-02): 把 Training.md 文档 翻译成了中文。
|
||||
|
||||
22
cog.yaml
Normal file
22
cog.yaml
Normal file
@@ -0,0 +1,22 @@
|
||||
# This file is used for constructing replicate env
|
||||
image: "r8.im/tencentarc/realesrgan"
|
||||
|
||||
build:
|
||||
gpu: true
|
||||
python_version: "3.8"
|
||||
system_packages:
|
||||
- "libgl1-mesa-glx"
|
||||
- "libglib2.0-0"
|
||||
python_packages:
|
||||
- "torch==1.7.1"
|
||||
- "torchvision==0.8.2"
|
||||
- "numpy==1.21.1"
|
||||
- "lmdb==1.2.1"
|
||||
- "opencv-python==4.5.3.56"
|
||||
- "PyYAML==5.4.1"
|
||||
- "tqdm==4.62.2"
|
||||
- "yapf==0.31.0"
|
||||
- "basicsr==1.4.2"
|
||||
- "facexlib==0.2.5"
|
||||
|
||||
predict: "cog_predict.py:Predictor"
|
||||
148
cog_predict.py
Normal file
148
cog_predict.py
Normal file
@@ -0,0 +1,148 @@
|
||||
# flake8: noqa
|
||||
# This file is used for deploying replicate models
|
||||
# running: cog predict -i img=@inputs/00017_gray.png -i version='General - v3' -i scale=2 -i face_enhance=True -i tile=0
|
||||
# push: cog push r8.im/xinntao/realesrgan
|
||||
|
||||
import os
|
||||
|
||||
os.system('pip install gfpgan')
|
||||
os.system('python setup.py develop')
|
||||
|
||||
import cv2
|
||||
import shutil
|
||||
import tempfile
|
||||
import torch
|
||||
from basicsr.archs.rrdbnet_arch import RRDBNet
|
||||
from basicsr.archs.srvgg_arch import SRVGGNetCompact
|
||||
|
||||
from realesrgan.utils import RealESRGANer
|
||||
|
||||
try:
|
||||
from cog import BasePredictor, Input, Path
|
||||
from gfpgan import GFPGANer
|
||||
except Exception:
|
||||
print('please install cog and realesrgan package')
|
||||
|
||||
|
||||
class Predictor(BasePredictor):
|
||||
|
||||
def setup(self):
|
||||
os.makedirs('output', exist_ok=True)
|
||||
# download weights
|
||||
if not os.path.exists('weights/realesr-general-x4v3.pth'):
|
||||
os.system(
|
||||
'wget https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/realesr-general-x4v3.pth -P ./weights'
|
||||
)
|
||||
if not os.path.exists('weights/GFPGANv1.4.pth'):
|
||||
os.system('wget https://github.com/TencentARC/GFPGAN/releases/download/v1.3.0/GFPGANv1.4.pth -P ./weights')
|
||||
if not os.path.exists('weights/RealESRGAN_x4plus.pth'):
|
||||
os.system(
|
||||
'wget https://github.com/xinntao/Real-ESRGAN/releases/download/v0.1.0/RealESRGAN_x4plus.pth -P ./weights'
|
||||
)
|
||||
if not os.path.exists('weights/RealESRGAN_x4plus_anime_6B.pth'):
|
||||
os.system(
|
||||
'wget https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.4/RealESRGAN_x4plus_anime_6B.pth -P ./weights'
|
||||
)
|
||||
if not os.path.exists('weights/realesr-animevideov3.pth'):
|
||||
os.system(
|
||||
'wget https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/realesr-animevideov3.pth -P ./weights'
|
||||
)
|
||||
|
||||
def choose_model(self, scale, version, tile=0):
|
||||
half = True if torch.cuda.is_available() else False
|
||||
if version == 'General - RealESRGANplus':
|
||||
model = RRDBNet(num_in_ch=3, num_out_ch=3, num_feat=64, num_block=23, num_grow_ch=32, scale=4)
|
||||
model_path = 'weights/RealESRGAN_x4plus.pth'
|
||||
self.upsampler = RealESRGANer(
|
||||
scale=4, model_path=model_path, model=model, tile=tile, tile_pad=10, pre_pad=0, half=half)
|
||||
elif version == 'General - v3':
|
||||
model = SRVGGNetCompact(num_in_ch=3, num_out_ch=3, num_feat=64, num_conv=32, upscale=4, act_type='prelu')
|
||||
model_path = 'weights/realesr-general-x4v3.pth'
|
||||
self.upsampler = RealESRGANer(
|
||||
scale=4, model_path=model_path, model=model, tile=tile, tile_pad=10, pre_pad=0, half=half)
|
||||
elif version == 'Anime - anime6B':
|
||||
model = RRDBNet(num_in_ch=3, num_out_ch=3, num_feat=64, num_block=6, num_grow_ch=32, scale=4)
|
||||
model_path = 'weights/RealESRGAN_x4plus_anime_6B.pth'
|
||||
self.upsampler = RealESRGANer(
|
||||
scale=4, model_path=model_path, model=model, tile=tile, tile_pad=10, pre_pad=0, half=half)
|
||||
elif version == 'AnimeVideo - v3':
|
||||
model = SRVGGNetCompact(num_in_ch=3, num_out_ch=3, num_feat=64, num_conv=16, upscale=4, act_type='prelu')
|
||||
model_path = 'weights/realesr-animevideov3.pth'
|
||||
self.upsampler = RealESRGANer(
|
||||
scale=4, model_path=model_path, model=model, tile=tile, tile_pad=10, pre_pad=0, half=half)
|
||||
|
||||
self.face_enhancer = GFPGANer(
|
||||
model_path='weights/GFPGANv1.4.pth',
|
||||
upscale=scale,
|
||||
arch='clean',
|
||||
channel_multiplier=2,
|
||||
bg_upsampler=self.upsampler)
|
||||
|
||||
def predict(
|
||||
self,
|
||||
img: Path = Input(description='Input'),
|
||||
version: str = Input(
|
||||
description='RealESRGAN version. Please see [Readme] below for more descriptions',
|
||||
choices=['General - RealESRGANplus', 'General - v3', 'Anime - anime6B', 'AnimeVideo - v3'],
|
||||
default='General - v3'),
|
||||
scale: float = Input(description='Rescaling factor', default=2),
|
||||
face_enhance: bool = Input(
|
||||
description='Enhance faces with GFPGAN. Note that it does not work for anime images/vidoes', default=False),
|
||||
tile: int = Input(
|
||||
description=
|
||||
'Tile size. Default is 0, that is no tile. When encountering the out-of-GPU-memory issue, please specify it, e.g., 400 or 200',
|
||||
default=0)
|
||||
) -> Path:
|
||||
if tile <= 100 or tile is None:
|
||||
tile = 0
|
||||
print(f'img: {img}. version: {version}. scale: {scale}. face_enhance: {face_enhance}. tile: {tile}.')
|
||||
try:
|
||||
extension = os.path.splitext(os.path.basename(str(img)))[1]
|
||||
img = cv2.imread(str(img), cv2.IMREAD_UNCHANGED)
|
||||
if len(img.shape) == 3 and img.shape[2] == 4:
|
||||
img_mode = 'RGBA'
|
||||
elif len(img.shape) == 2:
|
||||
img_mode = None
|
||||
img = cv2.cvtColor(img, cv2.COLOR_GRAY2BGR)
|
||||
else:
|
||||
img_mode = None
|
||||
|
||||
h, w = img.shape[0:2]
|
||||
if h < 300:
|
||||
img = cv2.resize(img, (w * 2, h * 2), interpolation=cv2.INTER_LANCZOS4)
|
||||
|
||||
self.choose_model(scale, version, tile)
|
||||
|
||||
try:
|
||||
if face_enhance:
|
||||
_, _, output = self.face_enhancer.enhance(
|
||||
img, has_aligned=False, only_center_face=False, paste_back=True)
|
||||
else:
|
||||
output, _ = self.upsampler.enhance(img, outscale=scale)
|
||||
except RuntimeError as error:
|
||||
print('Error', error)
|
||||
print('If you encounter CUDA out of memory, try to set "tile" to a smaller size, e.g., 400.')
|
||||
|
||||
if img_mode == 'RGBA': # RGBA images should be saved in png format
|
||||
extension = 'png'
|
||||
# save_path = f'output/out.{extension}'
|
||||
# cv2.imwrite(save_path, output)
|
||||
out_path = Path(tempfile.mkdtemp()) / f'out.{extension}'
|
||||
cv2.imwrite(str(out_path), output)
|
||||
except Exception as error:
|
||||
print('global exception: ', error)
|
||||
finally:
|
||||
clean_folder('output')
|
||||
return out_path
|
||||
|
||||
|
||||
def clean_folder(folder):
|
||||
for filename in os.listdir(folder):
|
||||
file_path = os.path.join(folder, filename)
|
||||
try:
|
||||
if os.path.isfile(file_path) or os.path.islink(file_path):
|
||||
os.unlink(file_path)
|
||||
elif os.path.isdir(file_path):
|
||||
shutil.rmtree(file_path)
|
||||
except Exception as e:
|
||||
print(f'Failed to delete {file_path}. Reason: {e}')
|
||||
@@ -1,5 +1,7 @@
|
||||
# Contributing to Real-ESRGAN
|
||||
|
||||
:art: Real-ESRGAN needs your contributions. Any contributions are welcome, such as new features/models/typo fixes/suggestions/maintenance, *etc*. See [CONTRIBUTING.md](docs/CONTRIBUTING.md). All contributors are list [here](README.md#hugs-acknowledgement).
|
||||
|
||||
We like open-source and want to develop practical algorithms for general image restoration. However, individual strength is limited. So, any kinds of contributions are welcome, such as:
|
||||
|
||||
- New features
|
||||
@@ -19,6 +21,7 @@ We like open-source and want to develop practical algorithms for general image r
|
||||
1. Create a PR
|
||||
|
||||
**Note**:
|
||||
|
||||
1. Please check the code style and linting
|
||||
1. The style configuration is specified in [setup.cfg](setup.cfg)
|
||||
1. If you use VSCode, the settings are configured in [.vscode/settings.json](.vscode/settings.json)
|
||||
10
docs/FAQ.md
Normal file
10
docs/FAQ.md
Normal file
@@ -0,0 +1,10 @@
|
||||
# FAQ
|
||||
|
||||
1. **Q: How to select models?**<br>
|
||||
A: Please refer to [docs/model_zoo.md](docs/model_zoo.md)
|
||||
|
||||
1. **Q: Can `face_enhance` be used for anime images/animation videos?**<br>
|
||||
A: No, it can only be used for real faces. It is recommended not to use this option for anime images/animation videos to save GPU memory.
|
||||
|
||||
1. **Q: Error "slow_conv2d_cpu" not implemented for 'Half'**<br>
|
||||
A: In order to save GPU memory consumption and speed up inference, Real-ESRGAN uses half precision (fp16) during inference by default. However, some operators for half inference are not implemented in CPU mode. You need to add **`--fp32` option** for the commands. For example, `python inference_realesrgan.py -n RealESRGAN_x4plus.pth -i inputs --fp32`.
|
||||
@@ -9,6 +9,8 @@
|
||||
- [Generate degraded images on the fly](#Generate-degraded-images-on-the-fly)
|
||||
- [Use paired training data](#use-your-own-paired-data)
|
||||
|
||||
[English](Training.md) **|** [简体中文](Training_CN.md)
|
||||
|
||||
## Train Real-ESRGAN
|
||||
|
||||
### Overview
|
||||
@@ -65,7 +67,7 @@ You can use the [scripts/generate_meta_info.py](scripts/generate_meta_info.py) s
|
||||
You can merge several folders into one meta_info txt. Here is the example:
|
||||
|
||||
```bash
|
||||
python scripts/generate_meta_info.py --input datasets/DF2K/DF2K_HR, datasets/DF2K/DF2K_multiscale --root datasets/DF2K, datasets/DF2K --meta_info datasets/DF2K/meta_info/meta_info_DF2Kmultiscale.txt
|
||||
python scripts/generate_meta_info.py --input datasets/DF2K/DF2K_HR datasets/DF2K/DF2K_multiscale --root datasets/DF2K datasets/DF2K --meta_info datasets/DF2K/meta_info/meta_info_DF2Kmultiscale.txt
|
||||
```
|
||||
|
||||
### Train Real-ESRNet
|
||||
@@ -164,7 +166,7 @@ You can finetune Real-ESRGAN on your own dataset. Typically, the fine-tuning pro
|
||||
|
||||
### Generate degraded images on the fly
|
||||
|
||||
Only high-resolution images are required. The low-quality images are generated with the degradation process described in Real-ESRGAN during trainig.
|
||||
Only high-resolution images are required. The low-quality images are generated with the degradation process described in Real-ESRGAN during training.
|
||||
|
||||
**1. Prepare dataset**
|
||||
|
||||
@@ -251,9 +253,9 @@ train:
|
||||
type: RealESRGANPairedDataset
|
||||
dataroot_gt: datasets/DF2K # modify to the root path of your folder
|
||||
dataroot_lq: datasets/DF2K # modify to the root path of your folder
|
||||
meta_info: datasets/DF2K/meta_info/meta_info_DIV2K_sub_pair.txt # modify to the root path of your folder
|
||||
meta_info: datasets/DF2K/meta_info/meta_info_DIV2K_sub_pair.txt # modify to your own generate meta info txt
|
||||
io_backend:
|
||||
type: disk
|
||||
type: disk
|
||||
```
|
||||
|
||||
We use four GPUs for training. We use the `--auto_resume` argument to automatically resume the training if necessary.
|
||||
271
docs/Training_CN.md
Normal file
271
docs/Training_CN.md
Normal file
@@ -0,0 +1,271 @@
|
||||
# :computer: 如何训练/微调 Real-ESRGAN
|
||||
|
||||
- [训练 Real-ESRGAN](#训练-real-esrgan)
|
||||
- [概述](#概述)
|
||||
- [准备数据集](#准备数据集)
|
||||
- [训练 Real-ESRNet 模型](#训练-real-esrnet-模型)
|
||||
- [训练 Real-ESRGAN 模型](#训练-real-esrgan-模型)
|
||||
- [用自己的数据集微调 Real-ESRGAN](#用自己的数据集微调-real-esrgan)
|
||||
- [动态生成降级图像](#动态生成降级图像)
|
||||
- [使用已配对的数据](#使用已配对的数据)
|
||||
|
||||
[English](Training.md) **|** [简体中文](Training_CN.md)
|
||||
|
||||
## 训练 Real-ESRGAN
|
||||
|
||||
### 概述
|
||||
|
||||
训练分为两个步骤。除了 loss 函数外，这两个步骤拥有相同数据合成以及训练的一条龙流程。具体点说：
|
||||
|
||||
1. 首先使用 L1 loss 训练 Real-ESRNet 模型，其中 L1 loss 来自预先训练的 ESRGAN 模型。
|
||||
|
||||
2. 然后我们将 Real-ESRNet 模型作为生成器初始化，结合L1 loss、感知 loss、GAN loss 三者的参数对 Real-ESRGAN 进行训练。
|
||||
|
||||
### 准备数据集
|
||||
|
||||
我们使用 DF2K ( DIV2K 和 Flickr2K ) + OST 数据集进行训练。只需要HR图像！<br>
|
||||
下面是网站链接:
|
||||
1. DIV2K: http://data.vision.ee.ethz.ch/cvl/DIV2K/DIV2K_train_HR.zip
|
||||
2. Flickr2K: https://cv.snu.ac.kr/research/EDSR/Flickr2K.tar
|
||||
3. OST: https://openmmlab.oss-cn-hangzhou.aliyuncs.com/datasets/OST_dataset.zip
|
||||
|
||||
以下是数据的准备步骤。
|
||||
|
||||
#### 第1步：【可选】生成多尺寸图片
|
||||
|
||||
针对 DF2K 数据集，我们使用多尺寸缩放策略，*换言之*，我们对 HR 图像进行下采样，就能获得多尺寸的标准参考（Ground-Truth）图像。 <br>
|
||||
您可以使用这个 [scripts/generate_multiscale_DF2K.py](scripts/generate_multiscale_DF2K.py) 脚本快速生成多尺寸的图像。<br>
|
||||
注意：如果您只想简单试试，那么可以跳过此步骤。
|
||||
|
||||
```bash
|
||||
python scripts/generate_multiscale_DF2K.py --input datasets/DF2K/DF2K_HR --output datasets/DF2K/DF2K_multiscale
|
||||
```
|
||||
|
||||
#### 第2步：【可选】裁切为子图像
|
||||
|
||||
我们可以将 DF2K 图像裁切为子图像，以加快 IO 和处理速度。<br>
|
||||
如果你的 IO 够好或储存空间有限，那么此步骤是可选的。<br>
|
||||
|
||||
您可以使用脚本 [scripts/extract_subimages.py](scripts/extract_subimages.py)。这是使用示例:
|
||||
|
||||
```bash
|
||||
python scripts/extract_subimages.py --input datasets/DF2K/DF2K_multiscale --output datasets/DF2K/DF2K_multiscale_sub --crop_size 400 --step 200
|
||||
```
|
||||
|
||||
#### 第3步：准备元信息 txt
|
||||
|
||||
您需要准备一个包含图像路径的 txt 文件。下面是 `meta_info_DF2Kmultiscale+OST_sub.txt` 中的部分展示（由于各个用户可能有截然不同的子图像划分，这个文件不适合你的需求，你得准备自己的 txt 文件)：
|
||||
|
||||
```txt
|
||||
DF2K_HR_sub/000001_s001.png
|
||||
DF2K_HR_sub/000001_s002.png
|
||||
DF2K_HR_sub/000001_s003.png
|
||||
...
|
||||
```
|
||||
|
||||
你可以使用该脚本 [scripts/generate_meta_info.py](scripts/generate_meta_info.py) 生成包含图像路径的 txt 文件。<br>
|
||||
你还可以合并多个文件夹的图像路径到一个元信息（meta_info）txt。这是使用示例:
|
||||
|
||||
```bash
|
||||
python scripts/generate_meta_info.py --input datasets/DF2K/DF2K_HR, datasets/DF2K/DF2K_multiscale --root datasets/DF2K, datasets/DF2K --meta_info datasets/DF2K/meta_info/meta_info_DF2Kmultiscale.txt
|
||||
```
|
||||
|
||||
### 训练 Real-ESRNet 模型
|
||||
|
||||
1. 下载预先训练的模型 [ESRGAN](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.1.1/ESRGAN_SRx4_DF2KOST_official-ff704c30.pth)，放到 `experiments/pretrained_models`目录下。
|
||||
```bash
|
||||
wget https://github.com/xinntao/Real-ESRGAN/releases/download/v0.1.1/ESRGAN_SRx4_DF2KOST_official-ff704c30.pth -P experiments/pretrained_models
|
||||
```
|
||||
2. 相应地修改选项文件 `options/train_realesrnet_x4plus.yml` 中的内容：
|
||||
```yml
|
||||
train:
|
||||
name: DF2K+OST
|
||||
type: RealESRGANDataset
|
||||
dataroot_gt: datasets/DF2K # 修改为你的数据集文件夹根目录
|
||||
meta_info: realesrgan/meta_info/meta_info_DF2Kmultiscale+OST_sub.txt # 修改为你自己生成的元信息txt
|
||||
io_backend:
|
||||
type: disk
|
||||
```
|
||||
3. 如果你想在训练过程中执行验证，就取消注释这些内容并进行相应的修改：
|
||||
```yml
|
||||
# 取消注释这些以进行验证
|
||||
# val:
|
||||
# name: validation
|
||||
# type: PairedImageDataset
|
||||
# dataroot_gt: path_to_gt
|
||||
# dataroot_lq: path_to_lq
|
||||
# io_backend:
|
||||
# type: disk
|
||||
|
||||
...
|
||||
|
||||
# 取消注释这些以进行验证
|
||||
# 验证设置
|
||||
# val:
|
||||
# val_freq: !!float 5e3
|
||||
# save_img: True
|
||||
|
||||
# metrics:
|
||||
# psnr: # 指标名称，可以是任意的
|
||||
# type: calculate_psnr
|
||||
# crop_border: 4
|
||||
# test_y_channel: false
|
||||
```
|
||||
4. 正式训练之前，你可以用 `--debug` 模式检查是否正常运行。我们用了4个GPU进行训练：
|
||||
```bash
|
||||
CUDA_VISIBLE_DEVICES=0,1,2,3 \
|
||||
python -m torch.distributed.launch --nproc_per_node=4 --master_port=4321 realesrgan/train.py -opt options/train_realesrnet_x4plus.yml --launcher pytorch --debug
|
||||
```
|
||||
|
||||
用 **1个GPU** 训练的 debug 模式示例:
|
||||
```bash
|
||||
python realesrgan/train.py -opt options/train_realesrnet_x4plus.yml --debug
|
||||
```
|
||||
5. 正式训练开始。我们用了4个GPU进行训练。还可以使用参数 `--auto_resume` 在必要时自动恢复训练。
|
||||
```bash
|
||||
CUDA_VISIBLE_DEVICES=0,1,2,3 \
|
||||
python -m torch.distributed.launch --nproc_per_node=4 --master_port=4321 realesrgan/train.py -opt options/train_realesrnet_x4plus.yml --launcher pytorch --auto_resume
|
||||
```
|
||||
|
||||
用 **1个GPU** 训练：
|
||||
```bash
|
||||
python realesrgan/train.py -opt options/train_realesrnet_x4plus.yml --auto_resume
|
||||
```
|
||||
|
||||
### 训练 Real-ESRGAN 模型
|
||||
|
||||
1. 训练 Real-ESRNet 模型后，您得到了这个 `experiments/train_RealESRNetx4plus_1000k_B12G4_fromESRGAN/model/net_g_1000000.pth` 文件。如果需要指定预训练路径到其他文件，请修改选项文件 `train_realesrgan_x4plus.yml` 中 `pretrain_network_g` 的值。
|
||||
1. 修改选项文件 `train_realesrgan_x4plus.yml` 的内容。大多数修改与上节提到的类似。
|
||||
1. 正式训练之前，你可以以 `--debug` 模式检查是否正常运行。我们使用了4个GPU进行训练：
|
||||
```bash
|
||||
CUDA_VISIBLE_DEVICES=0,1,2,3 \
|
||||
python -m torch.distributed.launch --nproc_per_node=4 --master_port=4321 realesrgan/train.py -opt options/train_realesrgan_x4plus.yml --launcher pytorch --debug
|
||||
```
|
||||
|
||||
用 **1个GPU** 训练的 debug 模式示例:
|
||||
```bash
|
||||
python realesrgan/train.py -opt options/train_realesrgan_x4plus.yml --debug
|
||||
```
|
||||
1. 正式训练开始。我们使用4个GPU进行训练。还可以使用参数 `--auto_resume` 在必要时自动恢复训练。
|
||||
```bash
|
||||
CUDA_VISIBLE_DEVICES=0,1,2,3 \
|
||||
python -m torch.distributed.launch --nproc_per_node=4 --master_port=4321 realesrgan/train.py -opt options/train_realesrgan_x4plus.yml --launcher pytorch --auto_resume
|
||||
```
|
||||
|
||||
用 **1个GPU** 训练：
|
||||
```bash
|
||||
python realesrgan/train.py -opt options/train_realesrgan_x4plus.yml --auto_resume
|
||||
```
|
||||
|
||||
## 用自己的数据集微调 Real-ESRGAN
|
||||
|
||||
你可以用自己的数据集微调 Real-ESRGAN。一般地，微调（Fine-Tune）程序可以分为两种类型:
|
||||
|
||||
1. [动态生成降级图像](#动态生成降级图像)
|
||||
2. [使用**已配对**的数据](#使用已配对的数据)
|
||||
|
||||
### 动态生成降级图像
|
||||
|
||||
只需要高分辨率图像。在训练过程中，使用 Real-ESRGAN 描述的降级模型生成低质量图像。
|
||||
|
||||
**1. 准备数据集**
|
||||
|
||||
完整信息请参见[本节](#准备数据集)。
|
||||
|
||||
**2. 下载预训练模型**
|
||||
|
||||
下载预先训练的模型到 `experiments/pretrained_models` 目录下。
|
||||
|
||||
- *RealESRGAN_x4plus.pth*:
|
||||
```bash
|
||||
wget https://github.com/xinntao/Real-ESRGAN/releases/download/v0.1.0/RealESRGAN_x4plus.pth -P experiments/pretrained_models
|
||||
```
|
||||
|
||||
- *RealESRGAN_x4plus_netD.pth*:
|
||||
```bash
|
||||
wget https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.3/RealESRGAN_x4plus_netD.pth -P experiments/pretrained_models
|
||||
```
|
||||
|
||||
**3. 微调**
|
||||
|
||||
修改选项文件 [options/finetune_realesrgan_x4plus.yml](options/finetune_realesrgan_x4plus.yml) ，特别是 `datasets` 部分：
|
||||
|
||||
```yml
|
||||
train:
|
||||
name: DF2K+OST
|
||||
type: RealESRGANDataset
|
||||
dataroot_gt: datasets/DF2K # 修改为你的数据集文件夹根目录
|
||||
meta_info: realesrgan/meta_info/meta_info_DF2Kmultiscale+OST_sub.txt # 修改为你自己生成的元信息txt
|
||||
io_backend:
|
||||
type: disk
|
||||
```
|
||||
|
||||
我们使用4个GPU进行训练。还可以使用参数 `--auto_resume` 在必要时自动恢复训练。
|
||||
|
||||
```bash
|
||||
CUDA_VISIBLE_DEVICES=0,1,2,3 \
|
||||
python -m torch.distributed.launch --nproc_per_node=4 --master_port=4321 realesrgan/train.py -opt options/finetune_realesrgan_x4plus.yml --launcher pytorch --auto_resume
|
||||
```
|
||||
|
||||
用 **1个GPU** 训练：
|
||||
```bash
|
||||
python realesrgan/train.py -opt options/finetune_realesrgan_x4plus.yml --auto_resume
|
||||
```
|
||||
|
||||
### 使用已配对的数据
|
||||
|
||||
你还可以用自己已经配对的数据微调 RealESRGAN。这个过程更类似于微调 ESRGAN。
|
||||
|
||||
**1. 准备数据集**
|
||||
|
||||
假设你已经有两个文件夹（folder）:
|
||||
|
||||
- **gt folder**（标准参考，高分辨率图像）：*datasets/DF2K/DIV2K_train_HR_sub*
|
||||
- **lq folder**（低质量，低分辨率图像）：*datasets/DF2K/DIV2K_train_LR_bicubic_X4_sub*
|
||||
|
||||
然后，您可以使用脚本 [scripts/generate_meta_info_pairdata.py](scripts/generate_meta_info_pairdata.py) 生成元信息（meta_info）txt 文件。
|
||||
|
||||
```bash
|
||||
python scripts/generate_meta_info_pairdata.py --input datasets/DF2K/DIV2K_train_HR_sub datasets/DF2K/DIV2K_train_LR_bicubic_X4_sub --meta_info datasets/DF2K/meta_info/meta_info_DIV2K_sub_pair.txt
|
||||
```
|
||||
|
||||
**2. 下载预训练模型**
|
||||
|
||||
下载预先训练的模型到 `experiments/pretrained_models` 目录下。
|
||||
|
||||
- *RealESRGAN_x4plus.pth*:
|
||||
```bash
|
||||
wget https://github.com/xinntao/Real-ESRGAN/releases/download/v0.1.0/RealESRGAN_x4plus.pth -P experiments/pretrained_models
|
||||
```
|
||||
|
||||
- *RealESRGAN_x4plus_netD.pth*:
|
||||
```bash
|
||||
wget https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.3/RealESRGAN_x4plus_netD.pth -P experiments/pretrained_models
|
||||
```
|
||||
|
||||
**3. 微调**
|
||||
|
||||
修改选项文件 [options/finetune_realesrgan_x4plus_pairdata.yml](options/finetune_realesrgan_x4plus_pairdata.yml) ，特别是 `datasets` 部分：
|
||||
|
||||
```yml
|
||||
train:
|
||||
name: DIV2K
|
||||
type: RealESRGANPairedDataset
|
||||
dataroot_gt: datasets/DF2K # 修改为你的 gt folder 文件夹根目录
|
||||
dataroot_lq: datasets/DF2K # 修改为你的 lq folder 文件夹根目录
|
||||
meta_info: datasets/DF2K/meta_info/meta_info_DIV2K_sub_pair.txt # 修改为你自己生成的元信息txt
|
||||
io_backend:
|
||||
type: disk
|
||||
```
|
||||
|
||||
我们使用4个GPU进行训练。还可以使用参数 `--auto_resume` 在必要时自动恢复训练。
|
||||
|
||||
```bash
|
||||
CUDA_VISIBLE_DEVICES=0,1,2,3 \
|
||||
python -m torch.distributed.launch --nproc_per_node=4 --master_port=4321 realesrgan/train.py -opt options/finetune_realesrgan_x4plus_pairdata.yml --launcher pytorch --auto_resume
|
||||
```
|
||||
|
||||
用 **1个GPU** 训练：
|
||||
```bash
|
||||
python realesrgan/train.py -opt options/finetune_realesrgan_x4plus_pairdata.yml --auto_resume
|
||||
```
|
||||
@@ -1,5 +1,7 @@
|
||||
# Comparisons among different anime models
|
||||
|
||||
[English](anime_comparisons.md) **|** [简体中文](anime_comparisons_CN.md)
|
||||
|
||||
## Update News
|
||||
|
||||
- 2022/04/24: Release **AnimeVideo-v3**. We have made the following improvements:
|
||||
@@ -15,14 +17,13 @@ We have compared our RealESRGAN-AnimeVideo-v3 with the following methods.
|
||||
Our RealESRGAN-AnimeVideo-v3 can achieve better results with faster inference speed.
|
||||
|
||||
- [waifu2x](https://github.com/nihui/waifu2x-ncnn-vulkan) with the hyperparameters: `tile=0`, `noiselevel=2`
|
||||
|
||||
- [Real-CUGAN](https://github.com/bilibili/ailab/tree/main/Real-CUGAN):
|
||||
we use the [20220227](https://github.com/bilibili/ailab/releases/tag/Real-CUGAN-add-faster-low-memory-mode) version, the hyperparameters are: `cache_mode=0`, `tile=0`, `alpha=1`.
|
||||
- [Real-CUGAN](https://github.com/bilibili/ailab/tree/main/Real-CUGAN): we use the [20220227](https://github.com/bilibili/ailab/releases/tag/Real-CUGAN-add-faster-low-memory-mode) version, the hyperparameters are: `cache_mode=0`, `tile=0`, `alpha=1`.
|
||||
- our RealESRGAN-AnimeVideo-v3
|
||||
|
||||
## Results
|
||||
|
||||
You may need to **zoom in** for comparing details, or **click the image** to see in the full size.
|
||||
You may need to **zoom in** for comparing details, or **click the image** to see in the full size. Please note that the images
|
||||
in the table below are the resized and cropped patches from the original images, you can download the original inputs and outputs from [Google Drive](https://drive.google.com/drive/folders/1bc_Hje1Nqop9NDkUvci2VACSjL7HZMRp?usp=sharing) .
|
||||
|
||||
**More natural results, better background restoration**
|
||||
| Input | waifu2x | Real-CUGAN | RealESRGAN<br>AnimeVideo-v3 |
|
||||
|
||||
68
docs/anime_comparisons_CN.md
Normal file
68
docs/anime_comparisons_CN.md
Normal file
@@ -0,0 +1,68 @@
|
||||
# 动漫视频模型比较
|
||||
|
||||
[English](anime_comparisons.md) **|** [简体中文](anime_comparisons_CN.md)
|
||||
|
||||
## 更新
|
||||
|
||||
- 2022/04/24: 发布 **AnimeVideo-v3**. 主要做了以下更新：
|
||||
- **更自然**
|
||||
- **更少瑕疵**
|
||||
- **颜色保持得更好**
|
||||
- **更好的纹理恢复**
|
||||
- **虚化背景处理**
|
||||
|
||||
## 比较
|
||||
|
||||
我们将 RealESRGAN-AnimeVideo-v3 与以下方法进行了比较。我们的 RealESRGAN-AnimeVideo-v3 可以以更快的推理速度获得更好的结果。
|
||||
|
||||
- [waifu2x](https://github.com/nihui/waifu2x-ncnn-vulkan). 超参数: `tile=0`, `noiselevel=2`
|
||||
- [Real-CUGAN](https://github.com/bilibili/ailab/tree/main/Real-CUGAN): 我们使用了[20220227](https://github.com/bilibili/ailab/releases/tag/Real-CUGAN-add-faster-low-memory-mode)版本, 超参: `cache_mode=0`, `tile=0`, `alpha=1`.
|
||||
- 我们的 RealESRGAN-AnimeVideo-v3
|
||||
|
||||
## 结果
|
||||
|
||||
您可能需要**放大**以比较详细信息, 或者**单击图像**以查看完整尺寸。 请注意下面表格的图片是从原图里裁剪patch并且resize后的结果，您可以从
|
||||
[Google Drive](https://drive.google.com/drive/folders/1bc_Hje1Nqop9NDkUvci2VACSjL7HZMRp?usp=sharing) 里下载原始的输入和输出。
|
||||
|
||||
**更自然的结果，更好的虚化背景恢复**
|
||||
|
||||
| 输入 | waifu2x | Real-CUGAN | RealESRGAN<br>AnimeVideo-v3 |
|
||||
| :---: | :---: | :---: | :---: |
|
||||
| |  |  |  |
|
||||
| |  |  |  |
|
||||
| |  |  |  |
|
||||
|
||||
**更少瑕疵，更好的细节纹理**
|
||||
|
||||
| 输入 | waifu2x | Real-CUGAN | RealESRGAN<br>AnimeVideo-v3 |
|
||||
| :---: | :---: | :---: | :---: |
|
||||
| |  |  |  |
|
||||
| |  |  |  |
|
||||
| |  |  |  |
|
||||
| |  |  |  |
|
||||
|
||||
**其他更好的结果**
|
||||
|
||||
| 输入 | waifu2x | Real-CUGAN | RealESRGAN<br>AnimeVideo-v3 |
|
||||
| :---: | :---: | :---: | :---: |
|
||||
| |  |  |  |
|
||||
| |  |  |  |
|
||||
|  |   |   |   |
|
||||
| |  |  |  |
|
||||
| |  |  |  |
|
||||
|
||||
## 推理速度比较
|
||||
|
||||
### PyTorch
|
||||
|
||||
请注意，我们只报告了**模型推理**的时间, 而忽略了读写硬盘的时间.
|
||||
|
||||
| GPU | 输入尺寸 | waifu2x | Real-CUGAN | RealESRGAN-AnimeVideo-v3
|
||||
| :---: | :---: | :---: | :---: | :---: |
|
||||
| V100 | 1921 x 1080 | - | 3.4 fps | **10.0** fps |
|
||||
| V100 | 1280 x 720 | - | 7.2 fps | **22.6** fps |
|
||||
| V100 | 640 x 480 | - | 24.4 fps | **65.9** fps |
|
||||
|
||||
### ncnn
|
||||
|
||||
- [ ] TODO
|
||||
@@ -2,12 +2,11 @@
|
||||
|
||||
:white_check_mark: We add [*RealESRGAN_x4plus_anime_6B.pth*](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.4/RealESRGAN_x4plus_anime_6B.pth), which is optimized for **anime** images with much smaller model size.
|
||||
|
||||
- [Anime Model](#anime-model)
|
||||
- [How to Use](#how-to-use)
|
||||
- [PyTorch Inference](#pytorch-inference)
|
||||
- [ncnn Executable File](#ncnn-executable-file)
|
||||
- [Comparisons with waifu2x](#comparisons-with-waifu2x)
|
||||
- [Comparisons with Sliding Bars](#comparisons-with-sliding-bars)
|
||||
- [How to Use](#how-to-use)
|
||||
- [PyTorch Inference](#pytorch-inference)
|
||||
- [ncnn Executable File](#ncnn-executable-file)
|
||||
- [Comparisons with waifu2x](#comparisons-with-waifu2x)
|
||||
- [Comparisons with Sliding Bars](#comparisons-with-sliding-bars)
|
||||
|
||||
<p align="center">
|
||||
<img src="https://raw.githubusercontent.com/xinntao/public-figures/master/Real-ESRGAN/cmp_realesrgan_anime_1.png">
|
||||
@@ -25,7 +24,7 @@ Pre-trained models: [RealESRGAN_x4plus_anime_6B](https://github.com/xinntao/Real
|
||||
|
||||
```bash
|
||||
# download model
|
||||
wget https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.4/RealESRGAN_x4plus_anime_6B.pth -P experiments/pretrained_models
|
||||
wget https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.4/RealESRGAN_x4plus_anime_6B.pth -P weights
|
||||
# inference
|
||||
python inference_realesrgan.py -n RealESRGAN_x4plus_anime_6B -i inputs
|
||||
```
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# Anime Video Models
|
||||
|
||||
:white_check_mark: We add small models that are optimized for anime videos :-)<br>
|
||||
More comparisons can be found in [anime_comparisons.md](docs/anime_comparisons.md)
|
||||
More comparisons can be found in [anime_comparisons.md](anime_comparisons.md)
|
||||
|
||||
- [How to Use](#how-to-use)
|
||||
- [PyTorch Inference](#pytorch-inference)
|
||||
@@ -34,9 +34,22 @@ The following are some demos (best view in the full screen mode).
|
||||
|
||||
```bash
|
||||
# download model
|
||||
wget https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/realesr-animevideov3.pth -P realesrgan/weights
|
||||
# inference
|
||||
python inference_realesrgan_video.py -i inputs/video/onepiece_demo.mp4 -n realesr-animevideov3 -s 2 --suffix outx2
|
||||
wget https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/realesr-animevideov3.pth -P weights
|
||||
# single gpu and single process inference
|
||||
CUDA_VISIBLE_DEVICES=0 python inference_realesrgan_video.py -i inputs/video/onepiece_demo.mp4 -n realesr-animevideov3 -s 2 --suffix outx2
|
||||
# single gpu and multi process inference (you can use multi-processing to improve GPU utilization)
|
||||
CUDA_VISIBLE_DEVICES=0 python inference_realesrgan_video.py -i inputs/video/onepiece_demo.mp4 -n realesr-animevideov3 -s 2 --suffix outx2 --num_process_per_gpu 2
|
||||
# multi gpu and multi process inference
|
||||
CUDA_VISIBLE_DEVICES=0,1,2,3 python inference_realesrgan_video.py -i inputs/video/onepiece_demo.mp4 -n realesr-animevideov3 -s 2 --suffix outx2 --num_process_per_gpu 2
|
||||
```
|
||||
|
||||
```console
|
||||
Usage:
|
||||
--num_process_per_gpu The total number of process is num_gpu * num_process_per_gpu. The bottleneck of
|
||||
the program lies on the IO, so the GPUs are usually not fully utilized. To alleviate
|
||||
this issue, you can use multi-processing by setting this parameter. As long as it
|
||||
does not exceed the CUDA memory
|
||||
--extract_frame_first If you encounter ffmpeg error when using multi-processing, you can turn this option on.
|
||||
```
|
||||
|
||||
### NCNN Executable File
|
||||
|
||||
@@ -14,6 +14,7 @@
|
||||
| [RealESRGAN_x2plus](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.1/RealESRGAN_x2plus.pth) | X2 | X2 model for general images |
|
||||
| [RealESRNet_x4plus](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.1.1/RealESRNet_x4plus.pth) | X4 | X4 model with MSE loss (over-smooth effects) |
|
||||
| [official ESRGAN_x4](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.1.1/ESRGAN_SRx4_DF2KOST_official-ff704c30.pth) | X4 | official ESRGAN model |
|
||||
| [realesr-general-x4v3](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/realesr-general-x4v3.pth) | X4 (can also be used for X1, X2, X3) | A tiny small model (consume much fewer GPU memory and time); not too strong deblur and denoise capacity |
|
||||
|
||||
The following models are **discriminators**, which are usually used for fine-tuning.
|
||||
|
||||
|
||||
@@ -3,6 +3,7 @@ import cv2
|
||||
import glob
|
||||
import os
|
||||
from basicsr.archs.rrdbnet_arch import RRDBNet
|
||||
from basicsr.utils.download_util import load_file_from_url
|
||||
|
||||
from realesrgan import RealESRGANer
|
||||
from realesrgan.archs.srvgg_arch import SRVGGNetCompact
|
||||
@@ -19,9 +20,18 @@ def main():
|
||||
type=str,
|
||||
default='RealESRGAN_x4plus',
|
||||
help=('Model names: RealESRGAN_x4plus | RealESRNet_x4plus | RealESRGAN_x4plus_anime_6B | RealESRGAN_x2plus | '
|
||||
'realesr-animevideov3'))
|
||||
'realesr-animevideov3 | realesr-general-x4v3'))
|
||||
parser.add_argument('-o', '--output', type=str, default='results', help='Output folder')
|
||||
parser.add_argument(
|
||||
'-dn',
|
||||
'--denoise_strength',
|
||||
type=float,
|
||||
default=0.5,
|
||||
help=('Denoise strength. 0 for weak denoise (keep noise), 1 for strong denoise ability. '
|
||||
'Only used for the realesr-general-x4v3 model'))
|
||||
parser.add_argument('-s', '--outscale', type=float, default=4, help='The final upsampling scale of the image')
|
||||
parser.add_argument(
|
||||
'--model_path', type=str, default=None, help='[Option] Model path. Usually, you do not need to specify it')
|
||||
parser.add_argument('--suffix', type=str, default='out', help='Suffix of the restored image')
|
||||
parser.add_argument('-t', '--tile', type=int, default=0, help='Tile size, 0 for no tile during testing')
|
||||
parser.add_argument('--tile_pad', type=int, default=10, help='Tile padding')
|
||||
@@ -39,39 +49,71 @@ def main():
|
||||
type=str,
|
||||
default='auto',
|
||||
help='Image extension. Options: auto | jpg | png, auto means using the same extension as inputs')
|
||||
parser.add_argument(
|
||||
'-g', '--gpu-id', type=int, default=None, help='gpu device to use (default=None) can be 0,1,2 for multi-gpu')
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# determine models according to model names
|
||||
args.model_name = args.model_name.split('.')[0]
|
||||
if args.model_name in ['RealESRGAN_x4plus', 'RealESRNet_x4plus']: # x4 RRDBNet model
|
||||
if args.model_name == 'RealESRGAN_x4plus': # x4 RRDBNet model
|
||||
model = RRDBNet(num_in_ch=3, num_out_ch=3, num_feat=64, num_block=23, num_grow_ch=32, scale=4)
|
||||
netscale = 4
|
||||
elif args.model_name in ['RealESRGAN_x4plus_anime_6B']: # x4 RRDBNet model with 6 blocks
|
||||
file_url = ['https://github.com/xinntao/Real-ESRGAN/releases/download/v0.1.0/RealESRGAN_x4plus.pth']
|
||||
elif args.model_name == 'RealESRNet_x4plus': # x4 RRDBNet model
|
||||
model = RRDBNet(num_in_ch=3, num_out_ch=3, num_feat=64, num_block=23, num_grow_ch=32, scale=4)
|
||||
netscale = 4
|
||||
file_url = ['https://github.com/xinntao/Real-ESRGAN/releases/download/v0.1.1/RealESRNet_x4plus.pth']
|
||||
elif args.model_name == 'RealESRGAN_x4plus_anime_6B': # x4 RRDBNet model with 6 blocks
|
||||
model = RRDBNet(num_in_ch=3, num_out_ch=3, num_feat=64, num_block=6, num_grow_ch=32, scale=4)
|
||||
netscale = 4
|
||||
elif args.model_name in ['RealESRGAN_x2plus']: # x2 RRDBNet model
|
||||
file_url = ['https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.4/RealESRGAN_x4plus_anime_6B.pth']
|
||||
elif args.model_name == 'RealESRGAN_x2plus': # x2 RRDBNet model
|
||||
model = RRDBNet(num_in_ch=3, num_out_ch=3, num_feat=64, num_block=23, num_grow_ch=32, scale=2)
|
||||
netscale = 2
|
||||
elif args.model_name in ['realesr-animevideov3']: # x4 VGG-style model (XS size)
|
||||
file_url = ['https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.1/RealESRGAN_x2plus.pth']
|
||||
elif args.model_name == 'realesr-animevideov3': # x4 VGG-style model (XS size)
|
||||
model = SRVGGNetCompact(num_in_ch=3, num_out_ch=3, num_feat=64, num_conv=16, upscale=4, act_type='prelu')
|
||||
netscale = 4
|
||||
file_url = ['https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/realesr-animevideov3.pth']
|
||||
elif args.model_name == 'realesr-general-x4v3': # x4 VGG-style model (S size)
|
||||
model = SRVGGNetCompact(num_in_ch=3, num_out_ch=3, num_feat=64, num_conv=32, upscale=4, act_type='prelu')
|
||||
netscale = 4
|
||||
file_url = [
|
||||
'https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/realesr-general-wdn-x4v3.pth',
|
||||
'https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/realesr-general-x4v3.pth'
|
||||
]
|
||||
|
||||
# determine model paths
|
||||
model_path = os.path.join('experiments/pretrained_models', args.model_name + '.pth')
|
||||
if not os.path.isfile(model_path):
|
||||
model_path = os.path.join('realesrgan/weights', args.model_name + '.pth')
|
||||
if not os.path.isfile(model_path):
|
||||
raise ValueError(f'Model {args.model_name} does not exist.')
|
||||
if args.model_path is not None:
|
||||
model_path = args.model_path
|
||||
else:
|
||||
model_path = os.path.join('weights', args.model_name + '.pth')
|
||||
if not os.path.isfile(model_path):
|
||||
ROOT_DIR = os.path.dirname(os.path.abspath(__file__))
|
||||
for url in file_url:
|
||||
# model_path will be updated
|
||||
model_path = load_file_from_url(
|
||||
url=url, model_dir=os.path.join(ROOT_DIR, 'weights'), progress=True, file_name=None)
|
||||
|
||||
# use dni to control the denoise strength
|
||||
dni_weight = None
|
||||
if args.model_name == 'realesr-general-x4v3' and args.denoise_strength != 1:
|
||||
wdn_model_path = model_path.replace('realesr-general-x4v3', 'realesr-general-wdn-x4v3')
|
||||
model_path = [model_path, wdn_model_path]
|
||||
dni_weight = [args.denoise_strength, 1 - args.denoise_strength]
|
||||
|
||||
# restorer
|
||||
upsampler = RealESRGANer(
|
||||
scale=netscale,
|
||||
model_path=model_path,
|
||||
dni_weight=dni_weight,
|
||||
model=model,
|
||||
tile=args.tile,
|
||||
tile_pad=args.tile_pad,
|
||||
pre_pad=args.pre_pad,
|
||||
half=not args.fp32)
|
||||
half=not args.fp32,
|
||||
gpu_id=args.gpu_id)
|
||||
|
||||
if args.face_enhance: # Use GFPGAN for face enhancement
|
||||
from gfpgan import GFPGANer
|
||||
|
||||
@@ -1,17 +1,327 @@
|
||||
import argparse
|
||||
import cv2
|
||||
import glob
|
||||
import mimetypes
|
||||
import numpy as np
|
||||
import os
|
||||
import queue
|
||||
import shutil
|
||||
import subprocess
|
||||
import torch
|
||||
from basicsr.archs.rrdbnet_arch import RRDBNet
|
||||
from basicsr.utils.logger import AvgTimer
|
||||
from basicsr.utils.download_util import load_file_from_url
|
||||
from os import path as osp
|
||||
from tqdm import tqdm
|
||||
|
||||
from realesrgan import IOConsumer, PrefetchReader, RealESRGANer
|
||||
from realesrgan import RealESRGANer
|
||||
from realesrgan.archs.srvgg_arch import SRVGGNetCompact
|
||||
|
||||
try:
|
||||
import ffmpeg
|
||||
except ImportError:
|
||||
import pip
|
||||
pip.main(['install', '--user', 'ffmpeg-python'])
|
||||
import ffmpeg
|
||||
|
||||
|
||||
def get_video_meta_info(video_path):
|
||||
ret = {}
|
||||
probe = ffmpeg.probe(video_path)
|
||||
video_streams = [stream for stream in probe['streams'] if stream['codec_type'] == 'video']
|
||||
has_audio = any(stream['codec_type'] == 'audio' for stream in probe['streams'])
|
||||
ret['width'] = video_streams[0]['width']
|
||||
ret['height'] = video_streams[0]['height']
|
||||
ret['fps'] = eval(video_streams[0]['avg_frame_rate'])
|
||||
ret['audio'] = ffmpeg.input(video_path).audio if has_audio else None
|
||||
ret['nb_frames'] = int(video_streams[0]['nb_frames'])
|
||||
return ret
|
||||
|
||||
|
||||
def get_sub_video(args, num_process, process_idx):
|
||||
if num_process == 1:
|
||||
return args.input
|
||||
meta = get_video_meta_info(args.input)
|
||||
duration = int(meta['nb_frames'] / meta['fps'])
|
||||
part_time = duration // num_process
|
||||
print(f'duration: {duration}, part_time: {part_time}')
|
||||
os.makedirs(osp.join(args.output, f'{args.video_name}_inp_tmp_videos'), exist_ok=True)
|
||||
out_path = osp.join(args.output, f'{args.video_name}_inp_tmp_videos', f'{process_idx:03d}.mp4')
|
||||
cmd = [
|
||||
args.ffmpeg_bin, f'-i {args.input}', '-ss', f'{part_time * process_idx}',
|
||||
f'-to {part_time * (process_idx + 1)}' if process_idx != num_process - 1 else '', '-async 1', out_path, '-y'
|
||||
]
|
||||
print(' '.join(cmd))
|
||||
subprocess.call(' '.join(cmd), shell=True)
|
||||
return out_path
|
||||
|
||||
|
||||
class Reader:
|
||||
|
||||
def __init__(self, args, total_workers=1, worker_idx=0):
|
||||
self.args = args
|
||||
input_type = mimetypes.guess_type(args.input)[0]
|
||||
self.input_type = 'folder' if input_type is None else input_type
|
||||
self.paths = [] # for image&folder type
|
||||
self.audio = None
|
||||
self.input_fps = None
|
||||
if self.input_type.startswith('video'):
|
||||
video_path = get_sub_video(args, total_workers, worker_idx)
|
||||
self.stream_reader = (
|
||||
ffmpeg.input(video_path).output('pipe:', format='rawvideo', pix_fmt='bgr24',
|
||||
loglevel='error').run_async(
|
||||
pipe_stdin=True, pipe_stdout=True, cmd=args.ffmpeg_bin))
|
||||
meta = get_video_meta_info(video_path)
|
||||
self.width = meta['width']
|
||||
self.height = meta['height']
|
||||
self.input_fps = meta['fps']
|
||||
self.audio = meta['audio']
|
||||
self.nb_frames = meta['nb_frames']
|
||||
|
||||
else:
|
||||
if self.input_type.startswith('image'):
|
||||
self.paths = [args.input]
|
||||
else:
|
||||
paths = sorted(glob.glob(os.path.join(args.input, '*')))
|
||||
tot_frames = len(paths)
|
||||
num_frame_per_worker = tot_frames // total_workers + (1 if tot_frames % total_workers else 0)
|
||||
self.paths = paths[num_frame_per_worker * worker_idx:num_frame_per_worker * (worker_idx + 1)]
|
||||
|
||||
self.nb_frames = len(self.paths)
|
||||
assert self.nb_frames > 0, 'empty folder'
|
||||
from PIL import Image
|
||||
tmp_img = Image.open(self.paths[0])
|
||||
self.width, self.height = tmp_img.size
|
||||
self.idx = 0
|
||||
|
||||
def get_resolution(self):
|
||||
return self.height, self.width
|
||||
|
||||
def get_fps(self):
|
||||
if self.args.fps is not None:
|
||||
return self.args.fps
|
||||
elif self.input_fps is not None:
|
||||
return self.input_fps
|
||||
return 24
|
||||
|
||||
def get_audio(self):
|
||||
return self.audio
|
||||
|
||||
def __len__(self):
|
||||
return self.nb_frames
|
||||
|
||||
def get_frame_from_stream(self):
|
||||
img_bytes = self.stream_reader.stdout.read(self.width * self.height * 3) # 3 bytes for one pixel
|
||||
if not img_bytes:
|
||||
return None
|
||||
img = np.frombuffer(img_bytes, np.uint8).reshape([self.height, self.width, 3])
|
||||
return img
|
||||
|
||||
def get_frame_from_list(self):
|
||||
if self.idx >= self.nb_frames:
|
||||
return None
|
||||
img = cv2.imread(self.paths[self.idx])
|
||||
self.idx += 1
|
||||
return img
|
||||
|
||||
def get_frame(self):
|
||||
if self.input_type.startswith('video'):
|
||||
return self.get_frame_from_stream()
|
||||
else:
|
||||
return self.get_frame_from_list()
|
||||
|
||||
def close(self):
|
||||
if self.input_type.startswith('video'):
|
||||
self.stream_reader.stdin.close()
|
||||
self.stream_reader.wait()
|
||||
|
||||
|
||||
class Writer:
|
||||
|
||||
def __init__(self, args, audio, height, width, video_save_path, fps):
|
||||
out_width, out_height = int(width * args.outscale), int(height * args.outscale)
|
||||
if out_height > 2160:
|
||||
print('You are generating video that is larger than 4K, which will be very slow due to IO speed.',
|
||||
'We highly recommend to decrease the outscale(aka, -s).')
|
||||
|
||||
if audio is not None:
|
||||
self.stream_writer = (
|
||||
ffmpeg.input('pipe:', format='rawvideo', pix_fmt='bgr24', s=f'{out_width}x{out_height}',
|
||||
framerate=fps).output(
|
||||
audio,
|
||||
video_save_path,
|
||||
pix_fmt='yuv420p',
|
||||
vcodec='libx264',
|
||||
loglevel='error',
|
||||
acodec='copy').overwrite_output().run_async(
|
||||
pipe_stdin=True, pipe_stdout=True, cmd=args.ffmpeg_bin))
|
||||
else:
|
||||
self.stream_writer = (
|
||||
ffmpeg.input('pipe:', format='rawvideo', pix_fmt='bgr24', s=f'{out_width}x{out_height}',
|
||||
framerate=fps).output(
|
||||
video_save_path, pix_fmt='yuv420p', vcodec='libx264',
|
||||
loglevel='error').overwrite_output().run_async(
|
||||
pipe_stdin=True, pipe_stdout=True, cmd=args.ffmpeg_bin))
|
||||
|
||||
def write_frame(self, frame):
|
||||
frame = frame.astype(np.uint8).tobytes()
|
||||
self.stream_writer.stdin.write(frame)
|
||||
|
||||
def close(self):
|
||||
self.stream_writer.stdin.close()
|
||||
self.stream_writer.wait()
|
||||
|
||||
|
||||
def inference_video(args, video_save_path, device=None, total_workers=1, worker_idx=0):
|
||||
# ---------------------- determine models according to model names ---------------------- #
|
||||
args.model_name = args.model_name.split('.pth')[0]
|
||||
if args.model_name == 'RealESRGAN_x4plus': # x4 RRDBNet model
|
||||
model = RRDBNet(num_in_ch=3, num_out_ch=3, num_feat=64, num_block=23, num_grow_ch=32, scale=4)
|
||||
netscale = 4
|
||||
file_url = ['https://github.com/xinntao/Real-ESRGAN/releases/download/v0.1.0/RealESRGAN_x4plus.pth']
|
||||
elif args.model_name == 'RealESRNet_x4plus': # x4 RRDBNet model
|
||||
model = RRDBNet(num_in_ch=3, num_out_ch=3, num_feat=64, num_block=23, num_grow_ch=32, scale=4)
|
||||
netscale = 4
|
||||
file_url = ['https://github.com/xinntao/Real-ESRGAN/releases/download/v0.1.1/RealESRNet_x4plus.pth']
|
||||
elif args.model_name == 'RealESRGAN_x4plus_anime_6B': # x4 RRDBNet model with 6 blocks
|
||||
model = RRDBNet(num_in_ch=3, num_out_ch=3, num_feat=64, num_block=6, num_grow_ch=32, scale=4)
|
||||
netscale = 4
|
||||
file_url = ['https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.4/RealESRGAN_x4plus_anime_6B.pth']
|
||||
elif args.model_name == 'RealESRGAN_x2plus': # x2 RRDBNet model
|
||||
model = RRDBNet(num_in_ch=3, num_out_ch=3, num_feat=64, num_block=23, num_grow_ch=32, scale=2)
|
||||
netscale = 2
|
||||
file_url = ['https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.1/RealESRGAN_x2plus.pth']
|
||||
elif args.model_name == 'realesr-animevideov3': # x4 VGG-style model (XS size)
|
||||
model = SRVGGNetCompact(num_in_ch=3, num_out_ch=3, num_feat=64, num_conv=16, upscale=4, act_type='prelu')
|
||||
netscale = 4
|
||||
file_url = ['https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/realesr-animevideov3.pth']
|
||||
elif args.model_name == 'realesr-general-x4v3': # x4 VGG-style model (S size)
|
||||
model = SRVGGNetCompact(num_in_ch=3, num_out_ch=3, num_feat=64, num_conv=32, upscale=4, act_type='prelu')
|
||||
netscale = 4
|
||||
file_url = [
|
||||
'https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/realesr-general-wdn-x4v3.pth',
|
||||
'https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/realesr-general-x4v3.pth'
|
||||
]
|
||||
|
||||
# ---------------------- determine model paths ---------------------- #
|
||||
model_path = os.path.join('weights', args.model_name + '.pth')
|
||||
if not os.path.isfile(model_path):
|
||||
ROOT_DIR = os.path.dirname(os.path.abspath(__file__))
|
||||
for url in file_url:
|
||||
# model_path will be updated
|
||||
model_path = load_file_from_url(
|
||||
url=url, model_dir=os.path.join(ROOT_DIR, 'weights'), progress=True, file_name=None)
|
||||
|
||||
# use dni to control the denoise strength
|
||||
dni_weight = None
|
||||
if args.model_name == 'realesr-general-x4v3' and args.denoise_strength != 1:
|
||||
wdn_model_path = model_path.replace('realesr-general-x4v3', 'realesr-general-wdn-x4v3')
|
||||
model_path = [model_path, wdn_model_path]
|
||||
dni_weight = [args.denoise_strength, 1 - args.denoise_strength]
|
||||
|
||||
# restorer
|
||||
upsampler = RealESRGANer(
|
||||
scale=netscale,
|
||||
model_path=model_path,
|
||||
dni_weight=dni_weight,
|
||||
model=model,
|
||||
tile=args.tile,
|
||||
tile_pad=args.tile_pad,
|
||||
pre_pad=args.pre_pad,
|
||||
half=not args.fp32,
|
||||
device=device,
|
||||
)
|
||||
|
||||
if 'anime' in args.model_name and args.face_enhance:
|
||||
print('face_enhance is not supported in anime models, we turned this option off for you. '
|
||||
'if you insist on turning it on, please manually comment the relevant lines of code.')
|
||||
args.face_enhance = False
|
||||
|
||||
if args.face_enhance: # Use GFPGAN for face enhancement
|
||||
from gfpgan import GFPGANer
|
||||
face_enhancer = GFPGANer(
|
||||
model_path='https://github.com/TencentARC/GFPGAN/releases/download/v1.3.0/GFPGANv1.3.pth',
|
||||
upscale=args.outscale,
|
||||
arch='clean',
|
||||
channel_multiplier=2,
|
||||
bg_upsampler=upsampler) # TODO support custom device
|
||||
else:
|
||||
face_enhancer = None
|
||||
|
||||
reader = Reader(args, total_workers, worker_idx)
|
||||
audio = reader.get_audio()
|
||||
height, width = reader.get_resolution()
|
||||
fps = reader.get_fps()
|
||||
writer = Writer(args, audio, height, width, video_save_path, fps)
|
||||
|
||||
pbar = tqdm(total=len(reader), unit='frame', desc='inference')
|
||||
while True:
|
||||
img = reader.get_frame()
|
||||
if img is None:
|
||||
break
|
||||
|
||||
try:
|
||||
if args.face_enhance:
|
||||
_, _, output = face_enhancer.enhance(img, has_aligned=False, only_center_face=False, paste_back=True)
|
||||
else:
|
||||
output, _ = upsampler.enhance(img, outscale=args.outscale)
|
||||
except RuntimeError as error:
|
||||
print('Error', error)
|
||||
print('If you encounter CUDA out of memory, try to set --tile with a smaller number.')
|
||||
else:
|
||||
writer.write_frame(output)
|
||||
|
||||
torch.cuda.synchronize(device)
|
||||
pbar.update(1)
|
||||
|
||||
reader.close()
|
||||
writer.close()
|
||||
|
||||
|
||||
def run(args):
|
||||
args.video_name = osp.splitext(os.path.basename(args.input))[0]
|
||||
video_save_path = osp.join(args.output, f'{args.video_name}_{args.suffix}.mp4')
|
||||
|
||||
if args.extract_frame_first:
|
||||
tmp_frames_folder = osp.join(args.output, f'{args.video_name}_inp_tmp_frames')
|
||||
os.makedirs(tmp_frames_folder, exist_ok=True)
|
||||
os.system(f'ffmpeg -i {args.input} -qscale:v 1 -qmin 1 -qmax 1 -vsync 0 {tmp_frames_folder}/frame%08d.png')
|
||||
args.input = tmp_frames_folder
|
||||
|
||||
num_gpus = torch.cuda.device_count()
|
||||
num_process = num_gpus * args.num_process_per_gpu
|
||||
if num_process == 1:
|
||||
inference_video(args, video_save_path)
|
||||
return
|
||||
|
||||
ctx = torch.multiprocessing.get_context('spawn')
|
||||
pool = ctx.Pool(num_process)
|
||||
os.makedirs(osp.join(args.output, f'{args.video_name}_out_tmp_videos'), exist_ok=True)
|
||||
pbar = tqdm(total=num_process, unit='sub_video', desc='inference')
|
||||
for i in range(num_process):
|
||||
sub_video_save_path = osp.join(args.output, f'{args.video_name}_out_tmp_videos', f'{i:03d}.mp4')
|
||||
pool.apply_async(
|
||||
inference_video,
|
||||
args=(args, sub_video_save_path, torch.device(i % num_gpus), num_process, i),
|
||||
callback=lambda arg: pbar.update(1))
|
||||
pool.close()
|
||||
pool.join()
|
||||
|
||||
# combine sub videos
|
||||
# prepare vidlist.txt
|
||||
with open(f'{args.output}/{args.video_name}_vidlist.txt', 'w') as f:
|
||||
for i in range(num_process):
|
||||
f.write(f'file \'{args.video_name}_out_tmp_videos/{i:03d}.mp4\'\n')
|
||||
|
||||
cmd = [
|
||||
args.ffmpeg_bin, '-f', 'concat', '-safe', '0', '-i', f'{args.output}/{args.video_name}_vidlist.txt', '-c',
|
||||
'copy', f'{video_save_path}'
|
||||
]
|
||||
print(' '.join(cmd))
|
||||
subprocess.call(cmd)
|
||||
shutil.rmtree(osp.join(args.output, f'{args.video_name}_out_tmp_videos'))
|
||||
if osp.exists(osp.join(args.output, f'{args.video_name}_inp_tmp_videos')):
|
||||
shutil.rmtree(osp.join(args.output, f'{args.video_name}_inp_tmp_videos'))
|
||||
os.remove(f'{args.output}/{args.video_name}_vidlist.txt')
|
||||
|
||||
|
||||
def main():
|
||||
"""Inference demo for Real-ESRGAN.
|
||||
@@ -26,9 +336,16 @@ def main():
|
||||
type=str,
|
||||
default='realesr-animevideov3',
|
||||
help=('Model names: realesr-animevideov3 | RealESRGAN_x4plus_anime_6B | RealESRGAN_x4plus | RealESRNet_x4plus |'
|
||||
' RealESRGAN_x2plus | '
|
||||
' RealESRGAN_x2plus | realesr-general-x4v3'
|
||||
'Default:realesr-animevideov3'))
|
||||
parser.add_argument('-o', '--output', type=str, default='results', help='Output folder')
|
||||
parser.add_argument(
|
||||
'-dn',
|
||||
'--denoise_strength',
|
||||
type=float,
|
||||
default=0.5,
|
||||
help=('Denoise strength. 0 for weak denoise (keep noise), 1 for strong denoise ability. '
|
||||
'Only used for the realesr-general-x4v3 model'))
|
||||
parser.add_argument('-s', '--outscale', type=float, default=4, help='The final upsampling scale of the image')
|
||||
parser.add_argument('--suffix', type=str, default='out', help='Suffix of the restored video')
|
||||
parser.add_argument('-t', '--tile', type=int, default=0, help='Tile size, 0 for no tile during testing')
|
||||
@@ -38,7 +355,9 @@ def main():
|
||||
parser.add_argument(
|
||||
'--fp32', action='store_true', help='Use fp32 precision during inference. Default: fp16 (half precision).')
|
||||
parser.add_argument('--fps', type=float, default=None, help='FPS of the output video')
|
||||
parser.add_argument('--consumer', type=int, default=4, help='Number of IO consumers')
|
||||
parser.add_argument('--ffmpeg_bin', type=str, default='ffmpeg', help='The path to ffmpeg')
|
||||
parser.add_argument('--extract_frame_first', action='store_true')
|
||||
parser.add_argument('--num_process_per_gpu', type=int, default=1)
|
||||
|
||||
parser.add_argument(
|
||||
'--alpha_upsampler',
|
||||
@@ -52,133 +371,27 @@ def main():
|
||||
help='Image extension. Options: auto | jpg | png, auto means using the same extension as inputs')
|
||||
args = parser.parse_args()
|
||||
|
||||
# ---------------------- determine models according to model names ---------------------- #
|
||||
args.model_name = args.model_name.split('.pth')[0]
|
||||
if args.model_name in ['RealESRGAN_x4plus', 'RealESRNet_x4plus']: # x4 RRDBNet model
|
||||
model = RRDBNet(num_in_ch=3, num_out_ch=3, num_feat=64, num_block=23, num_grow_ch=32, scale=4)
|
||||
netscale = 4
|
||||
elif args.model_name in ['RealESRGAN_x4plus_anime_6B']: # x4 RRDBNet model with 6 blocks
|
||||
model = RRDBNet(num_in_ch=3, num_out_ch=3, num_feat=64, num_block=6, num_grow_ch=32, scale=4)
|
||||
netscale = 4
|
||||
elif args.model_name in ['RealESRGAN_x2plus']: # x2 RRDBNet model
|
||||
model = RRDBNet(num_in_ch=3, num_out_ch=3, num_feat=64, num_block=23, num_grow_ch=32, scale=2)
|
||||
netscale = 2
|
||||
elif args.model_name in ['realesr-animevideov3']: # x4 VGG-style model (XS size)
|
||||
model = SRVGGNetCompact(num_in_ch=3, num_out_ch=3, num_feat=64, num_conv=16, upscale=4, act_type='prelu')
|
||||
netscale = 4
|
||||
|
||||
# ---------------------- determine model paths ---------------------- #
|
||||
model_path = os.path.join('experiments/pretrained_models', args.model_name + '.pth')
|
||||
if not os.path.isfile(model_path):
|
||||
model_path = os.path.join('realesrgan/weights', args.model_name + '.pth')
|
||||
if not os.path.isfile(model_path):
|
||||
raise ValueError(f'Model {args.model_name} does not exist.')
|
||||
|
||||
# restorer
|
||||
upsampler = RealESRGANer(
|
||||
scale=netscale,
|
||||
model_path=model_path,
|
||||
model=model,
|
||||
tile=args.tile,
|
||||
tile_pad=args.tile_pad,
|
||||
pre_pad=args.pre_pad,
|
||||
half=not args.fp32)
|
||||
|
||||
if args.face_enhance: # Use GFPGAN for face enhancement
|
||||
from gfpgan import GFPGANer
|
||||
face_enhancer = GFPGANer(
|
||||
model_path='https://github.com/TencentARC/GFPGAN/releases/download/v1.3.0/GFPGANv1.3.pth',
|
||||
upscale=args.outscale,
|
||||
arch='clean',
|
||||
channel_multiplier=2,
|
||||
bg_upsampler=upsampler)
|
||||
args.input = args.input.rstrip('/').rstrip('\\')
|
||||
os.makedirs(args.output, exist_ok=True)
|
||||
# for saving restored frames
|
||||
save_frame_folder = os.path.join(args.output, 'frames_tmpout')
|
||||
os.makedirs(save_frame_folder, exist_ok=True)
|
||||
|
||||
# input can be a video file / a folder of frames / an image
|
||||
if mimetypes.guess_type(args.input)[0].startswith('video'): # is a video file
|
||||
video_name = os.path.splitext(os.path.basename(args.input))[0]
|
||||
frame_folder = os.path.join('tmp_frames', video_name)
|
||||
os.makedirs(frame_folder, exist_ok=True)
|
||||
# use ffmpeg to extract frames
|
||||
os.system(f'ffmpeg -i {args.input} -qscale:v 1 -qmin 1 -qmax 1 -vsync 0 {frame_folder}/frame%08d.png')
|
||||
# get image path list
|
||||
paths = sorted(glob.glob(os.path.join(frame_folder, '*')))
|
||||
# get input video fps
|
||||
if args.fps is None:
|
||||
|
||||
import ffmpeg
|
||||
probe = ffmpeg.probe(args.input)
|
||||
video_streams = [stream for stream in probe['streams'] if stream['codec_type'] == 'video']
|
||||
args.fps = eval(video_streams[0]['avg_frame_rate'])
|
||||
elif mimetypes.guess_type(args.input)[0].startswith('image'): # is an image file
|
||||
paths = [args.input]
|
||||
video_name = 'video'
|
||||
if mimetypes.guess_type(args.input)[0] is not None and mimetypes.guess_type(args.input)[0].startswith('video'):
|
||||
is_video = True
|
||||
else:
|
||||
paths = sorted(glob.glob(os.path.join(args.input, '*')))
|
||||
video_name = 'video'
|
||||
is_video = False
|
||||
|
||||
timer = AvgTimer()
|
||||
timer.start()
|
||||
pbar = tqdm(total=len(paths), unit='frame', desc='inference')
|
||||
# set up prefetch reader
|
||||
reader = PrefetchReader(paths, num_prefetch_queue=4)
|
||||
reader.start()
|
||||
if is_video and args.input.endswith('.flv'):
|
||||
mp4_path = args.input.replace('.flv', '.mp4')
|
||||
os.system(f'ffmpeg -i {args.input} -codec copy {mp4_path}')
|
||||
args.input = mp4_path
|
||||
|
||||
que = queue.Queue()
|
||||
consumers = [IOConsumer(args, que, f'IO_{i}') for i in range(args.consumer)]
|
||||
for consumer in consumers:
|
||||
consumer.start()
|
||||
if args.extract_frame_first and not is_video:
|
||||
args.extract_frame_first = False
|
||||
|
||||
for idx, (path, img) in enumerate(zip(paths, reader)):
|
||||
imgname, extension = os.path.splitext(os.path.basename(path))
|
||||
if len(img.shape) == 3 and img.shape[2] == 4:
|
||||
img_mode = 'RGBA'
|
||||
else:
|
||||
img_mode = None
|
||||
run(args)
|
||||
|
||||
try:
|
||||
if args.face_enhance:
|
||||
_, _, output = face_enhancer.enhance(img, has_aligned=False, only_center_face=False, paste_back=True)
|
||||
else:
|
||||
output, _ = upsampler.enhance(img, outscale=args.outscale)
|
||||
except RuntimeError as error:
|
||||
print('Error', error)
|
||||
print('If you encounter CUDA out of memory, try to set --tile with a smaller number.')
|
||||
|
||||
else:
|
||||
if args.ext == 'auto':
|
||||
extension = extension[1:]
|
||||
else:
|
||||
extension = args.ext
|
||||
if img_mode == 'RGBA': # RGBA images should be saved in png format
|
||||
extension = 'png'
|
||||
save_path = os.path.join(save_frame_folder, f'{imgname}_out.{extension}')
|
||||
|
||||
que.put({'output': output, 'save_path': save_path})
|
||||
|
||||
pbar.update(1)
|
||||
torch.cuda.synchronize()
|
||||
timer.record()
|
||||
avg_fps = 1. / (timer.get_avg_time() + 1e-7)
|
||||
pbar.set_description(f'idx {idx}, fps {avg_fps:.2f}')
|
||||
|
||||
for _ in range(args.consumer):
|
||||
que.put('quit')
|
||||
for consumer in consumers:
|
||||
consumer.join()
|
||||
pbar.close()
|
||||
|
||||
# merge frames to video
|
||||
video_save_path = os.path.join(args.output, f'{video_name}_{args.suffix}.mp4')
|
||||
os.system(f'ffmpeg -r {args.fps} -i {save_frame_folder}/frame%08d_out.{extension} -i {args.input}'
|
||||
f' -map 0:v:0 -map 1:a:0 -c:a copy -c:v libx264 -r {args.fps} -pix_fmt yuv420p {video_save_path}')
|
||||
# delete tmp file
|
||||
shutil.rmtree(save_frame_folder)
|
||||
if os.path.isdir(frame_folder):
|
||||
shutil.rmtree(frame_folder)
|
||||
if args.extract_frame_first:
|
||||
tmp_frames_folder = osp.join(args.output, f'{args.video_name}_inp_tmp_frames')
|
||||
shutil.rmtree(tmp_frames_folder)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
|
||||
BIN
inputs/00017_gray.png
Normal file
BIN
inputs/00017_gray.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 68 KiB |
BIN
inputs/children-alpha.png
Normal file
BIN
inputs/children-alpha.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 268 KiB |
@@ -26,7 +26,17 @@ class RealESRGANer():
|
||||
half (float): Whether to use half precision during inference. Default: False.
|
||||
"""
|
||||
|
||||
def __init__(self, scale, model_path, model=None, tile=0, tile_pad=10, pre_pad=10, half=False):
|
||||
def __init__(self,
|
||||
scale,
|
||||
model_path,
|
||||
dni_weight=None,
|
||||
model=None,
|
||||
tile=0,
|
||||
tile_pad=10,
|
||||
pre_pad=10,
|
||||
half=False,
|
||||
device=None,
|
||||
gpu_id=None):
|
||||
self.scale = scale
|
||||
self.tile_size = tile
|
||||
self.tile_pad = tile_pad
|
||||
@@ -35,23 +45,46 @@ class RealESRGANer():
|
||||
self.half = half
|
||||
|
||||
# initialize model
|
||||
self.device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
|
||||
# if the model_path starts with https, it will first download models to the folder: realesrgan/weights
|
||||
if model_path.startswith('https://'):
|
||||
model_path = load_file_from_url(
|
||||
url=model_path, model_dir=os.path.join(ROOT_DIR, 'realesrgan/weights'), progress=True, file_name=None)
|
||||
loadnet = torch.load(model_path, map_location=torch.device('cpu'))
|
||||
if gpu_id:
|
||||
self.device = torch.device(
|
||||
f'cuda:{gpu_id}' if torch.cuda.is_available() else 'cpu') if device is None else device
|
||||
else:
|
||||
self.device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') if device is None else device
|
||||
|
||||
if isinstance(model_path, list):
|
||||
# dni
|
||||
assert len(model_path) == len(dni_weight), 'model_path and dni_weight should have the save length.'
|
||||
loadnet = self.dni(model_path[0], model_path[1], dni_weight)
|
||||
else:
|
||||
# if the model_path starts with https, it will first download models to the folder: weights
|
||||
if model_path.startswith('https://'):
|
||||
model_path = load_file_from_url(
|
||||
url=model_path, model_dir=os.path.join(ROOT_DIR, 'weights'), progress=True, file_name=None)
|
||||
loadnet = torch.load(model_path, map_location=torch.device('cpu'))
|
||||
|
||||
# prefer to use params_ema
|
||||
if 'params_ema' in loadnet:
|
||||
keyname = 'params_ema'
|
||||
else:
|
||||
keyname = 'params'
|
||||
model.load_state_dict(loadnet[keyname], strict=True)
|
||||
|
||||
model.eval()
|
||||
self.model = model.to(self.device)
|
||||
if self.half:
|
||||
self.model = self.model.half()
|
||||
|
||||
def dni(self, net_a, net_b, dni_weight, key='params', loc='cpu'):
|
||||
"""Deep network interpolation.
|
||||
|
||||
``Paper: Deep Network Interpolation for Continuous Imagery Effect Transition``
|
||||
"""
|
||||
net_a = torch.load(net_a, map_location=torch.device(loc))
|
||||
net_b = torch.load(net_b, map_location=torch.device(loc))
|
||||
for k, v_a in net_a[key].items():
|
||||
net_a[key][k] = dni_weight[0] * v_a + dni_weight[1] * net_b[key][k]
|
||||
return net_a
|
||||
|
||||
def pre_process(self, img):
|
||||
"""Pre-process, such as pre-pad and mod pad, so that the images can be divisible
|
||||
"""
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
basicsr>=1.3.3.11
|
||||
facexlib>=0.2.0.3
|
||||
gfpgan>=0.2.1
|
||||
basicsr>=1.4.2
|
||||
facexlib>=0.2.5
|
||||
gfpgan>=1.3.5
|
||||
numpy
|
||||
opencv-python
|
||||
Pillow
|
||||
|
||||
Reference in New Issue
Block a user