Generative AI
Generative AI¶
Junseo Park
Intro Related Work Method Experiment Conclusion
Core Idea
"Classification Operations in the Generative AI Model (생성 모델의 분류 작업)"
CVPR & ICCV Trend¶
2023 keyword
- Diffusion
- 3D
- Human pose, face, body, movements, gesture, etc.
- Image & Video generation
- NeRF
- Multimodal
2023 comment
- Image style transfer 분야는 흥미를 잃어간다.
- 특히 image style transfer using GAN 은 잘 안한다.
- Video/3D style transfer 는 가능
Image Synthesis¶
Controlling Image Diffusion Models/Personalization¶
reference paper-HyperDreamBooth
- DreamBooth
- Optimize entire T2I network weights
- 관심 주제를 설명하는 몇 장의 이미지에 대해 entire text-to-image model 을 fine-tuning 한다.
- 따라서 표현력이 풍부하며 세부적인 내용으로 주제를 캡쳐할 수 있다.
- Language drift & overfitting 을 완화하기 위해 class-specific prior preservation loss & class-specific prior image 를 도입했다.
- Textual Inversion
- Optimize text embedding
- LoRA
- Optimize low-rank approximations of weight residuals
- Custom Diffusion
- Optimize cross-attention layers
- SVDiff
- Optimize singular values of weights
- StyleDrop
- Use adapter tuning and finetunes a small set of adapter weights for style personalization
- Generation in any style with a single input image
- Style 특화 personalization. 다른 방법들은 painting style 에 제한되어 있지만, StyleDrop 은 a single style reference image 만을 사용하여 3D rendering, design illustration, sculpture 등 다양한 시각적 스타일을 시연한다.
- DreamArtist
- One-shot personalization techniques by employing a positive-negative prompt tuning strategy
- HyperNetwork
- Use an auxiliary neural network to predict network weights in order to change the functioning of a specific neural network
- ControlNet
- Propose ways to incorporate new input conditioning such as depth
- Specialist Diffusion
- Object 학습/ not style
- StyleBoost
- Style personalization with T2I diffusion models using DreamBooth
- vs StyleDrop & DreamBooth
Style Transfer¶
Traditional Method (e.g., CNN, GAN, ViT)¶
- 확인 필요
- StyleCLIP
- VQGAN-CLIP
- CLIPStyler
- StyleGAN
-
Style-NaDa
-
확인 완료
- AdaIN
- Neural Style Transfer
Diffusion-based model¶
- Diffusion-Enhanced PatchMatch
- Utilize Patch-based techniques with whitening and coloring transformations in latent space
- StyleDiffusion
- 해석가능하고 제어가능한 C-S disentanglement and style transfer 를 제안했다.
- CLIP image space 에서 C-S 를 해결하기 위해 CLIP-based style disentanglement loss 와 style reconstruction prior 를 도입했다.
- Inversion-based Style Transfer
- textual description \([C]\) 학습
- Inference 시에, \([C]\) 와 content image 의 inversion 을 통해 생성
- DreamStyler
- Advanced Texutal Inversion
- Using BLIP-2, image encoder
- Text 에 style binding
Super Resolution¶
Diffusion-based model¶
Image-to-Image Translation¶
- Pix2Pix
- Cycle-GAN
- Plug-and-Play