콘텐츠로 이동

Generative AI

Generative AI

Junseo Park

Intro
Related Work
Method
Experiment
Conclusion

Core Idea

"Classification Operations in the Generative AI Model (생성 모델의 분류 작업)"

CVPR & ICCV Trend

2023 keyword

  • Diffusion
  • 3D
  • Human pose, face, body, movements, gesture, etc.
  • Image & Video generation
  • NeRF
  • Multimodal

2023 comment

  • Image style transfer 분야는 흥미를 잃어간다.
    • 특히 image style transfer using GAN 은 잘 안한다.
  • Video/3D style transfer 는 가능

Image Synthesis

Controlling Image Diffusion Models/Personalization

reference paper-HyperDreamBooth

reference paper-StyleDrop

  • DreamBooth
    • Optimize entire T2I network weights
    • 관심 주제를 설명하는 몇 장의 이미지에 대해 entire text-to-image model 을 fine-tuning 한다.
    • 따라서 표현력이 풍부하며 세부적인 내용으로 주제를 캡쳐할 수 있다.
    • Language drift & overfitting 을 완화하기 위해 class-specific prior preservation loss & class-specific prior image 를 도입했다.
  • Textual Inversion
    • Optimize text embedding
  • LoRA
    • Optimize low-rank approximations of weight residuals
  • Custom Diffusion
    • Optimize cross-attention layers
  • SVDiff
    • Optimize singular values of weights
  • StyleDrop
    • Use adapter tuning and finetunes a small set of adapter weights for style personalization
    • Generation in any style with a single input image
    • Style 특화 personalization. 다른 방법들은 painting style 에 제한되어 있지만, StyleDrop 은 a single style reference image 만을 사용하여 3D rendering, design illustration, sculpture 등 다양한 시각적 스타일을 시연한다.
  • DreamArtist
    • One-shot personalization techniques by employing a positive-negative prompt tuning strategy
  • HyperNetwork
    • Use an auxiliary neural network to predict network weights in order to change the functioning of a specific neural network
  • ControlNet
    • Propose ways to incorporate new input conditioning such as depth
  • Specialist Diffusion
    • Object 학습/ not style
  • StyleBoost
    • Style personalization with T2I diffusion models using DreamBooth
    • vs StyleDrop & DreamBooth

Style Transfer

Traditional Method (e.g., CNN, GAN, ViT)

  • 확인 필요
  • StyleCLIP
  • VQGAN-CLIP
  • CLIPStyler
  • StyleGAN
  • Style-NaDa

  • 확인 완료

  • AdaIN
  • Neural Style Transfer

Diffusion-based model

  • Diffusion-Enhanced PatchMatch
    • Utilize Patch-based techniques with whitening and coloring transformations in latent space
  • StyleDiffusion
    • 해석가능하고 제어가능한 C-S disentanglement and style transfer 를 제안했다.
    • CLIP image space 에서 C-S 를 해결하기 위해 CLIP-based style disentanglement loss 와 style reconstruction prior 를 도입했다.
  • Inversion-based Style Transfer
    • textual description \([C]\) 학습
    • Inference 시에, \([C]\) 와 content image 의 inversion 을 통해 생성
  • DreamStyler
    • Advanced Texutal Inversion
    • Using BLIP-2, image encoder
    • Text 에 style binding

Super Resolution

Diffusion-based model


Image-to-Image Translation

  • Pix2Pix
  • Cycle-GAN
  • Plug-and-Play

Image Editing

Data Augmentation


Inpainting

Diffusion-based model


Question