Personalizing Text-to-Image Diffusion Models by Fine-Tuning Classification for AI Applications

Rafael Hidalgo; Nesreen Salah; Rajiv Chandra Jetty; Anupama Jetty; Aparna S. Varde

doi:10.1007/978-3-031-47721-8_44

Personalizing Text-to-Image Diffusion Models by Fine-Tuning Classification for AI Applications

Rafael Hidalgo
, Nesreen Salah
, Rajiv Chandra Jetty
, Anupama Jetty
, Aparna S. Varde^*

^*Kontaktforfatter

Montclair State University

Publikation: Kapitel i bog/rapport/konference-proceeding › Konferencebidrag i proceedings › Forskning › peer review

Abstract

Stable Diffusion is a captivating text-to-image model that generates images based on text input. However, a major challenge is that it is pretrained on a specific dataset, limiting its ability to generate images outside of the given data. In this paper, we propose to harness two models based on neural networks, Hypernetworks and DreamBooth, to allow the introduction of any image into Stable Diffusion, addressing versatility with minimal additional training data. This work targets AI applications such as augmenting next-generation multipurpose robots, enhancing human-robot collaboration, feeding intelligent tutoring systems, training autonomous cars, injecting subjects for photo personalization, producing high quality movie animations etc. It can contribute to AI in smart cities: facets such as smart living and smart mobility.

Originalsprog	Engelsk
Titel	Intelligent Systems and Applications - Proceedings of the 2023 Intelligent Systems Conference IntelliSys Volume 1
Redaktører	Kohei Arai
Antal sider	17
Forlag	Springer Science+Business Media
Publikationsdato	2024
Sider	642-658
ISBN (Trykt)	9783031477201
DOI	https://doi.org/10.1007/978-3-031-47721-8_44
Status	Udgivet - 2024
Udgivet eksternt	Ja
Begivenhed	Intelligent Systems Conference, IntelliSys 2023 - Amsterdam, Holland Varighed: 7. sep. 2023 → 8. sep. 2023

Konference

Konference	Intelligent Systems Conference, IntelliSys 2023
Land/Område	Holland
By	Amsterdam
Periode	07/09/2023 → 08/09/2023

Navn	Lecture Notes in Networks and Systems
Vol/bind	822
ISSN	2367-3370

Bibliografisk note

Publisher Copyright:
© 2024, The Author(s), under exclusive license to Springer Nature Switzerland AG.

Finansiering

and Disclaimer Dr. Aparna Varde acknowledges NSF grants 2018575 “MRI: Acquisition of a High-Performance GPU Cluster for Research & Education”, and 2117308 “MRI: Acquisition of a Multimodal Collaborative Robot System (MCROS) to Support Cross-Disciplinary Human-Centered Research & Education at Montclair State University”. She is a visiting researcher at Max Planck Institute for Informatics, Germany (ongoing from sabbatical). She is an Associate Director of the School of Computing, and an Associate Director of the CESAC: Clean Energy & Sustainability Analytics Center, Montclair State University. We make a disclaimer that the opinions presented here are extracted from online sources; and the content of this paper including the images is not meant to offend/hurt any national, ethnic, cultural, racial, religious and other groups. The images produced here are taken with the consent of the respective subjects. Any resemblance to anyone else is coincidental. This is a pilot study. Acknowledgments. Acknowledgments and Disclaimer Dr. Aparna Varde acknowledges NSF grants 2018575 “MRI: Acquisition of a High-Performance GPU Cluster for Research & Education”, and 2117308 “MRI: Acquisition of a Multimodal Collaborative Robot System (MCROS) to Support Cross-Disciplinary Human-Centered Research & Education at Montclair State University”. She is a visiting researcher at Max Planck Institute for Informatics, Germany (ongoing from sabbatical). She is an Associate Director of the School of Computing, and an Associate Director of the CESAC: Clean Energy & Sustainability Analytics Center, Montclair State University. We make a disclaimer that the opinions presented here are extracted from online sources; and the content of this paper including the images is not meant to offend/hurt any national, ethnic, cultural, racial, religious and other groups. The images produced here are taken with the consent of the respective subjects. Any resemblance to anyone else is coincidental. This is a pilot study.

Dokumenter og links

10.1007/978-3-031-47721-8_44

SDU link

Tjek adgangen til materialet i Syddansk Universitetsbiblioteks søgemaskine

Andre filer og links

Link to publication in Scopus

Citationsformater

Hidalgo, R., Salah, N., Chandra Jetty, R., Jetty, A., & Varde, A. S. (2024). Personalizing Text-to-Image Diffusion Models by Fine-Tuning Classification for AI Applications. I K. Arai (red.), Intelligent Systems and Applications - Proceedings of the 2023 Intelligent Systems Conference IntelliSys Volume 1 (s. 642-658). Springer Science+Business Media. https://doi.org/10.1007/978-3-031-47721-8_44

Hidalgo, Rafael ; Salah, Nesreen ; Chandra Jetty, Rajiv et al. / Personalizing Text-to-Image Diffusion Models by Fine-Tuning Classification for AI Applications. Intelligent Systems and Applications - Proceedings of the 2023 Intelligent Systems Conference IntelliSys Volume 1. red. / Kohei Arai. Springer Science+Business Media, 2024. s. 642-658 (Lecture Notes in Networks and Systems, Bind 822).

@inproceedings{e87d1a8181814bcdae6bf87ee7efd917,

title = "Personalizing Text-to-Image Diffusion Models by Fine-Tuning Classification for AI Applications",

abstract = "Stable Diffusion is a captivating text-to-image model that generates images based on text input. However, a major challenge is that it is pretrained on a specific dataset, limiting its ability to generate images outside of the given data. In this paper, we propose to harness two models based on neural networks, Hypernetworks and DreamBooth, to allow the introduction of any image into Stable Diffusion, addressing versatility with minimal additional training data. This work targets AI applications such as augmenting next-generation multipurpose robots, enhancing human-robot collaboration, feeding intelligent tutoring systems, training autonomous cars, injecting subjects for photo personalization, producing high quality movie animations etc. It can contribute to AI in smart cities: facets such as smart living and smart mobility.",

keywords = "ANN, Data mining, Image processing, Movie animations, Photo personalization, Stable diffusion, Text-to-image creation",

author = "Rafael Hidalgo and Nesreen Salah and \{Chandra Jetty\}, Rajiv and Anupama Jetty and Varde, \{Aparna S.\}",

note = "Publisher Copyright: {\textcopyright} 2024, The Author(s), under exclusive license to Springer Nature Switzerland AG.; Intelligent Systems Conference, IntelliSys 2023 ; Conference date: 07-09-2023 Through 08-09-2023",

year = "2024",

doi = "10.1007/978-3-031-47721-8\_44",

language = "English",

isbn = "9783031477201",

series = "Lecture Notes in Networks and Systems",

publisher = "Springer Science+Business Media",

pages = "642--658",

editor = "Kohei Arai",

booktitle = "Intelligent Systems and Applications - Proceedings of the 2023 Intelligent Systems Conference IntelliSys Volume 1",

address = "United States",

}

Hidalgo, R, Salah, N, Chandra Jetty, R, Jetty, A & Varde, AS 2024, Personalizing Text-to-Image Diffusion Models by Fine-Tuning Classification for AI Applications. i K Arai (red.), Intelligent Systems and Applications - Proceedings of the 2023 Intelligent Systems Conference IntelliSys Volume 1. Springer Science+Business Media, Lecture Notes in Networks and Systems, bind 822, s. 642-658, Intelligent Systems Conference, IntelliSys 2023, Amsterdam, Holland, 07/09/2023. https://doi.org/10.1007/978-3-031-47721-8_44

Personalizing Text-to-Image Diffusion Models by Fine-Tuning Classification for AI Applications. / Hidalgo, Rafael; Salah, Nesreen; Chandra Jetty, Rajiv et al.
Intelligent Systems and Applications - Proceedings of the 2023 Intelligent Systems Conference IntelliSys Volume 1. red. / Kohei Arai. Springer Science+Business Media, 2024. s. 642-658 (Lecture Notes in Networks and Systems, Bind 822).

Publikation: Kapitel i bog/rapport/konference-proceeding › Konferencebidrag i proceedings › Forskning › peer review

TY - GEN

T1 - Personalizing Text-to-Image Diffusion Models by Fine-Tuning Classification for AI Applications

AU - Hidalgo, Rafael

AU - Salah, Nesreen

AU - Chandra Jetty, Rajiv

AU - Jetty, Anupama

AU - Varde, Aparna S.

PY - 2024

Y1 - 2024

N2 - Stable Diffusion is a captivating text-to-image model that generates images based on text input. However, a major challenge is that it is pretrained on a specific dataset, limiting its ability to generate images outside of the given data. In this paper, we propose to harness two models based on neural networks, Hypernetworks and DreamBooth, to allow the introduction of any image into Stable Diffusion, addressing versatility with minimal additional training data. This work targets AI applications such as augmenting next-generation multipurpose robots, enhancing human-robot collaboration, feeding intelligent tutoring systems, training autonomous cars, injecting subjects for photo personalization, producing high quality movie animations etc. It can contribute to AI in smart cities: facets such as smart living and smart mobility.

AB - Stable Diffusion is a captivating text-to-image model that generates images based on text input. However, a major challenge is that it is pretrained on a specific dataset, limiting its ability to generate images outside of the given data. In this paper, we propose to harness two models based on neural networks, Hypernetworks and DreamBooth, to allow the introduction of any image into Stable Diffusion, addressing versatility with minimal additional training data. This work targets AI applications such as augmenting next-generation multipurpose robots, enhancing human-robot collaboration, feeding intelligent tutoring systems, training autonomous cars, injecting subjects for photo personalization, producing high quality movie animations etc. It can contribute to AI in smart cities: facets such as smart living and smart mobility.

KW - ANN

KW - Data mining

KW - Image processing

KW - Movie animations

KW - Photo personalization

KW - Stable diffusion

KW - Text-to-image creation

UR - https://www.scopus.com/pages/publications/85182514364

U2 - 10.1007/978-3-031-47721-8_44

DO - 10.1007/978-3-031-47721-8_44

M3 - Article in proceedings

AN - SCOPUS:85182514364

SN - 9783031477201

T3 - Lecture Notes in Networks and Systems

SP - 642

EP - 658

BT - Intelligent Systems and Applications - Proceedings of the 2023 Intelligent Systems Conference IntelliSys Volume 1

A2 - Arai, Kohei

PB - Springer Science+Business Media

T2 - Intelligent Systems Conference, IntelliSys 2023

Y2 - 7 September 2023 through 8 September 2023

ER -

Hidalgo R, Salah N, Chandra Jetty R, Jetty A, Varde AS. Personalizing Text-to-Image Diffusion Models by Fine-Tuning Classification for AI Applications. I Arai K, red., Intelligent Systems and Applications - Proceedings of the 2023 Intelligent Systems Conference IntelliSys Volume 1. Springer Science+Business Media. 2024. s. 642-658. (Lecture Notes in Networks and Systems, Bind 822). doi: 10.1007/978-3-031-47721-8_44

Personalizing Text-to-Image Diffusion Models by Fine-Tuning Classification for AI Applications

Abstract

Konference

Bibliografisk note

Finansiering

Dokumenter og links

SDU link

Andre filer og links

Fingeraftryk

Citationsformater