Useful resource-constrained picture technology and visible understanding: an interview with Aniket Roy

Within the newest in our collection of interviews assembly the AAAI/SIGAI Doctoral Consortium members, we caught up with Aniket Roy to search out out extra about his analysis on generative fashions for laptop imaginative and prescient duties.

Inform us a bit about your PhD – the place did you research, and what was the subject of your analysis?

I just lately accomplished my PhD in Pc Science at Johns Hopkins College, the place I labored beneath the supervision of Bloomberg Distinguished Professor Rama Chellappa. My analysis primarily centered on creating strategies for resource-constrained picture technology and visible understanding. Particularly, I explored how trendy generative fashions might be tailored to function effectively whereas sustaining sturdy efficiency.

Throughout my PhD, I labored broadly on the intersection of generative AI, multimodal studying, and few-shot studying. A lot of my work concerned designing strategies that allow fashions to be taught new ideas or carry out advanced visible duties with restricted knowledge or computational assets. This included analysis on diffusion fashions, customized picture technology, and multimodal illustration studying. General, my work goals to make superior imaginative and prescient and generative AI techniques extra adaptable, environment friendly, and sensible for real-world functions.

Might you give us an outline of the analysis you carried out throughout your PhD?

Throughout my PhD, my analysis broadly centered on bettering the adaptability, effectivity, and high quality of recent generative fashions for laptop imaginative and prescient duties. The fast progress in generative AI–significantly diffusion fashions and imaginative and prescient–language fashions–has created new alternatives to deal with long-standing challenges equivalent to knowledge shortage, controllable technology, and customized picture synthesis. My work aimed to develop strategies that enable these giant fashions to adapt successfully with restricted knowledge and computational assets whereas sustaining excessive visible constancy.

One line of my analysis addressed studying in data-constrained settings. For instance, I proposed FeLMi, a few-shot studying framework that leverages uncertainty-guided exhausting mixup methods to enhance robustness and generalization when solely a small variety of labeled samples can be found. Constructing on this concept of bettering coaching knowledge high quality, I additionally developed Cap2Aug, which introduces caption-guided multimodal augmentation. This strategy makes use of textual descriptions to information artificial picture technology, bettering visible variety whereas lowering the area hole between actual and generated knowledge.

Overview of Cap2Aug.

One other side of my analysis centered on bettering the perceptual high quality of photos generated by diffusion fashions. On this route, I proposed DiffNat, a plug-and-play regularization methodology primarily based on the kurtosis-concentration property noticed in pure photos. By incorporating this precept into diffusion fashions by way of a KC loss, the generated photos exhibit extra pure texture statistics and improved perceptual realism, which additionally advantages downstream imaginative and prescient duties.

A serious a part of my work explored personalization and environment friendly adaptation of enormous generative fashions. I launched DuoLoRA, a parameter-efficient framework for composing low-rank adapters that permits fine-grained management over content material and elegance with out requiring full retraining of the bottom mannequin. I additional prolonged personalization to zero-shot settings utilizing a training-free textual inversion strategy that permits arbitrary objects to be custom-made straight throughout technology. Lastly, I proposed MultiLFG, a frequency-guided multi-LoRA composition framework that makes use of wavelet-domain representations and timestep-aware weighting to allow correct and training-free fusion of a number of ideas in diffusion fashions.

Overview of DuoLoRA.

General, my analysis contributes towards constructing generative techniques which might be extra environment friendly, adaptable, and controllable, enabling high-quality picture technology and understanding even in data-limited or resource-constrained eventualities.

Was there a selected challenge or a facet of your analysis that was significantly attention-grabbing?

One challenge that I discovered significantly attention-grabbing throughout my PhD is DiffNat, which was revealed in TMLR 2025. Diffusion fashions have turn into the spine of many trendy generative AI techniques and have achieved spectacular leads to producing and enhancing real looking photos. Nonetheless, bettering the perceptual high quality and naturalness of generated photos stays an necessary problem.

Overview of DiffNat.

On this work, we launched a easy however efficient regularization method known as the kurtosis focus (KC) loss, which might be built-in into commonplace diffusion mannequin pipelines as a plug-and-play part. The concept was impressed by a statistical property of pure photos: when a picture is decomposed into totally different band-pass filtered variations–for instance utilizing the Discrete Wavelet Remodel–the kurtosis values throughout these frequency bands are usually comparatively constant. In distinction, generated photos usually present giant discrepancies throughout these bands. Our methodology reduces the hole between the best and lowest kurtosis values throughout the frequency parts, encouraging the generated photos to comply with extra pure picture statistics.

As well as, we launched a condition-agnostic perceptual steerage technique throughout inference that additional improves picture constancy with out requiring extra coaching alerts. We evaluated the strategy throughout a number of various duties, together with customized few-shot finetuning with textual content steerage, unconditional picture technology, picture super-resolution, and blind face restoration. Throughout these duties, incorporating the KC loss and perceptual steerage constantly improved perceptual high quality, measured by way of metrics equivalent to FID and MUSIQ, in addition to by way of human analysis.

What I significantly preferred about this challenge is that it connects classical picture statistics with trendy diffusion fashions. It reveals that comparatively easy statistical insights about pure photos can nonetheless play a strong function in bettering giant generative fashions.

What are your plans for constructing on the PhD – the place are you working now and what’s going to you be investigating subsequent?

Throughout my PhD, I found that I genuinely benefit from the strategy of analysis–particularly the second when an instinct or concept seems to work in apply. That strategy of exploring new concepts and pushing the boundaries of what we all know is one thing I discover very motivating.

To proceed pursuing this, I can be becoming a member of NEC Laboratories America as a Analysis Scientist. On this function, I hope to construct on my PhD work by creating new strategies for generative fashions and exploring how these fashions can work together with broader multimodal techniques. Particularly, I’m serious about advancing analysis on the intersection of generative fashions, imaginative and prescient–language–motion fashions, and embodied AI. Extra broadly, my objective is to contribute to the event of clever techniques that may perceive, generate, and work together with the visible world extra successfully, whereas additionally persevering with to push ahead the scientific understanding of those fashions.

I’m serious about how you bought into the sector. What impressed you to check laptop imaginative and prescient and machine studying?

My curiosity in laptop imaginative and prescient and machine studying began throughout my undergraduate research, once I took programs in sign processing and picture processing. I discovered these topics significantly fascinating as a result of they allowed you to experiment with algorithms and instantly see their results on photos. That visible and intuitive side made the sector very participating, and it helped me recognize how mathematical ideas can straight translate into significant visible outcomes.

On the similar time, I used to be additionally interested in how the human mind processes visible data—how we’re capable of acknowledge objects, perceive scenes, and interpret advanced visible alerts so effortlessly. That curiosity led me to wonder if we may design computational fashions that mimic points of human notion and allow machines to grasp visible knowledge in an analogous means.

A serious affect throughout this time was my professor, Dr. Kuntal Ghosh, who inspired me to suppose extra deeply about these issues and strategy them with a scientific mindset. His mentorship performed an necessary function in shaping my curiosity in analysis. Since then, that curiosity about visible notion and clever techniques has continued to drive my work in laptop imaginative and prescient and machine studying.

What was your expertise of the Doctoral Consortium at AAAI?

Sadly, I used to be not capable of attend the AAAI Doctoral Consortium in individual as a consequence of visa-related points. Nonetheless, a colleague kindly helped current my poster on my behalf in the course of the occasion. Regardless that I couldn’t be there bodily, I used to be very inspired by the response my work acquired. A number of researchers reached out to me after seeing the poster, and we had some very insightful discussions in regards to the concepts and potential future instructions of the analysis. In that sense, I nonetheless discovered the expertise fairly rewarding. The Doctoral Consortium is a good platform for sharing early-stage concepts, receiving suggestions from the group, and connecting with different researchers engaged on associated issues. I appreciated the chance to have interaction with individuals who had been within the work, and people interactions helped spark new views and collaborations.

Might you inform us an attention-grabbing (non-AI associated) truth about you?

Outdoors of analysis, I’m an enormous fan of music and stand-up comedy, and I actually get pleasure from touring at any time when I get the possibility. Exploring new locations, cultures, and views is one thing I discover refreshing—it’s a good way to recharge and keep curious in regards to the world past work. I additionally get pleasure from writing poetic satire occasionally, and I often carry out it. It’s a enjoyable inventive outlet that permits me to combine humor and storytelling, which is kind of totally different from the analytical nature of the analysis work I normally do.

About Aniket Roy

Aniket is at the moment a Analysis Scientist at NEC Labs America. He obtained his PhD from the Pc Science dept at Johns Hopkins College beneath the steerage of Bloomberg Distinguished Professor Prof. Rama Chellappa. Previous to that, he did a Grasp’s from Indian Institute of Know-how Kharagpur. He was acknowledged with the Finest Paper Award at IWDW 2016 and the Markose Thomas Memorial Award for the very best analysis paper on the Grasp’s degree. Throughout PhD, he explored domains of few-shot studying, multimodal studying, diffusion fashions, LLMs, LoRA merging with publications in main venues equivalent to NeurIPS, ICCV, TMLR, WACV, CVPR and likewise 3 US patents filed. Throughout his PhD, he additionally gained industrial expertise by way of a number of internships in Amazon, Qualcomm, MERL, and SRI Worldwide. He was awarded as an Amazon Fellow (2023-24) at JHU and chosen to take part in ICCV’25 and AAAI’26 doctoral consortium.