\( \text{Contents of this Presentation:}\)

\( \text{Theme of Research so far:}\)

Accepted first author paper in BOLD.

\( \textbf{Your Text Encoder Can Be An Object-Level Watermarking Controller}\)

Train Single Token Text Embeddings for Watermarking

\( \textbf{Your Text Encoder Can Be An Object-Level Watermarking Controller}\)

Latent Loss for Minimal Trajectory Adjustment to enable imperceptible watermarking

\( \textbf{Your Text Encoder Can Be An Object-Level Watermarking Controller}\)

Same Token Embedding can be used for both Full-image and Object-Level Watermarking

(utilizing Cross-Attention Maps)

\( \textbf{Your Text Encoder Can Be An Object-Level Watermarking Controller}\)

\( \textbf{Personalization through Trajectory Shifted Perturbations}\)

\( \textbf{Latent Diffusion Unlearning: Protecting against Unauthorized}\)

Shift Start of the Diffusion Denoising Trajectory for Latent-Level Unlearnable Sample.

\( \textbf{Personalization through Trajectory Shifted Perturbations}\)

\( \textbf{Latent Diffusion Unlearning: Protecting against Unauthorized}\)

Shortcut Diffusion Models for Unlearnable Perturbation Propagation

\( \textbf{Personalization through Trajectory Shifted Perturbations}\)

\( \textbf{Latent Diffusion Unlearning: Protecting against Unauthorized}\)

Artifact-Free

Unlearnable Samples

Visible Perturbation Overlay

\( \textbf{Personalization through Trajectory Shifted Perturbations}\)

\( \textbf{Latent Diffusion Unlearning: Protecting against Unauthorized}\)

Resistance to Strong DiffPure Attack

\( \textbf{Submitted First Author Papers - Under Review:}\)

\( \textbf{Controlling Hallucinations in Diffusion Models: A Case Study on Chess}\)

\( \textbf{Submitted to NeurIPS 2025:}\)

Variance Learning

and

Score Amplification

for

Hallucination Reduction

\( \textbf{Teaching Experience:}\)

\( \text{Course Instructor:} \textbf{ Computer Vision and Image Processing}\)

\( \text{Summer - 2025} \)

\( \text{Number of Students: } \textbf{71}\)

Generative Modeling Principles

unknown

• Generative modeling aims to learn a model \( p_\theta(x) \) that closely matches the real data distribution \( p_{\text{data}}(x) \) by minimizing their KL divergence or equivalently maximizing log-likelihood.

Forward Diffusion

Reverse Diffusion

Training via ELBO

Noise Matching Term

Training via ELBO

Noise Matching Term

Training via ELBO

Noise Matching Term

Parallel Decoding

Fixed-Time Sampling

Autoregressive Formulation in Traditional LLMs

Each token prediction depends on previous tokens.

Sequential sampling

No parallelism

Error compounding

Sequential sampling

Is the autoregressive paradigm the only viable path to achieving the intelligence exhibited by LLMs?

Research Question:

Large Language Diffusion Models

Pre-training

Supervised Finetuning

Sampling

Large Language Diffusion Models - Contributions

• Scalability
• In-Context Learning
• Instruction-Following
• Reversal Resoning
  • Outperforms GPT-4o

Large Language Diffusion Models - Loss

LLaDA’s core is a mask predictor: a model \( p_\theta(\cdot \mid x_t) \) that takes a masked sequence \( x_t \) and predicts all masked tokens (set \( M \)) simultaneously.

Pre-trained on 2.3 trillion tokens took 0.13 milion H800 GPU hours.

Large Language Diffusion Models - SFT Loss

\( L' \) - dynamic length response.

Large Language Diffusion Models - Inference

Initialization: The input sequence is constructed as [prompt tokens] + [MASK tokens] of target length. 

Block-wise Iterative Denoising: The generation proceeds block-by-block. At each step, the model predicts logits for all masked positions in the current block.

Confidence-based Remasking Loop: This denoising loop is repeated for a fixed number of steps. Enables parallel yet progressive refinement of the full sequence.

Evaluation Metrics

Scalability

Performance compared to baselines

Reversal Reasoning

Limitations: Inference Time (No KV Caching)

Accelerating Diffusion Large Language Models with SlowFast Sampling

LLaDA Acc: 50.19%, LLaMA: 41.09% (GSM8k)

LLaDA Acc: 75.21%, LLaMA: 71.42% (GSM8k)

Problem Statement

• Diffusion models are powerful generative models that learn to denoise Gaussian noise into image samples.
• However, training is inefficient, especially in few-step settings like Consistency Models or DDIM.

Motivation

Accelerate Diffusion Training

Key Insight

• The random pairing of images and noise (miscibility) leads to uninformative gradients and optimization difficulty.
• Goal: Improve training by structuring image-noise mapping using Immiscible Diffusion.

Posterior Under Miscibility:

Weak learning signal.

Noise Prediction in Vanilla DDIM:

where:

Immiscible Diffusion

Use linear assignment to optimally match each image in the batch with a noise sample:

• This ensures each image gets closer noise.
• Prevents global mixing.
• Implemented in 1 line using Hungarian algorithm (scipy)

Conditional Distribution After Assignment

\( f(\cdot) \): Decaying function (e.g., Gaussian)

Each image diffuses to a local region in noise space

Posterior Becomes Informative

The posterior is no longer uniform, it's peaked around assigned images.

Noise Prediction in Immiscible Diffusion

Model now learns to predict noise that corresponds to a local cluster of images.

Enables strong gradients even at high noise levels.

Batch Assignment Algorithm

Inputs:
- \( x_b \): batch of images  
- \( \epsilon_b \): batch of noise  
- \( t_b \): batch of diffusion steps  

Output:

Contributions

• Identified miscibility issue in noise-image mapping.
• Proposed immiscible diffusion using optimal assignment.
• Theoretically derived effects via posterior + expected noise.
• Demonstrated 1-line code fix with broad applicability.
• Achieved up to 3× speedup without compromising quality

Contributions

• Identified miscibility issue in noise-image mapping.
• Proposed immiscible diffusion using optimal assignment.
• Theoretically derived effects via posterior + expected noise.
• Demonstrated 1-line code fix with broad applicability.
• Achieved up to 3× speedup without compromising quality

Future Research on Large Language Diffusion Models

• User-specific adaptation
• Domain-specific knowledge
• Task tuning: Improve performance on summarization, question-answering, etc., in narrow settings.
• Memory/Contextual grounding: Make the model “remember” previous interactions or documents.

Personalization:

References:

• https://github.com/ML-GSAI/LLaDA
• https://github.com/Gen-Verse/MMaDA
• https://newsletter.maartengrootendorst.com/p/a-visual-guide-to-mamba-and-state
• https://github.com/hkproj/mamba-notes
• https://github.com/yhli123/Immiscible-Diffusion
• https://arxiv.org/abs/2410.03943