Notes

Some useful math results. Inverse Function Theorem Let $f(x)$ be a function that is invertible and differentiable. Let $y=f^{-1}(x)$ be the inverse function of $f(x)$. For all $x$ satisfying $f’(f^{-1}(x))\neq 0$, $$ \frac{dy}{dx}=\frac{d}{dx}(f^{-1}(x))=\frac{1}{f’(f^{-1}(x))}. $$ Alternatively, if $y=g(x)$ is the inverse of $f(x)$, then $$ g’(x)=\frac{1}{f’(g(x))}. $$ Sion’s minimax theorem Wiki: Sion’s minimax theorem Let $X$ be a compact convex subset of a linear topological space and $Y$ a convex subset of a linear topological space....

Diffusion Models Many recent papers and blogs [1-4] have provided a good overview of diffusion models. In this note, I will summarize some high-level understandings towards the diffusion models and write down some key steps in the derivations. Summary DDPM is parameterized as a Markov chain which means that the latent variables depend only on the previous (or following) timestep. However, DDIM shows that it is indeed not necessary. The forward prcess of DDPM is a chain of additive Gaussian noise channels and the variance of the noises are pre-defined, i....

This notes is about the compression algorithms including typical compression models and neural comression models. The length of a message representing some data is proportional to the entropy of this data. In compression tasks, our goal is to design a bitstream such that a uniquely decodable code whose expected length is as close as possible to the entropy of the original data. Background The goal of data compression is to represent a given sequence $a_1,\cdots, a_n$ produced by a source as a sequence of bits of minimal possible length....

The Legendre transform is an self-inverse transformation on real-valued convex functions of one real variable. The generalization of the Legendre transformation to affine spaces and non-convex functions is known as the convex conjugate (also called the Legendre–Fenchel transformation), which can be used to construct a function’s convex hull. Definition Let $f$ be a convex function. The Legendre transform of $f$ is defined as $$f^*(y)=\sup_x y^T x-f(x),$$ which is always well-defined when $f(x)$ is convex....

KKT Condition...