AI & Deep Learning
Deep Learning Reference Quick-reference formulas, tensor dimensions, activation functions, and optimizer update rules.
A concise, high-utility reference for formulas, tensor dimensions, activation functions, and optimizer update rules. Bookmark this page for fast lookup during research and implementation.
Given an input feature map of spatial dimension W W W K K K P P P S S S
O = ⌊ W − K + 2 P S ⌋ + 1 O = \left\lfloor \frac{W - K + 2P}{S} \right\rfloor + 1 O = ⌊ S W − K + 2 P ⌋ + 1 # PyTorch calculation out_dim = ((W - K + 2 * P) // S) + 1 O = ( W − 1 ) × S − 2 P + K + P out O = (W - 1) \times S - 2P + K + P_{\text{out}} O = ( W − 1 ) × S − 2 P + K + P out # PyTorch calculation out_dim = (W - 1 ) * S - 2 * P + K + P_out Applying a dilation factor D D D
K eff = K + ( K − 1 ) ( D − 1 ) K_{\text{eff}} = K + (K - 1)(D - 1) K eff = K + ( K − 1 ) ( D − 1 ) The cumulative receptive field at layer l l l
R F l = R F l − 1 + ( K l − 1 ) × S cum RF_l = RF_{l-1} + (K_l - 1) \times S_{\text{cum}} R F l = R F l − 1 + ( K l − 1 ) × S cum where S cum = ∏ i = 1 l − 1 S i S_{\text{cum}} = \prod_{i=1}^{l-1} S_i S cum = ∏ i = 1 l − 1 S i
Attention ( Q , K , V ) = softmax ( Q K T d k ) V \text{Attention}(Q, K, V) = \text{softmax}\left( \frac{QK^T}{\sqrt{d_k}} \right) V Attention ( Q , K , V ) = softmax ( d k Q K T ) V import torch import math def scaled_dot_product (q, k, v, mask = None ): d_k = q.size( - 1 ) scores = torch.matmul(q, k.transpose( - 2 , - 1 )) / math.sqrt(d_k) if mask is not None : scores = scores.masked_fill(mask == 0 , - 1e9 ) weights = torch.softmax(scores, dim =- 1 ) return torch.matmul(weights, v), weights Tensor Shape Description Input X X X [ B , L , d model ] [B, L, d_{\text{model}}] [ B , L , d model ] Batch size B B B L L L d model d_{\text{model}} d model Query Q Q Q K K K [ B , H , L , d k ] [B, H, L, d_k] [ B , H , L , d k ] Heads H H H d k = d model / H d_k = d_{\text{model}} / H d k = d model / H Value V V V [ B , H , L , d v ] [B, H, L, d_v] [ B , H , L , d v ] Value dimension d v = d model / H d_v = d_{\text{model}} / H d v = d model / H Attention Scores [ B , H , L , L ] [B, H, L, L] [ B , H , L , L ] Pairwise token correlation matrix Output [ B , L , d model ] [B, L, d_{\text{model}}] [ B , L , d model ] Linearly projected concatenated heads
Rotates consecutive pairs of query and key coordinates by frequency angles:
θ i = 10000 − 2 ( i − 1 ) / d \theta_i = 10000^{-2(i-1)/d} θ i = 1000 0 − 2 ( i − 1 ) / d R Θ , m d x m = ( x m ( 1 ) cos ( m θ 1 ) − x m ( 2 ) sin ( m θ 1 ) x m ( 1 ) sin ( m θ 1 ) + x m ( 2 ) cos ( m θ 1 ) ⋮ ) R_{\Theta, m}^d x_m = \begin{pmatrix} x_m^{(1)} \cos(m\theta_1) - x_m^{(2)} \sin(m\theta_1) \\ x_m^{(1)} \sin(m\theta_1) + x_m^{(2)} \cos(m\theta_1) \\ \vdots \end{pmatrix} R Θ , m d x m = x m ( 1 ) cos ( m θ 1 ) − x m ( 2 ) sin ( m θ 1 ) x m ( 1 ) sin ( m θ 1 ) + x m ( 2 ) cos ( m θ 1 ) ⋮
RoPE preserves relative distance: ⟨ R Θ , m q , R Θ , n k ⟩ \langle R_{\Theta, m} q, R_{\Theta, n} k \rangle ⟨ R Θ , m q , R Θ , n k ⟩ ( m − n ) (m - n) ( m − n )
Function Formula Output Range Derivative / Behavior ReLU max ( 0 , x ) \max(0, x) max ( 0 , x ) [ 0 , ∞ ) [0, \infty) [ 0 , ∞ ) 1 1 1 x > 0 x > 0 x > 0 0 0 0 GELU x Φ ( x ) ≈ 0.5 x [ 1 + tanh ( 2 / π ( x + 0.044715 x 3 ) ) ] x \Phi(x) \approx 0.5x[1 + \tanh(\sqrt{2/\pi}(x + 0.044715x^3))] x Φ ( x ) ≈ 0.5 x [ 1 + tanh ( 2/ π ( x + 0.044715 x 3 ))] [ − 0.17 , ∞ ) [-0.17, \infty) [ − 0.17 , ∞ ) Smooth probabilistic stochastic regularizer SiLU / Swish x ⋅ σ ( x ) = x 1 + e − x x \cdot \sigma(x) = \frac{x}{1 + e^{-x}} x ⋅ σ ( x ) = 1 + e − x x [ − 0.28 , ∞ ) [-0.28, \infty) [ − 0.28 , ∞ ) Self-gated non-monotonic activation SwiGLU Swish ( x W gate ) ⊙ ( x W up ) \text{Swish}(x W_{\text{gate}}) \odot (x W_{\text{up}}) Swish ( x W gate ) ⊙ ( x W up ) ( − ∞ , ∞ ) (-\infty, \infty) ( − ∞ , ∞ ) Modern transformer MLP standard (LLaMA) Softmax σ ( z ) i = e z i / τ ∑ j e z j / τ \sigma(z)_i = \frac{e^{z_i / \tau}}{\sum_j e^{z_j / \tau}} σ ( z ) i = ∑ j e z j / τ e z i / τ ( 0 , 1 ) (0, 1) ( 0 , 1 ) Normalizes logits to sum to 1; temperature τ \tau τ
m t = β 1 m t − 1 + ( 1 − β 1 ) g t m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t m t = β 1 m t − 1 + ( 1 − β 1 ) g t v t = β 2 v t − 1 + ( 1 − β 2 ) g t 2 v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2 v t = β 2 v t − 1 + ( 1 − β 2 ) g t 2 m ^ t = m t 1 − β 1 t , v ^ t = v t 1 − β 2 t \hat{m}_t = \frac{m_t}{1 - \beta_1^t}, \quad \hat{v}_t = \frac{v_t}{1 - \beta_2^t} m ^ t = 1 − β 1 t m t , v ^ t = 1 − β 2 t v t θ t = θ t − 1 − η λ θ t − 1 − η v ^ t + ϵ m ^ t \theta_t = \theta_{t-1} - \eta \lambda \theta_{t-1} - \frac{\eta}{\sqrt{\hat{v}_t} + \epsilon} \hat{m}_t θ t = θ t − 1 − η λ θ t − 1 − v ^ t + ϵ η m ^ t
In standard L2 regularization (Adam), weight decay is coupled to gradient scaling: g t + λ θ v t \frac{g_t + \lambda \theta}{\sqrt{v_t}} v t g t + λ θ θ − η λ θ \theta - \eta \lambda \theta θ − η λ θ
η t = η min + 1 2 ( η max − η min ) ( 1 + cos ( t T max π ) ) \eta_t = \eta_{\min} + \frac{1}{2}(\eta_{\max} - \eta_{\min})\left(1 + \cos\left(\frac{t}{T_{\max}}\pi\right)\right) η t = η m i n + 2 1 ( η m a x − η m i n ) ( 1 + cos ( T m a x t π ) ) Given variance schedule β 1 , … , β T \beta_1, \dots, \beta_T β 1 , … , β T α t = 1 − β t \alpha_t = 1 - \beta_t α t = 1 − β t α ˉ t = ∏ s = 1 t α s \bar{\alpha}_t = \prod_{s=1}^t \alpha_s α ˉ t = ∏ s = 1 t α s
q ( x t ∣ x 0 ) = N ( x t ; α ˉ t x 0 , ( 1 − α ˉ t ) I ) q(x_t \mid x_0) = \mathcal{N}\left(x_t; \sqrt{\bar{\alpha}_t} x_0, (1 - \bar{\alpha}_t) \mathbf{I}\right) q ( x t ∣ x 0 ) = N ( x t ; α ˉ t x 0 , ( 1 − α ˉ t ) I ) Sampling at arbitrary timestep t t t
x t = α ˉ t x 0 + 1 − α ˉ t ϵ , ϵ ∼ N ( 0 , I ) x_t = \sqrt{\bar{\alpha}_t} x_0 + \sqrt{1 - \bar{\alpha}_t} \epsilon, \quad \epsilon \sim \mathcal{N}(0, \mathbf{I}) x t = α ˉ t x 0 + 1 − α ˉ t ϵ , ϵ ∼ N ( 0 , I ) log p ( x ) ≥ E q ϕ ( z ∣ x ) [ log p θ ( x ∣ z ) ] − D KL ( q ϕ ( z ∣ x ) ∥ p ( z ) ) \log p(x) \ge \mathbb{E}_{q_\phi(z \mid x)}\left[\log p_\theta(x \mid z)\right] - D_{\text{KL}}\left(q_\phi(z \mid x) \parallel p(z)\right) log p ( x ) ≥ E q ϕ ( z ∣ x ) [ log p θ ( x ∣ z ) ] − D KL ( q ϕ ( z ∣ x ) ∥ p ( z ) ) For encoder output q ϕ ( z ∣ x ) = N ( μ , diag ( σ 2 ) ) q_\phi(z \mid x) = \mathcal{N}(\mu, \text{diag}(\sigma^2)) q ϕ ( z ∣ x ) = N ( μ , diag ( σ 2 )) p ( z ) = N ( 0 , I ) p(z) = \mathcal{N}(0, \mathbf{I}) p ( z ) = N ( 0 , I )
D KL = − 1 2 ∑ j = 1 J ( 1 + log ( σ j 2 ) − μ j 2 − σ j 2 ) D_{\text{KL}} = -\frac{1}{2} \sum_{j=1}^J \left(1 + \log(\sigma_j^2) - \mu_j^2 - \sigma_j^2\right) D KL = − 2 1 j = 1 ∑ J ( 1 + log ( σ j 2 ) − μ j 2 − σ j 2 ) kl_loss = - 0.5 * torch.sum( 1 + logvar - mu.pow( 2 ) - logvar.exp())