Core idea. External memory is a 2D matrix M of shape N×W (N locations, each a W-dimensional row vector). A controller LSTM emits interface parameters each timestep to position soft attention heads anywhere over the N rows — no constraint to read/write only the top.

Addressing. Two-stage: (1) content addressing — compute cosine similarity between a key vector and every memory row, apply softmax → content weights. (2) location addressing — optionally interpolate with the previous head position, then convolve with a learned shift kernel, then sharpen. Together they allow both associative lookup and sequential scanning.

Reading. The read head returns r = MTw — a weighted sum of memory rows using the attention weights w. This is fully differentiable.

Writing. The write head first erases: M ← M ⊙ (1 − w·eT), then adds: M ← M + w·aT. Erase vector e and add vector a are both output by the controller.

Multiple heads. There can be R read heads and W write heads, each with independent addressing. All reads happen before all writes each timestep.

Core idea. The DNC extends the NTM with three additional mechanisms: (1) a usage vector u tracking how much each location has been written to, (2) a temporal link matrix L recording the order in which locations were written, and (3) a freeness vector f that releases locations when read-weights dealloc them.

Allocation addressing. Instead of the NTM's shift convolution, the DNC allocates free locations by sorting them by usage. The least-used (most free) location gets the highest allocation weight. This prevents memory fragmentation without needing sequential structure.

Write addressing. A gated blend of content-based address and allocation address: ww = gw·(ga·a + (1−ga)·wc) where g_w is write gate and g_a is allocation gate.

Temporal links. After each write, the link matrix L updates: L[i,j] is the degree to which location i was written just after j. This enables forward and backward traversal through previously written sequences — the read head can "follow the chain."

Read modes. Each read head blends three orientations: content-based w_c, forward link π_f·Lw_r, and backward link π_b·L^Tw_r. Soft mode weights π are output by the controller.

Write before read. Within each timestep the DNC writes first, then reads. This is intentional: the read vector r_t is part of the current output y_t = W_y[h_t; r_t], so reading from the already-updated M_t lets the model store a value and immediately reflect it in the same step's output. The temporal link L_t must also be updated before reading so that forward/backward traversal includes the current write. The cost: write and read keys are both derived from the same h_t, so the model cannot react to the content it just wrote within the same step.

Core idea. Neural RAM (nRAM) models a register-based random-access machine. Memory consists of a fixed set of integer-indexed slots. The controller emits a discrete address (argmax or Gumbel-softmax over N locations) rather than a continuous attention distribution — trading differentiability for sharpness.

Registers. A small set of R registers hold working values. Each timestep the controller (1) loads from memory into a register, (2) applies an ALU-like function (add, subtract, copy, compare), and (3) stores a register value back to memory. The program is a sequence of such micro-operations.

Addressing. Content-free — addresses are positional integers, not similarity lookups. The controller directly outputs a distribution over [0…N−1] and selects an address. This means reading a value requires knowing (or learning) where it was stored, not what it contains.

Soft vs hard. Training uses a soft (weighted) relaxation of discrete addressing so gradients flow. At test time, the argmax is used — one location, zero cost. The gap between soft training and hard inference is smaller than in NTM because the distribution is already very sharp.

Use case. Best for learning algorithms that are intrinsically positional: sort by index, random swap, array reversal. Content-based lookup (e.g., find a value equal to a key) is harder without similarity addressing.

Core idea. Replace the flat integer address space with a Lie group manifold. Each memory cell is pinned to a point on the manifold (a group element g_k ∈ G). The read/write head also lives on the manifold. Moving the head means multiplying its current position by a learned group element δ — a group action, not an index increment.

Lie groups used. Typical choices: SO(2) — the circle of rotations in 2D (parameterised by one angle θ); ℝⁿ — translations (recovers NTM-style shift); SE(2) — rotation + translation (richer 2D plane). The group law encodes the geometry: on SO(2), "move by δ" means θ ← θ + δ mod 2π.

Attention. At each timestep, similarity between head position h and each cell address g_k is measured by the group metric d(h, g_k) = ‖log(h⁻¹g_k)‖. A softmax over −β·d(·) gives attention weights w_k. Small d → high weight.

Navigation. The controller outputs a "step" δ ∈ G (via a learned map). The head updates: h_t+1 = h_t · δ. On SO(2) this is just θ_t+1 = θ_t + δ. The model can learn complex traversal patterns — e.g., a zigzag (alternating +90°, −90°) to visit odd-indexed cells before even ones (oddFirst task).

Read & write. Once attention weights are computed from geometry, reading and writing are identical to NTM: r = Σ w_k·v_k; write via erase+add. The only change is how w is computed — geometry replaces cosine similarity.