# Deep Learning 学习笔记 04 – Energy-Based Model

## Deep Learning 学习笔记 04 – Energy-Based Model

Contents

### Hopfield Network

$$\Delta D = D(…,y^+,…) – D(…,y^-,…) = y^+\left (\sum_{i\ne j} w_{ij}y_j +b_i\right) – y^-\left(\sum_{i\ne j} w_{ij}y_j +b_i\right) > 0$$

### Stochastic Hopfield Network

$$P(y_i|y_{i\ne j}) = \frac{1}{1 + \exp(-(\sum_j w_{ij}y_j + 2b_i)/T)}.$$

#### 最大似然学习(maximum likelihood learning)

$$P(y)= \frac{\exp(\frac12 y^TWy)}{\sum_{y’} \exp(\frac12 y’^T W y’)}.$$

$$L(W) = \frac{1}{|P|} \sum_{y\in P} \frac{1}{2}y^TWy – \log \sum_{y’}\exp(\frac12 y’^TWy’).$$

$$\nabla_{w_{ij}}L(W) = \frac{1}{|P|} \sum_{y\in P} y_iy_j – \frac{1}{Z}\sum_{y’} \exp(\frac12 y’^TWy’)y_i’y_j’$$

$$\nabla_{w_{ij}}L(W) := \frac{1}{|P|} \sum_{y\in P} y_iy_j – \frac{1}{|\mathcal S|} \sum_{y’ \in \mathcal S} y_i’y_j’.$$

#### 隐藏神经元(with Hidden Neurons)

$$P(v) = \sum_h P(v,h) = \sum_{y = (h,v)} \frac{\exp(\frac12 y^TWy)}{\sum_{y’} \exp(\frac12 y’^T W y’)}$$

$$L(W) = \frac{1}{|P|} \sum_{v\in P}\log\left( \sum_{y=(v,h)}\exp(\frac{1}{2}y^TWy)\right) – \log \sum_{y’}\exp(\frac12 y’^TWy’).$$

$$\nabla_{w_{ij}}L(W) := \frac{1}{|P|} \sum_{v\in P} \mathbb E_{h}[y_iy_j] – \mathbb E_{\mathcal S} [y_i’y_j’].$$

### 受限玻尔兹曼机(Restricted Boltzmann Machine)

Stochastic Hopfield Network确实厉害，而且有一套很好的理论性质，但是它是建立在Gibbs Sampling上的。Gibbs Sampling虽然可以做到多项式时间的采样，但是仍然需要多轮之后才能趋向平稳分布(stationary distribution)，尤其是在训练的过程中，有两个需要采样的项。

//Hinton 牛逼（