# Restricted Boltzmann Machine and Deep Belief Network: Tutorial and Survey

@article{Ghojogh2021RestrictedBM, title={Restricted Boltzmann Machine and Deep Belief Network: Tutorial and Survey}, author={Benyamin Ghojogh and Ali Ghodsi and Fakhri Karray and Mark Crowley}, journal={ArXiv}, year={2021}, volume={abs/2107.12521} }

This is a tutorial and survey paper on Boltzmann Machine (BM), Restricted Boltzmann Machine (RBM), and Deep Belief Network (DBN). We start with the required background on probabilistic graphical models, Markov random field, Gibbs sampling, statistical physics, Ising model, and the Hopfield network. Then, we introduce the structures of BM and RBM. The conditional distributions of visible and hidden variables, Gibbs sampling in RBM for generating variables, training BM and RBM by maximum… Expand

#### References

SHOWING 1-10 OF 66 REFERENCES

Deep Boltzmann Machines

- Computer Science
- AISTATS
- 2009

A new learning algorithm for Boltzmann machines that contain many layers of hidden variables that is made more efficient by using a layer-by-layer “pre-training” phase that allows variational inference to be initialized with a single bottomup pass. Expand

An Efficient Learning Procedure for Deep Boltzmann Machines

- Medicine, Computer Science
- Neural Computation
- 2012

A new learning algorithm for Boltzmann machines that contain many layers of hidden variables is presented and results on the MNIST and NORB data sets are presented showing that deep BoltZmann machines learn very good generative models of handwritten digits and 3D objects. Expand

The Recurrent Temporal Restricted Boltzmann Machine

- Mathematics, Computer Science
- NIPS
- 2008

The Recurrent TRBM is introduced, which is a very slight modification of the TRBM for which exact inference is very easy and exact gradient learning is almost tractable. Expand

Efficient Learning of Deep Boltzmann Machines

- Mathematics, Computer Science
- AISTATS
- 2010

We present a new approximate inference algorithm for Deep Boltzmann Machines (DBM’s), a generative model with many layers of hidden variables. The algorithm learns a separate “recognition” model that… Expand

How to Center Deep Boltzmann Machines

- Mathematics, Computer Science
- J. Mach. Learn. Res.
- 2016

Analytically, centered and normal Boltzmann Machines (BMs) are different parameterizations of the same model class, such that any normal BM can be transformed to an equivalent centered BM/RBM/DBM and vice versa, and that this equivalence generalizes to artificial neural networks in general. Expand

Learning Deep Boltzmann Machines using Adaptive MCMC

- Computer Science
- ICML
- 2010

This paper first shows a close connection between Fast PCD and adaptive MCMC, and develops a Coupled Adaptive Simulated Tempering algorithm that can be used to better explore a highly multimodal energy landscape. Expand

A Fast Learning Algorithm for Deep Belief Nets

- Mathematics, Computer Science
- Neural Computation
- 2006

A fast, greedy algorithm is derived that can learn deep, directed belief networks one layer at a time, provided the top two layers form an undirected associative memory. Expand

Annealing and Replica-Symmetry in Deep Boltzmann Machines

- Physics, Mathematics
- 2020

In this paper we study the properties of the quenched pressure of a multi-layer spin-glass model (a deep Boltzmann Machine in artificial intelligence jargon) whose pairwise interactions are allowed… Expand

Multi-Prediction Deep Boltzmann Machines

- Computer Science
- NIPS
- 2013

The multi-prediction deep Boltzmann machine does not require greedy layerwise pretraining, and outperforms the standard DBM at classification, classification with missing inputs, and mean field prediction tasks. Expand

Greedy Layer-Wise Training of Deep Networks

- Computer Science
- NIPS
- 2006

These experiments confirm the hypothesis that the greedy layer-wise unsupervised training strategy mostly helps the optimization, by initializing weights in a region near a good local minimum, giving rise to internal distributed representations that are high-level abstractions of the input, bringing better generalization. Expand