The simplest neural network architecture—the perceptron—was inspired by biological research to understand the basis of mental processing in an attempt to represent the function of the brain with mathematical formulae. In this section, we will cover some of this early research and how it inspired what is now the field of deep learning and generative AI.

From tissues to TLUs

The recent popularity of AI algorithms might give the false impression that this field is new. Many recent models are based on discoveries made decades ago that have been reinvigorated by the massive computational resources available in the cloud and customized hardware for parallel matrix computations such as Graphical Processing Units (GPUs), Tensor Processing Units (TPUs), and Field-Programmable Gate Array (FPGAs). If we consider research on neural networks to include their biological inspiration as well as computational theory, this field is over a hundred...

While large research funding for neural networks declined until the 1980s after the publication of Perceptrons, researchers still recognized that these models had value, particularly when assembled into multilayer networks, each composed of several perceptron units. Indeed, when the mathematical form of the output function (that is, the output of the model) was relaxed to take on many forms (such as a linear function or a sigmoid), these networks could solve both regression and classification problems, with theoretical results showing that three-layer networks could effectively approximate any output.¹³ However, none of this work addressed the practical limitations of computing the solutions to these models, with rules such as the perceptron learning algorithm described earlier proving a great limitation to their applied use. A central problem was how to appropriately estimate the weights in the hidden layers of the network, which form the...

Up until now, we’ve primarily discussed the basics of neural networks by referencing feedforward networks, where every input is connected to every output in each layer. While these feedforward networks are useful for illustrating how deep networks are trained, they are only one class of a broader set of architectures used in modern applications, including generative models. Thus, before covering some of the techniques that make training large networks practical, let’s review these alternative deep models.

Networks for seeing: convolutional architectures

As noted at the beginning of this chapter, one of the inspirations for deep neural network models is the biological nervous system. As researchers attempted to design computer vision systems that would mimic the functioning of the visual system, they turned to the architecture of the retina, as revealed by physiological studies by neurobiologists David Hubel and...

In addition to image data, natural language text has also been a frequent topic of interest in neural network research. However, unlike the datasets we’ve examined thus far, language has a distinct order that is important to its meaning. Thus, to accurately capture the patterns in language- or time-dependent data, it is necessary to utilize networks designed for this purpose.

RNNs and LSTMs

Let’s imagine we are trying to predict the next word in a sentence, given the words up until this point. A neural network that attempted to predict the next word would need to take into account not only the current word but also a variable number of prior inputs. If we instead used only a simple feedforward MLP, the network would essentially process the entire sentence or each word as a vector. This introduces the problem of either having to pad variable-length inputs to a common length and not preserving any notion of correlation (that is, which...

While we will discuss this topic in more detail in Chapter 4, it is important to note that convolutional and recursive units have been replaced in many current applications by transformers, a type of architecture first described in 2017 (https://arxiv.org/abs/1706.03762). In a way, transformers combine the strengths of both recursive and convolutional networks.

Like convolutional networks, they compute the relative similarity between elements in a sequence or matrix; however, unlike convolutional networks, they perform this calculation between all elements rather than just locally. Like LSTMs, they preserve a context window through positional encoding elements, the all-to-all pairwise similarity (also known as self-attention), and pass through connections that resemble the memory units in LSTMs. However, unlike LSTMs, they can be computed in parallel, enabling more efficient training.

Figure 2.17 gives an overview of how this remarkable operation works; each...

So far in this chapter, we have discussed several examples in which better neural network architectures allowed for breakthroughs; however, just as (and perhaps even more) important is the optimization procedure used to minimize the error function in these problems, which “learns” the parameters of the network by selecting those that yield the lowest error. Referring to our discussion of backpropagation, this problem has two components:

• How to initialize the weights: In many applications historically, we see that the authors used random weights within some range, and hoped that the use of backpropagation would result in at least a locally minimal loss function from this random starting point. Whether the activation functions in the network had saturated or 0 values (increasing the likelihood of uninformative gradients during training of the model) was not considered.
• How to find the local minimum loss: In basic backpropagation...

In this chapter, we’ve covered the basic vocabulary of deep learning—how initial research into perceptrons and MLPs led to simple learning rules being abandoned for backpropagation. We also looked at specialized neural network architectures such as CNNs, based on the visual cortex, and recurrent networks, specialized for sequence modeling. Finally, we examined variants of the gradient descent algorithm proposed originally for backpropagation, which have advantages such as momentum, and described weight initialization schemes that place the parameters of the network in a range that is easier to navigate to a local minimum.

With this context in place, we are all set to dive into projects in generative modeling, beginning with the generation of MNIST digits using deep belief networks in Chapter 11, Neural Networks Using VAEs.

1. López-Muñoz, F., Boya, J., and Alamo, C. (2006). Neuron theory, the cornerstone of neuroscience, on the centenary of the Nobel Prize award to Santiago Ramón y Cajal. Brain Research Bulletin. 70 (4–6): 391–405. https://pubmed.ncbi.nlm.nih.gov/17027775/
2. Ramón y Cajal, S. (1888). Estructura de los centros nerviosos de las aves.
3. McCulloch, W.S. and Pitts, W. (1943). A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathematical Biophysics 5, 115–133. https://doi.org/10.1007/BF02478259
4. Rashwan, M., Ez, R., and Abd El reheem, G. (2017). Computational Intelligent Algorithms For Arabic Speech Recognition. Journal of Al-Azhar University Engineering Sector. 12. 886-893. 10.21608/auej.2017.19198. https://jaes.journals.ekb.eg/article_19198.html
5. Artificial neuron. Wikipedia. Retrieved April 26, 2021, from https://en.wikipedia.org/wiki/Artificial_neuron
6. Shackleton-Jones, N. (2019, May...