One of the earliest recurrent neural networks reported in literature was the auto-associator

independently described by Anderson (Anderson, 1977) and Kohonen (Kohonen, 1977) in 1977.

It consists of a pool of neurons with connections between each unit i and j, i 6= j.

The auto-associator network. All neurons are both input and output neurons, i.e., a

pattern is clamped, the network iterates to a stable state, and the output of the network consists of

the new activation values of the neurons.

All connections are weighted.

In 1982, Hopfield (Hopfield, 1982) brings together several earlier ideas concerning these

networks and presents a complete mathematical analysis based on Ising spin models (Amit,

Gutfreund, & Sompolinsky, 1986). It is therefore that this network, which we will describe in

this chapter, is generally referred to as the Hopfield network.

The Hopfield network consists of a set of N interconnected neurons which update

their activation values asynchronously and independently of other neurons. All neurons are

both input and output neurons. The activation values are binary. Originally, Hopeld chose

activation values of 1 and 0, but using values +1 and -1 presents some advantages discussed

below. We will therefore adhere to the latter convention.

The state of the system is given by the activation values1 y = (yk). The net input sk(t+1)

of a neuron k at cycle t+1 is a weighted sum

A simple threshold function is applied to the net input to obtain the new activation

value yi(t + 1) at time t + 1:

### Hopfield network as associative memory

A primary application of the Hopfield network is an associative memory. In this case, the

weights of the connections between the neurons have to be thus set that the states of the system

corresponding with the patterns which are to be stored in the network are stable. These states

can be seen as ‘dips’ in energy space. When the network is cued with a noisy or incomplete test

pattern, it will render the incorrect or missing data by iterating to a stable state which is in

some sense ‘near’ to the cued pattern.

i.e., if xp

j and xp

k are equal, wjk is increased, otherwise decreased by one (note that, in the original

Hebb rule, weights only increase). It appears, however, that the network gets saturated very

quickly, and that about 0:15N memories can be stored before recall errors become severe.

There are two problems associated with storing too many patterns:

- the stored patterns become unstable;
- spurious stable states appear (i.e., stable states which do not correspond with stored

patterns).

The first of these two problems can be solved by an algorithm proposed by Bruce et al. (Bruce,

Canning, Forrest, Gardner, & Wallace, 1986):

**Algorithm** 1 Given a starting weight matrix for each pattern xp to be stored and

each element xp

k in xp dene a correction εk such that

*Now modify wjk by Repeat this procedure until all patterns are
stable.*

It appears that, in practice, this algorithm usually converges. There exist cases, however, where

the algorithm remains oscillatory (try to find one)!