Information theoretical performance measure for associative memories and its application to neural networks

Three-state neural network: from mutual information to the Hamiltonian

The mutual information, I, of the three-state neural network can be obtained exactly for the mean-field architecture, as a function of three macroscopic parameters: the overlap, the neural activity and the activity-overlap, i.e., the overlap restricted to the active neurons. We perform an expansion of I on the overlap and the activity-overlap, around their values for neurons almost independent of the patterns. From this expansion we obtain an expression for a Hamiltonian which optimizes the retrieval properties of this system. This Hamiltonian has the form of a disordered Blume-Emery-Griffiths model. The dynamics corresponding to this Hamiltonian is found. As a special characteristic of such a network, we see that information can survive even if no overlap is present. Hence the basin of attraction of the patterns and the retrieval capacity is much larger than for the Hopfield network. The extreme diluted version is analyzed, the curves of information are plotted and the phase diagrams are built.