Efficient Information Representation and Propagation in Artificial Neural Networks

In the last few years, major milestones have been achieved in the field of artificial intelligence and neural networks. Part of these leaps forward can be explained by the ever-increasing amount of available training data and by the technological advances of modern computers, but these innovations alone cannot justify such unprecedented breakthroughs. In this thesis it will be discussed the idea that a driving factor of these improvements can be identified in innovations related to information representation and propagation. In particular, the majority of these achievements are linked to the development of new architectures rather than to improvements to existing ones, and these architectural innovations are almost always associated with novel information representation and propagation paradigms. In the following chapters, the concepts and motivations underlying these innovations will be discussed, and following those steps, some innovative architectures which use novel information representation and propagation schemes will be presented. In particular, a new form of bio-inspired information representation based on computational stigmergy will be presented and, after its mathematical formalization, two novel architectures based on it called Stigmergic Neural Network and Stigmergic Memory for Recurrent Neural Networks will be derived. Information propagation in sparse neural networks will also be discussed and the novel Mesh Neural Network and Competitive Joint Unstructured Neural Network architectures will be presented, formalized and discussed. It will be shown how some issues in the design of the backward signals of one of the most used reinforcement learning algorithm are the cause of major information loss, and a solution, with its mathematical proof, will be presented and discussed. Finally, the propagation of multimodal information in complex systems will be discussed and an innovative architecture called CLIP-Guided Generative Latent Space Search, capable of generating images from texts and vice versa via the orchestration and optimization of the information through generative and multimodal networks using an algorithm genetic will be presented.