Information is represented and transmitted in the brain by the joint activity of large groups of neurons. Understanding how information is “written” in these population patterns, and how it is read and processed, is a fundamental question in neuroscience. Yet, because of the huge number of potential activity patterns and complexity of natural stimuli, most of our understanding of the code relies on single neuron studies. We will extend and apply mathematical tools from information theory, machine learning, and physics, to overcome this ‘curse of dimensionality’ and build neural dictionaries relating activity and stimuli at an unparalleled resolution of hundreds of neurons. To identify the fundamental design principles of neural population codes we will study the spatial and spatio-temporal activity of hundreds of neurons from the retina, tectum, and cortical networks responding to naturalistic and artificial stimuli. Our primary goals are: (a) to characterize the encoding ‘codebooks’ of large populations of neurons, and the effect of network noise on encoding, and thus construct a thesaurus for neural populations, (b) use this thesaurus to develop new family of decoders of population activity which would be biologically plausible and accurate for natural stimuli, (c) characterize adaptation at the level of the code of networks of neurons, and the effect of learning on population neural codes, (d) explore “learnability” as a key feature of the neural code, and construct biologically plausible models of how the brain can learn to read population codes and compute, and (e) merge these ideas into a new mathematical framework that will connect the architecture of neural interaction networks and the properties of their neural codes. Our work will establish a new mathematical framework for studying the neural code, which will entail important implications for neural prostheses and brain machine interfaces, as well as brain-inspired learning algorithms.