The mind is not an unitary entity, nor is its physical substrate, the brain. Both can be divided into multiple components, some of which have been referred to as modules. Many controversies exist in cognitive science, psychology, and philosophy about the properties and the status of these modules. A compromise view is offered by an hypothesis of modularization which has two central tenets: (i) Genetic influences determine a weak non-modular organization of the mind and (ii) this map develops into a set of module-like compartments. Here we will test this hypothesis in the domain of visual object knowledge. Testable predictions are derived from a novel extension and integration of previous proposals (i) for the presence of non-modular maps (Op de Beeck et al., 2008, Nature Rev. Neurosci.), which are logical candidates for the starting point proposed in the modularization hypothesis, and (ii) for how maps might be transformed by further experience (Op de Beeck & Baker, 2010, Trends in Cognit. Sci.) into a strong compartmentalization for specific types of visual stimuli. We will determine whether the same rules govern modularization for face perception and reading, despite the very different evolutionary history of faces and word stimuli. We will apply well-known analysis tools from the psychology literature, such as multidimensional scaling, to the patterns of activity obtained by brain imaging, so that we can directly compare the structure and modularity of visual processing in mental space with the structure of “brain space” (functional anatomy). The combined behavioral and imaging experiments will characterize the properties of non-modular maps and module-like regions in sighted and congenitally blind adults and in children, and test specific hypotheses about how experience affects non-modular maps and the degree of modularization. The findings will reveal how the structure of the adult mind is the dynamic end point of a process of modularization in the brain.