Information Retrieval (IR) is a ubiquitous technology. The range of tasks and information collections IR systems are applied to is ever growing. However, the ability of researchers and developers to build high quality searching systems for new environments is not keeping up with the growth in potential application areas. As a consequence, search system quality is degrading: users of bespoke systems expect a Google-like experience of swift high quality retrieval; instead they commonly get slow sub-standard results. The key reason for this failing is the profound lack of testing resources that capture the unique and complex environments different retrieval systems are deployed in. The cost of building test resources is prohibitively expensive. Consequently, researchers are unable to assess the effectiveness of new ideas in such environments, search engine vendors struggle to tune their systems to the particular needs of their customers, customers looking to choose between competing search products do not have the resources to measure which system will better serve their needs. This project will address this problem by developing a new evaluation paradigm for search, which will, (1) establish an efficient and reliable test resource construction methodology, (2) develop a framework to adapt such resources to a wide variety of retrieval scenarios, and (3) extend the current evaluation paradigm to incorporate additional information about the different retrieval scenarios and users' activities. With such a methodology, researchers will develop and test a far richer set of approaches to search; companies will provide systems better suited for customer needs; organizations will be able to test more thoroughly their existing installed systems. Ultimately, developing the new paradigm will help ensure that the standard and quality of results common to web search will be available in all searching situations.