Data mining for scholarly journals: challenges and solutions for libraries

Abstract

As our global knowledge environment changes and the information to be found in scholarly journals becomes increasingly available in digital format, it is necessary to employ more and more sophisticated search and retrieval procedures to mine this knowledge. We have large holes in our globally accessible knowledge base as traditional web-crawlers cannot collect and assess all of the serially produced papers, articles and journals that exist. Many search engines only touch the surface and they cannot harvest potentially valuable information in the silos of the “deep web”. More comprehensive data mining is therefore essential if we are to effectively tap the knowledge often hidden in scholarly journals and databases. Data-mining models are being developed which aim to search all the global knowledge being produced--an essential goal that will aid in sharing and therefore accelerating global knowledge diffusion. Deep Web Technologies and World Wide Science.org are examples of ongoing efforts to assist in mining the rapidly increasing mass of serially produced scientific information. Knowledge can only be shared, advanced and accelerated if it is accessible and as users expect libraries to be ever more effective in gathering and utilizing knowledge they must serve the global community by offering the best access to and analysis of all information. This paper intends to contribute to a more comprehensive understanding of what information is potentially available and how to access and analyze it using the latest methods of information retrieval.