Web Information Retrieval

With increasingly higher numbers of non-English language web searchers the problems of efficient handling of non-English Web documents and user queries are becoming major issues for search engines. The main aim of this review paper is to make researchers aware of the existing problems in monolingual non-English Web retrieval by providing an overview of open issues. A significant number of

The information era has brought with it the wellknown problem of 'Information Explosion'. There are many and varied search engines on the Internet but it is still hard to locate and concentrate only on materials relevant to a specific task. Digital libraries, on the other hand, provide better services for focused discovery of relevant Web resources. However, digital libraries have been much less researched and implemented than search engines. The 'Katsir/Harvest' project laid the ground for our understanding that a new paradigm should to be developed the Harvested Digital Library (HDL). The contribution of this article is in presenting a new framework and harvesting model for constructing HDLs. The open harvesting architecture proposed here uses advanced information retrieval tools and provides a set of integrated DL services to its users. This model and architecture are discussed throughout the article, including description of the implemented Katsir system and disc...

With increasingly higher numbers of non-English language web searchers the problems of efficient handling of non-English Web documents and user queries are becoming major issues for search engines. The main aim of this review paper is to make researchers aware of the existing problems in monolingual non-English Web retrieval by providing an overview of open issues. A significant number of papers are reviewed and the research issues investigated in these studies are categorized in order to identify the research questions and solutions proposed in these papers. Further research is proposed at the end of each section.

Due to the numerous health documents available on the Web, information retrieval remains problematic with existing tools. This paper is positioned within the context of the CISMeF project (acronym of Catalogue and Index of French-speaking Medical Sites) and of a future Semantic Web. In CISMeF the resources are described using a set of metadata based on a structured terminology which "encapsulates" the MeSH thesaurus in its French version. Now, the objective is to migrate the CISMeF terminology, and thus the MeSH thesaurus, to a formal ontology, so as to get a more powerful search tool. The paper presents the very first stage and results of this ongoing project, aiming at migrating the MeSH to OWL. It reports on the first steps, which have presently been done, concerning the automatic transformation of the terminology into OWL-DL. First, the CISMeF terminology has been "formalized" in OWL. Then, the resulting OWL "ontology" has been imported under the Protégé editor which makes possible to check its consistency and its classification in using Racer. Finally, the paper concludes on the current results and encountered difficulties, and gives future work perspectives.

The objective of this paper is to analyze web logs using data mining so as to present users with more personalized web content. We classify users based on their internet usage patterns and for each class, maintain a cache of web documents. Searching is performed based on term set analysis and direction cosine distance approach. Finally a more personalized and

As more information becomes available on the World Wide Web (there are currently over 4 billion pages covering most areas of human endeavor), it becomes more difficult to provide effective search tools for information access. Today, people access web information through two main kinds of search interfaces: Browsers (clicking and following hyperlinks) and Query Engines (queries in the form of a set of keywords showing the topic of interest). The first process is tentative and time consuming and the second may not satisfy the user because ...

Vertical search engines and web portals are gaining ground over the general-purpose engines due to their limited size and their high precision for the domain they cover. The number of vertical portals has rapidly increased over the last years, making the importance of a topic-driven (focused) crawler evident. In this paper, we develop a latent semantic indexing classifier that combines link analysis with text content in order to retrieve and index domain specific web documents. We compare its efficiency with other well-known web information retrieval techniques. Our implementation presents a different approach to focused crawling and aims to overcome the size limitations of the initial training data while maintaining a high recall/precision ratio.

The set of information spaces collectively referred as Internet poses serious problems to information retrieval tasks. Content evolution of Internet spaces and documents is reviewed and distinctive features of web documents are empathized. Web search engines are classified according to their scope, functionalities and retrieval philosophy. A chapter is devoted to the characterization of Spanish web though the study of a random set of web sites, their quantitative composition and their qualitative features.
The analysis of search engines of the Spanish web begins with a study of coverage, methods of crawling, data schema and indexing mechanisms. Finally, eight search engines (AltaVista, EnlaWeb, Lycos, Olé/Terra, Ozú, Sol, Ya and Yahoo) were evaluated in retrieving information from Spanish web space. Indicators chosen were their relative coverage, specific offering, proportion of dead links and accessibility of Spanish websites. Performance was determined by relative recall and precision in retrieval during the first quarter in 2003. Search topics and relevance of results were determined by the end users. 12,4% of the searches led to dead links and 76% of the pages were returned by only a single system. System performance, expressed in terms of recall ranged from 7% (AltaVista) to 14% (Ozú) and precision between 9% (Sol) and 30% (Ozú). Only Yahoo displayed typical inverse relationship between recall and precision figures. The rest of the systems invariably showed an increase in precision figures starting with the second or third search result, suggesting problems with the sorting algorithm.

Abstract. Metasearch engines are increasingly becoming a very useful tool for Web information retrieval. Their success depends mainly on their rank aggregation (fusion) method, their interface and their total “sustainability”, meaning that the engine is updated regularly and will still be around after some time. In this paper we describe QuadSearch, an experimental metasearch engine that provides simultaneous access in four major conventional, crawler-based search engines. The heart of the new metasearch engine is based on two novel rank-based aggregation algorithms. Users can choose which rank aggregation algorithm to use and thus to adjust the results at their own needs. They can also alter the form of the result list in a more convenient and statistically enhanced way. The QuadSearch engine aims to combine speed, reliable rank aggregation method, “spam ” free results, and detailed and enriched information. A publicly accessible interface for the new metasearch engine can be found at

With the advent of the Internet, a new era of digital information exchange has begun. Currently, the Internet encompasses more than five billion online sites and this number is exponentially increasing every day. Fundamentally, Information Retrieval (IR) is the science and practice of storing documents and retrieving information from within these documents. Mathematically, IR systems are at the core based on a feature vector model coupled with a term weighting scheme that weights terms in a document according to their significance with respect to the context in which they appear. Practically, Vector Space Model (VSM), Term Frequency (TF), and Inverse Term Frequency (IDF) are among other long-established techniques employed in mainstream IR systems. However, present IR models only target generic-type text documents, in that, they do not consider specific formats of files such as HTML web documents. This paper proposes a new semantic-sensitive web information retrieval model for HTML ...

Content-Based Image Retrieval (CBIR) locates, retrieves and displays images alike to one given as a query, using a set of features. It demands accessible data in medical archives and from medical equipment, to infer meaning after some processing. A problem similar in some sense to the target image can aid clinicians. CBIR complements text-based retrieval and improves evidence-based diagnosis, administration, teaching, and research in healthcare. It facilitates visual/automatic diagnosis and decision-making in real-time remote consultation/screening, store-and-forward tests, home care assistance and overall patient surveillance. Metrics help comparing visual data and improve diagnostic. Specially designed architectures can benefit from the application scenario. CBIR use calls for file storage standardization, querying procedures, efficient image transmission, realistic databases, global availability, access simplicity, and Internet-based structures. This chapter recommends important and complex aspects required to handle visual content in healthcare.

The information era has brought with it the wellknown problem of 'Information Explosion'. There are many and varied search engines on the Internet but it is still hard to locate and concentrate only on materials relevant to a specific task. Digital libraries, on the other hand, provide better services for focused discovery of relevant Web resources. However, digital libraries have been much less researched and implemented than search engines. The 'Katsir/Harvest' project laid the ground for our understanding that a new paradigm should to be developed -the Harvested Digital Library (HDL). The contribution of this article is in presenting a new framework and harvesting model for constructing HDLs. The open harvesting architecture proposed here uses advanced information retrieval tools and provides a set of integrated DL services to its users. This model and architecture are discussed throughout the article, including description of the implemented Katsir system and discussion of future research directions. The future DLs will be knowledge rich in the sense that each DL contains relevant meta-information on its domain and employs advanced knowledge management techniques.

Queries and click-through data taken from search engine transaction logs is an attractive alternative to traditional test collections, due to its volume and the direct relation to end-user querying. The overall aim of this paper is to an-swer the question: How does ...

SARE-Bi es un sistema de gestión integral de documentos multilingües, que está basado en esquemas de descripción de metadatos que provienen de la anotación de corpora textuales (TEI), de la traducción asistida por ordenador (TMX) y de la localización de software (XLIFF). Todos estos modelos son dialectos de XML que se solapan y complementan de diversas maneras. Dichos estándares se han incorporado y adaptado a un conocido sistema de publicación web (Zope), resultando un entorno cooperativo en que usuarios con diferentes roles (redactores, traductores, administradores) pueden llevar a cabo el ciclo completo de generación y traducción de nuevos documentos, mediante la reutilización de los ya existentes en el sistema.

Information retrieval mechanisms from the web are a great need of the hour as the amount of the content is growing dynamically every day. There are many algorithms which have been proposed in literature mainly relying on the output of the search engines. These algorithms are either content based or snippet based and perform a clustered outcome re-ranking of the content for the user. This work proposes a hybrid approach to content clustering that combines the best of the web information retrieval methods and also uses the personal preference information of the users modeling a wide range of contexts. This work introduces a context mechanism of the users in the overall process and presents taxonomy of the methods to organize the output of the search engines. Experimental results are promising and show that this approach has great promise for a wide range of queries.

With the advent of the cloud computing, web servers, as the major channel in cloud computing, need to be redesigned to meet performance and power constraints. Considerable efforts have been invested in distributed web servers and web caching with dijJerent optimizing strategies, but few existing studies have been directly focused on improving the web server itself, not to mention complete hardware-favored web services. In this paper, we propose a novel architecture of web server and implement it on FPGA. After taking challeng with significant difficulties in design and implementation, we manage to complete an evaluation system which confirms that hardware-favored architecture brings higher throughput, lower power consumption as well as stand-alone web service functionalities due to direct pipelining execution of web service protocols in hardware without operating system

Development of methods for Information Retrieval based on conceptual aspects is vital to reduce the quantity of unimportant documents retrieved by the search engines. In this paper, a method for expanding user queries is presented, such that for each term in the original query, all of its synonyms by a certain meaning with maximum concept frequency are introduced. To measure the degree of concept presence in a document (or even in a document collection), a concept frequency formula is introduced. New fuzzy formulas are also introduced to calculate the synonymy degree between terms to manage with concepts (meanings). With them, even though a certain term does not appear in a document, some degree of its presence could be estimated based on its degree of synonymy with terms that do appear in the document. A polysemy index is also introduced in order to simplify the treatment of weak and strong words.

In (Computing with Words, Wiley, New York, 2001, p. 251; Soft Comput. 6 (2002) 320; Fuzzy Logic and The Internet, Physica-Verlag, Springer, Wurzburg, Berlin, 2003) we presented di erent fuzzy linguistic multi-agent models for helping users in their information gathering processes on the Web. In this paper we describe a new fuzzy linguistic multi-agent model that incorporates two information ÿltering techniques in its structure: a content-based ÿltering agent and a collaborative ÿltering agent. Both elements are introduced to increase the information ÿltering possibilities of multi-agent system on the Web and, in such a way, to improve its retrieval issues.

Language identification is an important task for web information retrieval. This paper presents the implementation of a tool for language identification in mono-and multi-lingual documents. The tool implements four algorithms for language identification. Furthermore, we present a n-gram approach for the identification of languages in multi-lingual documents. An evaluation for monolingual texts of varied length is presented. Results for eight languages including Ukrainian and Russian are presented. It could be shown that n-gram-based approaches outperform word-based algorithms for short texts. For longer texts, the performance is comparable. The evaluation for multi-lingual documents is based on real world web documents. Our tool is able to recognize the languages present in a document with reasonable accuracy.

Today, internet has become the most important source of information. People are highly accustomed to the use of internet for acquiring information which they need. Many times, it is revealed that, the information seeker does not get relevant information very easily due to the presence of non-relevant web pages. This paper addresses the problem of effective information retrieval from the web. In this paper, the notion of Web Information Retrieval using Automatic Multi-document Summarization is presented. The proposed work is blend of Web technology and Natural Language Processing. When user will fire the query, the system tries to fetch web pages from different web servers, and they are indexed as per the order of relevance. The degree of relevance is not determined by the how many times the keywords of query is present in the document but it is determined on the basis of semantic content of the document and the user query

The success of the semantic web depends largely on how well ontologies can be utilized and formulated. Interoperability between systems using different versions of the same ontology i s essential, and this implies the need for a regulated derivation of materialized ontology views (which can be considered a modified version of an ontology). This chapter applies the formalisms for such a derivation process to a practical example, emphasizing the possibility for automation, and also optimization to develop a high quality derived ontology.

We give an overview of our experience in utilizing several open source packages and composing them into sophisticated applications to solve several challenging problems as part of some of the research projects at the Knowledge Discovery & Web Mining lab at the Universe of Louisville. The projects have a common theme of knowledge discovery, however their application domains span a variety of areas. These areas range from mining Web data streams to mining Astronomy related image data, as well as Web information retrieval in social multimedia websites and e-learning platforms. As is already known, a significant proportion of the effort in any real life project involving knowledge discovery in data (KDD) is devoted to the early and final stages of KDD, i.e., the data collection and preprocessing, and the visualization of the results. Given the nature of the data in our projects, we expose our experience in handling text data and image data as part of the KDD process. In addition to the open source packages that we used, we will briefly present some of the stand-alone software that we developed in the lab, in particular a suite of software for clustering and for stream data mining.

Ambiguous queries constitute a significant fraction of search instances and pose real challenges to web search engines. With current approaches the top results for these queries tend to be homogeneous, making it difficult for users interested in less popular aspects to find relevant documents. While existing research in search diversification offers several solutions for introducing variety into the results, the majority of such work is predicated, implicitly or otherwise, on the assumption that a single relevant document will fulfill a user's information need, making them inadequate for many informational queries. In this paper we present a searchdiversification algorithm particularly suitable for informational queries by explicitly modeling that the user may need more than one page to satisfy their need. This modeling enables our algorithm to make a well-informed tradeoff between a user's desire for multiple relevant documents, probabilistic information about an average user's interest in the subtopics of a multifaceted query, and uncertainty in classifying documents into those subtopics. We evaluate the effectiveness of our algorithm against commercial search engine results and other modern ranking strategies, demonstrating notable improvement in multiple document scenarios.

With the advent of the Internet, a new era of digital information exchange has begun. Currently, the Internet encompasses more than five billion online sites and this number is exponentially increasing every day. Fundamentally, Information Retrieval (IR) is the science and practice of storing documents and retrieving information from within these documents. Mathematically, IR systems are at the core based on a feature vector model coupled with a term weighting scheme that weights terms in a document according to their significance with respect to the context in which they appear. Practically, Vector Space Model (VSM), Term Frequency (TF), and Inverse Term Frequency (IDF) are among other long-established techniques employed in mainstream IR systems. However, present IR models only target generic-type text documents, in that, they do not consider specific formats of files such as HTML web documents. This paper proposes a new semantic-sensitive web information retrieval model for HTML ...

Link analysis is the most important application of web structure mining and serves as a source of new knowledge in web information retrieval. Designing an effective link structure for customer interfaces is critical for the success of cybermalls. The user interface of a cybermall should provide friendly and pleasant shopping environments. A smart link structure that supports convenient moves and jumps serves to this end. We suggest the use of a customer tracking vector using the flow movement of customer interface. The main contribution of this paper is our unique approach to classify hypertext pages based on flow movement and extend the application of flow data for the design and structure of hypertext pages (nodes). This methodology help to create new links to obtain an efficient path to the right destination for customers when there are too many paths to choose from.

We give an overview of our experience in utilizing several open source packages and composing them into sophisticated applications to solve several challenging problems as part of some of the research projects at the Knowledge Discovery & Web Mining lab at the Universe of Louisville. The projects have a common theme of knowledge discovery, however their application domains span a variety of areas. These areas range from mining Web data streams to mining Astronomy related image data, as well as Web information retrieval in social multimedia websites and e-learning platforms. As is already known, a significant proportion of the effort in any real life project involving knowledge discovery in data (KDD) is devoted to the early and final stages of KDD, i.e., the data collection and preprocessing, and the visualization of the results. Given the nature of the data in our projects, we expose our experience in handling text data and image data as part of the KDD process. In addition to the open source packages that we used, we will briefly present some of the stand-alone software that we developed in the lab, in particular a suite of software for clustering and for stream data mining.

We present a dialogue system that enables the access in natural language to a web information retrieval system. We use a Web Semantic Language to model the knowledge conveyed by the texts. In this way we are able to obtain the associated knowledge necessary to perform the different analysis stages of natural language sentences. In the context of information retrieval, we aim to develop a system that, by increasing the interaction management capabilities, is able to achieve a better degree of cooperativeness and to reduce the average number of interactions needed to retrieve the intended set of documents. The documents in the IR system considered here are composed by the set of documents produced by the Portuguese Attorney General since 1940. These documents were analyzed and an ontology describing their structure and content was defined. Then, they were automatically parsed and a (partial) semantic structure was created. The ontology and the semantic content was represented in the OWL language. An example of a user interaction session is presented and explained in detail.

In this paper we propose a multi-agent architecture for web information retrieval using fuzzy logic based result fusion mechanism. The model is designed in JADE framework and takes advantage of JXTA agent communication method to allow agent communication through firewalls and network address translators. This approach enables developers to build and deploy P2P applications through a unified medium to manage agent-based document retrieval from multiple sources.

As an attempt to solve some contemporary web information retrieval problems, a construction of a cooperative multiagent system is proposed. This paper introduces the system and presents the use of a unique aggregation and fusion technique that is employed to attain reliable delivery performance. The intelligent methodology to fuse agents' decisions, the team consensus approach, models the interaction and bring a society into a consensus. After each agent in the group gathers information relevant to a user's query, the group engages in an uncertainty estimation stage. This process allows each agent to assess its self-uncertainty and the conditional uncertainties of other agents. The procedure facilitates the computation of a weighting scheme that operates recursively on information collected by these agents until the group reaches a consensus. Whenever a new task is received, the uncertainty estimates of agents are updated and used to compute a new weighting scheme.

As an attempt to solve some contemporary web information retrieval problems, a construction of a cooperative multiagent system is proposed. This paper introduces the system and presents the use of a unique aggregation and fusion technique that is employed to attain reliable delivery performance. The intelligent methodology to fuse agents' decisions, the team consensus approach, models the interaction and bring a society into a consensus. After each agent in the group gathers information relevant to a user's query, the group engages in an uncertainty estimation stage. This process allows each agent to assess its self-uncertainty and the conditional uncertainties of other agents. The procedure facilitates the computation of a weighting scheme that operates recursively on information collected by these agents until the group reaches a consensus. Whenever a new task is received, the uncertainty estimates of agents are updated and used to compute a new weighting scheme.

Contextual search tries to better capture a user's information need by augmenting the user's query with contextual information extracted from the search context (for example, terms from the web page the user is currently reading or a file the user is currently editing). This paper presents Y!Q-a first of its kind large-scale contextual search system-and provides an overview of its system design and architecture. Y!Q solves two major problems. First, how to capture high quality search context. Second, how to use that context in a way to improve the relevancy of search queries. To address the first problem, Y!Q introduces an information widget that captures precise search context and provides convenient access to its functionality at the point of inspiration. For example, Y!Q can be easily embedded into web pages using a web API, or it can be integrated into a web browser toolbar. This paper provides an overview of Y!Q's user interaction design, highlighting its novel aspects for capturing high quality search context. To address the second problem, Y!Q uses a semantic network for analyzing search context, possibly resolving ambiguous terms, and generating a contextual digest comprising its key concepts. This digest is passed through a query planner and rewriting framework for augmenting a user's search query with relevant context terms to improve the overall search relevancy and experience. We show experimental results comparing contextual Y!Q search results sideby-side with regular Yahoo! web search results. This evaluation suggests that Y!Q results are considered significantly more relevant. The paper also identifies interesting research problems and argues that contextual search may represent the next major step in the evolution of web search engines.

In this paper we investigate the use of stylistic features of Web texts in Portuguese to classify web pages according to users' needs, in order to improve Web Information Retrieval. We first describe a seven categories classification of users´ needs, which was the outcome of a qualitative analysis of two TodoBr logs (a major Brazilian search engine). We describe 46 shallow linguistic features, inspired by the works of Biber and Karlgren, and proceed describing the compilation of the corpus employed on the classifier training. Our aim is to obtain rules that can be applied on the classification of Web texts according to those seven users´ needs. Some experiments are reported, showing that it is possible, at least for some of the categories, to identify them reliably.

There is growing interest in accessing, relating, and combining data from multiple sources on the Web. Enormous amounts of heterogeneous information have been accumulated within corporations, government organization and universities. Such information continues to grow at an ever-increasing rate. This information comes from different subject areas, and comes in different formats: bitmap, plain text, binary, etc.

A common task in both Webmetrics and Web information retrieval is to identify a set of Web pages or sites that are similar in content. In this paper we assess the extent to which links, colinks and couplings can be used to identify similar Web sites. As an experiment, a random sample of 500 pairs of domains from the UK academic Web were taken and human assessments of site similarity, based upon content type, were compared against ratings for the three concepts. The results show that using a combination of all three gives the highest probability of identifying similar sites, but surprisingly this was only a marginal improvement over using links alone. Another unexpected result was that high values for either colink counts or couplings were associated with only a small increased likelihood of similarity. The principal advantage of using couplings and colinks was found to be greater coverage in terms of a much larger number of pairs of sites being connected by these measures, instead of increased probability of similarity. In information retrieval terminology, this is improved recall rather than improved precision.

In this paper we propose a multi-agent architecture for web information retrieval using fuzzy logic based result fusion mechanism. The model is designed in JADE framework and takes advantage of JXTA agent communication method to allow agent communication through firewalls and network address translators. This approach enables developers to build and deploy P2P applications through a unified medium to manage agent-based document retrieval from multiple sources.

Information retrieval on the Web is very different from retrieval in traditional indexed databases. This difference arises from: the high degree of dynamism of the Web; its hyper-linked character; the absence of a controlled indexing vocabulary; the heterogeneity of document types and authoring styles; the easy access that different types of users may have to it. Thus, since Web retrieval is substantially different from information retrieval, new or revised evaluative measures are required to assess retrieval performance using Web search engines. This paper suggests a number of different measures to evaluate information retrieval from the Web. The motivation behind each of these measures is presented, along with their descriptions and definitions. In the second part of the paper, application of these measures is illustrated in the evaluation of three search engines. The purpose of this paper is not to give the definite prescription for evaluating information retrieval from the Web, but rather to present some examples and to initiate a wider discussion of how to enhance measures of Web search performance.