JWE Abstracts 

Vol.3 No.3&4 December, 2004
Introduction (pp169-170)
        A. Spink and C. Watters
I.  Web Search
Why Is Web Search So Hard... to Evaluate? (pp171-181)
        D.E. Rose
Web search has several important characteristics that distinguish it from traditional information retrieval:  the often adversarial relationship between content creators and search engine designers, the nature of the corpus, and the multiplicity of user goals.  In addition to making the search task itself difficult, these characteristics make it particularly hard to evaluate search effectiveness.  In this paper, we examine these characteristics and then consider the problems with several different standard evaluation techniques.

The Effect of Specialized Multimedia Collections on Web Searching (pp182-199)
        B.J. Jansen, A. Spink and J. Pedersen
Multimedia Web searching is a significant information activity for many people. Major Web search engines are critical resources in people’s efforts to locate relevant online multimedia information. It is therefore important that we understand how searchers are utilizing these Web information systems in their quest to retrieve multimedia information to design effective Web systems in support of these information needs. In this paper, we report the results of a research study evaluating the effect of separate multimedia Web collections on individual searching behavior. The AltaVista search engine has an extensive multimedia collection and uses tabs to search specific collections. The motivating questions for this research are: (1) What are the characteristics of multimedia searching on AltaVista? and (2) What are the effects on Web searching of separate multimedia collections? The results of our research show that multimedia searching is complex relative to general Web searching. Searching specific multimedia collections has reduced the complexity of audio searching, but it has not had the same effect for image and video searching. Query length and Boolean usage rates are much higher for image searching, compared to general Web searching. We discuss the implications of the research findings for the design, development and evaluation of Web multimedia retrieval systems.

II.  Measuring Effectiveness
Measuring the Identification Capability of Acronyms on the World Wide Web: a Comparative Study (pp200-215)
        A. Skrop and S. Dominich
Many people in Hungary use the Web to obtain information from public institutions and organizations. Because these users typically do not know the URL of the desired institution’s home page, they use a Web search engine and the acronym of the institution is used as query to get there. Users prefer using acronyms because they usually do not know the full names of the institutions exactly. Acronyms are easy to remember and are extensively used in media and by people in everyday life. In this paper results from an analysis of the usefulness of the acronyms of Hungarian higher educational institutions present on the Web, i.e., the ability of acronyms to identify their own institutions is reported. The usefulness of acronyms of general institutions is used as a comparison. The working hypothesis is that higher educational acronyms are more effective than general acronyms. The study confutes the assumption and shows that the majority of acronyms of higher educational institutions are not effective in identifying their own institution. Causes are presented, and possible remedies are suggested.

Ranking Search Results by Web Quality Dimensions (pp216-235)
        J.C.C. Pun and F.H. Lochovsky
Currently, search engines rank search results using mainly link-based metrics. While usually most of the search results are relevant to a user's query, due to how the results are ranked, users often are still not totally satisfied with them. Using a proposed framework of web data quality, it is found that current search engines usually only consider a very small number of the dimensions of web data quality in their ranking algorithms. In this paper, a newly identified web data-quality dimension, appropriateness, which is based on the linguistic and visual complexity of a web page, is studied. It is computed using new metrics that classify web pages into three main appropriateness genres: scholarly, news/general interest and popular. Experiments have shown the effectiveness of the metrics in ranking web pages by whether they are appropriate to a user’s task and information needs.

Cybergenre: Automatic Identification of Home Pages on the Web (pp236-251)
        M. Shepherd, C. Watters and A. Kennedy
The research reported in this paper is part of a larger   project on the automatic classification of web pages by their genres.  The long term goal is the incorporation of web page genre into the search process to improve the quality of the search results.   In this phase, a neural net classifier was trained to distinguish home pages from non-home pages and to classify those home pages as personal home page, corporate home page or organization home page. In order to evaluate the importance of the functionality attribute of cybergenre  in such classification, the web pages were characterized by the cybergenre attributes of <content, form, functionality> and the resulting classifications compared to classifications in which the web pages were characterized by the genre attributes of <content, form>.  Results indicate that the classifier is able to distinguish home pages from non-home pages and within the home page genre it is able to distinguish personal from corporate home pages.  Organization home pages, however, were more difficult to distinguish from personal and corporate home pages. A significant improvement was found in identifying personal and corporate home pages when the functionality attribute was included.

Three Pillars for Congenial Web Searching – Continuous Evaluation for enhancing Web Search Effectiveness (pp252-280)
        M.Gnasa, M. Won and A.B. Cremers
In the context of large homogeneous retrieval systems, metrics have been established to evaluate the effectiveness with precision and recall. By contrast, measuring Web search effectiveness is a new challenge due to the heterogeneity of high-dynamic Web content. Currently, users select a Web search engine by their individual preferences, and the evaluation of effectiveness is a subjective measure defined by the user. Since there are different emphases for each single user, those user-defined measures cannot be quantified in a global way. Therefore, we propose a new Web search system, where the effectiveness is continuously evaluated by explicit user feedback in terms of a personalized ranking matrix. These local rankings can be evaluated according to different goals. First, accumulation leads to a wider base of ranked and validated results. Second, the aggregated ranking lists can be used to identify topics, as well as communities of interest. Finally, together with social aspects for community support, a framework for Congenial Web Search is defined.

III.  Visualization
Web-based Visualization Interface Testing: Similarity Judgments (pp281-297)
        S. Koshman
TouchGraph is a Web-based ranked similarity list browser that visualizes the relationship between the query and resulting item set as a graph.  TouchGraph provides visual analogs to Amazon’s recommendation feature based on item similarity and Google’s “similar to” pages.  TouchGraph may be able to assist diverse Web users, who have varying levels of knowledge on search topics, to visually select similar items to their query.  To examine this assumption further, this investigation asks: what are the effects of topic knowledge level on the similarity judgments generated by the users in comparison to the visualized system depictions?  Seventeen participants were asked to use TouchGraph for similarity matching of search output to the query and their results were compared to the items shown as most similar to the query by the visualization.  The results showed that participants rated their topic knowledge level quite low for most tasks, there was a high degree of participant-system item selection overlap, and a statistically significant relationship was found between knowledge level and node use for half of the tasks.  The subjective satisfaction data were positive for the TouchGraph interface.  The findings suggest that the TouchGraph visualization has the potential to enhance Web search effectiveness. This study aids in understanding better system design issues in regard to visualization-based tools for Web information retrieval. 

Toward Enabling Users to Visually Evaluate the Effectiveness of Different Search Methods (pp298-313)
        A. Spoerri
This paper explores how information visualization can provide insights into the effectiveness of different query formulations or the same query submitted to multiple search engines. Different queries or search methods are deemed more effective if the fusion of their results leads to a new result set that contains an increased number of relevant documents. The Meta­Crystal toolset can be used to visualize the degree of overlap and similarity between the results returned by different queries or en­gines. The work presented is guided by two working hypotheses: 1) documents found by multiple methods are more likely to be relevant; 2) high degrees of overlap and/or system­atic relationships between the ranked lists being compared will not lead to fusion results that contain more relevant documents. MetaCrystal visually identifies documents found by multiple queries or engines. The number and distribution patterns of documents found by multiple methods can be used as a visual measure of the fusion effectiveness. Examples, using Internet and TREC data, are presented that support both in a qualitative and quantitative way the working hypotheses.

Back to JWE Online Front Page