|
Vol.3 No.3&4 December,
2004
Introduction
(pp169-170)
A. Spink and C. Watters
I.
Web Search
Why Is Web
Search So Hard... to Evaluate? (pp171-181)
D.E. Rose
Web search has several important characteristics that
distinguish it from traditional information retrieval: the often
adversarial relationship between content creators and search engine
designers, the nature of the corpus, and the multiplicity of user
goals. In addition to making the search task itself difficult, these
characteristics make it particularly hard to evaluate search
effectiveness. In this paper, we examine these characteristics and then
consider the problems with several different standard evaluation
techniques.
The Effect of
Specialized Multimedia Collections on Web Searching
(pp182-199)
B.J. Jansen, A.
Spink and J. Pedersen
Multimedia Web searching is a significant information
activity for many people. Major Web search engines are critical
resources in people’s efforts to locate relevant online multimedia
information. It is therefore important that we understand how searchers
are utilizing these Web information systems in their quest to retrieve
multimedia information to design effective Web systems in support of
these information needs. In this paper, we report the results of a
research study evaluating the effect of separate multimedia Web
collections on individual searching behavior. The AltaVista search
engine has an extensive multimedia collection and uses tabs to search
specific collections. The motivating questions for this research are:
(1) What are the characteristics of multimedia searching on AltaVista?
and (2) What are the effects on Web searching of separate multimedia
collections? The results of our research show that multimedia searching
is complex relative to general Web searching. Searching specific
multimedia collections has reduced the complexity of audio searching,
but it has not had the same effect for image and video searching. Query
length and Boolean usage rates are much higher for image searching,
compared to general Web searching. We discuss the implications of the
research findings for the design, development and evaluation of Web
multimedia retrieval systems.
II. Measuring
Effectiveness
Measuring the
Identification Capability of Acronyms on the World Wide Web: a
Comparative Study
(pp200-215)
A. Skrop and S. Dominich
Many people in Hungary use the Web to obtain information
from public institutions and organizations. Because these users
typically do not know the URL of the desired institution’s home page,
they use a Web search engine and the acronym of the institution is used
as query to get there. Users prefer using acronyms because they usually
do not know the full names of the institutions exactly. Acronyms are
easy to remember and are extensively used in media and by people in
everyday life. In this paper results from an analysis of the usefulness
of the acronyms of Hungarian higher educational institutions present on
the Web, i.e., the ability of acronyms to identify their own
institutions is reported. The usefulness of acronyms of general
institutions is used as a comparison. The working hypothesis is that
higher educational acronyms are more effective than general acronyms.
The study confutes the assumption and shows that the majority of
acronyms of higher educational institutions are not effective in
identifying their own institution. Causes are presented, and possible
remedies are suggested.
Ranking
Search Results by Web Quality Dimensions
(pp216-235)
J.C.C. Pun and F.H.
Lochovsky
Currently, search engines rank search results using
mainly link-based metrics. While usually most of the search results are
relevant to a user's query, due to how the results are ranked, users
often are still not totally satisfied with them. Using a proposed
framework of web data quality, it is found that current search engines
usually only consider a very small number of the dimensions of web data
quality in their ranking algorithms. In this paper, a newly identified
web data-quality dimension, appropriateness, which is based on
the linguistic and visual complexity of a web page, is studied. It is
computed using new metrics that classify web pages into three main
appropriateness genres: scholarly, news/general interest and popular.
Experiments have shown the effectiveness of the metrics in ranking web
pages by whether they are appropriate to a user’s task and information
needs.
Cybergenre:
Automatic Identification of Home Pages on the Web
(pp236-251)
M. Shepherd, C. Watters and
A. Kennedy
The research reported
in this paper is part of a larger project on the automatic
classification of web pages by their genres. The long term goal is the
incorporation of web page genre into the search process to improve the
quality of the search results. In this phase, a neural net classifier
was trained to distinguish home pages from non-home pages and to
classify those home pages as personal home page, corporate home page or
organization home page. In order to evaluate the importance of the
functionality attribute of cybergenre in such classification, the web
pages were characterized by the cybergenre attributes of <content, form,
functionality> and the resulting classifications compared to
classifications in which the web pages were characterized by the genre
attributes of <content, form>. Results indicate that the classifier is
able to distinguish home pages from non-home pages and within the home
page genre it is able to distinguish personal from corporate home
pages. Organization home pages, however, were more difficult to
distinguish from personal and corporate home pages. A significant
improvement was found in identifying personal and corporate home pages
when the functionality attribute was included.
Three Pillars
for Congenial Web Searching – Continuous Evaluation for enhancing Web
Search Effectiveness
(pp252-280)
M.Gnasa, M. Won and A.B.
Cremers
In the context of large homogeneous retrieval systems,
metrics have been established to evaluate the effectiveness with
precision and recall. By contrast, measuring Web search effectiveness is
a new challenge due to the heterogeneity of high-dynamic Web content.
Currently, users select a Web search engine by their individual
preferences, and the evaluation of effectiveness is a subjective measure
defined by the user. Since there are different emphases for each single
user, those user-defined measures cannot be quantified in a global way.
Therefore, we propose a new Web search system, where the effectiveness
is continuously evaluated by explicit user feedback in terms of a
personalized ranking matrix. These local rankings can be evaluated
according to different goals. First, accumulation leads to a wider base
of ranked and validated results. Second, the aggregated ranking lists
can be used to identify topics, as well as communities of interest.
Finally, together with social aspects for community support, a framework
for Congenial Web Search is defined.
III. Visualization
Web-based
Visualization Interface Testing: Similarity Judgments
(pp281-297)
S. Koshman
TouchGraph is
a Web-based ranked similarity list browser that visualizes the
relationship between the query and resulting item set as a graph.
TouchGraph provides visual analogs to Amazon’s recommendation feature
based on item similarity and Google’s “similar to” pages. TouchGraph
may be able to assist diverse Web users, who have varying levels of
knowledge on search topics, to visually select similar items to their
query. To examine this assumption further, this investigation asks:
what are the effects of topic knowledge level on the similarity
judgments generated by the users in comparison to the visualized system
depictions? Seventeen participants were asked to use TouchGraph for
similarity matching of search output to the query and their results were
compared to the items shown as most similar to the query by the
visualization. The results showed that participants rated their topic
knowledge level quite low for most tasks, there was a high degree of
participant-system item selection overlap, and a statistically
significant relationship was found between knowledge level and node use
for half of the tasks. The subjective satisfaction data were positive
for the TouchGraph interface. The findings suggest that the TouchGraph
visualization has the potential to enhance Web search effectiveness.
This study aids in understanding better system design issues in regard
to visualization-based tools for Web information retrieval.
Toward
Enabling Users to Visually Evaluate the Effectiveness of Different
Search Methods
(pp298-313)
A. Spoerri
This paper explores how information visualization can
provide insights into the effectiveness of different query formulations
or the same query submitted to multiple search engines. Different
queries or search methods are deemed more effective if the fusion of
their results leads to a new result set that contains an increased
number of relevant documents. The MetaCrystal toolset can be used to
visualize the degree of overlap and similarity between the results
returned by different queries or engines. The work presented is guided
by two working hypotheses: 1) documents found by multiple methods are
more likely to be relevant; 2) high degrees of overlap and/or
systematic relationships between the ranked lists being compared will
not lead to fusion results that contain more relevant documents.
MetaCrystal visually identifies documents found by multiple queries or
engines. The number and distribution patterns of documents found by
multiple methods can be used as a visual measure of the fusion
effectiveness. Examples, using Internet and TREC data, are presented
that support both in a qualitative and quantitative way the working
hypotheses.
Back
to JWE Online Front Page |
