Pomerantz, J., Nicholson, S., & Lankes, R. D. (2003). Digital reference triage: An investigation using the Delphi method into the factors influencing question routing and assignment. The Library Quarterly 73(2). 103-120. 

DIGITAL REFERENCE TRIAGE:

FACTORS INFLUENCING QUESTION ROUTING AND ASSIGNMENT¹

Jeffrey Pomerantz,² Scott Nicholson,³ and R. David Lankes⁴

Abstract

This article describes a Delphi study conducted to determine factors that affect the process of routing and assigning reference questions received electronically by digital reference services, both to experts within the service and between services. Fifteen factors were determined, by expert consensus, to be important at the conclusion of this study. These fifteen factors are divided into three groups: 1) general factors, 2) factors in routing the question to an individual, and 3) factors when routing the question to another service. These factors were ranked in order of importance and grouped according to the recipient of the question. These fifteen factors need to be taken into account when automating the triage process. This article has laid out a methodology for investigating other digital reference processes so that those processes amenable to automation may be automated, and experts’ talents and time may be best used.

Introduction

Most digital reference services have a team of experts for assistance with incoming questions. This many-to-many relationship between experts and questions presents an optimization problem: how can the experts’ talents and time be best used in answering questions? One way to explore this is to ask those involved with routing and assigning questions (referred to here as triage) in digital reference services about the factors that influence this process. The goal of the current study was to identify factors that affect the process of routing and assigning reference questions. This research began by identifying factors acknowledged in the literature on desk and digital reference to affect this routing and assigning process. Next, a Delphi study was conducted to refine this list of factors through an iterative survey process with a panel of elite respondents.

Triage is a particularly important step in the process of providing digital reference service. Joanne Silverstein and R. David Lankes [1] and Michael McClennen and Patricia Memmott [2] point out that digital reference services inevitably receive questions that are outside the scope of the service. Some of these questions may be within the scope of another digital reference service; many digital reference services therefore forward their out-of-scope questions to other services for which the questions are in scope. This process has become formalized through consortial agreements between digital reference services in which the members may forward questions to one another. The Virtual Reference Desk network is such a consortium: in addition to being a digital reference service in its own right, the VRD is a clearinghouse to which network members may forward their out-of-scope and in-scope “overflow” questions. The VRD’s role is to forward those questions to the most appropriate service. By doing this, digital reference services are providing reference service beyond what any one individual service could provide without inconveniencing the patron.

This study begins from the general process model of asynchronous digital reference presented in Figure 1. This model is derived from Lankes [3] and the Virtual Reference Desk Project’s AskA Software specifications document [4]. This model consists of 5 steps:

1. Question Acquisition is a means of taking a patron’s questions from email, web forms, chat, or embedded applications.

2. Triage is the assignment and routing of a question to a digital reference service, and to a reference or subject expert within a service. This step may be automated or conducted via human decision support. Triage also includes the filtering of repeated questions or out-of-scope questions.

3. Answer Formulation includes factors for creating “good” answers such as age and cultural appropriateness. Answers are also sent to the user at this point.

4. Tracking is the quantitative and qualitative monitoring of repeat questions for trends. Tracking allows the identification of “hot topics,” and may indicate where gaps exist in the collection(s).

5. Resource Creation concerns the use of tracking data to build or expand collections and better meet users’ information needs.

Fig. 1: General Digital Reference Model [1]

This process model is presumed to be applicable to all asynchronous digital reference services, though different services employ variations of the processes at each step. Some services may even skip steps; for example, not all services may archive questions or answers to create resources. Additionally, some steps may be repeated, especially if this model is seen to span more than one service; for example, a triage center at one digital reference service may receive a question and route it to a different service, which may then route it to an expert.

As with any step in the general process model, triage may be performed in a variety of ways by different digital reference services. This study was exploratory in that it sought to identify the factors that affect digital reference services’ performance during the triage process. This study was guided by the hypothesis that there is a finite set of factors that are important to all digital reference services. A further hypothesis was that, like referrals performed at library reference desks, these factors can be grouped based on the nature of the service, the question, or other criteria. Thus, the research questions for this study are: 1) What factors are important to digital reference services when assigning and routing electronically submitted reference questions?; 2) Can these factors be ranked in order of importance?; and 3) Are there groups of factors that can be discovered?

Literature on Question Routing in Reference Services

The literature reviewed here comes from three areas: models of the question-routing process, factors in referring questions from physical library reference services, and factors used in triage decisions in virtual reference services.

A number of researchers have pointed to the existence of the filtering stage in the processing of digital reference questions. McClennen and Memmott [2] describe several roles played by participants in the digital reference process. These roles are similar to roles in the traditional reference process, but with “some new twists imposed by the digital environment” (The Model section, ¶ 2). One of these roles is the Answerer, which is “the customary role of the reference librarian,” that is, working at the (virtual) reference desk and answering questions. Another role is the Filterer, who filters questions by deciding which questions the service should accept, which can be answered with stock answers, and which are out of scope for the service. Additionally, Filterers decide which questions should be sent to which answerer and which should be forwarded to other services. This study is concerned with what criteria are significant for Filterers in making these internal and external triage decisions.

Two variations on the triage process are employed by different digital reference services [3]. The first variation is concerned with the agent that makes decisions in the triage process: either a human filterer or an automated process. In either case, criteria must exist for deciding how to assign and route questions. If the filterer is a human these criteria may be more heuristic, whereas if an automated process performs the filtering these criteria must be rigorously codified in software. The second variation on the triage process involves how a question is triaged to an answerer: questions are either assigned to specific answerers by the filterer (either human or automated) [5], or questions are stored in a “triage area” and self-selected by answerers [6].

The Collaborative Digital Reference Service (CDRS) is an example of a service that utilizes an automated filterer: a software algorithm that routes and assigns questions to other digital reference services. This algorithm assigns questions “on the basis of such data elements as hours of service, including time zones, subject strengths, scope of collections, types of patrons served, etc.” [7, How Does CDRS Work section, ¶ 2].

The Internet Public Library (IPL) is an example of a service that utilizes a human filterer and allows experts to self-select questions. McClennen and Memmott state that the IPL has “developed written policies and procedures, including guidelines for making the necessary decisions regarding which questions to accept, reject, or refer” [2, p. 146]. The existence of these policies and procedures demonstrates the necessity for digital reference services to establish criteria for the performance of triage, even when those performing triage are the service’s “most experienced staff” [p. 146].

This study is concerned with only human Filterers and the assignment and routing of questions to specific answerers. Filterers have the task of deciding how to triage incoming electronically submitted reference questions both among digital reference services, and to a reference or subject expert within a service.

Factors in Question Referral

The term “referral” is used here to indicate the practice of a reference librarian redirecting a patron to another reference service or organization that the referring librarian believes can better address the patron’s information need. Desk reference services, like digital reference services, receive questions that are outside the scope of the service or what can be answered using the library’s collection. Rather than simply turn the patron away without an answer, reference librarians may refer the patron to another reference service or organization for which the question is in scope. In this situation, the burden is placed on the patron to seek out the service or organization to which they were referred. This is different from triage, which is the assignment and routing of a question not only within a service, but also between digital reference services, and usually takes place without the participation of the patron. In the triage process, the burden is on the librarian to seek out an alternative service and forward the question.

While referrals are relatively common, but not necessary, in desk reference services, triage is a crucial step in the management of questions in digital reference services. There is only a small body of literature from desk reference that discusses factors in referring questions, while a considerably larger body of literature from digital reference discusses these factors. The literature from desk reference is concerned primarily with the reference interview, i.e., how to assess when the patron is satisfied with the information provided and when the librarian should refer the patron to another source [8 – 10]. The literature from digital reference is concerned primarily with assigning questions based on what is known about the patron, the available answerers, and the question itself [2; 3; 6; 7]. A list of thirty-four factors in triage was compiled from these two bodies of literature for use in this study.

Methodology

While the list of factors compiled from the literature is an appropriate starting point for investigating the process of triage, those performing digital reference triage must also be consulted before finalizing the list. One technique that allows a panel of experts to reach consensus on a list of factors is the Delphi method.

The Delphi Method was invented by Olaf Helmer and Norman Dalkey in the early 1950s as a technique for achieving a consensus of opinion among a group of experts, on a topic for which a more conventional data elicitation technique is unfeasible [11; 12]. While early Delphi studies such as [11] were attempts to predict and influence future trends, more recent Delphi studies have utilized the technique to explore complex issues in depth [13]. Delphi enables the researcher to structure the process of group communication among a group of elite respondents, so that the respondents may critically examine a complex problem more effectively than they could in person [12; 14].

The Delphi method resembles focus group methodology, in that respondents can raise issues to the group; Delphi is a more controlled methodology, however, in that all responses are channeled through the researcher. This allows anonymity to be preserved among the respondents. This is intended to prevent strong personalities from dominating the communication process, as may occur in focus groups. The researcher also has the responsibility of compiling the respondents’ comments, and providing these to the respondents, creating a feedback loop where the respondents receive the results from the previous round before participating in the next round. A Delphi study proceeds in rounds until a resolution is reached: either consensus is reached among the respondents or the respondents have exhausted all that they have to say about the problem at hand.

Selecting the Respondents

It is unclear how many digital reference services exist. Some of these services are affiliated with libraries – public, academic, special, or otherwise – and some are “standalone” services unaffiliated with any physical library or collection. Evaluation of desk reference services is a long-standing practice in libraries [8; 15]. There are, however, few studies evaluating digital reference services, and only recently have guidelines for this evaluation been proposed [16; 17; 18]. For this study, the researchers therefore decided to select a panel of elite respondents by seeking out filterers representing exemplary digital reference services. An exemplary service is defined as one that is “worthy of imitation” [3, p. 80]. As a result, the respondent panel may not be representative of all existing digital reference services. This study investigated the triage process as it is performed by services that are worthy models: they should serve as examples of “best practices” for both existing and new digital reference services.

Exemplary services were selected for participation in this study by relying on expert judgments of what constitutes an exemplary service. Each year, the Virtual Reference Desk (VRD) Project recognizes libraries and organizations that provide high quality digital reference service to their users (http://www.vrd.org/conferences/VRD2001/exemplary.shtml) by awarding the Exemplary Digital Reference Services Award. Seventeen services have been presented with this award: seven in 1998, four in 1999, three in 2000, and three in 2001. These seventeen services were selected for participation in this study. Additionally, the ten digital reference services that participate actively with the VRD Network in question exchange were selected for participation in this study. These services are exemplary as they adhere to the Facets of Quality, a set of standards in a variety of categories, intended to ensure quality responses and service, and user satisfaction [19]. Finally, the six studies that Lankes [3] selected as elite respondents for his study of exemplary K-12 digital reference services were selected for participation in this study. There was some overlap between these three lists of services, so the total number of services that were selected for participation in this study was twenty-four.

Some services out of these twenty-four were eliminated from the respondent pool because they allow answerers to select questions themselves. Other services declined to participate for reasons that they did not share with the researchers. Of the final respondent pool of fifteen services, all responded to rounds one and two, and twelve responded to round three.

The individual who is the point of contact at each selected digital reference service was sent an email asking him or her to participate in the study, and only one individual per service was sent this email. These points of contact are hereafter known as the Coordinators of their respective digital reference services. McClennen and Memmott state that, as in any other reference service, it is the role of the Coordinator to set policies and procedures [2, Coordinator section, ¶ 1]. Therefore, the Coordinators are in a unique position to know the factors affecting decisions made at every step in managing questions. To the researchers’ surprise, every Coordinator contacted responded to the survey (rather than delegating the survey to an employee). Thus each respondent represented a service, and there was only one respondent per service. Respondents were asked to answer the survey questions with the entire service in mind. The level of analysis for this study is therefore the service level instead of the individual or role.

The Survey

In a Delphi study, the researcher presents the respondents with a question or an issue for consideration. In this study, the issue was stated on the survey as follows: “We are interested in finding out about your service’s decision-making process about how incoming electronically submitted reference questions are assigned and routed. In the context of this decision-making process, please indicate whether each factor listed below is important or unimportant. If you don’t know or have no opinion, please select ‘No opinion.’”

This study was conducted as a Web-based survey listing the factors from the literature review broken into three groups: 1) General factors, 2) Factors in routing the question to an individual, and 3) Factors when routing the question to another service.

In a traditional Delphi study, the panel would provide the original list of factors. This was not done, however, as a list of factors could be and was derived from the literature on digital and desk reference services. Deriving the list of factors from the literature not only introduces into this study the expertise of scholars in desk and digital reference, but also provides a basis for generalizing works originally written for a desk reference setting to the digital reference environment.

Given the iterative and lengthy nature of Delphi studies, it can be difficult to get participants to commit to and carry through with participating in the entire study. Decreasing the number of iterations would increase the number of participants that would stay involved until the end of the study. Consequently, “round zero” of this Delphi study was the compilation from the literature on reference of factors in triage. The expert panel for this round was the authors of the literature from which the factors in triage were compiled.

During each round of the study, panelists were asked to rate each factor and add new factors. Low-scoring factors were then removed, suggestions added, and a new round begun. In the first round of the study thirty-four factors were listed, in round two there were twenty-five factors, and in round three there were nineteen factors (see Appendix A). In each round, the respondents were asked to vote whether each factor was important or unimportant to their service by checking a radio button on a web form for Important, Unimportant, or No opinion. A sample question is shown in figure 2.

Fig. 2: A sample question from the survey

In addition to voting on the importance of each factor, respondents were asked to suggest any additional factors that affect the triage process in their service that were not listed in the survey. This allowed for the possibility that the original list of thirty-four factors was not exhaustive. In addition if a factor was voted off the list, it could be reinstated if a respondent subsequently suggested it. In fact, eight factors were suggested by respondents that were not in the original list of thirty-four (see Appendix B)and one factor that was voted off in round one was added to the list by a panelist in round two.

Data Analysis

The method of data analysis used in this study was based on Scott Nicholson’s [20] study of academic research on the Web. In the data analysis, a vote for Important equals 1, a vote for Unimportant equals –1, and a vote for No opinion equals 0. The votes were totaled at the conclusion of each round to create a score for each factor. The factors that had a final score of zero or higher were retained; those factors that had a negative final score were dropped. The final scores from each round were analyzed and presented to the respondents at the conclusion of each round. Based on these results, a survey was constructed for the subsequent round, and the respondents were asked by email to fill it out. This process was repeated until the list of factors stabilized.

The original intention of the researchers was to continue the study until all factors had positive scores. However, the researchers decided to conclude the study after the third round because it was felt that another round would only confirm what was already apparent, and would be a further imposition on the respondents’ time. In the third round only two factors had negative scores: “User’s geographic location,” which had been voted off the list in round one, and “Profile of the user containing personal information,” which had been suggested in round two. These two were dropped, as were two factors with scores of zero. As there were no new suggestions, the list was finalized.

At the conclusion of round three, fifteen factors had been determined to be important as they had positive factor scores. In descending order of their scores, these factors were:

1. Subject area of the question

2. The service’s area(s) of subject expertise

3. The answerer’s area of subject expertise

4. Level / depth of assistance available from the service

5. Number of questions that may be forwarded to the service per unit of time, as set by consortium agreements

6. Response rate of the service

7. The answerer’s experience and skill in providing reference service

8. Past performance of the service in providing correct and complete answers

9. The service’s turnaround time for answering questions

10. Number of questions that your service may forward to other services per unit of time, as set by consortium agreements

11. Availability of sources to answer the question

12. The answerer’s experience and skill with providing customer service

13. Language of the question

14. Scope of the service’s collection

15. Type of question

These results are presented graphically in Figure 3.

Fig. 3: Positive Factor Scores, Round Three

The survey for this study presented the factors in three groups, according to the recipient of the routing. The fifteen factors fell into these groups as follows:

Factors that affect the triage process in general:

1. Subject area of the question

2. Availability of sources to answer the question

3. Language of the question

4. Type of question

Factors that affect the triage process when routing or assigning questions to an answerer:

1. The answerer’s area of subject expertise

2. The answerer’s experience and skill in providing reference service

3. The answerer’s experience and skill with providing customer service

Factors that affect the triage process when routing questions to another reference service:

1. The service’s area(s) of subject expertise

2. Level / depth of assistance available from the service

3. Number of questions that may be forwarded to the service per unit of time, as set by consortium agreements

4. Response rate of the service

5. Past performance of the service in providing correct and complete answers

6. The service’s turnaround time for answering questions

7. Number of questions that your service may forward to other services per unit of time, as set by consortium agreements

8. Scope of the service’s collection

Discussion

This study arose from a research agenda being pursued by the authors to investigate the steps in the general process model of digital reference presented above. This research agenda has also led to another study, investigating the paths that digital reference services take through the general process model, and the decisions that services make at different points [21]. This study has investigated one step in that process model in depth; other studies investigating other steps are needed to provide a more comprehensive basis for expanding the general process model of digital reference.

The authors’ research agenda is part of a larger research agenda, supported by the National Science Foundation’s National Science, Mathematics, Engineering, and Technology Education Digital Library (NSDL) project (http://www.ehr.nsf.gov/due/programs/nsdl/). The goal of the NSDL project is to establish a national digital library for science, technology, engineering, and mathematics education. The goal of the authors’ research is to design a more effective digital library. One of the ways in which this goal will be accomplished is by integrating human-intermediated digital reference service with the digital library environment. As the use of digital reference services increases, there is an increasing need for these services to “scale up” to handle an increasingly large number of questions. This scalability is directly affected by the amount of automation employed by the service: the more processes that are automated, the more of the human intermediaries’ time and effort can be dedicated to tasks that cannot yet be automated. There is, now more than ever, an increased and immediate need in digital reference services for automation.

Application of the Results to Triage Automation

Automating the triage process is something that very few digital reference services are currently doing [21]. In order to automate triage, a profile of answerers and digital reference services to which questions may be assigned is necessary, specifying factors such as the answerer’s or service’s name, days of availability, area of subject expertise, and whatever other criteria a service deems necessary. Lankes [22] describes the Question Interchange Profile (QuIP), a protocol for passing this type of profile with other information about a question. The results of this study provide fifteen pieces of information that such profiles need to contain about answerers and digital reference services. Some of these factors already exist in the QuIP element set, but some do not. Therefore, the results of this study provide factors that should be included in future revisions of QuIP and any other standard for profiling digital reference services and answerers.

This study also determined three factors intrinsic to the question itself that are important to the triage process: subject area, language, and type of the question. Currently, QuIP contains subject and language elements, but no element for question type. The results of this study indicate that question type should be included in future revisions of QuIP and other standards. A future direction for research will be to investigate whether these three factors have subsets and what those subsets are. It is easy to imagine a list of languages, of which one or more could be selected to describe a question (e.g., English, Dutch, Japanese). But several classification schemes of subjects exist: the Dewey Decimal Classification, the Library of Congress Subject Headings, and the ERIC Thesaurus, to name only a few. Which, if any, of the several existing schemes is the most appropriate to use for digital reference? Are different subject classification schemes appropriate for different types of digital reference services? Further, several classification schemes for question types exist, each designed for a different purpose. Some schemes address grammatical structure, classifying questions as types of Wh- questions (e.g., Who, What, When, Which, How) [23; 24]. Some schemes address the content of the desired answer (e.g., definition, comparison, quantification) [25; 26]. Some schemes designed to classify library reference questions address the nature of the reference transaction (e.g., ready reference, directional, reader’s advisory) [15; 27]. Other schemes designed to classify library reference questions address the type or genre of information source likely to contain an answer (e.g., a dictionary, a geographical source, a biographical source) [28]. Which, if any, of these existing schemes is the most appropriate to use for digital reference? Are different classification schemes of question types appropriate for different types of digital reference services, or should more than one scheme be used to form a faceted scheme? Additional research is required to answer these questions, which will enable the determination of what characteristics of a question need to be known in order for automated systems to perform triage.

Methodological Suggestions for Reference and Referral

This study investigated the factors that affect the triage process in digital reference. The thirty-four factors in triage that made up the original list in round one were compiled in part from literature on desk reference referrals. While this study does not claim to make any conclusions about desk reference, the same methodology used here may be used to investigate referrals made at reference desks. In some evaluations of reference transactions, only questions that are answered fully and correctly are considered to be successful [29]. According to these evaluation criteria, a referral is a failed reference transaction. A future direction for research will be to investigate the criteria that lead to a successful referral being made at a reference desk.

Conclusion

The goal of the research agenda that gave rise to this study is to design a more effective digital library service. One of the ways in which this may be accomplished is to integrate the digital reference service with the digital library environment to ensure that digital library users have a place to turn for assistance. However, as the use of digital libraries and digital reference services increases, these services must be able to scale up to handle the increased use. One way to handle additional questions without increasing staff is to automate portions of the digital reference process.

The identification of factors that affect these processes is the first step in addressing the problem of optimally utilizing experts’ talents and time in answering questions. For example, one unanswered question is how to determine how experts’ talents and time can be best used in answering questions. This question is only one aspect of a larger question which is determining what processes in digital reference may be automated and which must be performed by a human being. An automated system to perform triage must be able to take the same factors into consideration as a human filterer. Certain factors in the triage process need to be refined: for example, what is the most appropriate scheme for classification of subjects and question types given different digital reference services and different contexts? This study, however, investigated factors affecting only the triage process, and triage is just one process in the provision of digital reference. In order to address the optimization problem in digital reference, the other processes in digital reference must be investigated.

This article described a Delphi study conducted to determine the factors that affect the process of routing and assigning reference questions to answerers in digital reference services. This study refined a list of factors collected from a review of the literature on desk and digital reference through an iterative survey process with a panel of elite respondents. Fifteen factors were determined, by expert consensus, to be important at the conclusion of this study. These fifteen factors are divided into three groups: 1) general factors, 2) factors in routing the question to an individual, and 3) factors when routing the question to another service. This article has identified factors that need to be taken into account when automating the triage process. Finally, this article has laid out a methodology for investigating other digital reference processes so that those processes amenable to automation may be automated, and experts’ talents and time may be best used.

Appendix A: The Original 34 Factors in Triage Compiled from the Literature on Digital and Desk Reference, used for Round One of the Study

General factors:

1. Subject area of the question

2. Type of question

3. Need for query negotiation

4. Predicted difficulty of the question

5. Complexity of the question (one-part vs. multiple-part)

6. Length of the question

7. Availability of sources to answer the question

8. User’s affiliation (student at …, employee of …, member of …, etc.)

9. User’s geographic location

10. User’s planned use of the information provided

11. User’s prior search history

12. Date after which the user will not need or be able to use the information provided

13. Volume of questions submitted on a given day

14. Your service’s current turnaround time for answering questions

15. Format of the answer explicitly requested by the user (e.g., brief factual answer, a document, list of citations, etc.)

16. Format of answer that the question seems to indicate (e.g., brief factual answer, a document, list of citations, etc.)

17. Type of sources explicitly requested by the user (e.g., print sources, Internet sources, etc.)

18. Type of sources that the question seems to indicate (e.g., print sources, Internet sources, etc.)

When routing questions to an answerer:

19. The individual’s area of subject expertise

20. The individual’s educational background

21. The individual’s experience and skill in dealing with users from a particular community

22. The individual’s experience and skill with providing customer service

23. The individual’s experience and skill in providing reference service

24. The individual’s geographic location

When routing or forwarding questions to another reference service:

25. Budget of the service

26. Hours of availability of the service

27. Level / depth of assistance available from the service

28. Number of individuals who answer questions for the service

29. Number of questions received by the service per month/year/etc.

30. Past performance of the service in providing correct and complete answers

31. Scope of the service’s collection

32. The service’s area(s) of subject expertise

33. The service’s turnaround time for answering questions

34. Type of service

Appendix B: Factors in Triage Suggested by Respondents During the Course of the Three Rounds of the Study

1. Language of the question

2. Institutional service agreements

3. Number of questions that may be forwarded to the service per unit of time, as set by consortium agreements

4. Number of questions that your service may forward to other services per unit of time, as set by consortium agreements

5. “Profile” of the user containing personal information

6. Response rate of the service to which your service routes or forwards questions

7. “Sensitivity” of a question (e.g., for personal, PR, or other reasons)

8. User’s willingness to pay a fee for an answer to his/her question

References

1. Silverstein, Joanne L., and Lankes, R. David. “Digital Reference Services and Centers at the United States Department of Education: Analysis and Recommendations. Syracuse, NY: U.S. Department of Education, 1999. http://iis.syr.edu/projects/ed_gov_rpt.html (last accessed June 20, 2002).

2. McClennen, Michael, and Memmott, Patricia. “Roles in Digital Reference.” Information Technology and Libraries 20 (September 2001): 143-148. http://www.lita.org/ital/2003_mcclennan.html/ (last accessed June 20, 2002).

3. Lankes, R. David. Building and Maintaining Internet Information Services: K-12 Digital Reference Services. Syracuse, NY: ERIC Clearinghouse on Information & Technology, 1998.

4. Virtual Reference Desk Project. “Virtual Reference Desk AskA Software: Decision Points and Scenarios.” http://www.vrd.org/Tech/AskA-sw.PDF (last accessed June 20, 2002).

5. Bry, Lynn. “Simple and Sophisticated Methods for Processing Large Volumes of Question and Answer Information through the World Wide Web.” In Digital Reference Service in the New Millennium: Planning, Management, and Evaluation, edited by R. David LankesJohn W. III Collins and Abby S. Kasowitz. New York: Neal-Schuman Publishers Inc., 2000.

6. Lagace, Nettie, and McClennen, Michael. “Questions and Quirks: Managing an Internet-Based Distributed Reference Service.” Computers in Libraries 18 (February 1998): 24-27. http://www.infotoday.com/cilmag/feb98/story1.htm (last accessed June 20, 2002).

7. Kresh, Diane Nester. “Offering High Quality Reference Service on the Web: The Collaborative Digital Reference Service (CDRS).” D-Lib Magazine 6 (June 2000). http://www.dlib.org/dlib/june00/kresh/06kresh.html (last accessed June 20, 2002).

8. Crews, Kenneth D. “The Accuracy of Reference Service: Variables for Research and Implementation.” Library and Information Science Research 10 (1988): 331-355.

9. Lynch, Mary Jo. “Reference Interviews in Public Libraries.” Library Quarterly 48 (April 1978): 119-142.

10. Saxton, Matthew L., and Richardson Jr., John V. Understanding Reference Transactions. San Diego: Academic Press, 2002.

11. Dalkey, Norman, and Helmer, Olaf. “An Experimental Application of the Delphi Method to the Use of Experts.” Management Science 9 (1963): 458-467.

12. Linstone, Harold A., and Turoff, Murray, eds. In The Delphi Method, edited. Reading, MA: Addison-Wesley Publishing Co., Inc., 1975.

13. Kochtanek, Thomas R., and Hein, Karen K. “Delphi Study of Digital Libraries.” Information Processing and Management 35 (1999): 245-254.

14. Turoff, Murray, and Hiltz, Starr Roxanne. “Computer Based Delphi Processes.” In Gazing into the Oracle: The Delphi Method and Its Application to Social Policy and Public Health, edited by Michael Adler and Erio Ziglio: Jessica Kingsley Publishers, 1996. http://eies.njit.edu/~turoff/Papers/delphi3.html (last accessed June 20, 2002).

15. White, Howard D. “Measurement at the Reference Desk.” Drexel Library Quarterly 17 (Winter 1981): 3-35.

16. Kasowitz, Abby; Bennett, Blythe; and Lankes, R. David. “Quality Standards for Digital Reference Consortia.” Reference & User Services Quarterly 39 (2000): 355-363.

17. Lankes, R. David; Collins, John W. III; and Kasowitz, Abby S., eds. “The New Library Series.” In Digital Reference Service in the New Millennium: Planning, Management, and Evaluation, edited. New York: Neal-Schuman Publishers Inc., 2000.

18. White, Marilyn Domas. “Analyzing Electronic Question/Answer Services: Introduction and Analytical Framework.” CLIS Technical Report no. 99-02. College Park, MD: The College of Library and Information Services, University of Maryland, 1999. http://www.clis.umd.edu/research/reports/99/TR9902.PDF (last accessed June 20, 2002).

19. Virtual Reference Desk Project. “Facets of Quality for Digital Reference Services, Version 4.” http://www.vrd.org/facets-10-00.shtml (last accessed June 20, 2002).

20. Nicholson, Scott. “Creating a Criterion-based Information Agent through Data Mining for Automated Identification of Scholarly Research on the World Wide Web.” doctoral dissertation diss., University of North Texas, Denton, Texas, 2000.

21. Pomerantz, Jeffrey; Nicholson, Scott; Belanger, Yvonne; and Lankes, R. David. “The Current State of Digital Reference: Validation of a General Digital Reference Model through a Survey of Digital Reference Services.” (under review).

22. Lankes, R. David. “Question Interchange Profile, version 1.01D.” white paper. Syracuse, NY: ERIC Clearinghouse on Information & Technology, 1999. http://www.vrd.org/Tech/QuIP/1.01/1.01d.htm (last accessed June 20, 2002).

23. Belnap, Nuel D., Jr., and Steel, Thomas B., Jr. The Logic of Questions and Answers. New Haven, CT: Yale University Press, 1976.

24. Robinson, W. P., and Rackstraw, Susan J. A Question of Answers. Boston: Routledge & Kegan Paul, 1972.

25. Graesser, Arthur C.; McMahen, Cathy L.; and Johnson, Brenda K. “Question Asking and Answering.” In Handbook of Psycholinguistics, edited by Morton Ann Gernsbacher. San Diego: Academic Press, 1994.

26. White, Marilyn Domas. “Questions in Reference Interviews.” Journal of Documentation 54 (1998): 443-465.

27. Bunge, Charles A. “Factors Related to Output Measures for Reference Services in Public Libraries: Data from Thirty-six Libraries.” Public Libraries 29 (January/February 1990): 42-47.

28. Bopp, Richard E., and Smith, Linda C. Reference and Information Services: An Introduction. Englewood, CO: Libraries Unlimited, Inc., 2001.

29. Hernon, Peter, and McClure, Charles R. “Unobtrusive Reference Testing: The 55 Percent Rule.” Library Journal III (15 April 1986): 37-41.

FOOTNOTES

1. The researchers wish to thank the survey respondents for their patience and their contribution to this study, and the anonymous reviewers for valuable feedback that made this a better article. This material is based upon work supported by the National Science Foundation under Grant No. 05-11.

2. Jeffrey Pomerantz, Ph.D. Candidate, School of Information Studies, Syracuse University, Syracuse, New York 13244-4100. Telephone 315-443-4924; Fax 315-443-5448; E-mail jppomera@syr.edu.

3. Scott Nicholson, Assistant Professor, School of Information Studies, Syracuse University, Syracuse, New York 13244-4100. Telephone 315-443-1640; Fax 315-443-5806; E-mail srnichol@syr.edu.

4. R. David Lankes, Director, Information Institute of Syracuse, Syracuse University, Syracuse, New York 13244-4100. Telephone 315-443-1707; Fax 315-443-5448; E-mail rdlankes@ericir.syr.edu.