User satisfaction with referrals at a collaborative virtual reference service Virtual reference services, Reference services, Referrals, User satisfaction Vol. 11 No. 2, January 2006 Contents | Author index | Subject index | Search | Home User satisfaction with referrals at a collaborative virtual reference service Nahyun Kwon Assistant Professor, School of Library and Information Science, College of Arts & Sciences, University of South Florida, 4202 East Fowler Ave, CIS1040, Tampa, FL 33620-7800, USA Abstract Introduction. This study investigated unmonitored referrals in a nationwide, collaborative chat reference service. Specifically, it examined the extent to which questions are referred, the types of questions that are more likely to be referred than others, and the level of user satisfaction with the referrals in the collaborative chat reference service. Method. The data analysed for this study were 420 chat reference transaction transcripts along with corresponding online survey questionnaires submitted by the service users. Both sets of data were collected from an electronic archive of a southeastern state public library system that has participated in 24/7 Reference of the Metropolitan Cooperative Library System (MCLS). Results. Referrals in the collaborative chat reference service comprised approximately 30% of the total transactions. Circulation-related questions were the most often referred among all question types, possibly because of the inability of 'outside' librarians to access patron accounts. Most importantly, user satisfaction with referrals was found to be significantly lower than that of completed answers. Conclusion. The findings of this study addressed the importance of distinguishing two types of referrals: the expert research referrals conducive to collaborative virtual reference services; and the re-directional local referrals that increase unnecessary question traffic, thereby being detrimental to effective use of collaborative reference. Continuing efforts to conceptualize referrals in multiple dimensions are anticipated to fully grasp complex phenomena underlying referrals. Introduction The collaborative virtual reference service is an innovative reference service that can bring many benefits to both service users and libraries. Thanks to http://www.informationr.net/ir/11-2/paper246.html[6/21/2016 3:36:16 PM] User satisfaction with referrals at a collaborative virtual reference service Virtual reference services, Reference services, Referrals, User satisfaction collaborations among libraries that share their staff, technology and other resources, no longer are library patrons confined to the services and resources of their local libraries. Patrons now have the opportunity to ask questions of institutions in remote locations for specialized information that would not otherwise be available to them. In many collaborative services such as 24/7 Reference, patrons can access the service without being restricted by time and location. Thus, library patrons could obtain help for their information needs via the help of participating libraries in a reference consortium even when their local libraries are closed. From the library management perspective, libraries can deploy staff more efficiently by participating in a bigger pool and embrace otherwise more costly virtual reference services. Besides cost reduction, a member library of a consortium could answer difficult subject-based research questions more easily by referring the questions to subject specialists in other libraries in the consortium. However, it seems that a large proportion of questions asked in a collaborative service are simply referred back to the home libraries because they should be resolved at a local level (Coffman 2002; Helfer 2003). Unfortunately, there is little monitoring of further transactions to ensure whether the patrons received satisfying answers. Unmonitored referrals, originally defined by Dewdney and Ross (1994), refers to a situation where the reference staff gives the patron a call number or refers the patron to a source in the library thought to contain the answer but does not follow up or check to make sure that the source is not only found but actually answers the question. These unmonitored referrals seem to be frequently observed in a collaborative virtual reference setting when 'outside' library staff refer the patrons back to local libraries. If a large proportion of those referrals are made back to the local libraries, however, that would not be an effective allocation of resources. More importantly, the referrals, if unattended, can easily frustrate the users of collaborative virtual reference services. While reference questions are often answered by outside library staff, many patrons would still view the service as a local service. Whenever the questions require local resources and services to answer adequately, library staff have to refer patrons back to their local libraries to complete the answer. Would the patrons still feel satisfied with a delayed answer? In fact, patrons' frustration caused by being transferred from one point to another in the midst of referrals is not unfamiliar. This frustration is equivalent to our everyday experience from using automated customer services on the phone or asking directions on the street. Although some studies have attempted to identify the types of questions that are effectively answered in virtual reference, few have examined the types of questions that are more prone to referrals, or user satisfaction with referrals in the context of collaborative virtual reference. Perhaps this is because referral are felt to be legitimate answers, although the notion has yet to be validated. Dewdney and Ross (1994) pointed out the problem of a high proportion of unmonitored http://www.informationr.net/ir/11-2/paper246.html[6/21/2016 3:36:16 PM] User satisfaction with referrals at a collaborative virtual reference service Virtual reference services, Reference services, Referrals, User satisfaction referrals (42%) at the physical reference service desk. They maintained that reference staff tend to make unmonitored referrals as a way of circumventing the reference interview. Because few studies have examined the unmonitored referrals in collaborative virtual reference service settings, the present study investigates the extent of referrals and user satisfaction with referrals in these types of services. Specifically, the current study focuses on the following three questions: Research Question 1: To what extent are questions answered as referrals and to what extent are questions answered completely during collaborative virtual reference sessions? Research Question 2: Are certain types of questions referred more frequently than others? If so, what types of questions are they? Research Question 3: Are there any differences in user satisfaction levels betweentransactions answered with referrals and transactions answered with completed answers? The findings of the study will inform us about the question types that are frequently referred in the collaborative virtual reference service. The results will also inform of the extent to which referrals are made in collaborative virtual reference and the efficacy of the referrals as an acceptable answer category from the users' perspective. It will also help local libraries design their services for optimal question traffic management in the collaborative service. Literature review Collaborative virtual reference services Virtual reference services have become a critical component in current reference practice where the intent of libraries is to provide a fully functioning user service in the digital library environment. The rapid growth in the service has been made through both local and national efforts (Kresh 2002). Some critics question the cost-effectiveness of virtual reference services because of low usage and high cost (Coffman 2002; Coffman & Arret 2004). Based on a cost-benefit analysis, Coffman and Arret suggest the costs associated with software, training and staffing are 'a pretty expensive way to answer a question' and may not justify continuing the service. Others argue that virtual reference is a niche service that will merge into a part of reference services just as happened to telephone reference services in the 1930s (Kern 2004). In particular, responding to the negative view toward virtual references, many suggest joining a consortium as a way for those libraries that cannot afford a stand-alone virtual reference service (Bailey-Hainer 2005; Kresh 2002; Tenopir 2004). Examples of collaborative models include an international library network project, the Collaborative Digital Reference Service (CDRS), the http://www.informationr.net/ir/11-2/paper246.html[6/21/2016 3:36:16 PM] User satisfaction with referrals at a collaborative virtual reference service Virtual reference services, Reference services, Referrals, User satisfaction CLEVNET Library Consortium of Ohio public libraries, and AskColorado. Because of many benefits that collaborative service brings to both library management and user services, it is estimated that approximately seventy-six collaborative virtual reference services were in operation through the active participation of libraries of various types and sizes as of November 2004 (Sloan 2004). Referrals in collaborative virtual references Reference staff seem to make referrals for their patrons for various reasons. The cases can be divided roughly into two types: expert research and re-directional local referrals. Expert research referrals relate to the situations when the questions are referred to subject specialists who could answer with their high level of expertise and resources. Difficult research questions would fall into this type. The referrals in the other type, re-directional local referrals, relate to the situations when the answers require specific local knowledge or locally-restricted databases. Questions that would fall into this type include questions about local library events that may or may not be listed on the library's Website or patrons' requests to check overdue fines. Despite both being referral, each type of referral could have considerably different implications for reference service delivery. The former type of referral enables member libraries to coordinate human expertise and resources in order to answer the questions that are otherwise unanswerable. In this case, patrons are linked to richer resources and subject specialists. In contrast, re-directional local referrals can be ineffective or even result in a disservice to the patrons. That is, patrons who access the inter-institutional collaborative network may not necessarily understand the seamless nature of the service that goes beyond the walls of their local library. It is possible that people who access this service through their library portal would ask simple, circulation- related questions (e.g., overdue status), anticipating an immediate answer. However, many times, these questions cannot be completed by outside staff who do not have authority to access patron records. In this case, the patron should be referred back to her or his local library to get the final answer. If a high proportion of the questions submitted to collaborative services are referred to a local library for this latter reason, it becomes an ineffective deployment of resources for the local library. Having this concern, some libraries may decide to remain local for their virtual reference service, understanding that a high proportion of their reference questions relate to their library resources and services (Kibbee et al. 2002). In the current literature, there are not many empirical studies that have examined the proportion of referrals in reference services. Among the few, Ross and Nilsen (2000), in their unobtrusive reference service evaluation study, reported that 37% of the total reference transactions were unmonitored referrals. This finding is based on 261 reference transactions that occurred at physical reference desks in both public and academic library settings. http://www.informationr.net/ir/11-2/paper246.html[6/21/2016 3:36:16 PM] User satisfaction with referrals at a collaborative virtual reference service Virtual reference services, Reference services, Referrals, User satisfaction As libraries adopt collaborative virtual reference services, librarians tend to expect more research related referrals because collaboration and communication among institutions have become much easier and seamless in the online network. Interestingly, a rather reverse finding was reported in a recent study by Nilsen (2004) who compared the unmonitored referrals between physical and virtual reference settings. While more than one-third of the transactions have unmonitored referrals in the physical setting, the proportion was somewhat lower in the virtual setting, comprising 28% of a total of 40 transactions. In her updated study with more transactions, unmonitored referrals were reported as 38% of a total of 85 virtual reference transactions where chat reference comprised 25 transactions (Nilsen 2005). Since this small sample-size research appears to be the only one that investigated referrals in virtual reference setting, referrals in collaborative virtual reference are still largely unknown. Referrals by query type Are there any specific types of questions that result in more referrals than others in the collaborative virtual reference setting? In other words, would referrals be contingent on the types of questions? Indeed, most factual and subject-based research questions are expected to be answered during the session by most participating librarians, especially when the consortium is equipped with a shared online catalogue and consortia-shared databases. Among these questions, complex subject-based reference questions would be more likely to be referred to subject experts within the consortium. However, this type of question was reported to be very infrequent among the transactions. For example, Lee (2004) reported that research level questions comprised only 3% of all transactions in a virtual reference environment at an Australian academic library. A previous study suggests another type of question that would result in high referral rates. According to Kibbee et al. (2002), a service that generates a high proportion of local questions argues against the inter-institutional collaborative service because the questions would not be effectively answered by outside librarians in the consortium. This is because questions that involve local knowledge (e.g., history, events, or services) or questions that are answerable only by local staff (e.g., use of locally restricted databases or access to patron accounts) tend to result in more referrals back to local libraries. Yet, these speculations should be empirically tested to identify the types of questions that produce more referrals in a collaborative service. The results will serve as important managerial information for designing the practice and policies for virtual reference services. User satisfaction with referrals While the professional community generally encourages the use of referrals as an alternative to actual answers (Reference and User Services Association 2004), Dewdney and Ross (1994) revealed a pitfall of the referral practice by reporting its ineffectiveness from the users' perspective. In fact, this contention is supported http://www.informationr.net/ir/11-2/paper246.html[6/21/2016 3:36:16 PM] User satisfaction with referrals at a collaborative virtual reference service Virtual reference services, Reference services, Referrals, User satisfaction by a collaborative online chat reference use study (Kwon 2004). According to the finding, a mere suggestion of alternative resources or making referrals was not a positive predictor of user satisfaction. That is, unmonitored referrals do not warrant user satisfaction because the question might or might not be ultimately answered. A handful of studies has investigated the impact of referrals on reference effectiveness in traditional reference settings. In the study by Dewdney and Ross (1994), most patrons who received unmonitored referrals reported that they found nothing or very little helpful material when they had to search the sources to which reference staff referred them. In fact, Douglas (1988) categorized these referrals as one of the six types of reference service failures. Furthermore, Murfin and Bunge (1984) reported that the average reference success rate dropped from 69% to 22% when the librarian was busy and, as a result, made suggestions that were not followed up, instead of actually helping the patron with the search. Murfin and Bunge maintained that the practice, policy or necessity of directing the user rather than accompanying the user on the search was a factor that caused librarians to perform far below their potential. All of the above studies indicate apparent problems in the effectiveness of referrals in the traditional physical reference setting. Yet, there is still a question of how patrons would feel when their questions were referred as opposed to when their questions were answered completely. Furthermore, it is also a question how patrons would feel about referrals in a collaborative virtual reference setting. While many studies have examined the user satisfaction with virtual reference in general, few researchers have investigated the satisfaction of the patrons whose questions are answered as referrals in the collaborative virtual reference setting. The present study investigated this issue empirically. Method Setting and participants The present study examined chat reference services delivered through the Broward County public library system in Florida. As the largest library system in Florida, it has 33 regional and branch libraries. Since August 2002, the system has used the 24/7 Reference chat reference service delivered by the Metropolitan Cooperative Library System (MCLS), an association of libraries located in the greater Los Angeles area, funded by a Federal Library Services and Technology Act (LSTA) grant. The data analysed for the present study were on-line chat reference transactions initiated by the patrons of the Broward Country library system, along with online survey questionnaires, which the service users completed voluntarily. The questionnaire was devised and supplied by 24/7 Reference and popped up upon completion of each transaction. While the patrons were mostly the users of the library system, the reference staff who provided the service were from forty-nine http://www.informationr.net/ir/11-2/paper246.html[6/21/2016 3:36:16 PM] User satisfaction with referrals at a collaborative virtual reference service Virtual reference services, Reference services, Referrals, User satisfaction library systems across the United States participating in the MCLS 24/7 Reference programme. During the six-month research period between January and June 2004, a total of 1,667 transactions took place and 1,387 were analysable as being either completed or transferred transactions. In order to analyse the influence of referrals on user satisfaction, all 420 transactions that had a corresponding questionnaire that chat service users had submitted immediately after their session to report their satisfaction with the service were chosen. This amounted to 30.1% of the total analysable transactions. By having a chat reference session transcript with a matching user satisfaction survey questionnaire, the researcher was able to examine what question was asked in a reference session, whether the question was answered completely or referred, and to what extent the patron was satisfied with the answer or referrals she or he received. Measurement of research variables User satisfaction User satisfaction was assessed by three indicators: satisfaction with the answer, perceived staff quality, and willingness to use the service again. These indicators were measured by patrons' responses to three questionnaire items. First, satisfaction with the answer was measured by asking the question, 'Were you satisfied with the answer you received to your reference question?' Respondents answered the question by choosing one from 'Yes', 'Not sure' and 'No'. The result shows that, among the 417 respondents who answered this question, 65.2% reported that they were satisfied with the answer received, and 21.1% of the respondents were not sure whether they were satisfied or not. Finally, 12.6% of the respondents reported that they were not satisfied at all. Perceived staff quality, the second indicator of user satisfaction, was measured by the user response to the question, 'The quality of the library staff service in answering this request was _________'. The answer choices were 'Excellent', 'Good', 'Average' and 'Poor'. According to the finding (N = 416), 68.2% of the respondents answered that the librarians handling the reference questions were 'Excellent'. About 19.5% answered 'Good' and 11.3% answered 'Poor.' Finally, Willingness to use the service again, the third indicator of user satisfaction, was measured using the questionnaire item, 'Will you use this service again?' The answer choices were 'Very likely', 'Maybe' and 'Never'. 77.2% of users answered that they were very likely to use the service again 19.0% answered maybe, and only 3.8% said they would never use the service again (N = 417). The overall user satisfaction was computed by summing up the above three indicators (Footnote 1). Measurement research literature indicates that composite variables yield scores that are generally more valid and reliable than does a single http://www.informationr.net/ir/11-2/paper246.html[6/21/2016 3:36:16 PM] User satisfaction with referrals at a collaborative virtual reference service Virtual reference services, Reference services, Referrals, User satisfaction item (DeVellis 1991). In order to determine the reliability and validity of the scores yielded by the composite variable, a reliability test and a factor analysis were conducted, respectively. As a result, a Cronbach's alpha coefficient of .85 was obtained, indicating that the composite variable generated scores were reliable. The factor analysis revealed that the three items were represented by one factor, with structure-pattern coefficients of .868, .916, and .876. This factor explained 78.64% of the total variance. This high validity score confirms that the composite variable is measuring a single construct, suggesting that the composite variable is a valid measure of user satisfaction. Finally, the mean of the overall user satisfaction was 12.69 with a standard deviation of 3.44 in the range between a maximum value of 3 for 'highly dissatisfied' and the minimum value of 15 for 'highly satisfied.' Answer completeness Answer completeness refers to the level of completeness that library staff answers to the patron's inquiry during the reference transaction. A content analysis of 420 online chat reference transaction transcripts was conducted in order to code the levels of answer completeness. A total of 16 different levels of answer completeness were identified from the initial coding (See the first three columns of Table 1). Later, the initial sixteen levels of answer completeness were collapsed into four broader categories: completed answer, partial or no answer, referrals, and problematic ending (See the last three columns of Table 1). This reduction in the categories helped to remove some ambiguities in the micro-level coding. For example, when a transcript showed an abrupt ending, it was not clear whether it was patron's termination due to delay, librarian's premature ending, or technological failure. Thus, a broader category of 'problematic ending' incorporates all of these possibilities constituting an abrupt ending. A fewer number of answer categories also enables bivariate data analysis that is necessary to answer the research questions. Thus, the final coding for answer completeness categories were defined as follows: A transaction was coded as 'completed answer' if a transcript indicated that the librarian offered the patron direct answers or sources that may include answers to the patron's question and that the librarian ends the transaction with a proper closure (i.e., Librarians asks the patron if the question have been completely answered and the patron responds positively.). Additional sources of information may be provided to the patron along with the answer to the initial question. In particular, in order to determine a proper closure, specific behavioral items that are prescribed in the 'Guidelines for Behavioral Performance of Reference and Information Services Providers' (2004) were employed. It is the Guidelines that the Reference and User Services Association (RUSA) developed to train reference staff and to assess their performance. The following four items were specifically selected from the Guidelines: http://www.informationr.net/ir/11-2/paper246.html[6/21/2016 3:36:16 PM] User satisfaction with referrals at a collaborative virtual reference service Virtual reference services, Reference services, Referrals, User satisfaction 4.7: Asks the patrons if additional information is needed after an initial result is found (e.g., 'is there anything else I can help you with?') 5.1: Asks patrons if their questions have been completely answered (e.g., 'did you find what you needed?,''does this completely answer your question?') 5.2: Encourages the patrons to return if they have further questions (e.g., 'If you don't find what you are looking for, please come back and we'll try something else.') 5.9: Takes care not to end the reference interview prematurely. A transaction was coded as 'partial or no answer' if a transcript indicated that the patron's initial question was either partially answered or unanswered and no further reference to other sources or services was provided. A transaction was coded as 'referrals' when a transcript indicated that the patron's initial question was either partially answered or unanswered during the reference transaction but reference staff gave the patron information sources or contact information that may or may not contain the answer. The reference staff did not check if the sources had answered the patron's question. A transaction was coded as 'problematic ending' when a transaction was ended before the patron received the answer due to disconnection, delayed answer, or without clear reason. It also includes librarians' premature ending without proper closing remarks and system failure due to connection problems. Question types A content analysis of transaction transcripts was undertaken to identify the types of questions that are submitted to the collaborative virtual reference. A total of fourteen categories of question types emerged initially. Again, the fourteen categories were collapsed into five broader question types: simple, factual questions, subject-based research questions, resource access questions, circulation-related questions, and local library information inquiries. Below list the definitions of the five types of questions with examples: Simple, factual question refers to known-item search questions, directional questions, or a question of a factual nature that can be answered quickly by consulting only one or two reference tools (Garnsey & Powell 2000). Examples are 'Is the book Funny Laws and Other Zany Stuff by Sheryl Lindsell-Roberts available?', 'What does 'borrower status: restricted' mean?', and 'What is the correct way to write this name, in last name first order: Dr. James R. J. Smith, III?' Subject-based research question refers to a reference question requesting a particular kind and number of books or journal/magazine articles on a specific topic, teaching the steps of the research process, locating hard to find literature, and guidance with database choices (Marsteller & Mizzy 2003; Ward 2004). Examples are 'Can you tell me how to celebrate children's day in America?', 'I want to know the titles of 5 books that deal with the development of gender identity', and 'Where can I find a current article on the culture of any indigenous tribe?' Resource access question refers to an inquiry about access and use of library catalogues, databases or other library resources. Examples are 'How do I access the Oxford English Dictionary through the library Website?', 'I'm having trouble accessing my library account. Can you help?' and 'Where can I access NetLibrary to http://www.informationr.net/ir/11-2/paper246.html[6/21/2016 3:36:16 PM] User satisfaction with referrals at a collaborative virtual reference service Virtual reference services, Reference services, Referrals, User satisfaction read an e-book?' Circulation-related question, for the purposes of this study, refers to an inquiry about circulation policies (e.g., overdues, fines, renewal), online renewals, patron account checking, or other circulation related issues. The answer may or may not be readily found from the Website of the local library. Questions that fall in this category include 'Can you check which books are overdue in my record?', and 'I saw I have a 60 dollar fine. Is there a payment plan or forgiveness policy I could do to be able to check out books again?' Local library information inquiries, for the purposes of this study, refers to a question that involves information about a local library. Questions that fall in this category include 'Can I use a typewriter in Sunrise branch library?' 'I am looking for a job at this library', and 'Where should I e-mail comments about Thursday's 'Balancing the Books' seminar at the main library? There was no box into which we could anonymously drop the evaluation forms'. Inter-coder reliability Inter-coder reliability tests were undertaken to ensure the consistency of coding for both question types and answer completeness. The tests show the level of agreement between two independent coders. To test the inter-coder reliability, the primary researcher coded the entire 420 transactions. Subsequently, the second coder, a reference librarian who received training for coding the RUSA guideline categories, coded every fifth transaction of the 420 transactions (n = 84). This number comprised 20% of the total transactions, which is a recommended percentage for social science research (Neuendorf 2002). A widely used Cohen's Kappa (k) was used as the inter-coder reliability index for this study and was calculated using SPSS 13.0 (SPSS Inc. 2004). Cohen's Kappa coefficient was found to be 1.00 for question type and .75 for answer completeness, indicating high agreements between the two coders in classifying both variables. Results Proportion of referrals and answer completeness Research Question 1 examines 'To what extent are questions answered as referrals and to what extent are questions completely answered during collaborative virtual reference sessions?' Both Table 1 and Figure 1 show the results of answer completeness based on total transactions (N = 420). Initial coding Final Freq. Percent Freq. Percent category coding Answered 204 48.6 completely Completed Answered & 237 56.4 12 2.9 answers transferred Answered & referred 21 5.0 Ended with partial 12 2.9 answer Partial or http://www.informationr.net/ir/11-2/paper246.html[6/21/2016 3:36:16 PM]