Blogs, wikis and podcasts: the ‘evaluation bypass’ in action?

Andrew Booth Pick up any recent issue from our professional press, of which the September 2007 Library + Information Update is my current exemplar, and you are assailed by enthusiasm for the ‘Web 2.0 technologies’. Clearly these technologies, recently examined in an excellent overview within this journal,1 have tremendous potential to transform our library and information services from a physical locus to the hub of a virtual social network. I cannot help but be reminded, however, of a formative staging post on my personal road to evidence-based library and information practice (EBLIP)—namely reading an article in 1990 on the fad or phenomenon of that time, the CD-ROM database.2 Labelling CD-ROM a ‘transient technology’, those two agents provocateurs, Tony McSean and Derek Law, accurately captured the lemming-like tendency for librarians to rush into adoption of this new, but essentially unproven, delivery mechanism. Since that time I have frequently reflected on our personal and professional gravitation towards novel technologies—even recording my thoughts on my Psion Organizer! In the context of EBLIP, I have recently become absorbed in the concept of the ‘evaluation bypass’, first encountered within the context of innovative surgical procedures.3 Similar concerns within health information practice occasioned me to ask: ‘What questions do you and your colleagues ask when contemplating the introduction of some new technology or some innovative service? It is true that, on occasions, lack of time or other logistic constraints will conspire to cause what we might label an “evaluation bypass”. On such occasions ... we may move to uncritical adoption of a new technology or procedure. Hopefully, such instances are the exception rather than the rule.’4 In truth, that final sentence is more vain aspiration than firm conviction. Our adoption of information technologies is largely uncritical—as Muir Gray is reported to have asked—‘if information was a drug would it ever get licensed?’ At a team Strategic Away Day you discuss whether it is feasible to replace the once-successful monthly paper library newsletter with a library blog. You suspect that there has not been a formal evaluation of the comparative merits of these two alternatives. This gets you wondering what evidence base exists for the clutch of Web 2.0 technologies, namely blogs, wikis and podcasts. Is this an example of the ‘evaluation bypass’ in action? A rapid search was conducted of the four most relevant easily accessible databases—lisa, the free lista service from EBSCO, eric and the Web of Knowledge (isi). For inclusion in this cursory literature survey, articles should report either research or evaluation and examine an Intervention that is a constituent Web 2.0 technology. Evaluation could employ either quantitative or qualitative measures and could adopt one or more of the following Perspectives; library management, information professional or service user. Articles reporting effectiveness, as opposed to other outcomes of interest, preferably contain a Comparison (either between competing Web 2.0 technologies, e.g. blog versus wiki or between a Web 2.0 technology and either paper-based or digital precursor). For inclusion, research should be conducted in a library or related (in the widest possible interpretation) setting. Articles evaluating use of Web 2.0 technologies for increasing democratic participation or evaluating Wikipedia were specifically excluded. Keywords included blog*, podcast* and wiki* and the words ‘research’ and ‘evaluat*’ were particularly looked for as markers of evaluative studies.5 A more sensitive approach to retrieval would include ‘Web 2.0’ and the names of other less common constituent technologies, and would itemize a list of specific research designs (e.g. focus group, survey, questionnaire). Nevertheless, it was concluded that the risk of a relevant and useful abstract, not including one of the three most common Web 2.0 technologies, was minimal. As anticipated, there are several hundred references on the three principal Web 2.0 technologies. These are typically either descriptive articles or editorials. For example, of almost 200 references on the eric database, only one had been tagged as either a research or evaluative article. This is not to deny that descriptive accounts may be useful within the context of an overall decision. For example, Withers reports using a wiki among information desk staff at Miami University,6 while the article most immediately relevant to our scenario, Preparing for the Birth of Our Library blog,7 outlines ten considerations when implementing a library blog as a way of communicating with library patrons (Table 1). Blog* was the most commonly cited of the three technologies, as one might expect for a word that has been in existence since 1996 and which was voted the Merriam-Webster's Dictionary Word of the Year for 2004. On the isi Web of Science database, many evaluative works examined technical functionality of certain features and did not examine impact of the technology. These correspond to laboratory studies examining efficacy when our principal interest is how it works in practice—its effectiveness. Upon examining the characteristics of identified studies, we can confirm the paucity of the evidence base. An article that, from its title, sounds evaluative—‘The library blog: innovative idea or wasted words’8—merely reports a survey that showed that 75 per cent of students were satisfied with blogs that were posted and wished them to continue. Of course, within EBLIP such a survey constitutes a very low level of evidence. Key when examining such evidence is how the sample was identified and also what the comparator is—asking the famed ‘eight out of 10 cat owners’ whether they would eat Whiskas cat food would probably elicit difference responses depending on whether they had access to their usual diet or whether cat food was the only available source of sustenance! This 75% satisfaction rate is further perpetuated in an earlier article which, it transpires, reports exactly the same survey from the Paul J. Gutman Library, Philadelphia University.9 Also of relevance to our scenario, but of similarly low level evidence, is a report from Georgia State University on a blog, Science News, developed by their faculty librarians.10 This reports that ‘preliminary user statistics’ offer ‘encouragement that patrons are reading it’. In a more formal observational study, a set of blogs was monitored during September and October 2003 and descriptive statistics and content analysis were used for evaluation.11 The author found that ‘most of these blogs are excellent information hubs, i.e. provide short summaries (sometimes with a personal touch) with links to further information on major issues related to the topic of the blog’. Of course most library services are not exclusively concerned with delivery and support. Many librarians participate in information literacy training and in this context one of the more recently published evaluations looks potentially useful.12 Examining blogs, in terms of peer learning and peer support, on a module of an accredited library and information science (LIS) degree programme, again using content analysis, the authors found that they are ‘learning tools in their own right and not simply an option for providing information online’. In a similar context, but in one of few studies evaluating multiple technologies, an Open University study13 examined student feedback on each of four technologies (blogging, audio conferencing, instant messaging and Harvard's Rotisserie system) and, perhaps unsurprisingly given the nature of such case studies, this was overwhelmingly positive. Brief consideration of articles on Web 2.0 technologies leads to realization that a wider evaluation framework is required than that we typically use. In addition to the Content and Technical aspects typically evaluated for web resources, there is an important Social dimension. In a previously published review,14 I have observed that information resources that possess a ‘community of practice’ dimension require such an evaluative framework and, propitiously, this is reflected in the Web 2.0 literature. In line with the above, some evaluations examine the impact of blogs in preventing feelings of isolation and alienation, particularly within a distance learning environment, and such findings from teaching have obvious implications for virtual library services which often provide similar outreach at a distance.15 Podcasting has similarly been examined as a mechanism for reducing isolation and promoting inclusivity among distance learners.16 Student listeners saw podcasts as especially effective in clarifying and enhancing their understanding, reinforcing what they had learnt, and guiding the direction of their study efforts. Factors that are likely to determine choice and frequency of media use are immediacy of feedback, personal focus, transmission of cues, functionality, usability, and ease of use.17 Although these feature in the context of podcasting, they likely transfer to any Web 2.0 technology under consideration. The variable nature and quality of the evidence makes it unsuited to formal appraisal and conclusions from this brief overview can only be tentative and exploratory. Starting from the broadest principle of ‘first do no harm’, then, assuming that in maintaining the blog library staff do not neglect their current service responsibilities, instituting a library blog fulfils first principles.18 Certainly, the blog does carry inherent advantages over its virtual library predecessor, the static web page, a point substantiated by a comparative experiment which found that participants assigned to blogs perceived an organization's ‘conversational human voice’ to be greater than participants who were assigned to read traditional web pages.19 This led to a corresponding increase in trust, satisfaction and commitment. Given the temptation, not least among librarians, to identify a technological solution and then to look for a problem to solve with it, we do well to step back from our scenario and ask—is investing time in maintaining a newsletter-type blog the best possible use of this time? For example, our library service delivers a funding support service and so maintenance of a funding RSS feed and a swicki (search wiki—tailored search engine) might be a legitimate alternative activity. It is useful to step back and address a prerequisite background question—is this the best use for a blog? What are the alternatives? In this context, an overview of possible applications may prove enlightening.20 Additional food for thought is provided by a comparative article that concludes that, although wikis are currently the least popular conversational technology (from which podcasts are naturally excluded), they hold greatest potential for facilitating knowledge management needs.21 Consideration must also be given to how the eventual tool, whether it be blog or wiki, is to be structured.22 The above is not necessarily typical of the EBLIP process. I started from the assumption that there is little evaluative material for Web 2.0 technologies and was gratifyingly able to demonstrate this ‘evaluation bypass’ from my retrieved results. In defence of my approach, this process was initiated by a real-life scenario with which we, and many others, wrestle at the moment and I did not pick an obscure and acknowledged transient technology as an exemplar. I should also acknowledge that I identified several potentially useful items for implementation of our Library blog although none establishes that our approach might be effective (or even if it will be more successful than if I had not spent an afternoon reviewing the evidence base and had pushed blindly ahead!) Clearly blogs and wikis currently command significant planning time and effort—stepping for a moment outside the health sector indicative figures from the US school library sector in 2006 suggest that 29 per cent of respondents are currently using or plan to use blogs within the next year, with 18 per cent indicating similar plans for wikis.23 While such volume of projected use makes it likely that uptake will forge ahead regardless of the existence of, or even the findings from, evaluation, it does conclusively demonstrate that Web 2.0 technologies are currently being fast-tracked along an evaluation bypass. As Sharples and colleagues point out within the context of surgical interventions, the evaluation window for new technologies is very limited: ‘The ideal moment is between standardization of the new technique and its wide dissemination, but this is often a very brief moment given the understandable enthusiasm for the newest methods’.3 Such observations lead me to return to a plea, voiced previously in this feature, namely of evaluation alongside innovation.24 Only by putting in place mechanisms for capturing evaluative data as we introduce new technologies will we provide others with the opportunity to build upon what we are currently learning—and, after all, isn't knowledge transfer the business in which we are all inextricably involved?