Using research to justify your service: cause and effect

In his Keynote Address to the Health Libraries Group Conference, ‘To bury or to appraise’, in 1996 Muir Gray, erstwhile champion of health librarians, shared a vision of the ‘librarian epidemiologist’—the information specialist who combines professional skills with a knowledge of research design. At that time, this role was being modelled by Ann McKibbon and colleagues in Canada.1 Here in the UK, bearing in mind that this was a decade ago, those of us combining these two quite different skill sets were numbered on the fingers of one hand. Now, increasing numbers of librarians have a foothold in both domains. Some clinical librarians have registered on Masters courses in health research, scores of health librarians have shared in week-long evidence-based health-care events and over 200 colleagues have undertaken the PRECEPT/ADEPT programme,2 itself derived from the McMaster University ‘Panning for Gold’ model.3 Parallel developments exist across Europe, in North and South America and many other library networks across the world. This 1996 conference in Exeter examined, almost exclusively, the role of librarians in supporting the evidence-based practice of others. Subsequent years have seen increasing recognition of the importance of the same principles for our own professional practice. How can a knowledge of research design help a library manager to make an effective case for the instigation, continuation or, indeed, survival of their service? Those undertaking the ADEPT module on aetiology (causation) are introduced to principles of cause and effect—important precepts when preparing a case for a service. This column briefly ‘walks through’ these principles in the specific context of a library manager who is ‘fighting their corner’. We begin with a practitioner-based scenario: For some years you have been running a well-received library service, offering an intensive level of support and training in a clinical environment, within a modest hospital in the market town of Elucevan-Le-Stelle. Disaster strikes—in the most recent performance ratings your organization is awarded zero stars! A new Chief Executive is drafted in from elsewhere, Mr Bloodstone, a penny-pinching manager from the Kwik-Spend chain of supermarkets. Within 3 months Mr Bloodstone has worked his way down his list of the most prominent cost centres. He is now ready to target the library. You need to assemble evidence to support the continuation of your clinical support information service. Before addressing this scenario we must acknowledge the difficulties implicit in this challenge. Just prior to this year's Under One Umbrella Conference, I was contacted by an experienced Evidence-Based Healthcare developer who was preparing a guest presentation and an accompanying article4 by finding ‘robust (key word) evidence that librarians are effective’. Focusing on the specialist role of librarians as searchers, he had found the available evidence of poor quality (by accepted evidence-based standards) and thus inconclusive in proving the unique value of the health librarian. Caught in the open, beyond the refuge of composure and mature reflection, I rehearsed a ‘rag-bag’ of arguments, why librarians should conduct searching and why the evidence base is so poor, advanced with increasing desperation: The Cost-effectiveness/opportunity cost—librarians are cheaper and can therefore do it for cheaper/longer. Despite these unfortunate ‘bumper sticker’ connotations, it has been demonstrated that clinicians rarely spend more than 2 min pursuing an answer5 while clinical librarians often spend two or more hours using multiple techniques and sources. Most librarians do not perform the volume of searching to justify a specialist role. This parallels volume-outcome arguments in health care where surgeons need to undertake a minimum number of surgical procedures to maintain proficiency. The technology (i.e. electronic databases) is limited therefore librarian results are not appreciably better than those of clinicians.6 Armed with a single blunt knife, a butcher and surgeon achieve similar results. Equip both with a full range of surgical equipment and the differential performance (hopefully!) becomes more evident. Most librarians are generalists not specialists and therefore do not perform better than clinician searchers. ‘Consultant searchers’, those who perform extensive systematic review searching, are a more appropriate comparator than the majority of ‘general practitioner searchers’. Case mix considerations. Perhaps librarians receive problematic, in some ways untypical, questions and are therefore unable to demonstrate success as conclusively as for straightforward questions which are handled similarly by clinicians and specialist searchers. Absence of evidence is not evidence of absence. Possibly we have not conducted any rigorous studies to prove our value (yet!). Over-reliance on such designs as the critical incident technique (prone to selection and recall biases) means that few rigorous studies are conducted. Managers have, until now, typically required little more than ‘soundbites’ from satisfied customers as evidence of effectiveness. However, it is of only scant consolation that managers are even less evidence based than librarians! Other essential staff are not required to justify their existence. Where is the justification that general practitioners (GPs) are effective? Perhaps librarians, like GPs, can claim to occupy the category of ‘self evident’? Of course this is a potentially flawed and high risk strategy! The disconcerting aspect of the above is that these statements are advanced against a fairly complete knowledge of the evidence base. Even the most-often quoted study supporting the effects of literature searching on patient care is methodologically limited.7 Key to an effective case are considerations of association versus causation. If some reported outcome, e.g. an improvement in the searching skills of nurses, is happening anyway, regardless of whether or not there is a training programme in place, it is difficult to demonstrate to a manager that it is our training programme that causes this effect. At most we may demonstrate an association; where training programmes are in place the effect is more likely to occur or is likely to be more pronounced. Of course, if many supposed contributing factors are present together, making it impossible to isolate the specific contribution of the factor of interest, our argument is diluted. Sir Austin Bradford Hill, a noted statistician who researched the causes and effects of smoking, proposed nine ‘viewpoints’ when trying to establish cause and effect (does factor A cause phenomenon B?).8 Although these were originally developed within occupational medicine, these viewpoints are useful when justifying a service. It is important to bear in mind that satisfying one or more of these viewpoints does not conclusively establish cause and effect. Indeed the converse is true—failing to demonstrate one or more of these factors does not establish that a link does not exist. Nevertheless, the more lines of argument that are satisfied the stronger the case that a factor and phenomenon are causally related. This viewpoint addresses the question ‘how large is the effect?’. This is usually our first line of argument. It is generally true that, the more impressive our figures, the more likely that our service is having an impact. Two points of caution here: managers, particularly those with a commercial background, are less likely than other professional groups9 to be beguiled by relative figures on improvements (i.e. uptake of a service is said to double if it moves from 1 to 2% or equally from 50 to 100%). Also, because figures from early in the life of a service start from a low base they often demonstrate more dramatic increases than later when ‘market saturation’ levels are reached. Rather than simply reporting year-on-year increases in annual reports, it is helpful to include original baseline figures so that your case benefits from an attributed effect through the life cycle of the service. This viewpoint asks ‘Has the same association been observed by others, in different populations, using a different method?’ Your case is strengthened if you can demonstrate, either from published research studies or from a growing evidence base of operational service data, that effects observed in your service are being achieved by comparable services elsewhere. This implies that the general service model is robust and not a fortuitous ‘one-off’. Paradoxically, many librarians welcome the opportunity to demonstrate that their service performs better than comparable services. This is good for kudos and recognition but does correspondingly little for sustainability. The implication could be drawn that, when the current incumbent leaves service, achievements will be lost—far better to demonstrate that the service model itself is sound and worth ongoing investment. Related to this is evidence that establishes a specific influence of your service on the outcome being measured. So, if a change, either planned or unintentional, occurs to the service is its effect detected in the data you are collecting? Suppose, for example, you deliver an introductory literature searching session, year after year, to physiotherapy students. If, one year, through illness or service pressures, you are unable to deliver this session and performance in a related literature search assignment is correspondingly poorer you have evidence, albeit inconclusive, of a possible connection between your training and academic performance. This may then be used to justify continuation of this particular training session. Related to the above, and a key factor in justifying a service, is whether the cause precedes the effect. If improvements are taking place even before a service is established, our argument is correspondingly weaker than if introduction of a service clearly initiates an improvement or arrests, or reverses, a previous decline. Of course, the time lag between initiation of a service and measurement of an improvement must be plausible. If an improvement dates from when funding is agreed rather than when the service is actually implemented this weakens rather than strengthens our case! This terminologically challenging but basically simple concept asks ‘is there a dose response?’ In other words, if we do more of the activity we are measuring do we achieve better results? If we replace one hour of search training input with two hours, are our results at least doubly (if not more!) impressive? This helps point to an apparent relationship between our input and our achievement. Caution must again be exercised—is there a saturation point (‘overdosing on training’) where further input makes things worse rather than better? This viewpoint asks ‘Does it make sense?’ In other words, is it possible to explain why the particular input that you are making may be achieving the output that you are demonstrating. For example, if, on the one hand, you are training participants to cite bibliographic references, then it is plausible to suggest that this may be reflected in the quality of their bibliographies and reference lists. If, on the other hand, you are trying to establish that increasing numbers of inter-library loan (ILL) requests are leading to better patient care, then you need to establish several links in the chain. Are requests being made for the purpose of patient care? Are articles being received in time to make a difference? Are they being used? Do patients whose care is ILL supported have better outcomes than those whose care is not? Coherence refers to whether the evidence all fits together in a consistent way. Unlike most of the preceding items, this does not relate to the individual integrity of a single item of data. It concerns the ‘whole story’ that your various data items are telling. It is best modelled by asking questions along the lines of ‘If that is so then why … ?’ For example, ‘If you are saying that training in literature searching is increasing clinicians’ use of the medical literature then why have the numbers of inter-library loans decreased each year since you offered that training?’. Of course there are good, rational explanations for such apparent contradictions (e.g. availability of electronic journals, more judicious selection of relevant articles, et cetera). Nevertheless, in making your case you should anticipate such objections and construct your ‘story’ accordingly. While we frequently bemoan the quality of research evidence, we should nevertheless ensure that we are familiar with key studies to support an evidence-based case. These may either examine the effectiveness of a similar service in a practical context (as previously mentioned under ‘consistency of association’) or, more germanely, may be our equivalent of animal or laboratory-based studies. In the absence of investigations performed on rats, studies examining the efficacy of interventions on cohorts of docile students are considered acceptable! One colleague has identified dozens of experimental studies examining information literacy skills training interventions in students of all disciplines. This provides useful corroborative evidence for the likely effect of similar interventions performed specifically with health-care practitioners. Our final, and possibly least important, viewpoint—simply because it can only corroborate, not establish, a likely cause—is to examine whether an observed association is supported by similar associations. With such a limited evidence base it is crucial to draw upon supporting evidence from elsewhere. For example, a previous column examined the implications of a systematic review of generic training in evidence-based practice for our specific focus of literature searching.10 It hypothesised, by analogy, that integrated work-based training in literature searching may be more effective than stand-alone courses. Further analogies include studies of clinical informaticists to justify other models of question-answering services.11,12 Establishing analogies with wider postgraduate education may similarly be effective when targeting a Clinical Tutor while analogies to commerce may impact on our hard-nosed manager. Hopefully this brief discussion of cause and effect, in the context of making a case for library services, demonstrates two points. First, that it is potentially useful to transfer knowledge of research design principles, as used to support the practice of others, to our own evidence-based information practice. Second, that a judiciously chosen amalgam of locally-collected data (librarian observed) and findings from published research (research derived) may be used in conjunction with knowledge of our user or stakeholder population (user-reported) to make our case more effectively.13 While these nine viewpoints, as expressed by Bradford Hill, were never intended as formal criteria, they nevertheless provide a useful aide-memoire for any library manager charged with justifying their services. Whether advocating such an approach will itself have any effect is, of course, a topic for further exploration!