Column Editor

Will health librarians and related information workers ever work together to create an international network, such as the Cochrane Collaboration, dedicated to the purpose of preparing, maintaining and disseminating systematic reviews1 of the effects of health information services and systems? As we have remarked elsewhere ‘Information scientists may be equipped to scan the horizon but they possess silicon chips, not crystal balls’.2 Nevertheless, it is possible to take an informed look at developments in systematic reviews, together with the idiosyncratic characteristics of our own evidence base, and to assess where future prospects might lie. In previous issues this column has focused on obtaining funding for September 2000, and the critical appraisal of December 2000, primary research. In this issue we turn the spotlight onto secondary research, namely systematic review and synthesis. Before discussing the likelihood of a separate collaboration for health information science it is necessary to consider whether or not the Cochrane Collaboration would be an appropriate home for such an initiative. The Cochrane Library certainly contains a number of full-text reviews3, 4 or bibliographic references to topics falling within the wider domain of health information. A major stumbling block, identified in preliminary discussions with individuals involved in steering the Collaboration, has been that a Cochrane systematic review is required to be concerned primarily with an intervention's direct effects on health care outcomes. This admirably pragmatic tenet is no doubt aimed at protecting the Collaboration from becoming overly academic and detached from the considerations of health care delivery that should quite rightly be paramount. In considering to what extent the research reported in the health information literature focuses on patient-focused health outcomes, one would find that a large proportion demonstrates effects of information services or skills training on the knowledge of recipients, e.g. ‘I went on a medline training course and now I know about the explode and focus features’. A smaller but significant proportion examines the effects on the attitudes of participants, e.g. ‘Having attended a library open day I am now more likely to use the library services’. Still fewer reports concentrate on the effects on the observed behaviour (as opposed to self-reported behaviour!) of the subjects of a research study. Finally, an almost negligible amount focuses on whether the health of patients (i.e. their health care outcomes) actually benefits as a result of an information-related intervention. It is only this final category that would interest the Cochrane Collaboration. Such studies are less plentiful precisely because it is so difficult to prove such an effect. There are many confounding factors in the chain between delivering an information skills course or providing an electronic textbook and the benefit a patient might receive from the clinician's newly acquired skills or knowledge, and this makes it problematic to establish any genuine ‘cause and effect’ relationship. It is this, far more than the well-documented preference of the Cochrane Collaboration to focus on randomized controlled trials, that appears to pose the most significant obstacle to the widespread inclusion of health information topics in the Cochrane Library. Nevertheless developments in the recent years of the Cochrane Collaboration such as the recognition of health economics and qualitative research methods and the raised profile of the Effective Practice and Organization of Care (EPOC) group suggest that this position is not necessarily to be seen as an intransigent one. What characterizes these recent developments, however, is the prior existence of powerful lobby groups organized around established international communities of researchers. The health library and information community, academics and practitioners, need to consider seriously whether a similar sustained effort is required to secure recognition of its own potential contribution to the Collaboration. Some hope is offered by the inclusion in the Cochrane Library, under the auspices of EPOC, of a review protocol for instruction in critical appraisal,5 an intervention similar to our own core activity of instruction in literature searching. Health information professionals inhabit a ‘western front’ between the ‘hard’ applied science of medicine and the ‘softer’ social science of librarianship. Much of our activity is conducted within the domains of education (students of nursing, medicine and other professions) or of training (postgraduate education, continuing professional development, etc). We also find ourselves torn between the contrasting paradigms of the quantitative research espoused by the biomedical community and the qualitative approaches that are more common in the nursing and therapy professions and, indeed, so typical of our own research. Our ‘amphibious’ nature extends to us, at least at this preliminary stage, the prospect of involvement in another international initiative, the Campbell Collaboration. This recent sibling to the Cochrane Collaboration (http://campbell.gse.upenn.edu/) is a fledgling international network aimed at preparing, maintaining and promoting the accessibility of systematic reviews of the effects of social and educational policies and practices. It first met in February 2000 and it has been strongly supported by leading figures from the Cochrane Collaboration. The range of domains and outcomes to be considered within the activities of such an overarching organization is potentially much broader than that currently adopted by the Cochrane Collaboration. The downside of this might be a possible tendency for those with position and influence within the National Health Service to view such educational interventions as being removed from the main targets for their initiatives and funding. In short, placing the evidence base of health information squarely within the aegis of the Campbell Collaboration could result in a return to the assumption that health information work should call on the traditional reservoirs of postgraduate education funding rather than the newly opened streams associated with research and development or support to clinical care. Nevertheless any ‘flag of convenience’ likely to stimulate the development of an evidence base for health information services and systems should not be dismissed without serious investigation. ‘Ask not what the Cochrane/Campbell Collaborations can do for you—ask what you can do for the Collaborations’. This misquotation from J. F. Kennedy's inaugural address reminds us that involvement in one of these well-organized collaborations can only come once we have started to marshal our own information resources. Can our current evidence base sustain the rigorous methods required for systematic review and meta-analysis? A feasibility study conducted for the Health Libraries Group Research Working Party, the predecessor to the current LINC Health Panel Research Working Party, found that our evidence base is scattered across a number of sources and that it exhibits heterogeneity in the range of research designs and outcome measures, together with poor research methodology.6 This situation is exacerbated by poor indexing of research designs and methods and the prevalence of uninformative abstracts. If it is unlikely that many review questions from our domain will support a full-blown quantitative synthesis of results (meta-analysis), what might be the way forward? In a study that approximates most closely to the model espoused by the Cochrane Collaboration, physicians’ preferences for information sources are examined.7 In this review selected data from 12 studies published between 1978 and 1992 were compared, quantitatively aggregated and synthesized. The top five preferences from each study were ranked and then cross-study similarities in rankings were identified and summarized. This review may be flawed in that there is a simplistic assumption that rankings from different studies can be pooled as if a difference between 1st place and 2nd place in one study is equal to a difference between 1st and 2nd place in all the others. Nevertheless, it does provide a powerful demonstration of the power of synthesizing data in such a manner. Significantly, just as meta-analysis originated from social sciences before migrating to medicine and being enthusiastically adopted as its own, an alternative approach was derived from education and is known as meta-ethnography.8 This technique was originally used by its promulgators to synthesize qualitative data from a number of school inspection reports. In this way emerging themes from across reports could be identified and summarized. A three stage process is used that involves extracting themes from each individual report, tabulating all these themes into a single summary report and then finally establishing common categories and subcategories and equivalences across studies. So, for example, if the leadership characteristics of the headmaster were seen to be a major factor in the success of a number of schools, this would become a category for analysis. Individual characteristics (e.g. sense of humour, approachability etc) would then become subcategories. This approach could be applied across a body of related reports of health information research (e.g. all primary care information projects) to encapsulate our current knowledge and to identify future directions for research. If we can apply such a meta-ethnographic approach to our professional literature it is clear we could also use it to synthesize a myriad of related pilot projects or individual case studies that never make their way into formal publishing channels. If individual case studies of library projects were to support this level of analysis they would need to have fortuitously collected large amounts of supporting data—an uncommon characteristic of most local initiatives! This suggests a way forward that parallels an approach used by the Cochrane Collaboration, namely, collaborative overviews using prospective data collection. Some systematic reviews, notably those in cancer, are regularly updated by the ongoing results from large trials.9 This requires initial agreement regarding what data should be collected so as to ensure consistency across studies. This is best illustrated by an analogy from within our own field. Suppose that the various ‘clinical librarian’ projects currently springing up around the UK, or indeed the world, could agree on a minimum dataset to be used for their evaluation. This dataset might be based on criteria from a previous article.10 Alternatively it might be the result of a process of consensus. Each participating librarian would agree to collect at least the data required by the minimum dataset. They could, of course, collect any additional data that their local evaluation required. In this way each additional evaluation would not only draw strength from taking place within an acknowledged frame of reference but would, in turn, also contribute to the growth of the knowledge base. Sounds simple doesn’t it? This brief outline of the ways in which systematic reviews might relate to our field, indicates both the current situation and possible future directions. It is certainly possible to conduct systematic reviews in health information topics where randomized controlled trials exist. It is also possible to use meta-analytic techniques (not necessarily full-blown meta-analysis) to add value to an existing body of quantifiable research. Meta-ethnography offers the possibility of extracting common themes or hypotheses for further investigation from a number of related qualitative studies, either published or unpublished. Finally, agreement on common study protocols for initiatives at a local level that might contribute ultimately to an international body of evidence, as in our example from the clinical librarian movement, would seem to offer a practical mechanism for ongoing research and evaluation. Of course, to instigate such collaborative international activity also appears to require identification of an individual with a Celtic name (as in both Campbell and Cochrane)! Who are we to say that in years to come the prospect of a McKibbon Collaboration11 or of a Marshall Collaboration12 might not be realized?