Issues in methodological research: perspectives from researchers and commissioners

OBJECTIVES: (1) Methodological research has few well-defined tools and processes analogous to those available for reviews and data collection in substantive health technology assessment. (2) This project was set up to obtain researchers' and others' views on the innovative projects on research methodology under the NHS Health Technology Assessment Programme and the usefulness of the research. (3) The study was intended to span both epistemological and management issues. (4) The following issues were explored: (a) the degree to which researchers would feel constrained by the "Cochrane" approach to systematic reviews when undertaking reviews of a methodological nature; (b) whether methodological projects may require exceptional design and management arrangements, in view of their novelty, subjectivity and complexity; (c) whether researchers would seek out other methods, in addition to undertaking reviews of argument, as a means of extending their understanding of methodological issues (there may be three categories of research methods in methodology: reviews of methodological argument, studies that use the literature as a source of data, and research that collects new primary data); (d) whether the Methodology Programme overall can be considered a "success". METHODS: (1) Telephone interviews were carried out on researchers (one senior and one junior per project), resulting in 35 interviews from 19 of the 20 target projects. (2) A qualitative postal survey was sent to 12 people who had played a key role in the development of the Methodology Programme; replies were received from six of them. (3) Analysis was undertaken of the hit rates for 29 projects on the NCCHTA website by the end of February and the end of May 1999, comparing those concerned with methodology (n = 10) and those concerned with other issues (n = 19). RESULTS: UNDERTAKING METHODOLOGICAL RESEARCH: VIEWS OF RESEARCHERS: This section summarises the views of 35 researchers who were interviewed by telephone. RESULTS: UNDERTAKING METHODOLOGICAL RESEARCH: VIEWS OF RESEARCHERS: (THE NATURE OF METHODOLOGICAL REVIEWS): (1) There was a reluctance among researchers to use the term "systematic review" in the methodological context. (2) Practical problems in undertaking methodological reviews were found at every stage of the research process. (a) In the initial search stage, preplanned strategies were difficult to maintain, owing to the need to respond to the problems of too few or too many references. (b) At the analysis stage, most studies were not formally weighted, but there was implicit weighting in researchers' views of their merits or relevance. (c) It was often only at the synthesis stage that researchers could see clearly what their study was able to do; iteration was frequently necessary at this point. (d) It was difficult to form simple conclusions and recommendations beyond summaries of what was known in the field. (e) Dissemination activities were most often directed to other health service researchers, with some attention to NHS policy makers and research commissioners. RESULTS: UNDERTAKING METHODOLOGICAL RESEARCH: VIEWS OF RESEARCHERS (THE NEED FOR FLEXIBILITY): (1) Few researchers had amended their topic or methods once their research was under way, although some had made minor changes to their original plan, generally to refine the topic to fit the time or data available. (2) Changing a topic was seen as inappropriate unless checked with funders, but changes in research methods were viewed as reasonable because questions might be refined in the light of information gained or early thinking. RESULTS: UNDERTAKING METHODOLOGICAL RESEARCH: VIEWS OF RESEARCHERS (THE QUESTION OF BIAS): (1) Few researchers considered that this kind of research could be undertaken or presented in a wholly unbiased way because of the need to assess the research studied. (2) Objectivity was nonetheless seen as something that researchers should strive towards. Efforts to do so included presenting data clearly, separating findings from discussion, covering all points of view, setting out their own assumptions and values, and testing their ideas on others known to have differing views. (3) The formal peer-review process was not seen to have made a difference here, primarily because of the stage at which referees become involved. RESULTS: UNDERTAKING METHODOLOGICAL RESEARCH: VIEWS OF RESEARCHERS (PROJECT MANAGEMENT--TIMING AND TIME MANAGEMENT): (1) A majority of projects were completed within 3 months of their due date. Those studies completed roughly on time were considered to have efficient junior researchers and good project management, including clear deadlines for different stages of the research. (2) Some studies had severe problems of time management. Too much time tended to be spent on collecting and reading the literature and the writing stage was not always well planned. Referees' comments were also slow in coming. (ABSTRACT TRUNCATED)