Evaluating computerized health information systems: hardware, software and human ware: experiences from the Northern Province, South Africa

Despite enormous investment world-wide in computerized health information systems their overall benefits and costs have rarely been fully assessed. A major new initiative in South Africa provides the opportunity to evaluate the introduction of information technology from a global perspective and assess its impact on public health. The Northern Province is implementing a comprehensive integrated hospital information system (HIS) in all of its 42 hospitals. These include two mental health institutions, eight regional hospitals (two acting as a tertiary complex with teaching responsibilities) and 32 district hospitals. The overall goal of the HIS is to improve the efficiency and effectiveness of health (and welfare) services through the creation and use of information, for clinical, administrative and monitoring purposes. This multi-site implementation is being undertaken as a single project at a cost of R130 million (which represents 2.5 per cent of the health and welfare budget on an annual basis). The implementation process commenced on 1 September 1998 with the introduction of the system into Mankweng Hospital as the pilot site and is to be completed in the year 2001. An evaluation programme has been designed to maximize the likelihood of success of the implementation phase (formative evaluation) as well as providing an overall assessment of its benefits and costs (summative evaluation). The evaluation was designed as a form of health technology assessment; the system will have to prove its worth (in terms of cost-effectiveness) relative to other interventions. This is more extensive than the traditional form of technical assessment of hardware and software functionality, and moves into assessing the day-to-day utility of the system, the clinical and managerial environment in which it is situated (humanware), and ultimately its effects on the quality of patient care and public health. In keeping with new South African legislation the evaluation process sought to involve as many stakeholders as possible at the same time as creating a methodologically rigorous study that lived within realistic resource limits. The design chosen for the summative assessment was a randomized controlled trial (RCT) in which 24 district hospitals will receive the HIS either early or late. This is the first attempt to carry out an RCT evaluation of a multi-site implementation of an HIS in the world. Within this design the evaluation will utilize a range of qualitative and quantitative techniques over varying time scales, each addressing specific aims of the evaluation programme. In addition, it will attempt to provide an overview of the general impact on people and organizations of introducing high-technology solutions into a relatively unprepared environment. The study should help to stimulate an evaluation culture in the health and welfare services in the Northern Province as well as building the capacity to undertake such evaluations in the future.