A guide to the development, implementation and evaluation of

Cat. No. 9810223

NHMRC

National Health and Medical Research Council

A guide to the

development, implementation

and evaluation of

clinical practice guidelines

A guide to the development, implementation and evaluation of clinical practice guidelines

A guide to the

development, implementation

and evaluation of

clinical practice guidelines

Endorsed 16 November 1998

NHMRC

National Health and Medical Research Council

ISBN 1864960485

This work is copyright. Apart from any use as permitted under the Copyright Act 1968, no part

may be reproduced by any process without permission from AusInfo. Requests and enquiries

concerning reproduction and rights should be addressed to the Manager, Legislative Services,

AusInfo, GPO Box 1920, Canberra ACT 2601.

The strategic intent of the National Health and Medical Research Council (NHMRC) is to work with

others for the health of all Australians, by promoting informed debate on ethics and policy, providing

knowledge-based advice, fostering a high quality and internationally recognised research base, and

applying research rigour to health issues.

NHMRC documents are prepared by panels of experts drawn from appropriate Australian academic,

professional, community and government organisations. NHMRC is grateful to these people for the

excellent work they do on its behalf. This work is usually performed on an honorary basis and in

addition to their usual work commitments.

This document is sold through AusInfo Government Info Bookshops at a price which covers the

price which covers the cost of printing and distribution only. For publication purchases please

contact AusInfo on their toll-free number 132 447, or through their internet address:

http://www.ausinfo.gov.au/general/gen_hottobuy.htm

iii

CONTENTS

SUMMARY 1

Chapter 1 INTRODUCTION 9

1.1 Clinical practice guidelines 9

1.2 The evolution of guidelines in Australia 11

1.3 Endorsing externally developed guidelines 12

Chapter 2 THE GUIDING PRINCIPLES 13

2.1 Guideline development and evaluation 13

2.2 Using the best available evidence 14

2.2.1 The level and quality of evidence 15

2.2.2 The relevance of the evidence 16

2.2.3 The strength of the evidence 17

2.3 Using evidence in making recommendations

for treatment 17

2.4 A multidisciplinary approach that includes consumers 18

2.5 Flexibility and adaptability 19

2.6 Economic implications 19

2.7 Guideline dissemination and implementation 20

2.8 Evaluation 20

2.9 Revision 20

Chapter 3 GUIDELINE DEVELOPMENT 21

3.1 Determining the need for and scope of guidelines 21

3.2 Convene a multidisciplinary panel to oversee

the development of the guidelines 22

3.3 Define the purpose of and target audience for

the guidelines 23

3.4 Identify health outcomes 23

3.5 Review the scientific evidence 24

3.6 Formulate the guidelines 24

3.7 Formulate a dissemination and implementation strategy 25

CONTENTS

3.8 Formulate an evaluation and revision strategy 25

3.9 The guidelines themselves 25

3.10 Reporting on the guideline-development process 27

3.11 Assessing the guideline document 28

3.12 Consultation 28

Chapter 4 GUIDELINE DISSEMINATION AND

IMPLEMENTATION 31

4.1 Dissemination 33

4.1.1 Making the guidelines accessible 33

4.1.2 Publishing the guidelines 33

4.1.3 Informing the target audience of the

guidelines’ availability 34

4.2 Implementation 35

4.2.1 The use of opinion leaders and ‘champions’ 36

4.2.2 Endorsement by clinical groups 37

4.2.3 Practice visits from influential experts 37

4.2.4 Education of patients and patient-mediated intervention 37

4.2.5 Educational materials 38

4.2.6 Seminars and conferences 38

4.2.7 Reminder systems incorporated in clinicians’

daily work 38

4.2.8 Quality assurance and quality improvement 39

4.2.9 Local adaptation and incorporation 40

4.2.10 Incentives 41

4.2.11 The use of information technology 41

4.3 A systems approach 43

CONTENTS

Chapter 5 GUIDELINE EVALUATION AND REVISION 45

5.1 Evaluation 45

5.1.1 Evaluation of dissemination 46

5.1.2 Evaluation of changes in clinical practice

and health outcomes 46

5.1.3 Evaluation of the guidelines’ contribution

to changes in clinical practice and health outcomes 48

5.1.4 Evaluation of consumer guidelines 48

5.1.5 Economic evaluation 49

5.1.6 Reporting on the evaluation 49

5.2 Revision 50

Chapter 6 LEGAL CONSIDERATIONS 51

6.1 Liability of practitioners 51

6.2 Legal liability of guideline developers and

bodies supporting guideline development 52

6.3 Minimum legal requirements for guideline developers 52

Chapter 7 CONCLUSION 53

Appendix A Definitions 55

Appendix B Designation of levels of evidence 56

Appendix C Clinical practice guidelines flow chart 57

Appendix D Generic outcome measures 58

Appendix E Checklist for guideline development 59

Appendix F Process report 60

Appendix G Submissions received 62

Appendix H Economic appraisal information 64

Appendix I Implementation strategies that work 65

Appendix J National Health and Medical Research Council

standards for externally developed guidelines 66

Glossary 72

Bibliography 74

CONTENTS

Why these guidelines are needed

There has been a widespread move towards developing clinical practice

guidelines, which are designed to improve the quality of health care, to reduce

the use of unnecessary, ineffective or harmful interventions, and to facilitate the

treatment of patients with maximum chance of benefit, with minimum risk of

harm, and at an acceptable cost. Recent research has shown that clinical practice

guidelines can be effective in bringing about change and improving health

outcomes. But they are just one element of good medical decision making, which

also takes account of patients’ preferences and values, clinicians’ values and

experience, and the availability of resources.

There have been few widely accessible guides for groups seeking to develop

clinical practice guidelines. This document puts forward a method for developing

such guidelines in Australia.

Traditionally, guidelines have been based on the development of consensus

among experts, although this process has limitations and can lead to flawed

conclusions. There is now growing recognition that guidelines should be based,

where possible, on the systematic identification and synthesis of the best available

scientific evidence.

Clinical practice guidelines will be effective only if they are perceived to be

helpful and are actually used in clinical decision making. Many individuals and

groups found the earlier version of this document, published by the National

Health and Medical Research Council in 1995, very helpful. This revised edition

reflects concern that greater emphasis should be placed on guideline

implementation and evaluation.

Key principles for developing guidelines

There are nine basic principles for developing guidelines.

1. Processes for developing and evaluating clinical practice guidelines should

focus on outcomes. Outcome measures can range from survival rates to

quality-of-life attributes.

2. Clinical practice guidelines should be based on the best available evidence

and should include a statement about the strength of their recommendations.

Evidence can be graded according to its level, quality, relevance and strength.

The ‘level’ of evidence refers to the study design used to minimise bias: the

highest level involves a systematic review of randomised controlled clinical

trials. ‘Quality’ refers to the methods used to minimise bias in the design and

conduct of a study. ‘Relevance’ refers to the extent to which research findings

can be applied in other settings. The ‘strength’ of evidence relates to the

SUMMARY

Summary

2 A guide to the development, implementation and evaluation of clinical practice guidelines

magnitude and reliability of the treatment effect seen in clinical studies: strong

effects are more likely to be real and more likely to be clinically important.

Ideally, recommendations would be based on the highest level of evidence,

but this may be difficult to achieve in public health and social science

interventions.

3. The method used to synthesise the available evidence should be the strongest

applicable. Taking the evidence—of whatever level, quality, relevance or

strength—and turning it into a clinically useful recommendation depends on

the judgment, experience and good sense of the group developing the

guidelines. The fact of having evidence from a high-level study does not

automatically result in a good clinical recommendation.

4. The process of guideline development should be multidisciplinary and should

include consumers. If guidelines are to be relevant, those who are expected to

use them or to benefit from their use should play a part in their conception

and development. Involving a range of generalist and specialist clinicians,

allied health professionals, experts in methodology, and consumers will

improve the quality and continuity of care and will make it more likely that

the guidelines will be adopted.

5. Guidelines should be flexible and adaptable to varying local conditions.

They should include evidence relevant to different target populations and

geographic and clinical settings, take into account costs and constraints, and

make provision for accommodating the different values and preferences of

patients.

6. Guidelines should be developed with resource constraints in mind. They

should incorporate an economic appraisal, which may be helpful for choosing

between treatment options.

7. Guidelines are developed to be disseminated and implemented taking into

account their target audiences. They should also be disseminated in such a

way that practitioners and consumers become aware of them and use them.

8. The implementation and impact of guidelines should be evaluated.

9. Guidelines should be revised regularly.

Guideline development

When selecting guideline topics, there must be a clear problem or concern that

would be redressed if guidelines were developed. A multidisciplinary panel

consisting of representatives of all relevant groups should be convened. It should

clarify the purpose of the guidelines—such as specifying what conditions and

clinical problems they will cover—and identify the desired health outcomes.

Rigorous and systematic review of the scientific evidence is essential.

In formulating the guidelines, the panel should do the following:

• document the purpose for which the guidelines were developed;

• describe the natural history of the disease or condition in question and the

various treatments that are possible;

• identify situations where any recommendations might not apply;

• detail the probable outcomes;

• ensure that the guidelines are comprehensive and flexible;

• describe the support services that may be required for each potential

treatment;

• include information, for consumers and clinicians, on any special clinical

training or equipment that is needed;

• compare the costs associated with the various options;

• provide a statement of the scientific basis on which the guidelines were

developed and clearly specify the level, quality, relevance and strength of the

evidence on which the recommendations are based;

• document the uncertainty associated with any conclusions; and

• document the economic appraisals used in formulating the guidelines.

If consensus-based recommendations are to be issued, they should acknowledge

the desirability of developing evidence-based recommendations. Where non–

consensus based statements are issued, there should be clear reference to each

of the schools of thought and consumers should be made aware of the lack of

consensus.

Different versions of the guidelines should be developed for different audiences

—consumers, general practitioners, specialist nurses, and so on.

During the guideline development phase the panel should also develop a plan

for disseminating and implementing the guidelines and a plan to ensure that the

guidelines are evaluated properly and revised as necessary.

Once drafted, the guidelines document should be assessed to determine whether it

conforms to the principles outlined in this guide. The draft should be referred for

consultation to a wide range of interested parties—practising clinicians, clinical

colleges, allied health and professional organisations, consumer groups,

Commonwealth, State and Territory and local health authorities, industry groups,

and so on.

Summary

4 A guide to the development, implementation and evaluation of clinical practice guidelines

Disseminating and implementing the guidelines

The multidisciplinary panel should identify any barriers to acceptance and

implementation of the guidelines and work with members of target groups to

develop ways of overcoming these barriers.

Guidelines should be presented in a format and style suitable for their target

audience. Their cost should not constitute a barrier to access.

Strategies for dissemination and implementation will probably depend on the

nature of the guidelines and the target audience. There are a number of possible

strategies:

• compiling short summaries for use in professional or general publications or

brochures, as posters, on the Internet, as audio or video tapes, or on disk;

• involving potential users in developing the guidelines;

• using the media, professional journals and the publications of other groups,

such as consumer organisations, divisions of general practice, hospitals, area

health services and universities;

• using the communication links developed by clinical colleges and other

groups;

• asking respected clinical leaders to promote the guidelines;

• providing economic incentives;

• using the education processes of appropriate colleges and other groups;

• incorporating the guidelines in routine procedures, such as quality assurance,

within relevant organisations;

• using information technology;

• arranging for a credible health provider to visit practitioners in the clinical

setting;

• piloting the draft guidelines in order to facilitate their assessment;

• offering feedback on compliance with the guidelines;

• discussing the guidelines at conferences, seminars and other professional

meetings; and

• using the services of a communications professional.

Dissemination of the guidelines alone will not change practitioners’ behaviour.

Doctors and other clinicians are most likely to change their behaviour if they

themselves are involved in the change process and if that process involves

interventions that directly affect consultation between patient and practitioner.

A variety of approaches have been shown to change clinicians’ behaviour or

health outcomes, or both:

• using opinion leaders and clinical ‘champions’ in the media and

for marketing;

• endorsement by key clinical groups;

• practice visits from influential experts;

• educating patients;

• provision of educational materials;

• seminars and conferences;

• reminder systems incorporated in clinicians’ daily work;

• continuing quality assurance and data feedback;

• local adaptation and incorporation;

• local involvement in evaluation; and

• incentives.

There is emerging evidence that the new information technologies can help in

disseminating and implementing guidelines through improving databases, Internet

publication, computerised medical records, more interactive styles of learning,

and computerised prompting and decision-support systems. It is also easier to

update electronic guidelines.

Evaluation and revision

Evaluation of the guidelines should consider the following:

• how well were they disseminated? For example, how many copies were

mailed out?;

• is the general trend in clinical practice moving towards the guideline

recommendations?;

• have the guidelines contributed to any specific changes in clinical practice?

For example, compare clinical practice in areas where the guidelines have

been heavily promoted with practice in areas where they have not been

promoted;

• how have the guidelines affected consumers’ knowledge and understanding?;

and

• have health outcomes changed?

Summary

6 A guide to the development, implementation and evaluation of clinical practice guidelines

The guideline process should also be subjected to an economic evaluation.

A cost-effectiveness analysis would assess all costs incurred in providing the

guidelines and their impact.

In addition, the various sets of guidelines that are produced (for general

practitioners, consumers, and so on) should be evaluated for their relevance,

ease of access, clarity, how much information they contain, and general user

satisfaction.

A date should be set for revision of the guidelines. The National Health and

Medical Research Council recommends that this occur every three to five years

and more often where the subject matter or circumstances are prone to rapid

change.

Legal considerations

Liability of practitioners

Many practitioners are concerned about their potential legal liability if a patient

does not receive treatment as specified in clinical practice guidelines. It is

certainly possible that guidelines could be produced as evidence of what

constitutes reasonable conduct by a medical practitioner. The National Health

and Medical Research Council’s Health Advisory Committee considers that

practitioners who use guidelines will be afforded a measure of protection.

The case of Rogers v Whitaker (1992) 175 CLR 479 makes it clear that doctors

should give information about the risks of any treatment, especially risks that may

influence the patient’s decision. Patients should be provided with as much

information as they seek, and in a form that is appropriate to their culture and

level of education. They should be encouraged to make their own decisions.

Liability of guideline developers and bodies supporting guideline

development

If guidelines purport to be a definitive statement of the correct procedure, there

would be a greater risk of liability than if they were clearly stated to be a general

guide subject to the practitioner’s expert judgment in each case.

Minimum legal requirements for guideline developers

The potential for guidelines to be used as evidence in court depends on the

process used to develop them, the extent to which they are evidence-based, the

degree of consensus about them, and whether they are up to date.

Guideline developers are unlikely to be held liable for any negative consequences

of the implementation of guidelines. In general, guidelines should be summaries

of the evidence, should have an expiry date, should not be unduly prescriptive,

and should acknowledge areas where there is disagreement. An independent

review of the guideline development process is recommended.

Conclusions

The development, dissemination, implementation, evaluation and revision

processes described in this document are applicable to a wide range of clinical

interventions and disciplines and could also be applied to protocols relating to

the use of technology and pharmaceuticals.

Where possible, it is preferable, to use existing networks, facilities and

publications to disseminate, implement and evaluate guidelines, rather than

developing new processes. A systems approach is needed, which builds

guideline development, dissemination, implementation and evaluation into the

routine processes of care. Organisations such as hospitals, health care regions,

and divisions of general practice should establish an approach that integrates

guidelines into their local health care delivery processes. Such an approach

would involve sensible use of resources, developing sustainable infrastructure,

integrating available resources and strategies, and adapting to local needs.

The best approach to making sure guidelines work would appear to be a mix of

strategies that are suitable for local conditions and are developed in concert with

local clinicians, consumers and managers.

This document will be revised. For those seeking further information, a series of

specialised ‘tool kits’, currently being developed, will be available in mid-1999.

Users and reviewers of these guidelines are encouraged to contact the National

Health and Medical Research Council if they have comments—negative or

positive—to make. These comments will inform the revision process.

Users can contact the National Health and Medical Research Council by writing to:

Health Advisory Unit

Office of the National Health and Medical Research Council

Department of Health and Aged Care

GPO Box 9848 (MDP 50)

CANBERRA ACT 2601

Summary

8 A guide to the development, implementation and evaluation of clinical practice guidelines

Blank facing page to Chapter 1

The past decade has been marked by unprecedented interest in clinical

practice guidelines and the processes by which they can best be developed and

implemented. This worldwide interest has been prompted by concern about

unjustifiable variations in clinical practice for the same condition, the increasing

availability of new treatments and technologies, uncertainty about the

effectiveness of many interventions in improving people’s health, and a desire to

make the best use of available health resources.

Guidelines are being designed to improve the quality of health care and decrease

the use of unnecessary, ineffective or harmful interventions. In an era of

evidence-based medicine (Sackett et al. 1996), guidelines are becoming one of

the critical links between the best available evidence and good clinical practice.

Guidelines constitute one element of a systems approach to quality health care.

Despite the acknowledged need, there are few widely accessible, comprehensive

guides for groups seeking to develop clinical practice guidelines. This document

puts forward a method for developing clinical practice guidelines in Australia,

based on the best available models worldwide. The method has been trialed in

Australia in recent years and modified in the light of that experience; it is

applicable to a variety of conditions and procedures. In this document the term

‘clinical’ takes in all health care providers.

1.1 Clinical practice guidelines

Clinical practice guidelines are ‘systematically developed statements to assist

practitioner and patient decisions about appropriate health care for specific

clinical circumstances’ (Field & Lohr 1990). There is a move towards developing

such statements to assist clinicians in the management of specific conditions.

The procedures used to develop the statements are increasingly based on a

thorough evaluation of the evidence, including, when appropriate, meta-analysis

of published research studies on the outcomes of various treatment options,

rather than the consensus of expert panels. The statements are intended to be

‘a distillation of current evidence and opinion on best practice’ (Clover et al. 1995;

Grimshaw & Russell 1993).

Clinical practice guidelines are often referred to as algorithms, clinical pathways,

protocols and practice policies, although these differ from clinical practice

guidelines in that they are often much more prescriptive and not always based

on evidence.

CHAPTER 1

INTRODUCTION

Introduction

10 A guide to the development, implementation and evaluation of clinical practice guidelines

Clinical practice guidelines are one component of good medical decision-making,

which takes into account patients’ preferences and values, clinicians’ values and

experience, and the available resources. The guidelines’ main purpose is to

achieve better health outcomes by improving the practice of health professionals

and providing consumers with better information about treatment options.

Guidelines can inform consumers about risk factors and how to avert them; they

can be used to broaden the education of practitioners and the community, thus

contributing to quality assurance processes; and they can assist in the resolution

of legal disputes and ethical dilemmas.

Research has shown that clinical practice guidelines can be an effective means of

changing the process of health care and improving health outcomes (Grimshaw &

Russell 1993, EHCB 1994). A systematic review of many studies of guidelines’

effectiveness revealed strong evidence that they can change clinical practice.

Moreover, of those studies that assessed changes in consumer health, a majority

reported an improvement in health outcomes (EHCB 1994).

Guidelines vary, however, in the extent to which they produce improved health

outcomes. There are two main reasons for this. First, acceptance of a guideline is

likely to depend on the quality of the evidence on which it is based. Second, the

extent to which the potential health gain is realised from adherence to a guideline

will depend on how effectively the guideline is implemented.

Traditionally, guidelines have been based on consensus among experts. But this

method has its limitations. Expert opinion does not always reflect the state of

current medical knowledge (Antman et al. 1992). And, even where guidelines are

supported by literature surveys, if the medical literature has been analysed in an

unsystematic way biased conclusions can result (Woolf 1992 Vol 152; Mulrow

1994; EHCB 1994). In the past this has led to unnecessary delays in the

recommendation of effective interventions and delays in the withdrawal of

ineffective or harmful treatments (Antman et al. 1992; Advisory Group on Health

Technology Assessment 1992).

It is now acknowledged that guideline recommendations should be based on

systematic identification and synthesis of the best available scientific evidence.

Given the extensive research activity in some health care areas, identifying and

synthesising the available evidence can be a major undertaking.

The formulation and drafting of guidelines is only one part of the guideline

development process. Guidelines will be effective only if they are perceived to be

useful and are actually used in clinical decision making. It is therefore important

to ensure that clinicians are aware of the guidelines and that the guidelines are

incorporated in clinical practice. At present little is known about the relative

effectiveness of different dissemination and implementation strategies (Grimshaw

& Russell 1994), but the available evidence does suggest that implementation

strategies are most likely to succeed if they are relevant to the people the

guidelines are targeting and if they are integrated into the health care delivery

process (EHCB 1994).

Clinical practice guidelines should be linked to effective evaluation plans. This

involves documenting the beneficial and adverse consequences of different

interventions when applied in normal clinical practice and how their application

is viewed by the consumer.

Rather than being treated as a stand-alone item, clinical practice guidelines will

be most effective when incorporated fully in the health care system and used

alongside existing quality assurance activities such as continuing medical

education, accreditation, audit and certification.

Guidelines will also be most effective when they are part of a system in which

good data collection on outcomes is the norm and where the data collected are

used to inform the continuing development of the guidelines and their

dissemination and implementation. The challenge is to develop a systems

approach that can support guideline development, implementation and evaluation

as part of mainstream health care delivery.

Finally, it must be remembered that there are no ‘magic bullets’ or simple answers

when it comes to developing and implementing guidelines (Oxman et al. 1995).

The cycle of development, implementation, evaluation, and revision in the light of

new scientific evidence and consumer feedback will be central to the success of

future clinical guidelines.

1.2 The evolution of guidelines in Australia

The National Health and Medical Research Council has been piloting and funding

the process of clinical guideline development in areas such as early breast cancer,

coronary heart disease, unstable angina, stroke prevention, preterm birth,

depression in young people, uncomplicated lower urinary tract symptoms in men,

and diabetic retinopathy.

Many individuals and groups found the earlier version of this document,

produced in 1995, very helpful in developing guidelines. Feedback collected by

the National Health and Medical Research Council, from guideline developers,

shows overwhelming support for the idea of evidence-based guidelines. It also

shows general agreement about the inclusion of ‘decision trees’ in guidelines, as

well as the need for some kind of systems approach to make the guidelines work

effectively.

Introduction

12 A guide to the development, implementation and evaluation of clinical practice guidelines

There is, however, a strong feeling that much more attention should be given to

implementation and evaluation once guidelines have been developed. Many of

those involved in producing guidelines have become frustrated by the lack of

implementation.

Further, health care professionals’ acceptance of clinical practice guidelines has to

some degree been marred by lingering concern that the guidelines represent

‘cookbook’ medicine.

1.3 Endorsing externally developed guidelines

Many of the newly developed evidence-based guidelines have been produced

under the auspices of the National Health and Medical Research Council. It is

anticipated, however, that in the future many outside bodies will seek to produce

their own guidelines. Regulating the quality of such critically important processes

and endorsing externally developed guidelines pose challenges that are now

being tackled.

This chapter describes the nine guiding principles underlying the guideline

development process. They are as follows:

• processes for guideline development and evaluation should be outcome

focused;

• guidelines should be based on the best available evidence and should include

a statement about the strength of recommendations;

• the method used to synthesise the available evidence should be the strongest

applicable;

• the process of guideline development should be multidisciplinary and should

include consumers;

• guidelines should be flexible and capable of adapting to varying local

conditions;

• guidelines should be developed with resource constraints in mind;

• guidelines should be developed, disseminated and implemented taking into

account their target audiences;

• the validity and usefulness of the guidelines should be evaluated; and

• guidelines should be revised regularly.

2.1 Guideline development and evaluation

The process for guideline development should be aimed at identifying

interventions that will ensure the best possible health outcomes.

A health outcome has been defined as ‘a change in the health of an individual, a

group of people or population which is attributable to an intervention or a series

of interventions’ (AHMAC 1993). Donabedian describes outcomes as ‘what is

accomplished for patients’ and points out that they reflect the ‘contribution of all

those who provide care’ as well as the ‘appropriateness of the choice of a strategy

of care’ (1992b).

Outcome measures can range from survival rates to quality-of-life attributes.

Outcomes can be positive or negative and may differ according to population

group—for example, socio-economic group, gender, or current health or risk

factor status.

To date much of the evaluation of health care has centred around the process of

health care (whether clinicians conform to recommended practices), rather than

the outcomes (whether the recommended practices produce a change in health).

Whilst the ultimate goal of the guideline-evaluation process should be to establish

the degree to which the guidelines are effective in producing the health outcomes

CHAPTER 2

THE GUIDING PRINCIPLES

The guiding principles

14 A guide to the development, implementation and evaluation of clinical practice guidelines

sought, this may not be practical because of the difficulty of attributing outcomes

to the guidelines.

2.2 Using the best available evidence

The purpose of clinical practice guidelines is to encourage treatment that offers

individual patients maximum likelihood of benefit and minimum harm and is

acceptable in terms of cost. Recommendations contained in guidelines should be

based on the best possible evidence of the link between the intervention and the

clinical outcomes of interest.

The evidence on which a recommendation is based can be graded according to

level, quality, relevance and strength. Appendix A provides definitions of these

criteria, as used by the Health Advisory Committee of the National Health and

Medical Research Council.

Ideally, recommendations should be based on the highest level of evidence,

preferably a systematic review of high-quality randomised controlled clinical trials

that measure relevant outcomes and demonstrate a strong, clinically important,

beneficial effect of the intervention. It is important, though, to recognise that this

ideal may be difficult to attain in the case of public health and social science

interventions: these important areas of health care should not be disadvantaged

by the rigid application of a ‘hierarchy’ of evidence.

In many cases it may not be possible, or feasible, to evaluate a large-scale public

health intervention using a randomised controlled trial. Other forms of evidence

—such as well-designed controlled studies and time series analyses—may be the

most appropriate and feasible method. Although there is currently no agreed

separate grading for assessing the level of evidence in relation to public health

interventions, the primary objective is to strive for evidence derived from a study

design that is the most practical and feasible available in order to maximally

control for potential bias. In addition, evidence derived from a systematic review

of all the available studies that meet this criterion is obviously preferable to

evidence from a single study.

Until an agreed rating scale is developed to assess levels of evidence associated

with public health interventions, guideline developers are advised to use the

levels of evidence referred to in this document but to recognise that much of the

evidence currently available in relation to public health interventions will be

level III—see Appendix B. In this context it is important to note that the evidence

from public health interventions is often supported by strong biological data.

Where appropriate, guideline developers should include in the text a

statement on the level, quality, relevance and strength of the evidence and

recommendations made should reflect this statement.

2.2.1 The level and quality of evidence

The level of evidence and the quality of evidence need to be considered together.

‘Level of evidence’ refers to the study design used by investigators to minimise

bias (see Appendix B). Level I, the highest level, is generally accorded to

randomised clinical trials. A methodologically poor randomised trial (level II) may,

however, provide a weaker basis for a recommendation than a high-quality

observational study without randomisation, which can provide level III-2

evidence.

‘Quality of evidence’ refers to the methods used by investigators to minimise bias

in study design and in the conduct of a study. The types of bias and their possible

effects depend on the study type. There are now fairly well established criteria for

assessing the quality of randomised trials; they concern the degree to which

allocation to treatment groups is concealed from investigator and subject, whether

the study is double blind, and the completeness of follow-up of subjects.

Methods of assessing the quality of non-randomised studies (levels III-2 and III-3)

are less well established and the factors will probably vary depending on the

study type. In general, factors that are likely to influence the estimate of the effect

size are the methods used to select subjects for the trial, the comparability of the

treatment and control (if there is one) groups, the methods of measuring

outcomes, and the completeness of the follow-up.

The quality of systematic reviews also needs to be considered if they are being

used as the basis of guideline development. Standard methods for conducting and

reporting systematic reviews have been published (see Greenhalgh 1997).

Lack of empirical evidence

No formal level of evidence is attached to expert opinion, the findings of expert

working parties, or anecdotal information. In the absence of empirical evidence,

however, or where there is only poor-quality evidence, guidelines may in some

instances contain recommendations based on findings outside the levels-of-

evidence hierarchy. Such recommendations should be derived using a consensus

approach.

The guiding principles

16 A guide to the development, implementation and evaluation of clinical practice guidelines

If a consensus approach is not possible non-consensus practice statements can be

issued, but there should be clear reference to all schools of thought and

consumers should be made aware of the lack of consensus (see Appendix C).

2.2.2 The relevance of the evidence

‘Relevance of the evidence’ refers to the study question’s similarity to the clinical

question and the extent to which the findings from the study can be applied in

other clinical settings to different patients. The types of outcomes that are

measured and reported in a clinical trial determine whether all of the possible

benefits and harms associated with a treatment can be assessed properly—among

the factors that are important are the nature of the outcomes that have been

measured in the trial and the period over which the measurements have been

made.

It may be useful to consider outcomes as being of one type or a combination of

two or three types: ‘surrogate’ outcomes, ‘clinical’ outcomes, and ‘patient-relevant’

outcomes. In general, the types of outcomes that clinical trials measure and report

are either surrogate or clinical outcomes; these may or may not be of importance

to the patient.

A surrogate outcome is commonly a physiological variable; for example, serum

cholesterol concentration or blood pressure. There is a statistical association

between the surrogate outcome and the clinical outcome of interest; for example,

bone mineral density and fracture or measures of HIV viral load and progression

of AIDS. There is a biological and pathophysiological basis for believing that the

surrogate outcome is a primary determinant of the clinical outcome in the disease

being studied; for example, glycosylated haemoglobin measurements and diabetic

complications.

If the trials that form the basis of the recommendations are restricted to surrogate

outcomes it is important to determine that there is a relationship between these

surrogate outcomes and the clinical outcome of interest.

Clinical outcomes tend to be defined on the basis of the disease being studied.

In many cases the outcome that appears to the clinician to be of primary concern,

and that can be measured, is chosen; for example, survival in cancer, vertebral

fracture in osteoporosis, peptic ulcer healing and relapse, walking distance in

angina or claudication, or microbiological ‘cure’ of infection. But such an

approach does not necessarily capture all of the relevant outcomes from the

patient’s perspective: other factors that are often more difficult to measure may

be more relevant, in particular factors relating to improved quality of life.

One of the difficulties in measuring patient-relevant outcomes is that they may be

‘composite’ outcomes and include items that are not readily amenable to objective

measurement. They are likely to include a measure of a clinical effect, adverse

effects, and change in quality of life and may need to be individualised for each

treatment. In this situation the following types of outcomes should be considered:

• all-cause mortality;

• cause-specific mortality;

• changes in morbidity, subdivided according to type—for example, hospital

admission or nursing home requirements;

• side-effects of treatment—including adverse reactions to drug therapies

—and co-interventions that may be necessitated by the primary treatment; and

• disease-specific outcomes—including disease-specific quality-of-life measures.

Emphasis should be given to studies that measure clinical or patient-relevant

outcomes. Guideline developers should seek the advice of experts and patient-

support groups in determining what the most appropriate outcomes are. Many

clinical trials and studies do not consider all the outcomes, beneficial and harmful,

that are relevant to patients with a particular condition and to those who care for

them. Clinical endpoints—such as a small change in blood cholesterol in a ‘low-

risk’ patient—may not be as relevant as the side-effects of a particular treatment.

Equally, a condition that impairs the social functioning of a person might be very

effectively alleviated by a new intervention. Formal quality-of-life measures such

as the SF-36 often do not capture the information that is of real importance in

assessing the potential impact or value of a new technology.

2.2.3 The strength of the evidence

The strength of the evidence depends on the magnitude of the treatment effect

seen in the clinical studies. It also depends on how confident we are of the

observed effect—in other words, the size of the confidence interval—and the

extent to which the findings have been reproduced across a series of studies.

Strength of evidence is important for two main reasons. Strong effects are less

likely than weak effects to be the result of bias in the studies: they are more likely

to be real. And strong effects are more likely to be clinically important.

2.3 Using evidence in making recommendations for treatment

Evidence is necessary, but not sufficient, when making recommendations for

treatment. Taking the evidence—of whatever level, quality, relevance or

strength—and turning it into a clinically useful recommendation depends on the

judgment, experience and good sense of the guideline developers.

The guiding principles

18 A guide to the development, implementation and evaluation of clinical practice guidelines

Having evidence from a high-level study such as a randomised controlled trial

does not immediately mean that a good clinical recommendation will result.

Suppose we had one well-conducted randomised controlled trial on which to

base our recommendations: if the effects of the intervention were small and the

outcome measures were surrogates, rather than clinical outcomes, the evidence

would be relatively unhelpful.

In assessing the likely ratio of benefit to harm it is often helpful to produce a

simple ‘balance sheet’ in an attempt to estimate, for a ‘typical’ group of patients,

the number of subjects who will experience benefit or harm. All of the known

beneficial and harmful effects of the treatment should be included in the balance

sheet. In order to assess the applicability of the guideline advice to other groups

of patients, an attempt should be made to create a new balance sheet with revised

estimates of the number of beneficial and harmful effects.

For instance, if an effective preventive treatment (such as treatment to lower

cholesterol or the use of warfarin for non-rheumatic atrial fibrillation) is being

applied to a population at low risk of the outcome of interest, the estimated

number of benefits may not exceed the estimated number of harms by a

satisfactory margin. The guideline development group must make a judgment

about how wide this margin should be in order to justify a recommendation for

treatment.

Other factors warranting consideration when making recommendations are the

cost-effectiveness of the interventions, the financial implications for the health

budget of full implementation of the recommendations, and any inequity that may

arise if only certain sections of the community can afford to pay for the

interventions recommended.

2.4 A multidisciplinary approach that includes consumers

If guidelines are to be relevant, those who are expected to use them or to benefit

from their use should play a part in their conception and development.

The involvement of a range of specialist and generalist clinicians, allied health

professionals, experts in methodology (such as epidemiologists and health

economists), and consumers will improve the quality and continuity of care and

increase the likelihood of the guidelines’ adoption. These groups can make

significant contributions to the identification of relevant outcomes, values, target

audiences and guideline formats for different users. They can also play a critical

role in identifying potential obstacles to guideline implementation and in the

formulation of implementation strategies.

2.5 Flexibility and adaptability

Guidelines should provide a clear statement of the evidence but allow for

flexibility and adaptability in implementation. Thus, they should:

• provide evidence relevant to different target populations, which may vary in

terms of age, gender, ethnicity, diagnosis, disease severity, co-morbidity and

social support;

• provide evidence relevant to different geographic and clinical settings;

• take into account resource benefits, costs and constraints; and

• offer a means of accommodating different consumer values and preferences.

2.6 Economic implications

Guideline developers should be mindful of the resources required for

development and implementation of guidelines. Further, in adapting guidelines to

accommodate local conditions account should be taken of the following:

• limited treatment options;

• variations in the available equipment;

• variations in the training, experience and skills of existing and incoming

clinicians; and

• variations in the availability of staff—in terms of numbers and professions.

It is important that an economic appraisal be incorporated in guidelines. In many

circumstances this may be useful in guiding clinical decisions about treatment

options. There are three types of economic appraisal comparing two

interventions, or one intervention against a placebo:

• a cost-minimisation analysis, in which the direct costs of the proposed

intervention are compared, assuming equal efficacy but differences in

side-effects, or on the basis of the use of different patterns of care that

generate different costs;

• a cost-effectiveness analysis, in which the relative effectiveness (measured,

for example, by a reduction in blood pressure or days of disability averted)

and the relative direct costs of two interventions are compared using a ratio of

marginal cost to marginal effectiveness (for example, cost per life-year gained

or cost per day or disability averted); and

• a cost-utility analysis, in which the effectiveness measure derived in the cost-

effectiveness analysis is weighted by the recipient’s quality of life resulting

from each intervention (thus producing a measure such as marginal cost per

quality-adjusted year of life gained).

The guiding principles

20 A guide to the development, implementation and evaluation of clinical practice guidelines

2.7 Guideline dissemination and implementation

Guidelines are developed to be disseminated and implemented in such a way

that practitioners and consumers become aware of them and use them. This

will involve identifying potential obstacles to implementation and tailoring

implementation strategies to the particular contexts in which the guidelines are

to be introduced.

2.8 Evaluation

It is important to evaluate the implementation process to determine the extent to

which the guidelines affected practitioners’ knowledge and behaviour and what,

if any, factors contributed to non-compliance with the guidelines. These results

might inform the development of more effective implementation strategies.

It is also important to establish whether the introduction of the guidelines

produced the anticipated health outcomes. Thus, once the guidelines are

implemented, health outcomes should be routinely monitored wherever practical.

Outcomes associated with the introduction of evidence-based guidelines must be

evaluated since the efficacy of treatments delivered in strictly controlled trials may

not always be realised in day-to-day practice.

2.9 Revision

Because guidelines need to be based on the best available evidence, they should

be reviewed regularly and modified where necessary to take into account new

research, new technologies, and the results of evaluation of guideline outcomes.

This chapter describes the general steps involved in developing guidelines. The

steps are consistent with the principles discussed in Chapter 2. The process of

developing guidelines should run in tandem with formulating a strategy for their

dissemination and implementation and a plan for their evaluation and revision.

Chapter 4 deals with dissemination and implementation and Chapter 5 deals with

evaluation and revision.

It is important that guideline developers consider, at the beginning of the

guideline development process, what data will be needed. Data collection,

storage and retrieval must have due regard for principles of privacy and

confidentiality and should take into account matters associated with data

ownership.

It should be noted that the sequence of steps detailed in this chapter is intended

as a guide rather than a prescription for developing guidelines. Context may

require that the steps be taken in an order that differs from what is described

here. In addition, some early steps may be revisited at later stages in the process.

Appendix C illustrates the process of developing, disseminating, implementing,

evaluating and revising clinical practice guidelines.

3.1 Determining the need for and scope of guidelines

The body responsible for developing the guidelines should specify the problem

or objective and define what is involved in general terms. When selecting topics

there must be a clear problem that may well be resolved by the establishment and

dissemination of guidelines on what is the most appropriate practice. The

problem could relate to the extent of the health burden, cost, variations in

practice, or the availability of evidence.

Clinical practice guidelines are appropriate only if the problem or objective is

related to clinical decision making or decisions about the organisation of health

services. Guidelines may be developed, for instance, in response to an identified

variation in treatment among practitioners for the same condition. If this variation

is caused by lack of knowledge or information, the development of evidence-

based guidelines is an appropriate response. But variations in clinical practice

may be based on patients’ needs and sound evidence: such ‘legitimate variation’

can result from variations in morbidity rates and variations in consumer

preferences for different outcomes (Anderson & Mooney 1990, McPherson 1990).

Guidelines may not be necessary in such situations. Nor are they an appropriate

response to variations caused by unreliable data sources and random variations in

data. Furthermore, guidelines may not be needed if practice variations are a

consequence solely of resources or supply constraints (beds, technology, facilities,

specialists, and so on) or if clinical decisions are based on sound evidence.

CHAPTER 3

GUIDELINE DEVELOPMENT

Guideline development

22 A guide to the development, implementation and evaluation of clinical practice guidelines

Previously developed guidelines for redressing the same problem or a related one

should be assessed. If those guidelines are applicable to the current circumstance

or can be adapted, new guidelines may not be required. It is important, however,

to assess the previously developed guidelines according to the principles

discussed in this document, particularly in terms of the quality of the evidence on

which they are based. If the old guidelines have been implemented, establish

whether they were effective. Although they may be considered acceptable, it may

still be necessary to take steps to evaluate their effectiveness in clinical practice.

3.2 Convene a multidisciplinary panel to oversee the development

of the guidelines

The overseeing panel should consist of representatives of all relevant groups. The

panel’s precise composition will depend on the nature of the guidelines but

should be designed to encourage the expression of diverse interests. In general,

consideration should be given to involving the following groups, as appropriate:

• clinicians from all disciplines with relevant specialist expertise;

• clinicians with general expertise;

• other relevant health professionals;

• representatives of consumer groups;

• experts in research methods relevant to guideline development;

• health economists;

• public policy analysts;

• other relevant experts;

• industry representatives;

• bioethicists; and

• representatives of regulatory agencies.

To ensure that decision making takes full account of the consumer perspective,

the panel may need to establish a separate consumer focus group to identify all

the relevant consumer concerns.

Workshops for members of guideline development working parties may be

beneficial in bringing together multidisciplinary groups as well as consumers to

consider the need for clinical practice guidelines and the methods to be used in

their preparation, dissemination, implementation, evaluation and revision.

All members of guideline development groups should make declarations of their

interests available to the National Health and Medical Research Council or other

appropriate bodies. Both personal and non-personal interests should be declared;

for example, personal shares in companies, consultancies to companies, corporate

support for specific research, or general departmental activity (SIGN 1995).

3.3 Define the purpose of and target audience for the guidelines

Before proceeding, the panel should clarify the purpose of and target audience

for the guidelines. This will involve a careful specification of the following:

• the conditions and clinical problems that are at issue;

• the type of care providers for whom the guidelines are intended;

• the type of consumers for whom the guidelines are intended;

• a description of consumers not covered by the guidelines (where it might

otherwise be assumed that the guidelines are relevant to these consumers);

• the types of settings in which the guidelines will be employed; and

• the interventions to be evaluated.

The panel should also determine how many versions of the guidelines are

required. As a minimum requirement, separate consumer and practitioner

documents should be produced, but the panel will also need to determine

whether different documents are required for different care providers in different

clinical settings and for different consumers.

3.4 Identify health outcomes

The purpose of clinical practice guidelines is to improve health outcomes and to

encourage the appropriate use of resources. The appropriate outcome measures

will differ depending on what is under consideration. There is, however, a clear

difference between the following categories of measures:

• immediate or short-term measures such as mortality, morbidity, and treatment

complications;

• rates of relapse, complication, late morbidity and re-admission;

• return to work and physical and social functioning; and

• other measures such as quality of life, general health status and patient

satisfaction.

Guideline development

24 A guide to the development, implementation and evaluation of clinical practice guidelines

Each of these measures should be taken into account. Some measures will be

clinically defined and some will be socially defined and, again, both types should

be included. Appendix D outlines generic outcome measures that can be applied

to specific conditions and interventions.

3.5 Review the scientific evidence

If possible, a systematic literature review should be undertaken to establish the

benefits and harms of the possible interventions.

Methods for reviewing and evaluating the literature range from highly formal,

quantitative information syntheses (such as meta-analysis of randomised

controlled trials) to subjective synopses of observational data. The more formal

the review, the more credible and valid the resulting guidelines. The panel should

select the most rigorous and systematic review methods practicable according to

the approach outlined in Chapter 2 and the methods described in The Cochrane

Collaboration Handbook (Sackett 1994).

The guidelines should clearly state the methods used to review the literature.

There should be rigorous discussion of and debate about the literature, and any

gaps in it should be identified.

3.6 Formulate the guidelines

The guidelines should be constructed on the basis of the best available scientific

evidence of the probable outcomes associated with different clinical interventions.

The panel should consider the following:

• the natural history of the disease or condition in question, if appropriate;

• the probable outcomes of each intervention, taking into account the strength

of evidence associated with each, with preference given to empirical evidence

over expert judgment;

• the balance of benefits against risks;

• a comparison between the outcomes for alternative interventions, including

conservative or expectant management where appropriate, for the disease or

condition in question; and

• an economic appraisal of the best investment for the best health outcomes

(see Appendix E).

3.7 Formulate a dissemination and implementation strategy

During the guideline development phase the panel should also develop a plan for

disseminating and implementing the guidelines. Ideally, this plan will target all

potential users and define and recommend processes that will encourage them to

adopt and implement the guidelines. Such a plan may be developed in

consultation with the wider group of interested parties. Chapter 4 provides

detailed information about dissemination and implementation.

3.8 Formulate an evaluation and revision strategy

The panel should develop a strategy designed to ensure that the guidelines are

evaluated properly and revised when necessary.

This will involve specifying both short-term and long-term frameworks for

evaluation and identifying who will conduct the studies. The evaluation should

assess how effectively information has been disseminated to patients consenting

to the treatment in question as well as to practitioners providing the treatment.

It may also include a consideration of alternative evaluation procedures.

Most importantly, the evaluation will assess whether implementation of the

guidelines has led to improvements in health outcomes. Chapter 5 provides

detailed information about evaluation and revision.

3.9 The guidelines themselves

Once the dissemination and implementation and evaluation and revision strategies

have been developed, the panel should proceed to produce the guidelines.

The guidelines should be presented as clearly and concisely as possible, bearing

in mind that different formats may be appropriate for different types of guidelines

or different types of users. Guidelines may be presented as free text, as flow

charts or in any other format that facilitates comprehension. Abbreviations and

symbols should be consistent and easy to follow. Important terms and others that

might be misinterpreted should be defined.

The guideline document’s content will vary according to its specific purpose

—for example, whether it relates to diagnosis, choosing treatment options,

implementation of specific treatment, or managing an extended process of care.

In some cases an additional document with more explanation or in a different

format may be required for use by consumers. Consideration might also be given

to producing different guideline documents for specialist and non-specialist health

professionals and for different health care settings.

Guideline development

26 A guide to the development, implementation and evaluation of clinical practice guidelines

In general, the guidelines should do the following:

• document the purpose for which they were developed;

• describe each treatment option, including conservative or expectant

management, if appropriate;

• describe the natural history of the disease(s) or condition(s) under discussion,

where appropriate;

• detail the probable outcomes—the benefits and harmful side-effects specified

in terms of changes in physical, social or psychological function, changes in

pain and expected additional morbidity or mortality—associated with each

treatment option. The extent to which potentially harmful side-effects need to

be detailed in consumer documents should be determined in consultation

with consumers;

• be comprehensive and flexible enough to allow adaptation to the diverse

settings and circumstances of day-to-day clinical practice;

• identify the specifically known or generally expected exceptions—

circumstances in which the guidelines would not apply—or potential risks;

• indicate the probability (risk) of different outcomes occurring;

• identify the patient population to which they apply, bearing in mind possible

differences in expected outcomes for different populations;

• contain a description of the support services that may be required for each

treatment option, to allow consumers to assess the social consequences of the

options;

• include information for consumers and clinicians on any special training

needs for clinicians or equipment necessary for the safe delivery of the

treatment;

• provide a comparison of the costs associated with treatment options;

• provide a statement detailing the scientific basis on which the guidelines were

developed and explicitly note the strength of the evidence on which any

conclusions are based;

• document the uncertainty associated with any conclusion—uncertainty exists

where the evidence on outcomes is not strong and/or clinical opinion may

differ as to what outcomes may be;

• in the case of consensus-based guidelines, acknowledge the desirability of

developing evidence-based guidelines;

• in the case of non-consensus practice statements, make clear reference to

each of the schools of thought and direct consumer attention to the lack of

consensus. Note that the National Health and Medical Research Council has

issued guidelines on human experimentation—these should apply to

interventions for which non-consensus practice statements are issued; and

• document the economic appraisals used in formulating the guidelines.

3.10 Reporting on the guideline development process

The report should contain a description of the guideline development process,

the dissemination and implementation and evaluation and revision strategies,

and matters for future research.

The part of the report referring to guideline development should do the

following:

• identify the organisation under whose aegis the guidelines were developed

—who proposed, endorsed and funded the guidelines;

• define the purpose of the guidelines—for what audience(s) are the guidelines

intended?

• indicate if different formats were required for different users;

• list the individuals and groups involved in developing the guidelines;

• document the processes and nature of the scientific evaluation of the

evidence—for example, specify whether meta-analyses were used to analyse

the literature or if consensus formed part of the guideline development

process;

• provide a bibliography of materials used;

• contain a clear description of the scientific assumptions—whether relating

to physiological assumptions, methods of actions of drugs or processes,

expectations of deaths or morbidity following treatment, side-effects, and

so on—that were incorporated in the evidence;

• record the public policy and cost questions considered and the method by

which they were incorporated in the guidelines;

• identify the need for special training, equipment or facilities;

• describe the testing processes to which the guidelines were subjected;

• document the consultations undertaken; and

• record how consumer concerns were taken into account.

The part of the report dealing with dissemination, implementation, evaluation

and revision should contain the following details:

• the target audience and other interested parties;

• methods of consultation and dissemination;

• specific plans for particular circumstances—such as remote or rural locations

and links to institutional activities;

• the time frame for evaluation;

Guideline development

28 A guide to the development, implementation and evaluation of clinical practice guidelines

• the plan for assessing guideline implementation and barriers to

implementation;

• the plan for assessing the validity of the guidelines; and

• revision plans.

The part of the report dealing with matters for future research should contain the

following:

• suggestions for improving the guidelines and their capacity for enhancing

future care and optimising resource use;

• gaps in scientific knowledge or other information gaps—for example, costing

data—for future investigation; and

• organisational, funding or other ‘process’ factors that might enhance the

quality of outcomes expected from the guidelines’ implementation.

3.11 Assessing the guideline document

The adequacy of the guideline document itself should be evaluated by examining

whether it conforms to the principles outlined in this guide. Note that the Institute

of Medicine has developed a provisional instrument for prospectively assessing

the soundness of guidelines and the method by which they were developed

(Field & Lohr 1992). The Institute’s provisional assessment instrument is a formal

questionnaire that requires experts to rate the guidelines according to a range of

characteristics, such as their clinical applicability and flexibility, reliability and

reproducibility, validity and clarity.

The Scottish Intercollegiate Guidelines Network has also developed criteria for

appraising the guideline development process; it may be of value in the Australian

setting (SIGN 1995).

3.12 Consultation

The guidelines should be referred for examination to a wider group of interested

parties that may not have already been involved:

• practising clinicians;

• clinical colleges and allied health and professional organisations;

• consumer groups;

• Commonwealth, State and Territory and local health authorities; and

• industry groups.

29Guideline development

It may be useful to convene a conference of interested parties, provided this

forms only one part of the overall approach to guideline development and

provided the principles outlined in this document are incorporated in the

decision-making process.

In some instances an agency or group such as a professional college may disagree

with one or two of the recommendations in an otherwise acceptable document.

These points of disagreement should be noted in the guidelines.

The final guidelines should take into account comments received during the

consultations and in response to pilot testing.

30 A guide to the development, implementation and evaluation of clinical practice guidelines

Blank facing page to Chapter 4

Field and Lohr make the important point that ‘guidelines do not implement

themselves’ (1992). If guidelines are to be effective, their dissemination and

implementation must be vigorously pursued. If not, the time, energy and cost

devoted to the guidelines’ development will be wasted and potential

improvements in consumer health will be lost.

A multidisciplinary panel should oversee the various steps needed to disseminate

and implement the guidelines. The panel, which may be the same as the panel

responsible for developing the guidelines, should also identify any barriers to the

guidelines’ acceptance and implementation and work with members of target

groups to develop ways of overcoming these barriers. Identifying barriers to

change requires an understanding of sociological and psychological factors: it is

essential that the guideline development panel has expertise in these areas;

otherwise, inappropriate or ineffective methods of dissemination and

implementation may be advocated.

Although the guideline document may support a specific strategy for adapting

guidelines to local conditions, the guideline development panel may need to help

local groups with the adaptation process.

There is evidence that a range of dissemination and implementation strategies can

be effective (EHCB 1994). It is probable, however, that those strategies will

depend on the nature of the guidelines and which group is being targeted. There

are a number of possible strategies, such as the following:

• producing short summaries for use in a range of forums, including on the

Internet and websites;

• involving potential users in the development of the guidelines, directly or

through consultative mechanisms, or both, to promote ownership. The

consultation process on the draft guidelines can form part of the

dissemination strategy;

• using the media—local, regional or national, or a combination—to publicise

both the development process and the availability of the guidelines;

• using professional journals and magazines to inform people about guideline

development and promote the completed guidelines;

• using the communication links developed by clinical colleges, allied health

organisations, specialty societies, State and regional medical societies,

consumers, consumer groups, medical libraries and health education facilities;

• asking respected clinical leaders to promote the guidelines;

• providing economic incentives;

• using the educational processes of relevant colleges, professional

organisations and consumer groups, including conferences, workshops,

seminars and specialist journals;

CHAPTER 4

GUIDELINE DISSEMINATION AND IMPLEMENTATION

Guideline dissemination and implementation

32 A guide to the development, implementation and evaluation of clinical practice guidelines

• incorporating the guidelines in routine procedures—such as quality assurance

and review processes—of institutions and organisations that provide care;

• using information technology, particularly systems focusing on consultations

with individual patients;

• arranging for a credible health care provider to visit practitioners in the

clinical setting;

• piloting the draft guidelines in practices, clinics or hospitals, to facilitate

assessment of their relevance, applicability, comprehensiveness and flexibility;

and

• offering feedback on compliance with the guidelines.

Some of these strategies may be more effective when used in combination. There

is evidence that the more educational the dissemination strategy, the greater the

probability that the guidelines will be incorporated in clinical practice, provided

that dissemination is reinforced by an appropriate implementation strategy.

The initial dissemination strategy should define target audiences and design

accessible formats for the guidelines. Different versions of the guidelines should

be developed for different audiences—consumers, general practitioners, specialist

nurses, and so on. For example, the primary target audience for the National

Health and Medical Research Council’s guidelines on the management of early

breast cancer is consumers and specialists in breast surgery, medical oncology and

radiation oncology; the secondary target audience is general practitioners,

registered nurses involved in caring for women with breast cancer, funders,

managers, accreditors and others.

Implementation strategies should take into account the ways in which privacy and

confidentiality concerns might impede the implementation of guidelines.

Questions raised by the need to manage patient care across several locations

(hospitals, communities, nursing homes, and so on) or with the involvement of

several clinicians need to be considered.

In summary, the group producing the guidelines should inform the target

audience that the guidelines are being developed, let the audience know when

they are available, ensure that they are distributed, and work with the audience

on effective implementation.

4.1 Dissemination

4.1.1 Making the guidelines accessible

Guidelines should be presented in formats and styles suitable for their target

audience. Where the target audience is varied—as with, for example, the breast

cancer guidelines—the guidelines need to be presented in a range of formats and

styles suitable for each segment of the target audience. The cost of guidelines

should not present a barrier to access.

In their initial format, most guidelines will be large documents providing detailed

recommendations on diagnosis, investigation or treatment (or all of these things)

and providing evidence to support the recommendations.

Health professionals manage an array of conditions in a variety of circumstances,

so consideration should be given to the development of documents and formats

that are suitable for general practitioners, clinicians working in emergency

departments, ambulance officers, public health officials, quality assurance

managers, discharge planners, consumers and others.

4.1.2 Publishing the guidelines

Guidelines can be published as booklets, containing comprehensive information

and available from a central authority, or in summary form, providing information

about the main findings and recommendations and also available from a central

authority.

Summaries of the guidelines can also be published in other ways:

• in professional journals;

• in professional associations’ newsletters and magazines;

• in trade publications and industry newspapers;

• in institutional newsletters, such as those of divisions of general practice,

hospitals, consumer groups and area health services;

• in the popular media;

• as brochures—an example is the desk-top summary for General Practitioners

on the management of early breast cancer;

• as posters;

• in summary form or as full guidelines published on the Internet and linked to

websites appropriate for the target audience;

• as audio or video tapes; and

• on computer disks.

Guideline dissemination and implementation

34 A guide to the development, implementation and evaluation of clinical practice guidelines

In each case the language and style should be appropriate for both a general

audience and the target audience. Dissemination of guidelines will be improved if

more than one guideline format is used since, generally, multiple strategies yield

better results than individual strategies.

4.1.3 Informing the target audience of the guidelines’ availability

Ensuring that the target audience is aware of guideline development is a process

that should begin when consideration is being given to developing or revising

existing guidelines. The process is facilitated by advertising and wide publicity.

It is important to keep the target audience informed of progress, to alert them to

impending publication, to notify them when the guidelines are available, and to

distribute the guidelines to them.

A range of methods can be used to inform the target audience of the guidelines’

availability. These methods rely on informing clinicians directly; informing

consumers, who will request action based on the guidelines; and informing

managers, funders, accreditors and governments, who will all have their own

reasons for wanting the guidelines. Among strategies for informing the target

audience are the following:

• using national, regional and local media;

• publicity in trade publications and possibly writing articles for them;

• publicity through professional associations and their publications and possibly

writing articles for them;

• publicity in professional journals;

• publicity through consumer groups and their publications;

• contact with undergraduate and postgraduate educators;

• contact with undergraduate and postgraduate students;

• publicity through institutions such as colleges, hospitals, divisions of general

practice, area health services and universities;

• discussion at conferences, seminars and professional meetings;

• using ‘champions’ or local authorities to promote the guidelines or to be

interviewed about the guidelines in the media; and

• identifying ‘human interest’ stories for the media to highlight the need for and

benefits to be gained from the guidelines.

Although dissemination is an essential feature of guideline development, it alone

is not enough to change physicians’ behaviour. Most evaluations of guidelines

have shown that relying solely on printed material does not influence clinicians’

behaviour or health outcomes (Oxman et al. 1995). There is also evidence that

unsolicited mailing of guidelines does not influence clinicians’ behaviour,

although it can increase awareness of the guidelines (Grimshaw et al. 1995).

It may be appropriate to consult a professional communicator or seek expert help

from government, professional or consumer bodies.

In summary, dissemination involves making guidelines accessible, advertising

their availability, and distributing them widely. Strategies will vary according to

the target audience, but multiple strategies will ensure greater coverage than a

single strategy.

4.2 Implementation

Many studies have examined strategies for continuing medical education (see,

for example, Davis et al. 1995) and there is a considerable body of evidence on

which to draw. The most striking finding is that the simple dissemination of

guidelines is likely to have no impact at all on implementation (Oxman et al.

1995; Wise & Billi 1995).

Change will occur only if specific interventions designed to encourage it are used.

The interventions most likely to induce change are those that require the

clinicians’ participation in the change process (Wise & Billi 1995).

There is also evidence that the most effective implementation strategies are those

that have a direct effect on consultation between patient and health care

professional. Examples are restructuring medical records and patient-specific

reminders to incorporate guideline recommendations in decision making. In this

way, the guidelines become part of the process of care.

No single implementation strategy is effective in all circumstances for all people.

Successful implementation of guidelines requires that they become embedded in

many different aspects of the health care system and are informed by many

different aspects of the system. For more information about what implementation

strategies work, see Appendix I.

There is insufficient evidence to allow firm statements to be made about which

strategies work in which circumstances, and it is difficult to differentiate which

elements of strategies are the effective ones. Nevertheless, the following strategies

have been shown to be effective in changing clinicians’ behaviour or health

outcomes, or both, at least some of the time:

Guideline dissemination and implementation

36 A guide to the development, implementation and evaluation of clinical practice guidelines

• media marketing;

• the use of opinion leaders and ‘champions’;

• endorsement by clinical groups;

• practice visits from influential experts;

• education of patients;

• educational materials;

• seminars and conferences;

• reminder systems incorporated in clinicians’ daily work;

• continuing quality assurance and data feedback;

• local adaptation and incorporation;

• local involvement in evaluation; and

• incentives (Lomas & Haynes 1988; Lomas 1993, 1994; Oxman et al. 1995).

Each of these strategies requires different levels of input because of variations in

the degree of difficulty and in the cost of implementation. The implementation

plan should be developed to ensure that the most expensive implementation

strategies are used only when current practice differs from the recommendations.

Information technology, including the Internet and decision-support software, is

playing an increasingly important role in the implementation of guidelines, in

both summary and extended forms. A large body of work has already been

generated in this area and more is emerging.

4.2.1 The use of opinion leaders and ‘champions’

Highly regarded individuals—whether local specialists, general practitioners or

national figures—influence the practice of their peers (Lomas et al. 1991; Oxman

et al. 1995). They can deal with misconceptions about guidelines and help

clinicians observe the outcome of a particular innovation (Davis et al. 1995).

Consumer-support groups and well-recognised individual consumers can also

champion guidelines and guideline processes. Opinion leaders can be enlisted to

promote clinical practice guidelines at every opportunity, both formally and

informally.

37Guideline dissemination and implementation

4.2.2 Endorsement by clinical groups

If guidelines are endorsed by professional groups such as colleges and the

National Health and Medical Research Council they are more likely to be taken up

by clinicians. Endorsement by local groups is also important because the process

of endorsement means the guidelines will be examined closely by clinicians. This

may lead to adaptation to suit local circumstances, thus contributing to a sense of

ownership of the guidelines.

4.2.3 Practice visits from influential experts

The pharmaceutical industry employs a large number of marketing representatives

who visit clinicians in the practice setting and provide advice. This has been

shown to influence prescribing patterns (Todd 1995): meeting a sales

representative from a pharmaceutical company increases the likelihood that a

doctor will prescribe that company’s product (Chren & Landefeld 1994).

This approach has been adapted as academic detailing—where influential but

independent peers visit clinicians in their usual practice setting. Although

expensive, it has been shown to be effective (Oxman et al. 1995). In a typical

program, a clinical pharmacist armed with scientific data and simple memory aids

visits a clinician in his or her own practice and provides advice about a particular

matter, such as the appropriate use of non-steroidal anti-inflammatory drugs

(May et al. 1993). As noted, such programs have been shown to affect prescribing

(Todd 1995).

4.2.4 Education of patients and patient-mediated intervention

Providing patients with information and education—about guidelines or about

the condition under discussion—has been shown to change clinicians’ behaviour.

Educating patients raises their expectations, which then increases the demand on

clinicians, which improves performance. Information and education lead to

improved health outcomes when combined with other interventions, such as

academic detailing and clinician education (EHCB 1994; Oxman et al. 1995).

In developing education material for patients, and in considering the material’s

dissemination, it should be remembered that different groups of patients and

consumers have different needs.

38 A guide to the development, implementation and evaluation of clinical practice guidelines

4.2.5 Educational materials

Educational materials include the guidelines themselves, adaptations or summaries

of the guidelines, and articles about them in the general media, professional and

trade publications. Most studies have found that educational materials used alone

have little impact on professional behaviour and health outcomes (Oxman et al.

1995). Incorporation of educational materials in problem-based learning styles

has, however, been shown to be effective (B. Booth, pers. comm., 1996).

4.2.6 Seminars and conferences

Seminars and conferences requiring only passive attendance have not shown benefit.

Conferences requiring active participation through workshops, and with follow-up,

have been shown to influence clinicians’ behaviour (Oxman et al. 1995).

Based on the principles of adult learning and research evidence on the

effectiveness of continuing medical education, seminars and conferences should

be planned with the following principles in mind.

• the primary purpose of the activity is to improve the quality of patient care;

• the content of the activity demonstrates high clinical and ethical standards;

• clinicians participate in planning the activity;

• a learning needs assessment is conducted;

• the activity has clear learning objectives;

• the learning environment promotes fulfilment of the learning objectives; and

• the educational activity is evaluated (Royal Australian College of General

Practitioners 1995).

4.2.7 Reminder systems incorporated in clinicians’ daily work

Reminders—whether involving patient records or computerised prompting

—may influence clinicians’ behaviour (Grimshaw & Russell 1994; Oxman et al.

1995). These reminders provide a prompt that can be acted on immediately.

Their effectiveness probably comes partly from their immediacy.

Mailed reminder systems—such as those used for mammographic screening and

Pap smears—have also been shown to be effective (Lomas & Haynes 1988).

4.2.8 Quality assurance and quality improvement

Quality assurance and quality improvement activities have a complementary and

reciprocal relationship with clinical practice guidelines. Quality assurance activities

encourage the implementation of guidelines, and guidelines are a crucial

component of quality assurance activities.

Continuous clinical practice improvement claims to improve the quality of care by

bringing together research on variation of cost, access, quality and standardised

care. It requires a knowledge of processes and systems, human behaviour and an

approach to continuous learning.

Continuous clinical practice improvement generates valid statistical inferences

about the operational day-to-day elements in the process of clinical care. It

applies the scientific method to the day-to-day practice of health care. It also

analyses the content and timing of individual steps of the care process in order to

determine how to achieve superior health outcomes, for the least necessary cost,

over the continuum of a patient/client’s care.

There is evidence that the inclusion of evidence-based guidelines will strengthen

formal quality assurance activities such as clinical audit (Oxman et al. 1995). The

information in guidelines may also be used for developing utilisation reviews,

continuous quality improvement, total quality management and accreditation of

institutions or services, as well as providing a reference for interpreting data

feedback on clinical units’ practices and the health outcomes of their patients.

In effect, guidelines can become incorporated in routine processes of care. Using

guidelines as a basis, it is possible to develop hospital admission and discharge

policies, care pathways and referral protocols. These systems will also facilitate

evaluation of the uptake of guideline recommendations and their impact on

health outcomes.

Data collection systems measuring the impact of guidelines also allow comparison

between the evidence about outcomes gathered from clinical trials and the real

‘outcomes’ achieved in routine clinical practice. In this way both clinicians and

patients can become better informed about the risks and benefits of medical

interventions.

Guideline dissemination and implementation

40 A guide to the development, implementation and evaluation of clinical practice guidelines

4.2.9 Local adaptation and incorporation

The effectiveness of nationally developed guidelines may be enhanced if local

groups—such as clinicians at a particular hospital, a division of general practice

or a State faculty of a specialist group—adapt the guidelines for local use, while

taking into account geographic, demographic, resource and other factors (Oxman

et al. 1995). In one case studied, this process improved the scientific integrity of

the guidelines (Brown et al. 1995). The process of local adaptation exposes

clinicians to the guidelines, makes the guidelines more relevant to local

conditions, and encourages a feeling of ownership (Wise & Billi 1995).

Local adaptation could be assisted if the guideline publisher were to make the

document available on disk or on the Internet.

In adapting guidelines for local use, the core content in most circumstances could

include the following:

• identification of a specific patient population;

• details of the likely outcomes of care—benefits, adverse effects, changes in

physical, social or psychological functioning, changes in pain, and expected

changes to overall morbidity and mortality;

• indication of the probability of different outcomes occurring;

• a description of each treatment option available;

• identification of the specifically known or generally expected circumstances

where guidelines do not apply;

• identification of relevant support services;

• information on any special clinical training needs or equipment necessary for

safe treatment; and

• comparative costs associated with different treatments.

Care should be taken not to impair the scientific integrity of the primary

guidelines while making local adaptations. If local groups alter the guidelines in

the light of evidence not previously considered, they should inform the guideline

developers of the availability of that evidence.

4.2.10 Incentives

Guidelines are more likely to be implemented if clinicians have incentives such as

the following to implement them:

• financial incentives—for example, differential fees, prospective payment systems,

clinical budgeting, the removal of items not recommended by guidelines from

reimbursement schedules, and the provision of funds for retraining in specific

techniques recommended by the guidelines (Stocking 1993);

• personal satisfaction, compounded by recognition from peers, experts or

‘champions’;

• professional incentives such as accreditation or continuing medical education

points;

• invitations to attend professional conferences (Orlowski & Wateska 1992);

• the offer of increased protection against litigation (NHMRC 1995);

• government regulation; and

• the receipt of personalised, relevant data through the evaluation process.

It is important that there be no disincentives to implementing guidelines. For

clinicians, disincentives could be associated with workload, time, remuneration

and the need for resources and specialised skills and equipment. For consumers,

disincentives could be associated with costs directly related to the treatment itself

or costs associated with obtaining that treatment, such as the need to travel to

cities or specialised centres for treatment.

No single implementation strategy is effective enough to be recommended for use

on its own. Multiple implementation strategies are more likely to be effective

(Wensing & Grol 1994).

4.2.11 The use of information technology

Recent feedback from Australian guideline developers highlights the important

role of information technology in implementing clinical practice guidelines.

Advances in technology such as the Internet and improved systems of data

management and storage mean that information technology will occupy an

increasingly important place in future implementation strategies.

This technology is yet to be widely applied and evaluated. Associated with this is

the fact that information technology is not a ‘quick fix’. The infrastructure

necessary to make use of the technology takes time to establish and training and

education need to occur concurrently. Nevertheless, this should not be seen as an

insurmountable problem—rather, it should be seen a factor that warrants due

consideration.

Guideline dissemination and implementation

42 A guide to the development, implementation and evaluation of clinical practice guidelines

Data management is one aspect of information technology that should be

examined when assessing technology’s impact on the dissemination and

implementation of guidelines. Secure, reliable data and data-management

procedures are essential if the full potential of the opportunities offered by

information technology is to be realised.

Internationally, the use of information technology in relation to clinical practice

guidelines has advanced most rapidly in the United States. Guidelines have been

developed as part of computerised decision-support systems, largely as an

initiative of health insurance funds. Although guidelines incorporated in these

systems seek to improve health outcomes, a disproportionate emphasis, in the

health insurance fund context, is placed on the minimisation of costs.

The use of automated guidelines is receiving much attention. Few studies have

been done, but there is evidence to suggest that such guidelines are effective in

the implementation of clinical practice guidelines (Conroy & Shannon 1995).

Through this technology, the entire process of patient consultation can be

prompted step by step. It is also possible to progress through an appropriate

guideline module that not only guides decision making but also records and

saves the patient’s medical history. There is strong evidence that computer-based

clinical decision–support systems can enhance clinicians’ performance and

improve health outcomes (Johnston et al. 1994).

Clinicians are far more likely to make use of guidelines if the guidelines provide

information at the point of patient contact, not just on a bookshelf. It may be that

the immediacy is the attraction. Problems with revising guidelines are also more

readily solved when the guidelines are in electronic form.

Future directions in information technology seem set to include the Internet as a

major mechanism for the dissemination of clinical guidelines. People can gain

access to worldwide information and resources; they can locate guidelines that

pertain to their health and use this important source of information in joint

decision making with their health care professional. At present there are no

comprehensive guidelines spanning all the major areas of health, although some

corporations in the United States (such as Medaccess) are already providing

electronic copies of guidelines produced by the US Department of Health and

Human Services and the US Agency for Health Care Policy and Research. The

latter agency is planning to post all guidelines registered with it on the Internet.

Information technology is also very amenable to use as a tool for total quality

management. Evaluation and audit are important factors in developing and

maintaining quality guidelines, and information technology can provide ‘tagging’

of patients’ records in order to remind health care staff to deliver treatment in

keeping with appropriate guidelines (Forrest et al. 1996). The use of audit,

made more effective by computer records, also allows feedback to users such as

General Practitioners. This has been demonstrated to be an effective way of

inducing change in the clinical setting (Forrest et al. 1996).

In summary, developments in information technology make possible the

following:

• better database access to store, retrieve and interpret data;

• use of the Internet as a tool for dissemination and implementation of

professional and consumer guidelines;

• computerisation of medical records;

• computerised prompting, either as part of clinical practice or linked only to

accounting records;

• computerised decision-support systems; and

• more interactive styles of learning.

Although it will be some time before the full potential of information technology

for guideline dissemination and implementation is realised, the opportunities

cannot be overlooked.

Whatever strategies are used to disseminate and implement guidelines,

interventions most likely to induce change are those that require clinicians’

participation in the change process.

4.3 A systems approach

If guidelines are to have maximum impact they need to form one element of

an integrated quality-planning and improvement strategy, rather than being

developed and implemented in isolation. Other elements of such a strategy are

systematic review and revision, evidence-based policies and clinical pathways.

For maximum effectiveness, guidelines should be integrated with broader

activities—such as peer review, continuing medical education and quality

assurance, performance monitoring and accreditation—to promote and improve

the quality of care at the local level.

Guideline dissemination and implementation

44 A guide to the development, implementation and evaluation of clinical practice guidelines

Blank facing page to Chapter 5

5.1 Evaluation

Evaluation of clinical practice guidelines is essential. How will we know if

guidelines are worth developing, disseminating and implementing if we do not

know whether they make a difference to clinical practice and health outcomes?

The evaluation process is intended to assess the validity of the guidelines and the

effectiveness of their dissemination and implementation. Thus, it should yield

information about the extent to which the guidelines produce the health

outcomes identified in the guideline development stage. It should also produce

data on actual costs compared with the projected costs of producing the required

outcomes.

The dissemination and implementation strategies should be analysed in terms of

their effectiveness in reaching clinicians and consumers:

• the target groups’ awareness and understanding of the content of the

guidelines;

• the guidelines’ relevance to these groups;

• the extent to which barriers to implementation were overcome;

• the extent to which the guidelines affected or changed clinical practice; and

• whether the guidelines were sufficiently flexible for use in different local and

regional circumstances and whether they were interpreted consistently by

different providers.

Most importantly, the evaluation plan should be designed to assess whether the

guidelines improve health outcomes—that is, the guidelines’ validity.

Evaluation requires data on processes, practices and outcomes and these

requirements should be taken into account when guidelines are being developed.

Areas of concern in terms of confidentiality should be identified. The evaluation

plan should also specify areas where more extensive data collection is needed to

fill gaps in research identified during the development process and to examine

any specific concerns, such as differences in outcomes for different population

groups. The plan should seek to establish a process for dealing with these

matters.

Guidelines should be evaluated at least once every three years, although in

subject areas that are prone to rapid change this may need to occur more

frequently.

CHAPTER 5

GUIDELINE EVALUATION AND REVISION

Guideline evaluation and revision

46 A guide to the development, implementation and evaluation of clinical practice guidelines

In short, the evaluation of clinical practice guidelines has six components:

• an assessment of guideline dissemination;

• an assessment of whether or not clinical practice is moving towards the

guidelines’ recommendations;

• an assessment of whether or not health outcomes have changed;

• an assessment of whether or not the guidelines have contributed to any

changes in clinical practice or health outcomes;

• an assessment of the guidelines’ impact on consumers’ knowledge and

understanding; and

• an economic evaluation of the guideline process.

Groups involved in guideline development and implementation will be required

to evaluate at least some of these components, but it is unlikely that an individual

group would evaluate all six. In the following sections the components are

examined in turn, with changes in clinical practice and health outcomes

considered together. Guideline developers will also be required to produce a

report on the evaluation process.

5.1.1 Evaluation of dissemination

Evaluation of guideline dissemination is very common and relatively

straightforward.

How many copies of the guidelines have been requested? How many copies of

the guidelines have been mailed out? Of those that were disseminated, how many

have been received? opened? read in part? read in full? understood?

How many posters were sent out? How many were displayed in appropriate and

prominent places? How many articles have been published, or interviews

broadcast, about the guidelines? How many consumer inquiries have there been?

These criteria can be used for assessing whether or not a dissemination strategy

has achieved its goals. An indication of the response rates that might be expected

can be obtained from the marketing literature.

5.1.2 Evaluation of changes in clinical practice and health outcomes

Ideally, guideline developers would have had access to a national data set that

provided evidence of the need for clinical practice guidelines and baseline

information for evaluation purposes. The guideline developers themselves may

also have collected relevant data, ideally concerned with both clinical practice and

health outcomes.

Repeated data analysis and collection should be carried out after the guidelines

are disseminated and implemented. Such analysis will allow for continual

assessment of changes in patterns of practice in relation to the guidelines and a

redefinition of target areas for further promotion. It will also permit an assessment

of whether or not health outcomes are improving.

Institutions should also collect data that can be usefully compared against national

data. These data can be used in clinical audit and other quality assurance activities

and will facilitate assessment of how the guidelines are affecting the routine care

being delivered by clinicians and the effects of that care on the health of their

patients.

Where possible, existing national, State and Territory and local data sources

should be used, for both pre-guideline and post-guideline applications. In other

circumstances customised surveys and other initiatives will be required. In many

cases the local data collected can be fed back directly into clinical practice.

In providing data for guideline-evaluation purposes, some clinicians may feel it is

necessary to seek protection from disclosure of this kind of information under

quality assurance legislation. Such legislation exists at Commonwealth and State

and Territory levels.

A wide variety of organisations monitor health care and health outcomes. For

example, the Australian Council on Healthcare Standards has developed clinical

indicators that may provide guidance. The clinical colleges are also collecting data

for this purpose.

In some cases it will not be possible to measure health outcomes or the desirable

outcomes will be such that immediate feedback is not possible. In these cases it is

reasonable to use more immediate proxy measures of outcomes.

For example, mammographic screening programs will be judged effective only

if they reduce mortality from breast cancer. Because of the lag time between

initiation of screening and the anticipated reduction in mortality, it is reasonable

to use a number of proxy measures in the first instance. Among the suitable

proxies are tumour size and the degree of lymph node involvement at the time of

diagnosis. Should these measures show the scale of improvement expected from

the results of the original field trials, it will be reasonable to assume that

mammographic screening is on track in terms of providing a benefit to those who

take part in the program.

Guideline evaluation and revision

48 A guide to the development, implementation and evaluation of clinical practice guidelines

5.1.3 Evaluation of the guidelines’ contribution to changes in clinical

practice and health outcomes

Section 5.1.2 discusses evaluation of the improvement in clinical practice and

health outcomes, which is the goal of clinical practice guidelines. Such

assessment, though, does not take account of the contribution guidelines may

have made to those improvements.

It is important that the impact of guidelines be assessed. These assessments

require complex study designs and will need to be carried out by researchers in

collaboration with clinicians, rather than by clinicians alone.

There is a strong recent history of guideline evaluation. A systematic review of

87 studies of the effectiveness of guidelines revealed evidence of changed

clinical practice in 81 studies (93 per cent). Of the 17 studies that examined

health outcomes, 12 studies (71 per cent) found a measurable improvement

(EHCB 1994).

Among possible methods for evaluation of guidelines are the following:

• comparing changes in clinical practice or health outcomes, or both, in areas of

exceptionally high guideline promotion with changes in areas of exceptionally

low guideline promotion;

• comparing health outcomes in areas of exceptionally high guideline uptake

with outcomes in areas of exceptionally low guideline uptake—focus group

testing can be useful to elucidate factors that have influenced this uptake.

When evaluating the guidelines it is important to focus on the guidelines

themselves, rather than on the clinicians or other service providers. If there is little

change in practice, or little adherence to the guidelines, this may be a

consequence of a wide range of factors, among them the guideline development,

dissemination and implementation process.

5.1.4 Evaluation of consumer guidelines

Consumer guidelines need to be evaluated in addition to the professionals’

guidelines.

A number of different approaches to the evaluation of consumer guidelines

should be considered:

• the accessibility of the guidelines;

• the clarity and lucidity of the guidelines;

• the acceptability of the amount of information contained in the guidelines,

particularly compared with the amount of information contained in the

professionals’ guidelines;

• the relevance of the guidelines to different groups of consumers; and

• the overall level of consumer satisfaction (or dissatisfaction) with the

guidelines.

In all cases the evaluation needs to be carried out with extensive consumer input

and direction.

Examination of the relationship between doctors and the pharmaceutical industry

shows that clinicians who perform any research into, or evaluation of, a drug are

more likely as a consequence to use that drug (Chren & Landefield 1994).

Generalising from this experience, it would be expected that clinicians involved in

evaluating guidelines are more likely to use those guidelines.

5.1.5 Economic evaluation

A number of cost factors relating to guidelines need to be known:

• process costs—the costs of development, dissemination and implementation;

• pre-implementation costs—the costs associated with existing practice patterns,

including testing, pharmaceutical and surgical interventions; and

• post-implementation costs—the costs and cost savings resulting from

observance of the guidelines. For example, the guidelines may result in the

elimination of unnecessary tests or procedures, which may result in lower

overall costs as well as better health outcomes.

Ideally, where a desired outcome is determined, a cost-effectiveness analysis

should be done. Such analyses require an assessment of all costs incurred in

providing the guidelines and the cost-related impact of the guidelines.

Such economic analyses can also assist by providing a comparison of guideline-

dissemination and -implementation strategies. In this way it will be possible to

assess which strategies are more cost-effective, in addition to improving clinical

practice and health outcomes.

5.1.6 Reporting on the evaluation

The results of the evaluation, including any flaws found in the guidelines and

their usefulness or otherwise for groups, should be included in a report on the

evaluation. Wherever possible, indicators of avoidable outcomes should be

provided. Among the avoidable outcomes might be the following:

Guideline evaluation and revision

50 A guide to the development, implementation and evaluation of clinical practice guidelines

• unforeseen consequences of the guidelines;

• outcomes that would suggest that the guidelines were not being followed and

that would be useful in monitoring the extent to which guidelines were being

implemented; and

• outcomes that would suggest that the guidelines were not being implemented

correctly or that the quality of care was poor.

This information could be used in quality assurance processes.

5.2 Revision

When guidelines are developed a date and strategy should be set for their

revision. The National Health and Medical Research Council recommends that

revision take place at least every three to five years and more often where the

subject matter or circumstances are prone to rapid change (NHMRC 1995).

A number of steps are required for revision:

• a multidisciplinary group, from disciplines similar to those of the guideline

development group, should assess the guidelines to see whether there is any

new evidence that should be incorporated;

• the group should also assess what has been learnt from the evaluation of the

dissemination and implementation strategies and incorporate suggested

improvements in the further dissemination and implementation of guidelines;

• the group should draw on current practice and experience and on national

data that have been informed by the guidelines. Revision should be a

coordinated activity, extending beyond the academic literature on clinical

practice and health outcomes to incorporate experience, local knowledge and

regional and national data; and

• guidelines should state the date of their development and the anticipated

revision date.

6.1 Liability of practitioners

Many practitioners are concerned about their potential legal vulnerability if the

patient does not receive treatment specified in clinical practice guidelines. It is

certainly possible that guidelines could be produced as evidence of what

constitutes reasonable conduct by a medical practitioner for the purposes of

assessing whether the practitioner’s duty of care had been breached in a medical

negligence action.

It is the view of the Health Advisory Committee, which oversaw the preparation

of this document (see Appendix F) that the existence of clinical practice

guidelines will provide a measure of protection for practitioners who use the

guidelines.

In a recent Australian court decision it was held that the Bolam principle—usually

summarised as the idea that a doctor is not negligent if they act in accordance

with the behaviour of a reasonable body of their peers—is not the only principle

to be applied. In Rogers v Whitaker (1992) 175 CLR 479 the High Court held that a

practitioner had a duty to inform a patient of the risk associated with medical

procedures and that the court would assess, on the facts of each case, whether

the conduct of the practitioner was reasonable. The High Court pointed out that

there is a distinction to be drawn between determining whether a medical

practitioner has been negligent in the particular form of the treatment (where

professional opinion will be influential) and determining whether a practitioner

has been negligent in the provision of all relevant information to a patient.

The Rogers v Whitaker case makes it clear that doctors should give information

about the risks of any treatment, especially those that may influence a patient’s

decision. The National Health and Medical Research Council advises all involved

in health care to take note of the spirit of the law as well as the letter of the law.

Patients should be provided with as much information as they seek, and in a form

that is appropriate to their culture and level of education. Doctors should give

advice but not coerce: patients should be encouraged to make their own

decisions. The use of guidelines should lead to better informed medical decision

making by both practitioners and patients.

CHAPTER 6

LEGAL CONSIDERATIONS

Legal considerations

52 A guide to the development, implementation and evaluation of clinical practice guidelines

6.2 Legal liability of guideline developers and bodies supporting

guideline development

Normally, a general publication of information, even where negligently collated,

does not give rise to liability because the author does not owe a duty of care to

the general public at large, although the guideline issuer could be held liable if a

relevant close relationship can be established between them and the person who

suffers a loss.

If guidelines purport to be a definitive statement of the correct or appropriate

procedure there would be a greater risk of liability than if the guidelines are

expressly stated to be provided as a general guide subject to the medical

practitioner’s expert judgment in each case.

Guideline developers need to demonstrate that they have taken reasonable steps

to ensure that the guidelines were properly prepared and that they are based on

the best information available at the time of publication. The guidelines should

clearly state that they are not a definitive statement of the correct procedure;

rather, they constitute a general guide to be followed, subject to the medical

practitioner’s judgment in each case. They should also state the date the

guidelines have been issued and make it clear that the information is correct only

to that date.

If these procedures are followed it is unlikely that the guideline developers will

be exposing themselves to the risk of legal liability.

6.3 Minimum legal requirements for guideline developers

The potential for guidelines to be used as evidence in courts of law depends on

the process used to develop them, the extent to which they are evidence-based,

the degree of consensus about them, and whether they are up to date.

Guideline developers are unlikely to be held liable for any negative consequences

of implementation of the guidelines, particularly if the processes of preparation

and the limitations of the guidelines are clearly described. In general, the

following principles apply:

• guidelines should be predominantly summaries of the evidence;

• guidelines should have an expiry date;

• an independent review of the guideline development process is

recommended;

• areas of disagreement in the guidelines should be acknowledged; and

• guidelines should not be unduly prescriptive and must allow for cases that

call for management that differs from what is recommended.

The development, dissemination, implementation, evaluation and revision

processes described in this document are applicable to a wide range of clinical

interventions and disciplines and could also be applied to protocols relating to the

use of technology and pharmaceuticals. Users and reviewers of these guidelines

are encouraged to contact the National Health and Medical Research Council if

they have comments—positive or negative—to make.

The emerging guideline processes are highly complex and depend not only on

integration of the activities of all the people involved but also on individual and

organisational behaviour change. With the exception of one or two systematic

reviews of guideline implementation, the research literature is largely anecdotal,

being based on the experiences and impressions of participants and observers.

The research evidence that does exist, however, shows that properly developed

clinical practice guidelines can lead to improvements in both clinical practice and

health outcomes. But developing the guidelines alone is not enough: they must

be disseminated; there must be strategies for implementation; and resources must

be set aside for evaluation of the guidelines’ impact, which will highlight areas

requiring revision, improvement or further development.

In disseminating, implementing and evaluating guidelines, it is preferable, where

possible, to use existing networks, facilities and publications rather than to

develop new processes. Not only is this less costly: guidelines provided through

existing channels and capable of being incorporated in normal practice are more

likely to be accepted as part of the routine than anything that requires new

structures or changed modes of practice.

What is needed is a systems approach that builds guideline development,

implementation and evaluation into the routine processes of care. Organisations

such as hospitals, health care regions and divisions of general practice could

establish such an approach, integrating the guidelines into their local health care

delivery processes. It is important that these activities be coordinated between

different groups. A systems approach would be based on four main principles:

• sensible use of resources;

• sustainable infrastructure;

• integration of available resources and strategies; and

• adapting to local needs.

This document provides a range of strategies—from the personal to the general,

from those of low intensity to those of high intensity. As a general rule, one-to-

one approaches—that is, ‘champions’ or academic detailing—are most effective,

but they are also the most expensive.

CHAPTER 7

CONCLUSION

Conclusion

54 A guide to the development, implementation and evaluation of clinical practice guidelines

The best approach to making sure guidelines work would appear to be a mix of

strategies suitable for local conditions and developed in concert with local

clinicians, consumers and managers. There is no single method for all conditions

but all the strategies discussed in this document, if used selectively and in

combination, can work. Although effort is required, evidence-based guidelines

can and will contribute to improved practice and health outcomes.

The guidelines process itself is a continuum and it is expected that this document

will be revised as new information becomes available. Since the objective of

implementing clinical practice guidelines is to ensure the provision of the best

possible patient care and the best possible outcomes from clinical interventions,

the incorporation of new knowledge is fundamental to the guidelines’

effectiveness.

Best clinical practice ensures that the most appropriate resources are used in the

most effective way. This document is designed to contribute to that process and

to better health outcomes for all Australians.

The recommended process is endorsed by the National Health and Medical Research Council and is

based on the general approach to improving health outcomes agreed to by Australian Ministers

responsible for health. It is strongly influenced by the 1992 US Institute of Medicine report,

Guidelines for Clinical Practice: from development to use, and the work of the US Agency for Health

Care Policy and Research, the US Preventive Services Task Force, the UK Clearing House for

Information on the Assessment of Health Outcomes and the Cochrane Collaboration, plus the

Australian National Health Strategy document entitled Making It Better (Harvey 1991) and the wider

body of research conducted in the area.

Level of evidence Study design used as an indicator of the degree

to which bias has been eliminated by design

(see Appendix B).

Quality of evidence The quality of the methods used by investigators to

minimise bias in a study design.

Relevance of evidence A term encompassing the closeness of the study

question to the clinical question, which is determined

by the relevance of the outcome measures used and the

applicability of study results to other treatments, settings

and patients.

Strength of evidence The magnitude, precision and reproducibility of the

intervention effect; includes the effect size, confidence

interval, P value, and the exclusion of clinically

unimportant effects. In the case of non-randomised

studies, additional factors such as biological plausibility,

biological gradient and temporality of associations may

be considered.

APPENDIX A

DEFINITIONS

Appendix

56 A guide to the development, implementation and evaluation of clinical practice guidelines

I evidence obtained from a systematic review of all relevant

randomised controlled trials.

II evidence obtained from at least one properly designed randomised

controlled trial.

III-1 evidence obtained from well-designed pseudo-randomised

controlled trials (alternate allocation or some other method).

III-2 evidence obtained from comparative studies with concurrent

controls and allocation not randomised (cohort studies), case-

control studies, or interrupted time series with a control group.

III-3 evidence obtained from comparative studies with historical control,

two or more single-arm studies, or interrupted time series without a

parallel control group.

IV evidence obtained from case series, either post-test or pre-test and

post-test.

These levels of evidence ratings have been adapted from US Preventive Services Task Force (1989),

Guide to clinical preventive services: an assessment of the effectiveness of 169 interventions, (ed M

Fisher), Williams and Williams, Baltimore, Appendix A, p388.

APPENDIX B

DESIGNATION OF LEVELS OF EVIDENCE

APPENDIX C

CLINICAL PRACTICE GUIDELINES FLOW CHART

Appendix

Is the topic related to clinical decision-making

Define topic

Are there suitable existing guidelines?

Convene a multidisciplinary panel

Identify health outcomes and barriers to change

Review scientific evidence of efficacy of

interventions in relation to outcomes

Is there Level I-IV evidence in

respect of each recommendation?

Develop evidence-based

recommendations or update

existing recommendations

Consultation and pilot testing

Disseminate and implement

Evaluate and revise

Is there consensus?

Develop consensus-

based recommendations

that indicate lack of

clear evidence but

acknowledge consensus

Make brief non-consensus

statement (state options and

acknowledge uncertainty)

stop

yes

58 A guide to the development, implementation and evaluation of clinical practice guidelines

APPENDIX D

GENERIC OUTCOME MEASURES

Process-based

Quantity outcome Satisfaction

of life measures Quality of life with care

Physical Social Mental

Mortality Re-admission Measures of Measures of Measures of Patient

rates impairment social support depression satisfaction

surveys

Avoidable Relapses Disease- Measures of Measures

premature specific disability of social

mortality measures adjustment

Pain scales

Measures of

functional

status

Multi-dimensional health status profiles and

indexes cover many of these dimensions in

a single instrument

Source: UK Clearing House for Information on the Assessment of Health Outcomes (1993).

Health-related quality-of-life

dimensions of health

Measures of handicap

The following checklist is adapted from Woolf (1992):

• selection of topic;

• selection of panel members and chairperson;

• clarification of objectives;

• assessment of projected health outcomes (benefits and harms);

• assessment of scientific evidence;

• assessment, where appropriate, of cost-effectiveness;

• assessment of expert opinion;

• assessment of appropriateness;

• drafting of document;

• consultation;

• pre-testing/piloting;

• recommendations for dissemination, implementation, evaluation

and revision; and

• recommendations for research.

APPENDIX E

CHECKLIST FOR GUIDELINE DEVELOPMENT

Appendix

60 A guide to the development, implementation and evaluation of clinical practice guidelines

Background

The National Health and Medical Research Council published the first edition of

Guidelines for the Development and Implementation of Clinical Practice

Guidelines in 1995. This publication aimed to distil the best aspects of the

considerable guideline development work and research being done in Australia

and overseas. In September 1997 the National Health and Medical Research

Council’s Health Advisory Committee undertook to review the document to take

account of new research on guideline development and to include new emphases

on implementation and system strategies.

The Working Party

A Working Party consisting of Professor Chris Silagy, Professor George Rubin,

Professor David Henderson-Smart and Mr Paul Gross undertook to revise the

document with the assistance of Professors David Henry and Paul O’Brien and

technical writers Mr Ray Moynihan and Ms Melissa Sweet.

The Working Party that developed the first edition consisted of Ms Helen Lapsley

(Chair), Mr Roy Harvey, Ms Hilda Bastian, Emeritus Professor Tom Reeve,

Professor Chris Silagy, Dr Ian Steven and Associate Professor Peter Woodruff.

The process

Most of the Working Party’s work was done out of session, with meetings being

used primarily to plan and clarify the task and to analyse and respond to matters

raised in consultations.

The report was updated by synthesising the latest evidence on guideline

development and making revisions in accordance with feedback from interested

parties who had used the document to develop guidelines. A technical writer was

contracted to incorporate the revisions and finalise the editing.

The final draft was approved by members of the Health Advisory Committee and

endorsed for publication by the National Health and Medical Research Council.

Consultations

Organisations and individuals were asked for their views about the usefulness of

the first edition of the Guidelines. A questionnaire was sent to about 120

individuals and organisations, including the following:

• members of the Working Party who developed the initial document;

APPENDIX F

PROCESS REPORT

• all chairs and members of former working parties developing other guidelines

issued by the National Health and Medical Research Council in the previous

three years;

• peak consumer organisations; and

• relevant branch heads and medical advisers within the Commonwealth

Department of Health and Aged Care.

Submissions were received from 25 organisations and individuals during this first

stage of public consultations (see Appendix G).

The second stage of public consultation took place during April 1998. The draft

revised document was circulated to the individuals and organisations just listed

and to the senders of submissions. In addition, the release of the draft revised

document was advertised in the national media and the Commonwealth Gazette.

Twenty-three submissions were received during the second stage of consultation

(see Appendix H).

All submissions were carefully analysed by the Working Party and the draft

document was amended accordingly.

Dissemination

These revised guidelines will be distributed to:

• clinical colleges;

• academic departments;

• allied health organisations;

• State and Territory health departments;

• public policy makers;

• hospitals;

• consumer and patient groups;

• health economists; and

• professional journals.

They will also be published electronically.

Evaluation and revision

These guidelines reflect existing knowledge and practices at the time of their

publication. This is, however, a rapidly changing area, and as new evidence

becomes available the guidelines will require further revision in order to remain

valid.

Appendix

62 A guide to the development, implementation and evaluation of clinical practice guidelines

October 1997

Professor Hugh Taylor Royal Victorian Eye and Ear Hospital

Dr Lawrence Brunello Royal Australian College of Obstetricians and

Gynaecologists

Associate Professor Royal Melbourne Hospital

Stephen Davis

Mr L. Apolony Royal Australasian College of Radiologists

Professor Peter Phelan Royal Australian College of Paediatrics

Dr C. Aroney The Prince Charles Hospital

Professor R.C. Bennett Royal Australasian College of Surgeons

Ms J. Noonan New South Wales

Associate Professor Queensland

Peter Woodruff

Dr Anthony Arklay University of Queensland

Dr Roger Goucke Sir Charles Gairdner Hospital

Professor Austin and Repatriation Medical Centre

Geoffrey Donnan

Ms Merle Fullerton Northern Rivers ME/CFS/FM Support Association Inc.

Dr Thelma Hunter Australian Capital Territory

Dr David Wong Australasian College of Dermatologists

Dr John Vinen Royal North Shore Hospital

Dr Craig Martin Health Department of Western Australia

Dr Michael Bollen Royal Australian College of General Practitioners

Dr Laurence Lau Royal Australasian College of Radiologists

Mr Charles Curran Department of Health and Aged Care

Dr Robert Broadbent Royal Australia and New Zealand College of

Psychiatrists

Dr Caroline Crowther University of Adelaide

Dr Colinh McLeod Royal College of Pathologists

Dr Jermey Anderson Monash Medical Centre

Dr Andrew Wilson New South Wales Health Department

APPENDIX G

SUBMISSIONS RECEIVED

May 1998

Professor Hugh Taylor Royal Victorian Eye and Ear Hospital

Professor Justin O’Day St Vincents Medical Centre

Dr John Vinen Royal North Shore Hospital

Dr Cathy McCarthy University of Melbourne

Dr Nicholas Clark Turning Point

Professor R. Bennett Royal Australasian College of Surgeons

Dr Michael Stanford Royal Melbourne Hospital

Dr Peter van Asperen Australian College of Paediatrics

Ms Mary Draper Royal Melbourne Institute of Technology

Ms Sophie Hill

Professor University of New South Wales

Kevin McConkey

Dr Andrew Wilson New South Wales Health Department

Dr Jenny Doust Australasian Cochrane Centre

Ms Eileen Wilson Australian National University

Professor Paul Martin Australian Psychological Society

Dr Sally Redman National Breast Cancer Centre

Dr Julie Bines Royal Childrens Hospital, Melbourne

Dr Colin McLeod Royal College of Pathologists of Australia

Ms Merle Fullerton Northern Rivers ME/CFS/FM Support Association Inc.

Mr Alan Bansemer Health Department of Western Australia

Professor Tom Reeve Australian Cancer Network

Dr John Campbell Royal Australian College of Obstetricians and

Gynaecologists

Dr Ron Tomlins Royal Australian College of General Practitioners

Professor Chris Del Mar University of Queensland

Appendix

64 A guide to the development, implementation and evaluation of clinical practice guidelines

When comparing the cost-effectiveness of alternative interventions, it is essential

that an appropriate comparator be selected. Ideally, this would result in a ‘head to

head’ study comparing the relative benefits and the relative costs of the various

treatment options. In practice, though, a treatment ‘option’ may consist of a

combination of several types of treatment, as is the case in the treatment of cancer

patients. Furthermore, cost data may not be available for new treatments.

Economic appraisal must take such matters into account when examining

alternatives. Nonetheless, if the evidence indicates that alternative interventions

differ markedly in cost but produce similar health outcomes and have similar

levels of acceptability to consumers and practitioners, it is reasonable that

guidelines recommend the use of the least costly alternative.

There are many difficulties in incorporating economic appraisals in clinical

practice guidelines. One of the most obvious is the scarcity of accurate cost data

for many clinical interventions. But, because of the wider resource consequences

of clinical decisions, it is important that these and other potential problems be

dealt with rather than side-stepped and that economic data be collected when

guidelines are being developed and evaluated.

APPENDIX H

ECONOMIC APPRAISAL INFORMATION

Strategies used to influence clinical behaviour

Target Individuals Populations

Patient-centred Patient participation Shopping centre screening

Patient education

Mailed reminders

Media campaigns

Public education

Educational Traditional continuing Evidence-based guidelines

medical education

Innovative continuing Journal publication of

medical education critical appraisal methods

—academic detailing,

‘champions’,

individual instruction

Advanced academic Other forms of ‘official

training in effective and policy’

efficient care

Commercial detailing

Commercial advertising

Task force reports

Administrative Reminder systems

Government regulation

Computerised medical Feedback of practice

records

pattern data

Peer review Second opinion programs

Criteria setting Review committees

Peer assessment

Clinical audit

Economic Clinical budgeting

Differential fee charges

Decentralised management Prospective payment

systems

Removal of items from

reimbursement schedules

Strategies that have been shown to be effective.

Source: Adapted from Lomas and Haynes (1988).

APPENDIX I

IMPLEMENTATION STRATEGIES THAT WORK

Target group and strategy

Appendix

66 A guide to the development, implementation and evaluation of clinical practice guidelines

APPENDIX J

NATIONAL HEALTH AND MEDICAL RESEARCH COUNCIL

STANDARDS FOR EXTERNALLY DEVELOPED GUIDELINES

Introduction

The National Health and Medical Research Council is keen to see the

development of clinical practice guidelines by expert bodies that have identified a

need for such guidelines. Where such guidelines are likely to have national

application, the Council encourages guideline developers to adhere to national

standards for development.

The Council is prepared to endorse guidelines that meet its standards for

guideline development. These standards are set out in this appendix.

Guideline developers should bear in mind that some clinical practice guidelines

already exist, both in Australia and overseas, so it may be possible to adapt

existing guidelines. Should this be done, developers still need to comply with the

standards set out in this appendix in order to obtain National Health and Medical

Research Council’s endorsement. If guidelines do not adhere to these standards or

principles they will not receive National Health and Medical Research Council’s

endorsement. It is expected that guidelines endorsed by the Council would be of

national significance and not simply relate to a local matter.

In the first instance, it would be expected that external guideline developers

would notify the Council of their intention to develop guidelines. Developers

would then be advised of the expected National Health and Medical Research

Council’s standards and a situation where two separate groups are developing

similar guidelines could be avoided (where this information is known to the

Council). If the guideline development process is expected to deviate significantly

from National Health and Medical Research Council’s standards, guideline

developers are encouraged to discuss their requirements with the Council.

Developers are asked to submit a final draft of the guidelines—not a final, printed

‘glossy’—for the Council’s consideration.

National significance

Developers should give some supporting rationale explaining their choice of

topic. The underlying premise is that a topic would be of national significance

and may relate to topics such as the following:

• significant variations in clinical practice for the health care intervention;

• high cost;

• high health burden;

• evidence to support recommendations for practice; and

• an initial assessment of available literature or known guidelines.

Standards for externally produced guidelines

1. Multidisciplinary panel

It is essential that guideline development be overseen by a multidisciplinary

panel. Such a panel will consist of expert clinicians in the field, experts in

research and health economics, public policy analysts, allied health practitioners

and consumer representatives.

In keeping with the multidisciplinary approach, the guideline development

process should show evidence of wider consultation, which is also consistent with

the spirit of the National Health and Medical Research Council’s requirements for

public consultation.

2. Scope and coverage of guidelines

The purpose of the guidelines, including their scope and target audience, should

be clearly defined.

3. Identification of health outcomes

Outcome measures need to be clearly identified—these can be clinically or

socially defined (see Appendix D for a suggested typology).

4. Systematic review of evidence

A systematic review of evidence should include clear statements of the following:

• the aim or aims of the review;

• data sources used to identify evidence;

• data-collection methods—with inclusion and exclusion criteria; and

• characteristics of the primary studies—including methodological quality.

The review should also include:

• a synthesis of the evidence, using meta-analytic techniques where applicable;

• ratings of the level of quality, relevance and strength of the evidence (see

Appendixes A and B); and

• a summary of the evidence, including its implications for future research and

clinical practice.

Appendix

68 A guide to the development, implementation and evaluation of clinical practice guidelines

5. Consultation

There should be adequate consultation in the development of the guidelines.

Consultation can vary—for example, targeted consultations, public advertisements,

local workshops, and other face-to-face meetings—and developers can select the

most appropriate mechanisms.

In considering externally developed guidelines, the Health Advisory Committee of

the National Health and Medical Research Council will require developers to

provide a short report of the consultation process, showing who was consulted

and how comments were handled.

6. Guidelines

Quality of evidence

• The actual guidelines should include supporting statements of the quality of

evidence for key decision points (see Appendix B).

• Where scientific opinion or consensus is lacking this should be stated and

details of the differences in expert opinion should be provided.

Flexibility and adaptability

• Guidelines should be flexible and adaptable and take account of differences

in target populations, geographical and clinical settings, resources, and

patients’ values and preferences.

Resource considerations

• Cost information that would assist, where necessary or appropriate, in

choosing between interventions should be provided; this may involve cost-

effectiveness studies.

7. Revising

The guidelines should contain a statement to the effect that they are based on the

best available knowledge at a specific time.

There should be regular monitoring for health outcomes and new information.

The need for review should be stated explicitly, noting that a guideline should be

revised at least once every three years. Preferably, the guidelines should be

reviewed annually.

8. Implementation plan

An implementation plan should be developed. It should target all potential users

and take into account questions of privacy and confidentiality, different patient

care settings, and funding and organisational barriers.

9. Evaluation plan

A comprehensive evaluation plan should be developed, identifying how and

when the guidelines should be reviewed. The plan should include the

mechanisms for revising the guidelines set out in Section 7.

Consideration should be given to probable costs, the involvement of interested

parties, and the management of the evaluation.

Proposed measures for evaluating both the guideline development process and

the outcomes of guideline implementation should be included.

10. Endorsement

Externally developed documents would receive time-limited National Health and

Medical Research Council endorsement (similar to hospital accreditation

processes), the aim being to encourage external bodies to regularly review,

evaluate and revise their documents and recommendations. This principle can be

seen as a system of continuing quality assurance.

11. Legal considerations

In keeping with advice provided by the Attorney-General’s Department, a legal

disclaimer should be included in all guidelines. The following is a suggested form

of words:

This document is a general guide to appropriate practice, to be followed

only subject to the medical practitioner’s judgment in each individual case.

The guidelines are designed to provide information to assist decision making

and are based on the best information available at the date of publication.

12. Approvals process

The approvals process will involve the following steps:

• notification by the guideline developer to the Secretariat of the Health

Advisory Committee of the National Health and Medical Research Council.

This notification will include provision of information noted in the

introductory section and Section 1 of this appendix;

Appendix

70 A guide to the development, implementation and evaluation of clinical practice guidelines

• registration of ‘notice to develop’ by the Health Advisory Committee

Secretariat;

• regular progress reports from the developer to the Health Advisory Committee

Secretariat;

• draft guidelines submitted to the Health Advisory Committee, which will

review the guideline development process. The maximum time from the

notice of intent, to provision of the final draft, should not exceed 18 months;

• establishment of a review panel to review the guideline development process

and the content of guidelines. The review panel will be established by the

Health Advisory Committee. The number of members will be limited to a

maximum of five people although, in exceptional circumstances, the Chair of

the Health Advisory Committee may approve a variation to this;

In particular, the review panel will consider submissions received in relation

to the second stage of the consultation process and provide advice to the

sponsoring organisation and the Health Advisory Committee about

amendments that should be made to the draft guidelines. The review panel

will be strictly time-limited in its activities. It will be multidisciplinary,

recognising the intended audience for the guidelines and will bring together

expertise in systematic review and the subject matter under consideration; in

addition, one member of the review panel will be a consumer representative;

• provision of the review panel’s report to the Health Advisory Committee. Note

that, other than in exceptional circumstances, the review process should not

exceed two months; and

• consideration by the Health Advisory Committee and a response provided to

the guideline developer. If positive, the Committee will recommend to the

National Health and Medical Research Council that the guidelines be

endorsed.

13. Assessment by the review panel

Draft guidelines submitted will be assessed against the criteria just specified. They

will have to meet all criteria to be recommended for endorsement by the National

Health and Medical Research Council.

Draft guidelines that do not meet one or more of the criteria will be returned to

the guideline developer along with written reasons for the return.

14. Appeals process

Should the guideline developer disagree with the review panel’s assessment, the

developer is encouraged to write to the Chair of the Health Advisory Committee,

setting out the reasons for disagreement.

It would be useful if the guideline developer included any additional evidence or

information that would help the Committee respond.

The Health Advisory Committee Secretariat will provide advice on this process.

15. Correspondence

All correspondence should be addressed to:

Health Advisory Unit

Office of the National Health and Medical Research Council

Department of Health and Aged Care

GPO Box 9848 (MDP 50)

Canberra ACT 2601

Appendix

72 A guide to the development, implementation and evaluation of clinical practice guidelines

‘That strategy of action which maximises the potential

health benefits valued by informed individuals or

populations after considering the likely outcomes, their

probabilities and their costs’ (Harvey 1991, p. 19).

‘Systematically developed statements to assist practitioner

and patient decisions about appropriate health care for

specific clinical circumstances’ (Field & Lohr 1990, p. 8).

The intervention, drug or therapy most likely to be used as

a replacement or substitute by practitioners or prescribers.

Clinical practice guidelines based on a guidelines

consensus of expert opinion.

In this document, a broad term covering a variety of

consumer groups with general and specific health interests

and self-help and support groups.

A comparative analysis of alternative courses of action in

terms of both costs and consequences; includes cost

effectiveness, cost-benefit and cost utility analyses.

‘The extent to which an intervention does more good than

harm for the patient when used under “normal”

circumstances’ (Last 1987, p. 299).

‘The extent to which an intervention does more good than

harm for the patient when applied under “ideal”conditions’

(Last 1987, p. 299).

Clinical practice guidelines based on a systematic review

of scientific data.

‘A person who by reason of experience and knowledge is

entitled to give an opinion upon facts ascertained by

himself or herself’ (MacNalty 1978).

‘A change in the health of an individual, a group of people

or population which is attributable to an intervention or a

series of interventions’ (AHMAC 1993).

‘Systematically developed statements that can be used to

assess the appropriateness of specific health care

decisions, services and outcomes’ (Field & Lohr 1990,

p. 8).

GLOSSARY

Appropriate care

Clinical practice

guidelines

Comparator

Consensus-based

guidelines

Consumer groups

Economic appraisal

Effectiveness

Efficacy

Evidence-based

guidelines

Expert

Health outcome

Medical review criteria

A systematic review that employs statistical methods to

combine and summarise the results of several studies’

(Cook & Guyatt 1994, p. 1327).

Statements formulated but in relation to which there is

disagreement between the parties concerned.

In this document, people using any clinical service.

In this document, a systematic approach to assessing the

care provided, identifying opportunities for improvement

and providing a mechanism for such improvement. Quality

assurance should consider medical records, consumer

interest and cooperation, administrative processes, staff

development, equipment selection and maintenance, and

clinical practice guideline development, dissemination,

implementation, evaluation and revision.

Research study where ‘participants are allocated at random

to receive one of two or more alternative forms of care’

with the aim of creating ‘unbiased treatment groups for

comparison’ (Advisory Group on Health Technology

Assessment 1992, p. 10).

‘Guidelines are reproducible and reliable:

1. if—given the same evidence and methods for guidelines

development—another set of experts produces

essentially the same statements; and

2. if—given the same circumstances—the guidelines are

interpreted and applied consistently by practitioners

(other parties)’ (Field & Lohr 1992, p. 30).

‘Authoritative statements of:

1. minimum levels of acceptable performance or results;

2. excellent levels of performance or results; or

3. the range of acceptable performance of results’ (Field &

Lohr 1990, p. 8).

Application of a rigorous scientific approach to the

preparation of a review article.

‘Practice guidelines are valid if, when followed, they lead

to the health and cost outcomes projected for them, other

things being equal’ (Field & Lohr 1990, p. 10).

Glossary

Meta-analysis

Non–consensus

based statements

Patients

Quality assurance

Randomised

controlled trial

Reliable/reproducible

guidelines

Standards of quality

Systematic review

Valid guidelines

74 A guide to the development, implementation and evaluation of clinical practice guidelines

BIBLIOGRAPHY

Advisory Group on Health Technology Assessment (1992) ‘Draft Guidelines on

Health Technology Assessment for the Australian Health Technology Advisory

Committee’ National Health Services, London. Adapted from the report,

Assessing the Effects of Health Technologies: principles, practice and proposals.

AHMAC 1993. ‘Health Outcomes Seminar: AHMAC Sunshine statement’,

Communique, February, unpublished.

Anderson, T.F. & Mooney, G. (1990) ‘Medical Practice Variations: where are we?’

in T.F. Anderson & G. Mooney (eds), The Challenges of Medical Practice

Variations, Macmillan, London, pp. 1–15.

Antman, E.M. et al. (1992) ‘A Comparison of Meta-analyses of Randomised Control

Trials and Recommendations of Clinical Experts’, Journal of the American

Medical Association, vol. 268, pp. 240–8.

Bardsley, M. & Cole, J. (1991) ‘Measured Steps to Outcomes’, Health Service

Journal, 17 October, pp. 18–20.

Brown, J.B., Shye, D. & McFarland, B. (1995) ‘The Paradox of Guideline

Implementation: how AHCPR’s depression guideline was adapted at Kaiser

Permanente Northwest Region’, Journal on Quality Improvement, vol. 21,

pp. 5–21.

Chren, M-M. & Landefeld, C.S. (1994) ‘Physicians’ Behaviour and their Interactions

with Drug Companies: a controlled study of physicians who requested

additions to a hospital drug formulary’, Journal of the American Medical

Association, vol. 271, no. 9, pp. 684–9.

Clover, K. et al. (1995) Project 5: the development of consensus guidelines, Hunter

Access to Surgery Research Project, Hunter Area Health Service, May.

Conroy, M. & Shannon, W. (1995) ‘Clinical Guidelines: their implementation in

general practice’, British Journal of General Practice, vol. 45, pp. 371–5.

Cook, D.J. & Guyatt, G.H. (1994) ‘The Professional Meta-analyst: an evolutionary

advantage’, Journal of Clinical Epidemiology, vol. 47, pp. 1327–9.

Davis, D.A., Thomson, M.A., Oxman, A.D. et al. (1995) ‘Changing Physician

Performance: a systematic review of the effect of continuing medical

education strategies’, Journal of the American Medical Association, vol. 274,

pp. 700–6.

Donabedian, A. (1992) ‘The Lichfield Lecture: quality assurance in health care—

consumers’ role’, Quality in Health Care, vol. 1, pp. 1–5.

Donabedian, A. (1992) ‘The Role of Outcome in Quality Assessment and

Assurance’, Quality Review Bulletin, November, pp. 356–60.

Drummond, M.F. (1980) Principles of Economic Appraisal in Health Care, Oxford

University Press, Oxford, UK.

Dunn, E., Norton, P. et al. (1994) ‘Disseminating Research/Changing Practice’,

Research Methods for Primary Care, vol. 6, Sage Publications, Thousand Oaks,

US.

Effective Health Care Bulletin (EHCB) (1994) ‘Implementing clinical practice

guidelines. Can guidelines be used to improve practice?’, Effective Health Care

Bulletin, no. 6, University of Leeds, UK.

Ellwood, P.M. (1988) ‘Shattuck Lecture: outcomes management—a technology of

patient experience’, New England Journal of Medicine, vol. 318, pp. 1549–56.

Field, M.J. & Lohr, K.N. (eds) (1992) Guidelines for Clinical Practice: from

development to use, Institute of Medicine, National Academy Press,

Washington, DC.

Field, M.J. & Lohr, K.N. (eds) (1990) Clinical Practice Guidelines: directions for a

new program, Institute of Medicine, National Academy Press, Washington, DC.

Fisher, M. (ed.) (1989) Guide to Clinical Preventive Services: an assessment of the

effectiveness of 169 interventions, US Preventive Services Task Force, Williams

and Wilkins, Baltimore, Maryland.

Forrest, D., Hoskins, A. & Hussey, R. (1996) ‘Clinical Guidelines and their

Implementation’, Postgraduate Medical Journal, vol. 72, pp. 19–22.

Greenhalgh, T. (1997) ‘Papers that Summarise Other Papers—systematic reviews

and meta-analysis, British Medical Journal, vol. 315, pp. 672–5.

Grimshaw, J.M. & Russell, I.T. (1994) ‘Achieving Health Gain through Clinical

Guidelines II: ensuring guidelines change medical practice’, Quality in Health

Care, vol. 3, pp. 45–52.

Grimshaw, J.M. & Russell, I.T. (1993) ‘Effect of Clinical Guidelines on Medical

Practice: a systematic review of rigorous evaluation’, The Lancet, vol. 342,

pp. 1317–22.

Grimshaw, J.M., Freemantle, N., Wallace, S. et al. (1995) ‘Developing and

Implementing Clinical Practice Guidelines’, British Medical Journal, vol. 4,

pp. 55–64.

Haines, A. & Feder, G. (1992) ‘Guidance on Guidelines—writing them is easier

than making them work’, British Medical Journal, vol. 305, pp. 785–6.

Bibliography

76 A guide to the development, implementation and evaluation of clinical practice guidelines

Harvey, R. (1991) Making It Better, National Health Strategy background paper

no. 8, Department of Health and Family Services, Canberra.

Health Ministers Forum (1994) National Health Policy, State Health publication

no. P(P) 94-156.

Health Services Research Group (1992) Outcomes and the Management of Health

Care, Canadian Medical Association Journal, vol. 147, pp. 1775–80.

Johnston, M.E., Langton, K.B., Haynes, B. et al. (1994) ‘Effect of Computer-based

Clinical Support Systems on Clinician Performance and Patient Outcome: a

critical appraisal of research’, Annals of Internal Medicine, vol. 120, pp. 135–

42.

Kane, R.L. & Lurie, N. (1992) ‘Appropriate Effectiveness: a tale of carts and

horses’, Quality Review Bulletin, vol. 18, no. 10, pp. 322–6.

Last, J.M. (1987) Public Health and Human Ecology, Appleton & Lange, East

Norwalk, Connecticut.

Lomas, J. (1994) ‘Teaching Old (and not so old) Docs New Tricks’, in E. Dunn,

P. Norton et al. (eds), Disseminating Research/Changing Practice, Research

Methods for Primary Care, vol. 5, Sage Publications, Thousand Oaks, US.

Lomas, J. (1993) ‘Retailing Research: increasing the role of evidence in clinical

services for childbirth’, The Millbank Quarterly, vol. 71, no. 3, pp. 439–75.

Lomas, J. (1991) ‘Words without Action? The Production, Dissemination and

Impact of Consensus Recommendations’, Annual Review of Public Health,

vol. 12, pp. 41–65.

Lomas, J. et al. (1989) ‘Do Practice Guidelines Guide Practice? The Effect of a

Consensus Statement on the Practice of Physicians’, New England Journal of

Medicine, vol. 321, pp. 1306–11.

Lomas, J., Enkin, M., Anderson, G.M. et al. (1991) ‘Opinion Leaders vs Audit and

Feedback to Implement Practice Guidelines: delivery after previous caesarean

section’, Journal of the American Medical Association, pp. 2202–7.

Lomas, J. & Haynes, R.B. (1988) ‘A Taxonomy and Critical Review of Tested

Strategies for the Application of Clinical Practice Recommendations: from

“official” to “individual” clinical policy’, American Journal of Preventive

Medicine, vol. 4, pp. 77–97.

MacNalty, A.S. (ed.) (1978) Butterworth’s Medical Dictionary, 2nd edn

(M. Critchley, Editor-in-chief), Butterworths, London.

May, F., Gilbert, A., Hurley, E. et al. (1993) ‘A Drug and Therapeutics Information

Service for Community Medical Practitioners’, Australian Prescriber, vol. 16,

suppl. 1, pp. 49–51.

McCormick, K.A., Moore, S.R. & R.A. Siegel (eds) (1994) Clinical Practice

Guideline Development: methodology perspectives, Publication no. 95-0009,

Agency for Health Care Policy and Research, US Department of Health and

Human Services.

McGlyn, E.A., Kosecoff, J. & Brook, R.H. (1990) ‘Format and Conduct of

Consensus Development Conferences’, International Journal of Technology

Assessment in Health Care, vol. 6, pp. 450–69.

McPherson, K. (1990) ‘Why do Variations Occur?’ in T.F. Andersen and G. Mooney

(eds), The Challenges of Medical Practice Variations, Macmillan, London,

pp. 16–35.

Mulrow, C.D. (1994) ‘Rationale for Systematic Reviews’, British Medical Journal,

vol. 309, pp. 597–9.

Mulrow, C.D. (1987) ‘The Medical Review Article: state of the science, Annals of

Internal Medicine, vol. 106, pp. 485–8.

National Health and Medical Research Council (NHMRC) (1995) Guidelines for the

Development and Implementation of Clinical Practice Guidelines, 1st edn,

Australian Government Publishing Service, Canberra.

NSW Department of Health (1995) Evaluation Checklist for Evidence-based

Guidelines, 1st edn, Department of Health, North Sydney.

Opit, L.J. (1991) ‘The Measurement of Health Service Outcomes’, in Holland et al.

(eds) Oxford Text Book of Public Health, 2nd edn, vol. 3, Oxford University

Press, Oxford, pp. 159–307.

Orlowski, J.P. & Wateska, L. (1992) ‘The Effects of Pharmaceutical Firm

Enticements on Physician Prescribing Patterns: there’s no such thing as a free

lunch’, Chest, vol. 102, no. 1, pp. 270–3.

Oxman, A.D., Thomson, M.A., Davis, D.A. et al. (1995) ‘No Magic Bullets: a

systematic review of 102 trials of interventions to help health care

professionals deliver services more effectively or efficiently’, Journal of the

Canadian Medical Association, vol. 153, pp. 1423–31.

Royal Australian College of General Practitioners (RACGP) (1995) ‘Adjudication

Criteria for Continuing Medical Education’, RACGP, unpublished.

Bibliography

78 A guide to the development, implementation and evaluation of clinical practice guidelines

Royal College of General Practitioners (RCGP) (1995) The Development and

Implementation of Clinical Guidelines, Report of the Clinical Guidelines

Working Group, RCGP, London.

Sackett, D.L. (1995) ‘Applying Overviews and Meta-analyses at the Bedside’,

Journal of Clinical Epidemiology, vol. 48, pp. 61–6.

Sackett, D.L. (ed.) (1994) The Cochrane Collaboration Handbook, (CD Rom) –

Update Software, Oxford, UK, Issue 1 1999

Sackett, D.L., Rosenberg, W.C., Muir Gray, J.A. et al. (1996) ‘Evidence-based

Medicine: what it is and what it isn’t’, British Medical Journal, vol. 312,

pp. 71–2.

Scottish Intercollegiate Guidelines Network (SIGN) (1995) Criteria for Appraisal

for National Use, Pilot edition, SIGN, September.

Simes, R.J. & Glasziou, P.P. (1992) ‘Meta-analysis and Quality Evidence in the

Economic Evaluation of Drug Trials’, Pharmaco Economics, vol. 1, pp. 282–91.

Slavin, R.E. (1995) ‘Best Evidence Synthesis: an intelligent alternative to meta-

analysis’, Journal of Clinical Epidemiology, vol. 48, pp. 9–18.

Stocking, B. (1993) ‘Implementing the Findings of Effective Care in Pregnancy and

Childbirth in the United Kingdom, The Millbank Quarterly, vol. 71, pp. 497–

521.

Todd, C. (1995) ‘Is There a Role for Academic Detailing in Today’s Healthcare

System?’ Pharmacoresources, vol. 38, pp. 3–5.

UK Clearing House for Information on the Assessment of Health Outcomes (1993)

Issues in Health Outcome Measurement: outcomes briefing’, Introductory issue,

Nuffield Institute for Health, Leeds, Spring.

US Agency for Health Care Policy and Research (AHCPR) (1993) Program Note:

clinical practice guidelines development, AHCPR publication no. 93-0023, US

Department of Health and Human Services.

US Agency for Health Care Policy and Research (AHCPR) (1991) Report to

Congress: progress of research on outcomes of health care services and

procedures, AHCPR publication no. 91-0004, US Department of Health and

Human Services.

Wensing, M. & Grol, R. (1994) ‘Single and Combined Strategies for Implementing

Changes in Primary Care: a literature review’, International Journal of Quality

in Health Care, vol. 6, pp. 115–32.

Wise, C.G. & Billi, J.E. (1995) ‘A Model for Practice Guideline Adaptation and

Implementation: empowerment of the physician’, Journal on Quality

Improvement, vol. 21, pp. 465–76.

Woolf, S.H. (1992) ‘Practice Guidelines, a New Reality in Medicine—11: methods

of developing guidelines’, Annals of Internal Medicine, vol. 152, pp. 946–52.

Woolf, S.H. (1992) ‘Practice Guidelines, a New Reality in Medicine—1: recent

developments’, Annals of Internal Medicine, vol. 150, pp. 1811–18.

Bibliography

80 A guide to the development, implementation and evaluation of clinical practice guidelines

Blank facing page to NHMRC back page

The National Health and Medical Research Council

The National Health and Medical R esearch Council (NHMR C) is a statutory authority within the por tfolio

of the Commonw ealth Minister for Health and Aged car e, established b y the National Health and Medical

Research Council Act 1992. The NHMRC advises the A ustralian community and Commonw ealth; State and

Territory Governments on standar ds of individual and public health, and suppor ts research to improve those

standards.

The NHMRC advises the Commonw ealth Government on the funding of medical and public health r esearch

and training in A ustralia and suppor ts many of the medical adv ances made b y Australians.

The NHMRC also develops guidelines and standar ds for the ethical conduct of health and medical r esearch.

The Council comprises nominees of Commonw ealth, State and Territory health authorities, pr ofessional and

scientific colleges and associations, unions, univ ersities, business, consumer gr oups, welfare organisations,

conservation groups and the A boriginal and Torres Strait Islander Commission.

The Council meets four times a y ear to consider and make decisions on r eports prepared by committees and

working parties follo wing wide consultation on the issue under consideration.

A regular publishing pr ogram ensures that Council ’s recommendations ar e widely available to governments,

the community , scientific, industrial and educational gr oups.

The Council publishes extensiv ely in the follo wing areas:

• Aged car e • Infection contr ol

• Child health • Men’s health

• Clinical practice • Mental health

• Communicable diseases • National Health and M edical Research Council

• Dentistr y • Nutrition

• Drugs and poisons • Public Health

• Drugs and substance abuse • Research

• Environmental health • Technology Assessment

• Ethics • Women’s health

A list of curr ent publications is av ailable from:

The Publications O fficer

ONHMRC

MDP 50

GPO Box 9848

Canberra ACT 2601

Phone: (02) 6289 1430 (24 hour answering machine)

Toll free: 1800 020 103 (toll-free)

Fax: (02) 6289 1351

E-mail: nhmr[email protected] v.au

Internet: http://www.health.gov.au/nhmr c