Untitled

Document status:

Final Draft

Allen + Clarke has been

independently certified as

compliant with ISO9001:2015

Quality Management Systems

Version and date:

V 2.4 09/11/2018

Author(s):

Jacqui Haggland, Vince Catherwood, Pounamu

Aikman

Filing Location:

W:\MFAT\Evaluation for the Sāmoa Education

Sector Program\Phase 3 Analysis and Reporting

Peer / technical

review:

Mathea Roorda, Ned Hardie-Boys

Verification that QA

changes made:

Jacqui Haggland

Proof read:

Ryan Angus

Formatting:

Jacqui Haggland

Design

Salman Abbasnejad

Final QA check and

approved for release:

Jacqui Haggland

Cover photographs: Top left and bottom right photos are from the ‘Paul Soliai Foundation Sāmoa Tour 2014’,

by the U.S. Embassy in New Zealand, used under CC BY-ND 2.0. All other photos are those of Allen + Clarke.

Evaluation of the Sāmoa Education Sector Support Programme – Final Report iii

ABOUT ALLEN + CLARKE

Allen and Clarke Policy and Regulatory Specialists Limited (Allen + Clarke) is a consultancy firm

based in Wellington, New Zealand. We specialise in policy and programme development and

implementation, research and evaluation, business change, operational management and risk, and

secretariat and programme support services. A key component of our work is undertaking

reviews and developing and implementing policies that improve outcomes for the public.

Founded in 2001, the company is owned and managed by senior staff and has a team of

approximately forty-eight senior evaluation and policy practitioners, analysts and project support

staff. Our company works extensively for a range of government agencies in New Zealand, and

international clients and non-government organisations in the Pacific and Asia. More information

about our work can be found on our website: www.allenandclarke.co.nz

Jacqui Haggland, who led this work for Allen + Clarke, has significant experience in policy, research

and evaluation. Jacqui has a background in secondary teaching and spent four years working for

the New Zealand government in adult literacy and numeracy operational policy. Jacqui’s

evaluation and policy work has supported improvements in education (including early childhood

and international education in New Zealand), international development, public health, social

development, and sport. Jacqui’s Pacific experience has included project management of the

evaluation of the New Zealand Aid Programme’s support to the Curriculum Development Division

of the Ministry of Education and Human Resources Development in the Solomon Islands, as well

as conducting in-country legislative review and stakeholder consultation in the Cook Islands

legislation on behalf of WHO in the area of tobacco control.

Vince Catherwood joined Allen + Clarke for this assignment. Vince has had an extensive

background of experience and expertise over the last seventeen years in development work in the

education sector through his company Vince Catherwood & Associates Ltd. He has a background

of assisting in the development and monitoring of country-wide education strategic plans for

several Pacific countries. He has well-developed evaluation and monitoring skills, having

conducted a number of education reviews for both the New Zealand Ministry of Foreign Affairs

and Trade and the Australian Agency for International Development. Vince has also worked

previously as a consultant in Sāmoa and in nine other Pacific countries on education assignments.

ACKNOWLEDGEMENTS

Allen + Clarke is grateful to the evaluation participants who made themselves available for

interviews, many at short notice and during a busy time of the year. Your experiences and ideas

shared were invaluable to the evaluation process. We are thankful to the Ministry of Foreign

Affairs and Trade and the Department of Foreign Affairs teams for their time, enabling access to

information, and for their participation in the evaluation discussions.

DISCLAIMER

Data referred in this report were provided by various stakeholders. The evaluation team was not

able to validate these data and are not responsible for any misrepresentations in the data

presented.

CONTENTS

ABOUT ALLEN + CLARKE III

ACKNOWLEDGEMENTS III

DISCLAIMER III

CONTENTS IV

GLOSSARY V

TABLES VI

FIGURES VI

EXECUTIVE SUMMARY 1

Recommendations 5

1. BACKGROUND AND CONTEXT 6

1.1. About the ESSP 6

1.2. Evaluation purpose 7

1.3. Earlier independent review of the ESP and ESSP 7

1.4. Evaluation methodology 7

2. FINDINGS 9

2.1. Relevance 9

2.2. Effectiveness 11

2.3. Efficiency 23

2.4. Impact 27

2.5. Sustainability 29

2.6. Cross-cutting issues 32

3. LESSONS LEARNED 34

4. EVALUATION CONCLUSIONS 35

5. RECOMMENDATIONS 36

REFERENCES 37

APPENDIX A: TERMS OF REFERENCE 40

APPENDIX B: STAKEHOLDER ENGAGEMENT LIST 53

APPENDIX C: AIDE MEMOIRE – EVALUATION OF THE SĀMOA EDUCATION SECTOR SUPPORT

PROGRAMME – INTERIM FINDINGS 57

APPENDIX D: RESPONSE TO ESP MID-TERM REVIEW 62

APPENDIX E: KEY PERFORMANCE INDICATORS: EDUCATION SECTOR SUPPORT PROGRAMME 65

APPENDIX F: ESSP FUNDING TIMELINES 68

APPENDIX G: LINKAGES BETWEEN KEY EVALUATION SUB-QUESTIONS AND SECTIONS OF THE DRAFT

EVALUATION REPORT 69

APPENDIX H: STUDENT ACHIEVEMENT IN LITERACY AND NUMERACY 73

Evaluation of the Sāmoa Education Sector Support Programme – Final Report v

GLOSSARY

ACD Aid Coordination Division

ACEO Assistant Chief Executive Officer

CEO Chief Executive Officer

CMAD Curriculum Materials and Assessment Division

CT Consumer Tax DAC

DAC Development Assistance Committee (of the OECD)

DFAT Department of Foreign Affairs and Trade (Australia)

DP Development Partner

ECE Early Childhood Education

EMIS Education Management Information System

ESAC Education Sector Advisory Committee

ESCD Education Sector Coordination Division

ESP Education Sector Plan

ESP ll Education Sector Programme ll

ESSP Education Sector Support Programme

ESWG Education Sector Working Group

JFA

KPI

Joint Funding Agreement

Key Performance Indicator

GoS Government of Sāmoa

IA Implementing Agency (MESC, SQA and NUS)

IVP Independent Verification Process

M&E Monitoring and Evaluation

MCIL Ministry of Commerce, Industry and Labour

MESC Ministry of Education, Sports and Culture

MFAT Ministry of Foreign Affairs and Trade (New Zealand)

MoF Ministry of Finance (GoS)

MoU Memorandum of Understanding

MSS Minimum Service Standards

MTEF Medium-Term Expenditure Framework

NTDF National Teacher Development Framework

NUS National University of Sāmoa

OECD Organisation for Economic Co-operation and Development

PSC Public Service Commission

PSET Post-Secondary Education and Training

SAT/WST Sāmoan Tala

SBS Sector Budget Support

SEN Student Enrolment Number

SOD School Operations Division

SPELL Sāmoa Primary Education Literacy Level Test

SQA Sāmoa Qualifications Authority

SSFGS Sāmoa School (Primary) Fee Grant Scheme

SWAp Sector Wide Approach

TA Technical Assistance

TDAD Teacher Development and Advisory Division

TOC Team of Consultants

TVET Technical and Vocational Education and Training

TABLES

Table 1: OECD DAC Criteria and associate Sāmoa ESSP KEQs 8

Table 2: ESSP Budget and Actual Funds Received, 2015-2019 Currency: Sāmoan tala, rounded to $000s

Table 3: Percentage Achievement against KPIs 1, 2, 4 and 5 by Gender - 2018 IVP report 32

Table 4: MTR recommendations - have they been addressed and what is their ongoing relevance 62

Table 5: SPELL 1 Percentage of Students at Risk - at Beginner level 74

Table 6: SPELL 2 Percentage of Students at Risk - at Beginner level 74

Table 7: SPECA: Percentage of students at the ‘beginning’ level/ level 1 75

Table 8: Year 12 SSC Results: Pass Rates 75

Table 9: Year 13 SSLC Results 76

FIGURES

Figure 1: Advantages and Disadvantages of sector budget support funding modality 16

Figure 2: Alignment of GoS Budget and ESSP timelines 68

Figure 3: Timing of ESSP funding availability 68

Evaluation of Sāmoa Education Sector Support Programme

An evaluation of the Sāmoa Education Sector Support

Programme (ESSP) was undertaken from August to

November 2018. The purpose of the evaluation was

to assess the relevance, effectiveness, efﬁciency,

impact, and sustainability of the ESSP on the broader

Sāmoan education sector, to examine cross-cutting

issues such as gender and inclusive education, and

to inform any further commitments by Australia and

New Zealand to the programme. An Evaluation Plan

was developed and approved by the New Zealand

Ministry of Foreign Affairs and Trade (MFAT) in the

ﬁrst phase, followed by two weeks of ﬁeld work in

Sāmoa in the second phase, and the development of

the draft and ﬁnal reports in the third phase.

Executive

Summary

The evaluation found that the ESSP was broadly

relevant to the Sāmoa education sector. The ESSP has

a clear strategic framework aligned with Australian/

New Zealand international development policy

and Government of Sāmoa (GoS) development

objectives, and the educational objectives of

education providers. The outcomes sought in the

Education Sector Plan 2013-2018 (ESP) and included

in the ESSP, set very ambitious targets to achieve

within a relatively short timeframe. The broad scope

of the ESP and the ESSP encompasses the entire

education sector, which justiﬁed the involvement of

the three Implementing Agencies (IAs): the Ministry

of Education, Sports and Culture (MESC), the Sāmoa

Qualiﬁcations Authority (SQA), and the National

University of Sāmoa (NUS).

RelevancePurpose

The intended outcomes of ESSP, which relate to

improved access to, and participation in, education,

improved learning outcomes, and improved access to

employment, all remain fundamentally relevant and

important. The additional activities made possible by

the ESSP investment have addressed the priorities of

target groups such as schools and other education

providers, and have targeted key issues facing the

education sector. However, the connection between

theory and practice, and therefore between planning

and expected outcomes, was overly ambitious in the

original design of the programme.

The collection of data, management of information,

and reporting against key performance indicators

(KPIs) have all improved to some extent during the

course of the ESSP. Because there are inefﬁciencies

in the way information is managed within the

Sāmoa education system, there is a need for a more

integrated and comprehensive data management

system that is used by all staff in the IAs. A monitoring

and evaluation (M&E) framework is being used to

report on a much broader range of indicators than

are included in the ESSP (for example, additional

ESP indicators). The M&E Framework needs to be

streamlined, simpliﬁed, and updated more regularly.

Similarly, the joint funding agreement (JFA) between

GoS and the donors requires the GoS to update the

Medium-Term Expenditure Framework (MTEF) in

February each year as part of the monitoring process.

Governance

and

Coordination

The governance structure of ESSP, the Education

Sector Advisory Committee (ESAC) and the Education

Sector Working Group (ESWG) is effective. There

were problems with the implementation of the

governance structure in the ﬁrst two years, but

signiﬁcant progress has been made recently. The

Education Sector Coordination Division (ESCD) is

now better placed to support the ESWG to provide

the required analysis before reports are forwarded

to the ESAC for decision-making. Major factors

enabling or constraining the achievement of ESSP

objectives include the commitment and hard work of

IA staff (an enabling factor), and the weak capacity

of staff to monitor, analyse and report on data, the

lack of timeliness in reporting, and staff turnover

(three constraining factors).

Effectiveness

Monitoring

and

Evaluation

Evaluation of Sāmoa Education Sector Support Programme

A mixed-mode sector budget support modality has

been used as a funding mechanism. The evaluation

found that this budget support modality is broadly

appropriate and its retention in the future was

supported by the sector. Modiﬁcations would be

needed to some details of the funding modality if

the programme was to continue in the future. The

misalignment between the ESSP timelines and GoS

Budget timelines needs to be addressed, so that one

process (based on the GoS Budget processes) is

used. A weakness in the programme initiation was the

delay by Development Partners (DPs) in establishing

the programme. However, the MFAT and Australian

Department of Foreign Affairs and Trade (DFAT)

posts in Apia have had close involvement with ESSP,

including participation in the governance framework,

and their staff have provided useful policy dialogue

and advice. Feedback from the interviews with

education sector participants also suggested there

were delays by the Ministry of Finance (MoF) in

releasing funding to the IAs which may have had an

impact on the underspend.

The ﬁxed tranche disbursement of the budget support

component of the funding modality (currently 70

percent) is appropriate, although the evaluators

question the effectiveness of reducing the amount of

the ﬁxed tranche component, as funding predictability

is important for planning purposes. In this context,

the process indicators associated with the ﬁxed

tranche disbursement, and the KPIs associated with

the performance-based funding tranche, need to

Efﬁciency and

the Funding

Modality

be reviewed for the next iteration of the ESSP. The

Independent Veriﬁcation Process (IVP) was generally

supported, and should continue in any future ESSP,

although those interviewed recognised that it had

taken time for the education sector in Sāmoa to

appreciate that timely data collection, analysis and

reporting was a pre-requisite to success in using a

performance-based system of funding. In this respect,

although there was considerable early communication

about the sector budget support modality, some of

the initial information may have been misunderstood

or misinterpreted by the education sector in Samoa,

and, with the advantage of hindsight, it is clear that

communications by both donors and the GoS could

have been better.

The implementation of ESSP by the GoS has been

less efﬁcient than desired, but improvements are

occurring. Results against the nine KPIs linked to

performance-linked contributions have shown some

improvement over three years, although considerable

scope remains for efﬁciency improvements. The need

for training and capacity development to address

weaknesses in staff capacity is an issue across the

education sector. Investment in a programme of

staff capacity development is needed to improve

the analysis of data and reporting against key

performance objectives in order to provide evidence

that progress towards long-term outcomes is being

made.

Cross-cutting issues such as gender equality and

inclusive education (IE) have been identiﬁed and

addressed with mixed success in the ESSP. The

performance of boys has fallen behind that of girls

and is an ongoing issue that needs to be carefully

monitored. The use of ring-fenced funding within the

ﬁxed tranche component of the ESSP to support IE

was successful in ensuring progress was made. The

use of technical assistance, supported by access to

specialist volunteers, was also helpful in supporting

staff to develop a programme of future work. The

incorporation of internationally agreed questions in

the annual school census to support more accurate

identiﬁcation of students with disabilities has led to a

much better information base with respect to children

with disabilities. However, this work is in its early

stages and the full impact is not yet known.

Impact and

Sustainability

The ESSP has provided IAs with a larger funding pool

from which they can address signiﬁcant initiatives such

as upskilling teachers or Post-Secondary Education

and Training (PSET) tutors, and these investments can

be expected to bring results over time. The efforts

to build staff capacity need to continue, with more

emphasis on using technical assistance to build

local capacity. In the short to medium-term, the

sustainability of the programme is dependent upon

continued support from MFAT and DFAT.

Cross-cutting issues

Evaluation of Sāmoa Education Sector Support Programme

All things considered, evaluation of the achievement

of ESSP goals after three years was too soon. A

further three to ﬁve years is needed in order to align

the intended outcomes of the ESP with the goals of

the ESSP, and to allow the Sāmoa education sector to

put into practice the learning that has been achieved

since 2016 and to experience the full beneﬁts of the

new sector wide approach that is being implemented.

While there has been positive learning about

management of the ESSP, there have also been

challenges. The evaluators remain concerned about

the low level of student achievement in Sāmoa, but

recognise that ESSP investment has targeted areas

such as teacher performance and the upgrading

of PSET tutors’ skills, and that these initiatives are

intended, over time, to lift student performance.

While access to education has been generally good

in Sāmoa in the past, it is apparent that education

agencies need to continue to monitor access to

education, and to improve their data collection and

analysis in this space.

While the desired achievement of outcomes over

the course of the ESSP have not eventuated, the

evaluators are conﬁdent that there is every prospect

that the planned outcomes related to improving

student learning and building the capacity of the

education system can be achieved over time. An

important area of ongoing focus is ensuring that

the measurement of progress towards identiﬁed

education outcomes is supported by the collection

and reporting of evidence against relevant KPIs.

Lessons Learned

Evaluation

Conclusions

Evaluation of Sāmoa Education Sector Support Programme

Recommendations

1 Continue to use a sector budget

support modality to deliver any future ESSP-

type funding with the following conditions:

better alignment between the ESSP reporting

and planning timelines and the established

timelines of the GoS Budget process;

expert guidance in the development of

performance indicators for any future ESP,

and selection for inclusion in an ESSP,

to ensure the KPIs are SMART (speciﬁc,

measurable, achievable, relevant and time-

bound), and that the required evidence

is able to be collected and reported;

accountability requirements must

be satisﬁed by regular reporting

against appropriate KPIs; and

the ‘ﬁxed tranche’ component of the

education sector budget support modality

should be delivered annually in total if a

reasonable proportion of any ﬁxed process

indicators are met.

2 Continue to use an Independent Veriﬁcation

Process for a small number of agreed KPIs, with the

following conditions:

include clear guidelines on how the individual

KPIs for the sector relate to the release of the

performance funding component; and

specify the steps for the release of funding,

including whether any IA is entitled to a share

of the funding disbursed.

3 Prepare a capacity development plan for

staff of GoS agencies with involvement in all levels

of the education sector (e.g., MESC, SQA, NUS)

to ensure those working in these agencies have

the necessary skills to support future planning and

reporting. This plan would sit alongside the National

Teacher Development Framework, and include

capacity development plans for the ECE, PSET/TVET

and IE sectors.

4 Improve the processes for engaging

technical assistance to ensure timely recruitment and

engagement with a focus on capacity development

in priority areas. For example, resourcing a reading

specialist in the Education Faculty of NUS to help

ensure teachers (participating in pre-service and

in-service training) have the skills required to teach

reading.

5 Support current work to develop a Sāmoa

Education Data Quality Improvement Plan, and

support implementation of the plan in order to

strengthen data collection and analysis, monitoring,

and reporting.

6 Update, rationalise and simplify the M&E

Framework to ensure that it can be used effectively,

and consider whether it gives the necessary focus

to the range of activities across the education sector

(e.g., ECE, IE, primary, secondary, formal, and

informal PSET and TVET).

7 Improve the integration of MTEF update

processes with other reporting to ensure the necessary

updates actually occur.

1. BACKGROUND AND CONTEXT

This section provides context on the Sāmoa Education Sector Support Programme (ESSP),

detailing the purpose of this evaluation, its methodology, and an outline of the structure of this

report.

1.1. About the ESSP

The ESSP is a three-year activity between MFAT and DFAT designed to specifically support and

finance implementation of elements of the Education Sector Plan 2013-2018 (ESP). The ESP

outlines how the Implementing Agencies (IAs) (the Ministry of Education, Sports and Culture

(MESC), the Sāmoa Qualifications Authority (SQA), and the National University of Sāmoa (NUS))

will work together as a sector, building on previous Sector Wide Approach programmes (SWAp),

and a number of large donor funded projects. The ESSP commenced in October 2015 (DFAT) and

March 2016 (MFAT), and will be completed in December 2018. The total value of the investment

is SAT $35,865,685. The goals of the ESSP (not presented here) are taken from the ESP.

The strategic outcomes of the ESSP are:

increased numbers of three to five-year olds participating in Early Childhood Education;

improved literacy and numeracy scores at Years 4 and 6;

increase in percent of children commencing Year 1 Primary and completing Year 8;

increased numbers of children with special education needs in inclusive classroom

settings with Individual Education Plans;

increase in participation rates and outcomes of Year 12 examinations;

increase in transition rates to post-secondary education; and

increase in the employment rate of graduates of post-secondary education and training.

In particular, the ESSP investment aims to:

• strengthen Sāmoa’s new approach to the development of the education sector as a

coherent whole;

• support implementation of reform policies designed to improve learning outcomes and

skills development linked to realistic employment expectations;

• strengthen government systems for sustainable achievement of improved learning

outcomes and skills development for employment; and

• support achievement of Sāmoa’s National Goals and Strategies, through a well-educated

and skilled Sāmoan society.

1.1.1. Monitoring the achievement of ESSP objectives

The Joint Funding Agreement (JFA) between GoS, MFAT and DFAT as the development partners

was developed to support achievement of the ESP objectives. The ESSP was therefore constructed

at the outset from the ESP, while the Independent Verification Process (IVP) was drawn from the

ESP Monitoring and Evaluation (M&E) Framework. The ESSP investment priorities are therefore

drawn from the ESP. This process was driven by the GoS, with strong direction from the Ministry

of Finance (MoF) for a sector budget support funding modality. The IAs were therefore

responsible for development of the indicators in the ESP. The donors were not responsible for

Evaluation of the Sāmoa Education Sector Support Programme – Final Report 7

developing the Key Performance Indicators (KPIs). The evaluators understand there was

extensive consultation about which of these KPIs should be selected as part of the performance-

based tranche, i.e., which ones should be highlighted to provide a comprehensive picture of the

health of the Sāmoa education system and how the ESP was being implemented. The MFAT and

DFAT as the Development Partners (DPs) noted that the KPIs were outcomes-related, and asked

the IAs whether the KPIs were realistic, and whether the IAs were sure these KPIs were the ones

that should be focused on. DPs recognised at the time that the KPIs were very ambitious, but they

were identified in consultation with the GoS.

1.2. Evaluation purpose

This evaluation was designed to assess the likely relevance, efficiency, effectiveness,

sustainability, and short and medium-term impacts of the ESSP on the broader Sāmoan education

sector. Cross-cutting issues (gender and inclusive education) were also assessed. The evaluation

will inform any further commitments of Australia and New Zealand and the funding modalities

and assessment frameworks used. The Government of Sāmoa (GoS) also indicated an interest in

grasping the impact of the ESSP at the grassroots levels amongst communities, schools, and

employers.

1.3. Earlier independent review of the ESP and ESSP

The ESSP evaluation follows on from the 2016 Mid-Term Review (MTR) of the ESP and the three

Independent Verification Process (IVP) reviews undertaken in 2016, 2017 and earlier in 2018. In

reference to these earlier independent reviews of the ESP (MTR) and ESSP (IVP), the role of this

evaluation was not to revalidate their findings. Rather, the ESSP evaluation Terms of Reference

(see Appendix A: Terms of reference) noted the evaluation was expected to build on the MTR

findings, alongside assessments from the intervening period, to respond to key evaluation

questions. Comment on the MTR, and discussion of achievement and ongoing relevance of the

MTR recommendations, is included in Appendix D: Response to ESP Mid-Term Review

The ESSP evaluation was also expected to analyse the effectiveness of the IVP including whether

is this something that donors would wish to continue in any future ESSP. Analysis of the IVP

findings and effectiveness is discussed within the report and specifically in section 2.2.7.

1.4. Evaluation methodology

The evaluation comprised three phases.

Phase One determined the focus and scope of the evaluation and included a document review

and stakeholder engagement tool design. Documents were provided by MFAT Wellington and

MFAT Sāmoa across Phases One and Two. Phase One concluded with the Evaluation Plan and a

tentative schedule of in-country fieldwork, which was agreed to by MFAT.

Phase Two consisted of in-country fieldwork in Sāmoa (6-19 September 2018) and focused on

implementing the evaluation according to the plan agreed to in Phase One. This involved

conducting 37 face-to-face interviews with over 70 people across GoS agencies, Development

Partners, and education providers. At the end of Phase Two, the in-country team held an Interim

Findings Workshop and provided an Aide Memoire, which presented emerging issues, sought

verification of facts and assumptions, and discussed the feasibility of the initial recommendations.

Phase Three involved data analysis and drafting of this Evaluation Report.

This evaluation adopted a mixed-method approach, gathering information through stakeholder

interviews and review of relevant documentation. These data were then analysed against the key

evaluation questions and sub-questions identified in the Evaluation Plan. The evaluation was

undertaken using the Organisation for Economic Co-operation and Development (OECD)

Development Assistance Committee (DAC) criteria to underpin and focus the assessments being

made.

Table 1: OECD DAC Criteria and associated Sāmoa ESSP KEQs

DAC Criteria Key evaluation question for the Evaluation of the Sāmoa ESSP

Relevance To what extent is the design of the ESSP relevant to the key issues facing the education

sector in Sāmoa?

Effectiveness To what extent has the ESSP has been effective in improving the quality of learning and

enhancing educational access and opportunities?

Efficiency To what extent is the implementation of the ESSP being managed efficiently?

Impact What is the likely impact of the ESSP?

Sustainability Are the benefits of the ESSP likely to be sustainable?

The evaluation team also considered the extent to which cross-cutting issues (gender equality and

inclusive education) have been identified and pursued in the implementation of the ESSP. The

evaluative judgements made about the ESSP under the above criteria were tested with key

stakeholders in MFAT, DFAT, GoS agencies, and the Sāmoan education sector more broadly.

Structure of this report

The remainder of this report is structured as follows: Section 2 outlines the evaluation’s findings,

organised under the DAC criteria of relevance, effectiveness, efficiency, impact, sustainability, as

well as a consideration of cross-cutting issues; Sections 3 and 4 provide lessons learned and

evaluation conclusions; and Section 5 presents a list of recommendations. Linkages between the

key evaluation sub-questions set out in the Evaluation Plan and the various sections of the draft

report are set out in Appendix G: Linkages between Key Evaluation sub-questions and sections of

the draft evaluation report.

Evaluation of the Sāmoa Education Sector Support Programme – Final Report 9

2. FINDINGS

This section outlines the findings of the evaluation of the ESSP based on a review of existing policy

and practice. Stakeholder interviews, operational documents, and published reports informed the

evaluation. Interim findings and preliminary recommendations were presented to stakeholders

at the end of the field mission in Sāmoa on 18 and 19 September 2018, and feedback from that

process has been incorporated into this draft report.

2.1. Relevance

This section seeks to address the key evaluation question, “To what extent is the design of the ESSP

relevant to the key issues facing the education sector in Sāmoa?”

2.1.1. ESSP objectives, scope and design

The evaluation found the scope of the ESSP encompasses the whole education sector, is well

aligned with Development Partners and Government of Sāmoa policies and objectives, addresses

key issues and priorities of target groups, and justifies the involvement of the three Implementing

Agencies.

The five broad strategic goals, the related seven strategic outcomes, and the four aims of the ESSP

are highly relevant, well-aligned with Development Partner (DP) and GoS policies and objectives,

and are supported by the Sāmoa education sector. Very few respondents referred to the original

investment design of the ESSP, although people referred to the Joint Funding Agreement (JFA) or

the ESP.

The theory of change for the ESSP Investment Design was based on the premise that supporting

the implementation of Sāmoa’s ESP would be more efficient and effective than working through a

parallel process (for example, provision of funding for specific projects). This approach was

generally supported by GoS agencies; however, problems arise when these goals, strategic

outcomes, and aims are translated into more specific and practical sub-sector objectives and

activities that require measurement.

The time frame for evaluation of the ESSP goals was too short. The need for a longer-term

approach is recognised by a number of development partner stakeholders, particularly those that

were involved at the design stage. The original intention in designing the programme was to align

the ESSP completion date with the end date of the ESP. However, given the complexities of a

budget support approach, the ESSP time frame (effectively two years, given the initial setting-up

delays) was too short for a realistic evaluation of what could be achieved under this new model.

A longer time frame such as an additional three to five years is needed in order to align the

intended outcomes of the ESSP with the goals of the ESP, and to evaluate effectively the impact of

the sector budget support approach. A longer time frame would also provide space for the

education sector to put into practice the learning that has been achieved since 2016, and would

allow the full benefits of the new approach that is being implemented to become more visible.

The GoS agencies have faced significant challenges in implementing the programme. One key

difficulty was inadequate appreciation of the downstream implications of adopting a new sector

budget support approach where delivery of funding was linked to meeting key performance

indicators (KPIs). Under this new approach, there was insufficient engagement with, and

commitment by, the broader education sector in Sāmoa to provide input to the initial design of

the ESSP performance indicators which would be used to measure progress of the programme. A

complementary problem was a lack of staff capacity to manage the necessary analysis and

reporting of data against KPIs.

The scope of ESP is wide, as it encompasses the whole education sector including:

• early childhood education (ECE);

• primary schooling;

• secondary schooling; and

• post-secondary education and training (PSET), including University-based education,

and technical and vocational education and training (TVET).

The comprehensive coverage of all education sectors by the ESP had an impact upon the ESSP

because it shares the same goals with the ESP. However, the scope of the ESSP is narrower, since

it focuses on the relatively specific aims of the ESSP investment. This specific focus is appropriate,

since the available additional funding through the ESSP needed to be targeted if it was to make a

difference. The monitoring and evaluation (M&E) framework is based on the ESP and, although

used for monitoring the ESSP, is wide in scope. Targeting of ESSP objectives occurs with the use

of the nine selected KPIs as part of the performance-based Independent Verification Process (IVP).

The evaluators considered whether the scope of the ESSP, while narrower than the ESP, should

be narrower still, so that more resources are targeted to selected high priority areas. They

concluded that each of the stated ESSP outcomes remain a priority for the Sāmoa education sector,

and that the current scope of the ESSP remains appropriate and is therefore adequate for its

intended purpose as it allows GoS to focus ESSP funding on priority initiatives within this scope.

2.1.2. Activities made possible by the ESSP

The evaluation found that additional activities made possible by the ESSP investment have

addressed the priorities of target groups such as schools and other education providers and have

targeted key issues facing the education sector.

The additional activity made possible by the ESSP investment has been significant, even though

schools and other providers may not have been aware of the source of the additional funding.

Examples of such activities include teacher development, teacher appraisal initiatives, work to

support literacy and numeracy objectives as part of implementing Minimum Service Standards

(MSS), and support from the PSET Support Fund to upskill trainers and build better links to

employment in the PSET/TVET sector. While the design of the ESSP is broadly aligned with the

educational objectives of education providers, there is nevertheless an ongoing need for more

effective communication of programme objectives to schools, PSET providers and the wider

community.

Evaluation of the Sāmoa Education Sector Support Programme – Final Report 11

2.2. Effectiveness

This section seeks to address the key evaluation question, “To what extent has the ESSP been

effective in improving the quality of learning and enhancing educational access and opportunities?’’

2.2.1. Improving Quality and Access

Efforts through the ESSP to improve the quality of and access to education have had mixed

success. The positive focus on professional development is expected to lead to an improved quality

of teaching in the long-term. Implementing Agencies in Sāmoa need to continue to monitor and

to improve their data collection and analysis so that evidence of progress can be captured and

reported.

Clear evidence of improved student performance over time has not been demonstrated, although

there are some indications of recent improvements. The IVP assessed the KPIs

related to literacy

(English and Sāmoan) and numeracy of Year 4 and Year 6 students on three occasions (July 2016,

April 2017, and March 2018). The literacy KPIs at these two levels of primary schooling were

recorded as ‘not achieved’ in July 2016, but were ‘partially achieved’ in April 2017 and March

2018. The IVP evaluators noted in the 2018 report that the extent of improvement in literacy over

a one-year period raised questions about the comparability of the data in the 2016/17 Annual

Review Report (ARR) compared to the baseline data set in 2015.

The numeracy KPIs for Year 4

and Year 6 students were recorded as ‘not achieved’ in the July 2016 and April 2017 IVP reports,

but were ‘partially achieved’ in the March 2018 report. The IVP evaluators noted in the 2018

report that the extent of overall improvement in numeracy was quite low.

The KPI relating to the

number of schools meeting MSS related to literacy and numeracy was ‘not achieved’ in July 2016

but was recorded as ‘achieved’ in April 2017 and March 2018. Overall, the evidence of the impact

of the ESSP on education quality in Sāmoa is limited and ambiguous.

Some other reports indicate low levels of student learning achievement. For example, the Sāmoa

Early Grade Reading Assessment (SEGRA) in 2017 noted that only 16 percent of Year 3 students

can read fluently and understand what they read. Trends over time in the results of the SPELL1

and SPELL2 tests at Year 4 and Year 6 primary levels demonstrate a high and increasing

proportion of students (especially boys) categorised as ‘at risk’ in English literacy and numeracy.

These results raise concerns about the ability of these students to engage with the curriculum at

higher levels of schooling. Furthermore, the results of senior examinations in Years 12 and 13

indicate high numbers of students have achieved only at ‘beginner’ levels, and will either struggle

with the demands of formal and informal PSET/TVET, or are not well prepared to make the

transition to employment. A more detailed analysis of student achievement in Sāmoa is presented

in Appendix H: Student Achievement in Literacy and Numeracy.

In addition to monitoring student achievement indicators, other indicators of quality in education

have been examined in order to assess the impact of the ESSP. In the July 2016 IVP report, KPI 7

(percentage of teachers meeting teacher performance standards) was assessed as ‘not achieved’.

However, there has been a definite improvement in the subsequent two IVP reports (April 2017

and March 2018) where KPI 7 has been assessed as ‘achieved’. This improvement reflects the

support from the ESSP for quality improvements via assisting more teachers to meet the required

See Appendix H: Student Achievement in Literacy and Numeracy

von Dadelszen, Jane, & Pongi, Visesio. (2018) Sāmoa Education Sector Independent Verification Report

(Draft), p. 9.

Ibid, p. 11.

performance standards. Teacher development has been occurring in the primary and secondary

school sectors, and the new appraisal system for teachers is beginning to show positive results.

The PSET Support Fund (managed by SQA) has supported the upgrading of skills of tutors in the

PSET area. There have also been successes in developing quality assured qualifications, but there

is much more to be done to link skill development to employment outcomes. The focus through

the ESSP on literacy, numeracy and teacher quality has been commendable, but there is still room

for improvement in these areas.

In recent years, access to primary education has been generally very good for both girls and boys

in Sāmoa (see Table 3 in Section 2.6.1). The transition rate from primary to secondary education

has also increased recently, although participation in secondary education favours girls. However,

data from the 2016 National Census indicate a worrying trend in attendance rates: 6 percent of

children in the primary school age range 5-12 are not attending school, with over 10 percent of

13 to 17-year-olds not attending school. The IVP reported on two KPIs that measure access to

education: KPI 8, the primary cohort completion rate (percentage of children commencing Year 1

primary and completing Year 8 by gender) and KPI 5, the transition rate from Year 13 to formal

PSET. The July 2016 IVP report noted that KPI 8 was ‘not achieved’, but the April 2017 and March

2018 reports indicated the KPI was ‘partially achieved’. These last two reports verified that males

achieved the minimum target (1 percent increase), but the target achieved for females declined.

The July 2016 IVP report noted that the baseline for KPI 5 (the transition from Year 13 to formal

PSET) was still being established, and consequently KPI 5 was ‘not achieved’ in that year. KPI 5

was rated ‘achieved’ in the April 2017 report, but ‘not achieved’ in the March 2018 report. The

evaluators that conducted the IVP stated that the data would be of more value if all transitions

from school to PSET (and not just transitions from Year 13 at secondary school to the foundation

year at NUS) could be tracked and measured.

This analysis illustrates that agencies in Sāmoa

need to continue to monitor access to education, and to improve their data collection and analysis.

2.2.2. Governance

The governance structure for the ESSP (Education Sector Advisory Committee and Education

Sector Working Group) is effective, and significant progress has been made recently to use this

structure better. Active participation by all Education Sector Advisory Committee members is

required to support effective governance of any future ESSP.

The Education Sector Advisory Committee (ESAC) and Education Sector Working Group (ESWG)

provide the primary system of governance. The structure is effective, and significant progress has

been made recently to use this structure better. Efforts to convene regular meetings of both ESAC

and ESWG, improve meeting records and submit documents at a sufficient period prior to

meetings have helped improve efficiency and participation. The ESAC includes the sector heads of

the three IAs and representation from a number of other stakeholders, including key Ministries

such as the Ministry of Finance (MoF), the Ministry of Public Enterprises (MPE) and the Ministry

for Commerce, Industry and Labour (MCIL), representatives of Mission Schools, and the DPs. The

ESWG includes officials who are responsible for coordination and working with the ESSP

taskforces, IAs, DPs, and other stakeholders to meet requirements for monitoring, evaluation and

annual reporting.

von Dadelszen, Jane, & Pongi, Visesio. (2018) Sāmoa Education Sector Independent Verification Report

(Draft), p. 12.

Evaluation of the Sāmoa Education Sector Support Programme – Final Report 13

Coordination between IAs has occurred through the ESAC at a high level and through the ESWG,

which undertakes prior analysis and scrutiny of reports. The ESAC has provided high-level

guidance at a strategic level and has been able to monitor risks effectively. Although this system

of governance now appears to be generally effective, ESAC and ESWG may need to periodically

review the descriptions of their roles and functions, as set out in the JFA, in the light of actual

practice, to ensure that there is clarity about their respective responsibilities.

The DPs have been involved on the ESAC at an advisory level, to ensure the programme has been

implemented in accordance with the funding requirements. Their presence has had both benefits

(e.g., policy engagement and independent advice) and challenges (e.g., questioning of inefficient

practices). The donor role on the ESAC assisted transparency around what was being

implemented. MFAT and DFAT felt that they had meaningful input into the direction of the ESSP,

that they were able to assist effective policy dialogue, and present a consistent and unified

approach at ESAC.

Prior to May 2018, the governance structure was not used effectively, and decisions, tasks and

agreements did not follow agreed processes. Minutes of meetings up to early 2018 provide

evidence of dissatisfaction with Education Sector Coordination Division (ESCD) management.

This dissatisfaction was a major issue over a considerable period since the inception of the

programme. As at October 2018, these issues have been resolved, and a positive and constructive

approach is now evident. Papers to the ESAC are expected to have been through the ESWG before

being presented to ESAC. In the past this did not always happen, with the result that reports have

not always included a full analysis of progress and risks. Respondents reported that this has

improved recently and the necessary prior analysis and timely presentation of reports from ESCD

(including the M&E reports) is now allowing ESWG and ESAC to fulfil their roles more effectively.

Several of those interviewed in the PSET sector placed emphasis on developing a more effective

approach to TVET in Sāmoa, in order to improve skill development and connections between the

education sector, industry and employment. Australia, in particular, wishes to see a more coherent

and coordinated approach to TVET. NUS has developed a TVET strategy, and SQA has an approved

policy document on TVET. Positive steps are underway to integrate these documents into a

national TVET strategy, including input from MESC who have responsibility for TVET provision in

secondary schools. There are many other post-secondary education and training providers in

Sāmoa in addition to NUS. MCIL has a key role in relation to the apprenticeship scheme and want

to build the capacity of its apprenticeship inspectors. An interview with MCIL personnel focused

on working with NUS and industry, in order to align practical training in the workplace with the

curriculum. Revision of the ESSP could include consideration of the ESAC membership to ensure

it provides appropriate representation of those with an interest in employment skills

development, such as other TVET providers.

There are seats available for Mission and Private school representatives on the ESAC. The

evaluators understand that initially the mission schools were more active on the ESAC and did

some innovative work around quality. This situation may have changed recently, as, at the time of

the field visit, the Mission Schools appeared to be less engaged at the governance level. It was

suggested this is because these schools have less direct connection with GoS, apart from the

Government grants they receive. However, there was only limited consultation with Mission

School representatives on the ESAC, since the second Mission School representative was travelling

NUS are currently the only PSET/TVET provider on the ESAC. Respondents noted while NUS are the

largest single provider of PSET in Sāmoa, the combined enrolments at all other PSET/TVET providers are

greater than that of NUS.

during the evaluators’ field visit and could not be interviewed. Since approximately 25 percent of

students in Sāmoa are taught in Mission schools, consideration by ESAC of the quality of education

provision within these schools is necessary.

2.2.3. Sector Coordination

Lack of coordination has been a weakness in implementation in the past, resulting in late

reporting and missed deadlines. Appointment of a new Education Sector Coordinator has seen

improved coordination and evidence of a more coherent approach to the development of the

education sector.

Cooperation among the three IAs is necessary to coordinate education activities and reporting

where agency input is shared, and respondents suggested this has been an area of significant

improvement in recent times. The fact that this cooperation between agencies, led by the focal

points within the three IAs, is now occurring is a step in the right direction. This new approach is

a welcome change from the siloed approach within the three IAs that was previously reported by

stakeholders in this evaluation and the earlier ESP MTR. Improved coordination is evidence of a

more coherent approach to the development of the education sector. More needs to be done to

ensure that coordination among the three IAs continues in order to integrate education polices

and processes at a high level. A coordinated whole-of-Government approach needs to be adopted

for sectors such as TVET where links are needed beyond the three IAs with employers, and with

labour and economic development focused Government agencies such as MCIL. The professional

dialogue among staff within and across different agencies has created a better understanding of

education standards, provided a forum for staff to reflect on their own capacity, and emphasised

the need for collaboration between different parts of the education sector.

The evaluation found that the ESCD is now better placed to support the ESWG to provide the

required analysis before reports are forwarded to the ESAC for decision-making. In particular,

positive feedback was received from respondents about the new appointee to the role of

Education Sector Coordinator within ESCD. The coordinator has brought effective coordination

skills to the role and achieved a positive impact in a short time, although more time will be needed

before the full impact of a fully staffed and functioning ESCD can be assessed.

The ESAC has intervened in the past when reporting has not been up to standard and made

decisions to ensure that educational outcomes are reported using a strengths-based approach

rather than a deficit-based approach. For example, following advice from the IVP consultants,

changes to indicators relating to literacy were introduced to provide evidence on the proficiency

of Sāmoan children (a strengths-based concept) rather than focus on ‘at risk’ children (a deficit-

based concept). Some of these decisions had unintended outcomes, including the need to reset

baselines, and consequently meant that full reporting against some KPIs was delayed by a year.

Furthermore, ESAC and MoF expect quarterly reporting against the M&E Framework and the joint

funding agreement (JFA) between GoS and the donors requires the GoS to update the Medium-

Term Expenditure Framework (MTEF) in February each year as part of the monitoring process.

Timeliness in reporting and in updating these monitoring tools has been a problem to date.

Constraining factors include the limited capacity of staff to monitor, analyse and report on data,

the lack of timeliness in reporting, and staff turnover. Staff turnover at all levels in the education

sector has meant that new people have been appointed in nearly every leadership role since the

implementation of ESSP began, and consequently time was needed for new people to become

familiar with a programme that they had no direct role in designing. This slowed the

establishment and implementation of the ESSP, although progress is now increasingly evident.

Evaluation of the Sāmoa Education Sector Support Programme – Final Report 15

2.2.4. Data and Information Management

A lack of a single education information management system and a disconnect between the

systems in use across the three IAs are weaknesses of the ESSP implementation and reporting.

Significant work is required in this area to support more effective data security and the ability to

utilise the data available for reporting and planning purposes.

Most of the ESSP KPIs were outcomes focused and, due to weaknesses in data collection and

analysis, it proved difficult for the IVP to verify to what extent these outcomes-focused indicators

were achieved. The first IVP presented a learning experience for the IAs in the practicalities of

collating data from multiple sources, bringing it together, and synthesising the reporting against

KPIs in a timely way. Reflection from staff from the three IAs on this process included the

realisation that poor planning and inadequate coordination had led to a situation where

performance was rated in the first IVP report as ‘not-achieved’ against seven of the nine KPIs

selected for monitoring the education system. Lessons have been learned from that process and

data collection and reporting against the KPIs has since improved, although further work in this

area is still required.

Two recent reviews have commented on education data management in Sāmoa, noting that the

quality of data captured by MESC is poor.

The evaluators understand that some staff in MESC

continue to use Excel spreadsheets on personal computers for storing important educational data.

While an Education Management Information System (EMIS) exists, it is not sufficiently developed

nor widely used by staff. This presents several risks for GoS, both in terms of data security and the

ability to utilise the data for reporting and planning purposes. In addition, MESC and SQA still

report their annual Education Statistical Digest (MESC) and PSET Statistical Bulletin (SQA)

separately. NUS appear to some extent to sit apart from the other two IAs in its reporting. The

evaluators consider this separation of reporting is because currently there is no education sector-

wide approach to reconcile all the data sources.

This situation is a result of poor data management practices coupled with an absence of sound

data cleaning and verification practices. User feedback and consultation is a critical element

missing in the data production process. The Sāmoa Bureau of Statistics does not have a sufficient

role in the quality control and production of education statistics. The evaluation found that the

dissemination of education data suffers from several shortcomings. Currently each IA is

developing its own technical infrastructure in silo without taking into consideration the need for

a sector-wide approach to education statistics. Any work to develop an Education Data Quality

Improvement Plan should be supported, and implementation of the plan prioritised to strengthen

data collection and analysis, monitoring and reporting. Associated with this work is the need to

develop the capacity of staff in capturing, analysing and reporting on data. It is important to

acknowledge that the long-term goals of the ESSP have to remain the paramount concern, and that

management and analysis of information should not distract the sector from its investment in

achieving these goals. Nevertheless, effective information management and analysis remains a

critical tool in implementing the programme effectively and reporting evidence of outcomes

achievement.

UNESCO Institute of Statistics. (2018) Sāmoa Education Data Quality Assessment Report, and WizConsult.

(2017) Situational Analysis for Creating Education Sector Management Information System &

Recommendations.

2.2.5. Monitoring and Evaluation

An M&E Framework exists and is being used to report on a much broader range of indicators than

are included in the ESSP. The M&E Framework needs to be updated, streamlined and simplified.

An M&E Framework exists and is being used to report on a much broader range of indicators than

are included in the ESSP (for example, additional ESP indicators). The decision to report on all the

activities in the ESP was reasonable, as reporting on the education sector needs to be

comprehensive. Reporting against the ESSP KPIs is required under the M&E Framework currently

in use. The many progressive modifications to the KPIs have made the M&E Framework

unnecessarily complex. Because the current M&E Framework includes the various changes in

KPIs (in strikethrough and/or different coloured text) that have occurred over a period of three

years, it is hard to follow and is less usable than is optimal. Some parts of the M&E framework

include deleted indicators which need to be removed. There are strengths and weaknesses in the

way monitoring and evaluation has occurred, partly related to the information management

issues discussed above. The Annual Review Reports (ARR) are useful and essential tools which

require the IAs to coordinate their reporting.

2.2.6. The Funding Modality

The mixed-mode funding modality for the ESSP, including the 70 percent fixed tranche sector

budget support and 30 percent performance-based tranche is appropriate. However,

improvements are required to Budget processes to ensure all available efficiencies are achieved.

A mixed mode funding modality was used for ESSP. This mixed-mode modality includes sector

budget support based on fixed process indicators for 70 percent of the funds (more or less a sector

wide approach), combined with performance-linked contributions assessed via the IVP for the

remaining 30 percent of the funding, plus a separate ring-fenced or ear-marked funding amount

for inclusive education (IE). In effect, three funding modality types have been used. In addition,

funds were made available for technical assistance to support the ESSP. The advantages and

disadvantages with this mixed mode method of funding outlined in Figure 1 below.

Figure 1: Advantages and Disadvantages of sector budget support funding modality

Evaluation of the Sāmoa Education Sector Support Programme – Final Report 17

The sector budget support approach requires alignment between the GoS budget timelines and

the ESSP timelines in order to be effective and to achieve the anticipated benefits described above.

ESSP funding priorities and allocations should be dealt with as part of the established GoS budget

process. The implication is that any new ESSP in the future should align its dates much better with

the GoS Budget processes by allowing sufficient time for ESAC and the MoF to do their work. This

alignment was not present in practice in the Sāmoan approach to management of the ESSP. Figure

2 in Appendix F: ESSP Funding Timelines shows the misalignment in diagrammatic form.

Under the performance-based IVP mechanism, 30 percent of the ESSP funding was released (or

not) based on GoS’s tracking towards nine pre-determined KPIs subject to an independent

assessment of a minimum trend of one percentage point towards the ESSP’s 2018 targets. The

first IVP report in 2016 found that seven of the nine KPIs targets had not been met, one target had

been partially met, and one target had been achieved. Consequentially, full funding was dispersed

against only one of the KPIs and 50% against one other KPI. Funding was withheld against the

remaining seven KPIs. The 2017 IVP Report saw an improvement where four of the nine KPI

targets had been fully met, two were partially met and three KPIs were not achieved. In 2018 the

IVP Report recommended full disbursement against three KPIs, partial disbursement against four

KPIs and no disbursement against two KPIs.

Perceived advantages of the performance-based fund (30 percent of the total funds) assessed by

an IVP against nine selected KPIs include an incentive to the GoS to earn additional funds through

good performance, and a transparent focus on a limited set of high GoS priorities. Disadvantages

include a reduction in funds if the KPIs are assessed as ‘not achieved’, and the need to ensure staff

have the necessary capacity to administer and report on the KPIs.

Advantages of the ‘ring-fenced’ fund (for inclusive education) include more transparency in the

use of funds, and an ability for donors to track with precision exactly where the funds are being

spent. Disadvantages of a ring-fenced fund include less flexibility by the GoS to reallocate funds to

other priority areas if there are administrative or other delays in spending the funds on what was

intended.

The main challenge for the Government of Sāmoa education sector in using the budget sector

support funding modality was that it was a new approach, and that it has taken considerable time

for the Implementing Agencies and their staff (and to some extent the Ministry of Finance and

Development Partners) to understand and appreciate what sector budget support really implies,

and what the downstream practical implications of adopting this approach to funding are.

Areas of concern in relation to funding which were raised by education sector representatives

who were consulted included the following: a lack of transparency for education players about

what funds were actually available; the out-of-date MTEF; the different approaches for approval

of payments; and the absence of detailed acquittals against the annual workplans. Some of these

issues (e.g., the out-of-date MTEF) need to be addressed by the ESCD. Other issues (e.g., better

communication about approval of payments, and transparency and communication about the

funding available) are matters that need to be addressed by the MoF.

It is difficult in many sectors of the Sāmoan education system to separate out the effect of GoS

budget funding from the effect of additional funds delivered through the ESSP, because the 70%

fixed funding tranche is not tagged to achievement of specific KPIs (although a report on the 12

specified process KPIs is required). A very detailed analysis, such as that outlined in the OECD

Guidelines Evaluating Budget Support: Methodological Approach, coupled with preparatory

financial analysis such as the recent draft 2018 World Bank Sāmoa Public Expenditure Review:

Education Sector (not yet released), would be required to provide definitive answers as to what

contribution towards achieving outcomes was made by regular GoS budget funding, and what

contribution was made by ESSP funding. That kind of detailed analysis was not possible in the

timeframe and scope of this evaluation.

Sector budget support means that DPs provide donor funds through the ESSP which are expected

to then be merged with GoS funds and spent according to the ESP. The merging of additional donor

funds with GoS funds from its regular budget (implied by direct sector budget support) makes

disaggregation and tracking of the additional donor funds difficult. For example, improvement of

literacy is a key GoS priority which it funds via the education budget, and literacy improvement is

also an aim of the ESSP. The ESSP included ring-fenced funding from Australia targeted at IE.

Although it is difficult to assess the relevance, effectiveness and efficiency of ESSP without detailed

financial analysis to disaggregate the impact of donor funding from ongoing GoS funding, the

evaluators have identified some specific areas of ESSP investment. Examples include:

• support for upgrading of teacher skills and qualifications;

• support (through the PSET Support Fund managed by SQA) for upskilling of tutors in the

PSET sector;

• support for programme and professional development at NUS;

• support for the work of task forces on goals such as ECE and IE;

• investment in making education provision in schools more relevant; and

• investment in developing capacity such as Principal leadership and other in-service

training courses.

There is a need for better alignment of key milestones in the GoS annual Budget processes and

procedures, and the ESSP milestones set out in the JFA. A sector budget support modality should

include integration between GoS budget and ESSP timelines. In practice two separate processes

appear to have been operating to date: the GoS annual budget round (to determine the regular

annual funding budget for MESC, SQA and NUS from GoS resources, according to the established

GoS timetable) and a separate ESSP annual funding round, according to the JFA dates. This

duplication of activity created additional work for both the IAs and for MoF and meant that an

expected advantage of sector budget sector support (elimination of unnecessary duplication) was

not achieved. This misalignment is illustrated in Figure 2 in Appendix F: ESSP Funding Timelines.

As an example, currently the ESAC is expected to meet in January (according to the JFA), but in

practice it has met in March. The purpose of the January ESAC meeting is to ensure that the MTEF

is developed with guidance from ESAC, and that the Annual Budget is updated in February each

year in preparation for Budget bids and forward estimates due to MoF before the end of March

each year. Due to delays in scheduling ESAC meetings in January, and delays in reporting to ESAC,

there has in the past been intense pressure to finalise ESSP financial arrangements in mid-March

at a time when the MoF is very busy. Essentially, the ESSP dates specified in the JFA need to be

brought forward by about two months to ensure a better integration between the standard GoS

budget processes and timelines, and the ESSP timelines. The annual timeline of the ESSP

requirements set out in the JFA such as the Annual Sector Reviews (November), ESAC meeting

(January) and updating of the MTEF and Annual Budget (February) needs to be changed in any

future plan to be better aligned in practice with the deadlines expected for the GoS annual budget

process. Better alignment of GoS budget timelines and anticipated ESSP dates would allow more

timely assessment of budget bids and would improve efficiency.

Evaluation of the Sāmoa Education Sector Support Programme – Final Report 19

The evaluators consider the fixed tranche disbursement of the budget support component of the

funding modality (70 percent) is appropriate. The main benefit of this approach is that it

theoretically ensures GoS systems are used, thereby reducing duplication and transaction costs

and satisfying accountability requirements (although as discussed above this is not necessarily

what is happening in practice).

Staff in IAs were emphatic in their belief that sector budget support was a funding modality that

supported GoS processes, and they wished to see this approach continued. It was argued that

sector budget support gives GoS agencies and divisions more autonomy than project-based

approaches to funding. While there is general agreement about the merits of a funding approach

that uses GoS processes and procedures to achieve its goals by aligning planning and reporting

processes and avoiding unnecessary duplication, more work is needed to communicate to the

sector a clearer understanding of how a sector budget support modality operates and the

associated accountability requirements to receiving funding through this modality.

Some stakeholders questioned the fairness of reducing the amount of the fixed tranche

component, as funding predictability is important for planning purposes, based on the GoS self-

assessment of progress against the 12 fixed process indicators, and the DP’s assessment of the

likelihood of an underspend in the current financial year. It appears that current practice is to

treat the ‘fixed’ amount as a performance-based fund (to some extent), and the evidence indicates

that the amount has been reduced in some years (e.g., in 2017-18).

One purpose of the ‘fixed tranche’ funding is to provide the education sector with funding

predictability from year to year. The evaluators consider that a reasonably favourable report by

the GoS as part of the Annual Review Report against the 12 process indicators should result in all

of the funds available being paid to the education sector. It is unreasonable to expect the education

sector to plan effectively if they do not have ongoing certainty about their funding. It is

acknowledged that the JFA includes clauses

to the effect that if there is the likelihood of an

underspend, the DPs will consider a reduction in the subsequent fixed tranche funding. Only in

circumstances where the underspend is large should these clauses in the JFA be invoked to

withhold funding in this way. In this context, the process indicators associated with the fixed

tranche disbursement need to be reviewed for the next iteration of the ESSP. The KPIs associated

with the performance-based funding tranche also need to be reviewed in the next iteration of the

ESSP.

A perceived advantage of the performance-based approach using the IVP was that it would

provide a funding inventive for the education sector. In practice, no funding incentive has been

evident. This situation has arisen partly because of a lack of transparency in management of, and

communication about, the funding by the MoF, and partly because education staff are motivated

not by purely financial considerations but by the desire to do a good job and meet education

objectives. There are advantages in continuing the performance-based approach, since education

sector participants found that the IVP experience has improved their management and

implementation of KPIs, and their understanding of the need for effective data collection and

analysis processes. Managing a performance-based approach to funding is a challenging exercise

in all countries, not just in Sāmoa, and is an area where help and technical expertise may need to

be brought in to assist. This issue is further developed in the next section.

Author Unknown. (2015) Investment Design for Sāmoa Education Sector Support Program (ESSP), July

2015 to June 2018. Ministry of Foreign Affairs and Trade. Pages 28 and 36.

The use of ring-fenced funding for IE was a transparent approach that successfully ensured that

funding was targeted to relevant activities that supported the mainstreaming in regular

classrooms of children with disabilities. Activities that were supported include training of teacher

aides who would provide support in regular classrooms to children with disabilities. DPs like the

transparency of ring-fenced funding since it provides clear accountability for ESSP funding. It

would be good to see the GoS invest more of its funding to help to really ensure sustainability for

IE objectives within the education sector.

The ‘ring-fenced’ funding component is essentially project-based funding. The evaluators

considered that a move to place more emphasis on a project-based funding approach would

undermine the relationship between the donors and the GoS. There was always an intention from

the donor perspective that ESSP would be part of a long-term approach. It is for this reason that

the evaluators have recommended that the sector budget support modality should continue to be

supported as the major mechanism for delivery of funding to support the education sector.

Funding has been available as part of ESSP to support the procurement of technical support for

ESSP accountability and quality assurance purposes, and to support the ESCD and other mutually

determined areas where required. Technical assistance, where it has been called upon, has had

positive results. Valuable technical assistance has been provided, for example, through the

Strategic Planning Adviser role. The person undertaking this role has provided strategic advice to

ESCD on implementation of the ESSP, and positive assistance with the development of cross-

agency reporting templates to support efficient and effective reporting of outputs and outcomes.

Useful technical assistance has been provided in other areas (for example, IE) by international

advisers with technical expertise. There is further scope for using technical assistance (both

international and local) in targeted areas (for example, reading or numeracy specialists at the

education faculty of NUS to assist in the training of teachers, or technical expertise in the design

and development of new key performance indicators) in order to support and strengthen the

implementation of the ESSP.

2.2.7. Independent Verification Process

IAs, ESAC, and other stakeholders interviewed are generally supportive of the IVP. However, the

selected KPIs were not SMART (specific, measurable, achievable, relevant and time-bound) and

this led to difficulties in providing evidence of achievement. In any performance-based component

of a new ESSP that is developed in future, consideration should be given to including process

indicators that demonstrate measurable steps towards achieving outcomes, as well as outcomes-

focused KPIs.

The IVP was designed as a mechanism to verify the performance-based tranche (30 percent) of

the funding provided under ESSP. On balance, the people interviewed supported the IVP and felt

there had been positive learning from the exercise. Many saw it as an opportunity to learn from

external consultants and develop staff capacity on the management of performance-based

systems and this had improved data collection and analysis procedures across the board. The

people interviewed from key agencies and the ESAC suggested that the IVP provided good

motivation to work together and design and implement the systems required to provide evidence

against the nine ESSP KPIs.

The support for the IVP occurred despite the obvious disappointment in not successfully

achieving all nine ESSP KPIs. The results from the initial IVP in 2016 were particularly

disappointing but results improved in the 2017 and 2018 reports. Feedback from the IVP has

helped create a better awareness of the multiple sources of data that need to be coordinated, and

Evaluation of the Sāmoa Education Sector Support Programme – Final Report 21

the need for effective system-wide information management systems to collate relevant

information and report in a timely way on KPIs. However, there was little evidence available that

withholding funding provided incentives to IAs to improve performance.

Learning did occur through mistakes that were made, but it was not until the first IVP had been

completed that stakeholders appreciated the importance of identifying the sources of data,

gathering the required information, and reporting in a timely way. In some cases, these lessons

were too late for effective reporting against the KPIs over the course of the ESSP.

The KPIs selected originally for the performance-based assessment component of the funding are

open to criticism for not being easily measurable or achievable, e.g., they are not what would be

considered SMART indicators – specific, measurable, achievable, relevant and time-bound. More

consultation was needed at the ESSP design stage as to whether the data sources were available

that would allow the IAs to provide evidence for the achievement of the KPIs. However, collation

of data for IVP should not have been extra work, rather it is part of delivering education

programmes. The evaluators consider significant modifications to the current processes and KPIs

are needed if a performance-based component is to be included in any new ESSP.

The education sector is expected to work collaboratively and collectively to achieve the KPIs

through a sector-wide approach. Unfortunately, there is some evidence the three IAs continue to

see the indicators as ‘belonging to’ individual agencies. The intention of the performance-based

component of sector budget support approach is that achieving KPIs generates funding for the

education sector as a whole and does not deliver funding specifically to the agency that may be

primarily responsible for gathering and reporting on the data. The initial presentation of the KPIs

(see Appendix E: Key Performance Indicators: Education Sector Support Programme), where lead

agencies are named against specific KPIs, may have contributed to this perception. It is important

for the three IAs to realise that new funding should be secured as part of the annual GoS budget

process, and that reporting against the KPIs is an accountability requirement for the sector, rather

than a revenue-generating exercise for an individual IA. The evaluators suggest one approach to

support improved coordination is that the performance funding received for meeting individual

KPIs could be targeted at sector initiatives that are expected to support achievement of the KPIs

that have not yet been achieved.

Better communication about the implications of the sector budget support modality and the IVP

would have assisted stakeholders to understand what they needed to do in collating and analysing

information. Clarity is needed about the release of the performance-based funds under a sector

budget support approach. While there were initial misunderstandings about how the

performance-based funding tranche would be released if KPIs were met, IAs now have a better

understanding that the sector budget support approach means they need to cooperate, and that

this process generates funding for the education sector, rather than funds that are allocated to

individual IAs.

Indicators for the performance-based funds focus on outcomes-focused KPIs. It is generally

acknowledged in the education sector that achieving outcomes such as improved student

achievement takes time, and that little progress may be observable over a restricted time frame

such as that of the ESSP. An approach that includes some process KPIs would enable smaller

quantifiable steps towards larger goals to be measured. An example of a process indicator might

be one that measures the quality, accuracy and timeliness of delivery of annual Government grant

funding to schools and other education providers. Delivery of funding is a key step in helping

education providers to meet broader student learning outcomes.

2.2.8. Communications and Guidelines for Schools, ECE and PSET/TVET Providers

Communications and guidelines to education providers have an operational focus and there is

little awareness by providers about the ESSP funding, its purpose and expected outcomes.

Feedback from the interviews indicated that there was a lack of provider insight about the ESSP

beyond the awareness that, “New Zealand and Australia provide funding for education”. There was

no evidence of guidelines for schools or other education providers that relate specifically to the

ESSP (for example explaining what the ESSP is, how it supports the education sector, or what

dimensions of the education sector are targeted). Individual providers (beyond NUS) had little to

no knowledge of the ESSP, and certainly no idea of how that funding might have been relevant to

them on an individual provider basis. Schools, however, were aware and appreciative of the school

(fee) grant funds delivered to each school to assist them with their school operations but did not

link those funds explicitly to ESSP. That money was now sourced from GoS core Budget, even

though the initial impetus and funding for the school grants scheme came from New Zealand and

Australia. This feedback is consistent with a sector budget support modality where the GoS

systems are being successfully used.

There are other creative ways (beside official guidelines) in which visibility for the contribution

made by donors to support the education sector might be communicated to a wider audience in

Sāmoa. These communication models could include, for example, media releases, advertisements

in newspapers and on television, and speeches by High Commissioners at significant events.

The MSS guidelines produced by MESC are well designed and provide helpful advice and guidance

to schools. These guidelines include a component that helps schools with their development

planning. However, there is close centralised supervision of the MSS, including some apparent

role duplication as MESC staff and school inspectors both have a role in monitoring the MSS.

Monitoring of the MSS could be streamlined, with less ‘hands on’ approval by MESC staff and more

delegation to Principals/Head Teachers as managers of schools who could take more

responsibility for their own school development. High-level monitoring responsibility of the MSS

could be transferred to school inspectors.

A new integrated approach to delivery of school grants is being developed in Sāmoa. The existing

school grants administration manuals will need to be reviewed as part of the implementation of

the new integrated approach.

Evaluation of the Sāmoa Education Sector Support Programme – Final Report 23

2.3. Efficiency

This section seeks to address the key evaluation question “To what extent is the implementation of

the ESSP being managed efficiently?”

2.3.1. Implementation

The implementation of the ESSP by the Government of Sāmoa has been less efficient than has been

anticipated, but improvements are occurring. Delays in delivering school grant funding directly

to schools is evidence of inefficiency within Government of Sāmoa systems. Despite these

inefficiencies, the ESSP has clearly added value to the Sāmoa education sector.

The implementation of the ESSP by the GoS has been less efficient than has been anticipated, but

improvements are occurring. Part of the reason for initial inefficiency was the very ambitious

nature of the ESP to which the ESSP is linked. Challenges from the outset of the ESSP, related to

implementation of a SWAp based on a new (to education sector) budget support modality and of

coordinating the activities of the three IAs, meant that the programme made a slower start than

expected.

There has been some inconsistency in implementation of the ESSP as compared with its original

design. The stakeholders informed us the practical application of the first IVP caused the Sāmoa

education sector to appreciate that its planning and information management systems needed

overhaul and strengthening to demonstrate in a timely way that the KPIs had been achieved.

Delayed or late reporting and initial failure of the ESCD to follow the required procedures were

indications of this inefficiency. Added value through reduced transaction costs and reduced

duplication are not yet apparent, but these benefits should become evident in future, given the

better understanding of what sector budget support for education means in practice.

Results against the nine ESSP KPIs have shown steady improvement over the last three years,

although considerable scope remains for efficiency improvements. For example, the teacher

appraisal process has been proceeding positively, and the relevant KPI (percentage of teachers

meeting teacher performance standards) was achieved in the 2018 IVP report. In the same report,

the KPI relating to the number of schools meeting MSS for literacy and numeracy was achieved.

Similarly, the KPI for the number of accredited courses provided by PSET providers was also

achieved. Four other KPIs in the 2018 IVP report were partially achieved, while two KPIs were

not achieved. Without exception, all those interviewed were committed to the improvement of

education in Sāmoa. The passion and commitment of staff in the education sector has been a key

contributor to the achievement of ESSP outcomes.

The need for training and capacity development is recognised as a key issue across the education

sector. This includes the quality of teaching, since the quality of student performance in the

classroom is dependent upon the quality of the teacher. The Faculty of Education at NUS and the

teacher development division of MESC acknowledge the need to strengthen the skills of teachers

currently in the profession, and have been implementing programmes to upgrade the

qualifications of teachers to degree level (from diploma level), and to provide relevant in-service

training. Teachers in schools have been given in-service training in internal assessment processes

and procedures. It is too soon, however, to assess whether this training is leading to improved

student achievement in schools. The teacher appraisal system is being implemented successfully.

The PSET Support Fund is providing opportunities for tutors to upgrade their skills. In-Country

Training is being delivered by NUS in the post-compulsory sector via their Oloamanu Centre for

Professional Development and Continuing Education.

Feedback from stakeholders was clear that they felt the ESSP has added value to the Sāmoa

education sector. The assistance and support provided includes provision of technical assistance

(TA), support and advice through participation on the governance bodies, and help with managing

for results. The evaluators met with the technical assistant who was engaged to support the ESCD,

and concluded that the direct implementation support given through such activities as provision

of ongoing advice on the ESSP, and the development of reporting templates, had assisted

considerably in reducing transaction costs and avoiding duplication. Useful expertise and advice

had been contributed by contracted technical advice in relation to MESC’s IE policy. Direct support

was also provided through the engagement of technical experts to assist with the IVP. There was

general acknowledgment that value had been added through the independence and technical

expertise that these skilled people had brought to bear on the ESSP.

Feedback from schools visited indicated that 2018 school grants had not yet been delivered to

school bank accounts by mid-September. The delay in delivering funds was explained as a result

of the planned introduction of the new integrated approach to school grant delivery. Nevertheless,

Principals/Head Teachers commented that schools were facing difficult cash flow issues as a

result of the late delivery of school grant funds. The impact on schools of the unacceptable delay

in receiving funding is that other sources of revenue (such as the fees paid by parents) had to be

used to pay outstanding monthly bills such as water and power bills. An inability to meet deadlines

for delivering funds to educational institutions raises questions about the efficiency of

implementation of the ESSP.

2.3.2. Development Partner Management Efficiency

Delays in the initial delivery of funding and misalignment between ESSP and Government of

Sāmoa Budget process timelines have caused inefficiencies. Despite these delays, Development

Partner engagement with the ESSP Implementing Agencies has been positive.

The ESP was a five-year programme starting in 1 July 2013. The ESSP was expected to be a three-

year programme covering August 2015 to December 2018. As shown by Figure 3 in Appendix F:

ESSP Funding Timelines there was some delay in signing the ESSP JFA by DPs (JFA signed 8

October 2015 by Australia, 21 March 2016 by New Zealand) and consequently delivery of funding

was late (not delivered on 1 August 2015). Late signing of the JFA impacted upon the ability of the

GoS to make an early start in implementing the programme, and further limited the ability of GoS

to understand and come to terms with the implications of the new sector budget support funding

modality. The practical effect of the delay in receiving funding from donors was to reduce the

effective time frame of the ESSP to two years rather than the three years that was originally

anticipated.

Personnel from the MFAT and DFAT posts in Sāmoa have had close involvement with ESSP, and

have maintained good communication with the three IAs through their participation in the ESAC.

The support and advice provided to ESAC by DPs has been appreciated. The Head Offices of MFAT

Wellington and DFAT Canberra have only been involved at a relatively high level. On balance, the

support of the DPs has helped to promote more efficient implementation of the ESSP through a

process of policy dialogue and through monitoring of progress.

Evaluation of the Sāmoa Education Sector Support Programme – Final Report 25

2.3.3. Systems Efficiency

The decision to deliver the bulk of ESSP funding through a sector budget support mechanism was

appropriate. However, there are several initiatives that would improve the efficiency of ESSP –

specifically initiatives focused on strengthening the education system and the capacity of those

working within it.

Some examples of areas where more efficient systems would improve performance were

provided to the evaluators. For example, the procurement process was perceived to cause delays

in implementation of some activities, especially those managed by MESC. The process for

obtaining approval from MoF for procurement was considered cumbersome.

An MTEF for the ESSP has been sighted by the evaluators. However, it appears to exist in multiple

versions, is relatively complex, and does not appear to have been recently updated. Reporting to

the ESWG and ESAC as part of quarterly and annual review reports should provide an updated

MTEF as part of the financial reporting. The evaluators understand that a recent appointment has

been made to the ESCD with responsibility for updating and reporting against the MTEF. It is

expected having this additional capacity in the team will support improvements in this area.

The total value of the ESSP over its life was up to a maximum of SAT$35,865,685. In summary, the

GoS has actually received SAT$28,893,258 over the four financial years of the ESSP from the total

budget. Table 2 sets out the budget for ESSP and the funding actually received by the GoS is set

out by ESSP financial years.

Table 2: ESSP Budget and Actual Funds Received, 2015-2019 Currency: Sāmoan tala, rounded to $000s

FY2015-16 FY2016-17 FY2017-18 FY2018-19 Total

Budget Actual Budget Actual Budget Actual Budget Actual

Budget Actual Difference

8,001.3

7,116.7

8,498.4

8,437.8

6,338.2

2,112

24,937.4

24,065.2

872.2

3,642.1

619.1

3,616.2

2,007

3,669.9

2,202

10,928.3

4,828

6,100.3

8,001.3

7,116.7

12,140.5

9,117.5

12,054

8,345.2

3,669.9

4,314

35,865.7

28,893.2

6,972.5

• Row A – Annual fixed tranche amount based on process indicators (70 percent)

• Row B – Performance-based tranche amount subject to achievement of KPIs (30 percent)

• Row C – Total of fixed and performance-based tranches (Rows A and B added together)

A proportion (SAT$6,972.5k or 19.4 percent) of the potential funding available has not been

delivered to the education sector. This figure is the difference between the total budget figure of

SAT$35,865.7k and the actual funding delivered (SAT$28,893.2k). Row B in Table 2 shows the

amount of funding that was withheld due to not achieving all the targets (the nine KPIs) associated

with the performance-based component. Another reason was that funds from a previous

education programme (ESPII) were unspent, and were reallocated, with a consequent offset

reduction in the money made available for the education sector under ESSP. The evaluators do

have a concern, however, that some of the fixed tranche funds were withheld in 2017/2018. It is

difficult for education providers to plan expenditure without predictability of funding. The

reasons for withholding fixed tranche funds arise possibly because of uncommitted expenditure

and services not being delivered to an anticipated timeline, or possibly because financial

Note: Small differences in addition may be a result of rounding.

administrators have taken steps to ‘recover’ unspent funds. Nevertheless, the withholding of

funds that were signalled in the ESSP budget is an indication of some inefficiency in the system.

Timely financial reporting by the ESCD using an updated MTEF, and timely financial reporting by

MoF back to the education sector, would assist overall efficiency.

The evaluators are also concerned that at this stage, near the end of the programme, the data from

the MoF shows that SAT$6,564,607 of the funds allocated remains unspent. This figure is an

indication of inefficiency in the system. Some of the unspent funds may need to be returned to

donors because these funds may be part of the 30 percent performance-based tranche where KPIs

have not been met and funding was consequently not released to the education sector. However,

MoF could be invited by ESCD to identify if there are any unspent funds available in the ESSP

budget that could, with donor agreement, either be released before the end of 2018, or be carried

over into 2019 for possible disbursement.

On balance the evaluators have concluded that the decision to deliver the bulk of ESSP funding

through a sector budget support mechanism was appropriate. Potentially this arrangement

should have been more efficient. The inefficiency that was evident, such as inadequate prior

analysis and late reporting, arose partly because of a lack of understanding by the IAs of what the

new SWAp represented by the sector budget support modality entailed, and partly from a lack of

staff capacity in prior planning, coordination and delivery of the requirements for timely

information and reports.

There are several initiatives that would improve the efficiency of ESSP. These include a targeted

programme of capacity development for staff of IAs and teachers and tutors in education

providers, better integration of GoS budget and ESSP timelines, and better planning, preparation

and management of data so that reporting can occur in a timely way. Attention to timeliness of

reporting by earlier prior planning and thorough analysis would assist the governance bodies who

would not have to return reports for further work. Another step that would improve efficiency is

simplification of the M&E Framework and the MTEF, and regular use of these tools in reporting to

ESWG and ESAC. Reform of information management systems, and improvement and regular use

by staff in the three IAS of a redesigned EMIS would assist in more efficient management of data.

Evaluation of the Sāmoa Education Sector Support Programme – Final Report 27

2.4. Impact

This section seeks to address the key evaluation question “What is the likely impact of the ESSP?”

Any discussion of impact of the ESSP needs to be understood within the timeframe of the

programme and the related ability to demonstrate improvements in educational outcomes within

this short timeframe. The very ambitious nature of some of the anticipated ESP outcomes across

the broad range of the entire education sector (for example, measurable improvements in literacy

and numeracy skills) have been unrealistic to achieve in the relatively short duration of the ESSP

programme (just over three years). It may take at least a generation (twenty-five years or more)

before education sector initiatives (such as improvements in teacher qualifications and capacity)

can be shown to lead to quantifiable improvements in student performance.

2.4.1. Impact of funding via sector budget support

Delivery of the funding via sector budget support has allowed the Implementing Agencies to use

the ESSP to support activities that will support the long-term achievement of Education Sector

Plan goals and sector outcomes.

The evaluators understand that the availability of ESSP funding also made it possible for the IAs

to design and pilot new initiatives that would otherwise have been outside of the reach of the

available GoS funding for education. Two such initiatives are an e-initiative led by MESC and NUS

that has included training for teachers to use e-learning, and the SQA delivered PSET trainer

development programme. In both cases, these programmes are targeted at upskilling the

education sector to deliver education programmes that are fit for purpose in a changing education

environment.

The distribution of ESSP funding via a sector budget support approach, rather than via project

specific funding, has allowed the IAs to utilise the available funds to focus on activities that they

consider will support the long-term achievement of ESP goals and sector outcomes. A clear

example of this is the roll out of activities aimed at supporting teachers to deliver higher quality

education. As noted above, this has included professional development of teachers and principals

to strengthen the education sector which, if appropriately managed, will have a positive impact

on the medium to long term educational outcomes.

It has not been possible for the evaluators to make a definitive determination whether the ESSP

funding would have had more impact had they been used in alternative ways (e.g., more targeted

funding on provision of resources and teacher training to improve literacy and numeracy). The

impact of education investment is always going to be long-term. Attempts to evaluate education

system improvement on a short-term basis are risky, and may lead to false conclusions based on

uneven, partial or inaccurate data.

2.4.2. Impact on families, schools and communities

The direct impact of ESSP funding on families, schools and communities is difficult to specifically

identify as ‘nature’ of budget sector support is that the source of the funding is not ‘visible’ to those

on the frontline.

For most families any impacts would be difficult to identify as specifically linked to the ESSP

funding, although the school grant funds have had the effect of eliminating or reducing school fees.

As noted above, activities targeted at strengthening the education sector and education system

are more likely to have a positive impact on Sāmoan families as they lead to improvements in

education quality and increase in learning outcomes. In saying this, the impact is likely to have

been noticeable for those families of children with disabilities (see section 2.6.2).

The support that the ESSP funding has provided to the teacher qualification upgrade programme

is one noticeable activity that has had a direct impact on schools in respect to the requirement for

their staff to undertake professional development after school hours during term time in addition

to the MESC run professional development that occurs in the school holidays. Beyond this, the

school representatives we spoke to attributed progress in the way funding is allocated to their

schools to either the school grant scheme or other forms of external donation (e.g., community

fundraising initiatives or donations from donors for large capital projects funded by JICA or the

Chinese Government). The ‘nature’ of sector budget support is that the source of the funding is

not ‘visible’ to those receiving it. This creates some difficulties for donors, in that they are unable

to clearly identify what their funding was spent on.

Mission education organisations are concerned about the financial pressure they face in funding

salaries of teachers in schools and tutors in PSET institutions. There is no GoS funding provided

directly for salaries of teachers in Mission schools, although Mission schools qualify for a

Government Grant and (in the schools’ sector) receive access to the school (fee) grants that

available to all schools. Nevertheless, Mission schools in Sāmoa can only afford to pay lower

salaries to teachers in comparison with Government schools, and therefore suffer consequent

regular migration of their staff to positions in Government schools with higher paid salaries. The

result is a lack of qualified teachers in some Mission schools, and regular turnover of staff. This

situation in Sāmoa compares unfavourably with the situation in other Pacific countries such as

Fiji or the Solomon Islands, where Mission school teachers receive salary payments from their

respective Governments.

Evaluation of the Sāmoa Education Sector Support Programme – Final Report 29

2.5. Sustainability

This section seeks to address the key evaluation question, “Are the benefits of the ESSP likely to be

sustainable?” To improve the long-term impact of the ESSP funding, activities should be focused

on strengthening the education sector (for example, the skills of individuals and teams working

within education) and the education system (for example, the tools and resources that are used

by those working in the sector). Several activities identified in the ESP have been undertaken by

the three IAs over the period of the ESSP, and can be seen as focused on building the education

sector and the system that supports it. However, there is still further work to be done.

2.5.1. Capacity building in the education sector

The ESSP has been used to fund initiatives to support capacity growth across the education sector

in Sāmoa (including upgrading teacher qualifications and provide for taskforces and technical

advisory services). Further planning and targeted capacity initiatives are required to ensure the

sustainability of ESSP investments.

Initiatives supported by ESSP funding undertaken over 2016-2018 have included a focus on

capacity building of government officials, principals, teachers and those in governance and/or

management roles within education providers across the full spectrum of the education sector

(ECE, Primary, Secondary, formal and informal PSET/TVET, and IE).

One initiative aimed at harnessing the expertise of people across the education sector in Sāmoa

was to establish taskforces in six key areas – literacy, numeracy, teacher development,

assessment, ECE and IE. In some cases, these taskforces have been supported by the procurement

of technical assistance through ESSP funding set aside for this purpose. It was reported that the

taskforces have met with mixed success. It is disappointing that more has not been achieved by

the literacy, numeracy and assessment taskforces over this period as they are key areas for

improving learning outcomes and further work is required in these areas. The IE taskforce has

been successful in developing policies and plans for the future. This work is discussed further in

section 2.6.2 and should be further integrated into any new ESP and ESSP.

It is noted that there was no evidence that the MTR recommendation to appoint people to the two

key positions at MESC which have leadership responsibility for literacy and numeracy had been

achieved. The absence of leadership and direction continues to have an impact on the ability of

the sector to achieve the expected outcomes in these areas.

The Teacher Development taskforce had a role in revising the National Teacher Development

Framework (NTDF), a key step in supporting the ongoing capacity development of the teacher

workforce to achieve sustainability of the ESSP funded outcomes. The implementation of the

NTDF is key to the targeted upskilling of teaching staff, however, the impact of this unknown as it

has not yet been rolled out.

The ECE taskforce has made significant progress, including leading the development of an ECE

curriculum which is key to the professionalisation of this sector. Historically ECE was managed

and delivered at arm’s length from GoS agencies and most ECE delivery was managed at the

community level, often supported by village churches. IA involvement in the taskforce has seen

more GoS agency input in the ECE space. Furthermore, the one indicator in the M&E framework

for ECE (assessment of school readiness of children completing early childhood education) has

been removed due to a lack of available data.

The evaluators understand that there is some political support for making ECE compulsory in

Sāmoa from age four. The evaluators saw little evidence that the ECE sector is ready for this

requirement. Development of an evidence-based ECE policy (including provision of sufficient

well-trained ECE staff), an implementation plan for the roll out of the ECE curriculum, and

guidance on what is required for ECE facilities to meet the MSS are necessary pre-conditions

before any requirement for ECE attendance comes into force.

Except for the TA support for the ESCD, the targeted TA funding has mostly been used to supply

short term capacity for individual projects such as the Review of Extended school hours

completed in July 2018. While this addresses immediate needs for specific skills, it can be argued

that this is not the most effective approach for the long-term sustainability of the ESP activities.

The evaluators understand the TA guidelines include a reference to capacity building, however

due to delays in task definition and work scoping, only a few of the 19 approved requests for TA

have been filled. An improvement to the processes for engaging TA support is required to ensure

timely recruitment and engagement of experts to provide project and capacity development

support in priority areas.

In speaking with representatives from the three IAs, there was some concern that their staff do

not have sufficient skills to make best use of the education data collected and are unable to

complete the level of analysis required to advise the ESWG and support decision making by the

ESAC. Therefore, it is recommended that this capacity development plan should include activities

to improve the quality of analysis and reporting across the education sector.

2.5.2. Strengthening the education system

The tools and resources used in delivering education (including assessment) and the processes

used by Implementing Agencies to complete monitoring and reporting activities are not always

fit for purpose and more work is required to support the ongoing sustainability of the systems in

use.

In addition to raising the skills of individuals and teams working within the education sector, it is

important they are supported by an education system that is fit for purpose. This includes the

tools and resources used in delivering teaching and learning (including assessment) and the

policies and processes used by the IAs that allow them to complete the necessary monitoring and

reporting activities. This is not currently the case and more work is required. Additional activities

are required to support the achievement of Goal 4 of the ESP “Improved sector coordination of

research, policy and planning development” and the associated sector outcome “Analysis of

research findings, evaluations and monitoring evidence increasingly used to inform policy and

planning across the sector”.

The evaluation team examined the reports outlining the findings of two recent reviews of how

information is managed across the education sector.

More urgency is needed on the project to

develop and implement an EMIS that can be used to collate data to support planning and reporting

within a SWAp. The evaluation team consider this to be a priority project to ensure IAs are able to

make better use of the data available to report learning outcomes.

One key area where the three IAs have previously struggled to collaborate is TVET policy and

planning. SQA has taken the lead and has developed a National Technical and Vocational Education

and Training Policy (2018-2023). This is a positive development. This policy is an initiative of GoS

through the ESAC and the SQA to improve access to TVET, as well as enhancing the relevance,

UNESCO Institute of Statistics. (2018) Sāmoa Education Data Quality Assessment Report, and WizConsult.

(2017) Situational Analysis for Creating Education Sector Management Information System &

Recommendations.

Evaluation of the Sāmoa Education Sector Support Programme – Final Report 31

quality assurance and coordination of TVET systems. NUS has developed a parallel document, a

Technical and Vocational Education (TVET) Strategy for Sāmoa 2018-2022. While NUS is only one

of several providers of TVET in Sāmoa, it is nevertheless the major provider of PSET. It will be

important for the future of TVET development in Sāmoa that these two different policy strategies

are integrated into a coherent national approach that recognises the need for a whole-of-

Government approach. It is for the GOS to determine if ESAC is the right forum for development,

management and support of TVET policies and processes in Sāmoa, or whether an alternative,

such as a well-functioning TVET Oversight Committee (like the idea proposed by NUS), is more

appropriate. Determining the appropriate response to future TVET development is a matter that

needs to be settled as part of the design process for a new ESP and a new ESSP.

The CEOs for SQA and MESC, and the Vice-Chancellor for NUS, informed the evaluation team that

they are currently collaborating on a joint TVET Policy and Strategy that will include secondary

and post-secondary provision of TVET. This type of collaboration should be encouraged to

strengthen the system, supporting this important area of provision, so that a more coherent and

sustainable approach is in place in the future. Similarly, the three agencies should be encouraged

to work together on any future policy and plans around IE and ECE provision to ensure

consideration of provider quality and teacher training are developed using a SWAp.

2.6. Cross-cutting issues

This section seeks to address the key evaluation question “To what extent have cross-cutting issues

(gender equality and disability inclusive development) been identified and pursued in the ESSP?”

2.6.1. Gender equality

Currently male students have lower achievement in core areas (literacy and numeracy) at lower

levels of schooling and are less likely to transition to formal Post-Secondary Education and

Training. More work is needed to ensure that enough appropriate education opportunities are in

place to ensure that boys’ education needs are being met at primary, secondary and PSET levels.

During the ESSPs period of operation it appears that gender equality has not been identified as an

area of focus beyond data collection and analysis by gender. Historically, gender equality in

education has focused on access to educational opportunities for females. Sāmoa is achieving well

in this area; however, there is evidence of other issues related to gender equality.

There are four KPIs within ESSP that report achievement results by gender:

• KPI 1: Percentage of Year 4 and 6 children meeting a minimum of Level 3 for Literacy

(English and Sāmoan) by gender;

• KPI 2: Percentage of Year 4 and 6 children meeting a minimum of Level 3 or better for

Numeracy by gender;

• KPI 4: Percentage of children commencing Year 1 Primary and completing Year 8, by

gender; and

• KPI 5: Transition rate from Year 13 to formal PSET.

There are poorer results for males than females in three of these KPIs (literacy, numeracy,

transition to formal PSET). KPIs 1 and 2 show that females are consistently outperforming males

in these key result areas.

As shown by KPI 4, access to, and completion of, primary education is

consistent across both genders. However, access to formal PSET is significantly lower for males.

In contrast to the data included in the latest IVP, the latest data available for PSET enrolments and

graduations show the gross enrolment numbers in formal PSET in 2017 were 1,190 for females,

and 1,436 for males. The graduation ratio from formal PSET in 2017 was 39.5 percent for females

and 38.2 percent for males.

This data has not been verified in the same way that the IVP data

has. Further analysis is required to identify reasons for these gaps.

Table 3: Percentage Achievement against KPIs 1, 2, 4 and 5 by Gender - 2018 IVP report

KPI 1:

Year 4

English

Literacy

KPI 1:

Year 6

English

Literacy

KPI 1:

Year 4

Sāmoan

Literacy

KPI 1:

Year 6

Sāmoan

Literacy:

KPI 2:

Year 4

Numeracy

KPI 2:

Year 6

Numeracy

KPI 4:

Primary

completion

rates

KPI 5:

Transition

to formal

PSET

Male 32% 36% 48% 52% 37% 50% 80% 37%

Female

48% 54% 62% 69% 46% 58% 81% 52%

It is also worth noting that Sāmoa College which has an entrance exam at Year 9 and takes the 130 top

performing students typically enrols two girls for every boy.

Sāmoa Qualifications Authority. (2018) Sāmoa Post School Education and Training (PSET): Statistical

Bulletin 2017. SQA, p. 3.

Evaluation of the Sāmoa Education Sector Support Programme – Final Report 33

The results shown in Table 3 above, are taken from the 2018 IVP results report. Further work is

required to ensure that there are sufficient appropriate education opportunities in place to ensure

that boys’ needs are being met through formal and informal education.

2.6.2. Inclusive education

Ring-fenced funding for inclusive education has been successful. The development of an Inclusive

Education Policy Implementation Plan 2017-2020 and associated tools for more accurate

identification of people with disabilities have supported the inclusive education sector to be better

prepared for future mainstreaming of people with disabilities in education.

The ESSP evaluation included a review of the relevance, effectiveness, efficiency, impact and

sustainability of the funding of SAT$6,126,185 across FY2015-16 to FY2017-18 from DFAT

specifically for IE disbursed through the sector budget support mechanism and ring-fenced by

MoF from the 70 percent tranche of the sector budget support mechanism. Overall, the evaluation

determined that the approach to ring-fence the funding for IE has been successful. While there is

still much work to be done, the sector is better prepared for future mainstreaming of people with

disabilities in education than it was in 2016.

Following the release of the SIEDP evaluation report in late 2016, MESC and DFAT prepared a

management response to the findings and recommendations outlined in the evaluation report

including agreed actions (November 2017). This response followed the release of the Inclusive

Education Policy Implementation Plan 2017-2020 in June 2017.

The policy also includes an M&E plan with draft indicators against which progress will be

measured on an annual basis. The indicators identified are expected to guide reporting by service

providers; the IE Taskforce will monitor progress against these indicators. Further work is needed

to ensure these indicators are SMART. Once approved, the Implementation Plan will be submitted

to the ESCD for incorporation into the M&E Framework.

The evaluation team understands that that a memorandum of understanding (MOU) was signed

between MESC, and the IE Service Providers and Special Schools in March 2018. This saw the

achievement of one of the IE Policy Implementation Plan Outcomes as the main purpose of the

MOU is to clarify the current and planned services that are central to the IE Policy. It also clarifies

which services MESC will support each special school and provider to deliver as part of the IE

policy. This is an indication of a partnership between MESC and the IE service providers and

special schools, as well as DPs in implementing the IE policy.

The evaluators understand that work is underway to revise MESC’s Annual School Survey by

incorporating the Washington Group questions to support more accurate identification of

students with disabilities. An initial implementation of this new approach to disability

identification (undertaken by Senese in March 2018) saw a jump from 140 to 270 students with

disabilities identified in mainstream schooling. The evaluators understand that the Washington

Group questions are also used to inform the Sāmoan Census and alignment between these two

approaches to data collection about people with disabilities should help to strengthen cross-

sector (i.e. health, education, and women, community and social development) activities to

support Sāmoans living with disabilities. More work is needed to identify and assist people with

disabilities in the PSET sector.

It was noted that students with disabilities are not always getting timely access to the health

specialists they need. Whilst the provider Senese does its best, it is not able to provide the

specialist services required (e.g., speech therapy, educational psychology, or physical therapy). It

was suggested that by the time the assistance becomes available the child has often been moved

on to a special school or has been removed from schooling altogether.

Stakeholders identified that one benefit of incorporating IE within the wider MESC work

programme has been an increase in the perceived legitimacy of the providers working in this area.

This is particularly important for Senese who interact directly with schools that have students

with disabilities enrolled. It was reported that in the past Senese have had difficulties getting

access to students within the classroom. However, Principals now appear to have a better

understanding of their work.

3. LESSONS LEARNED

The key lessons learned from the evaluation of the ESSP to inform future iterations of DP funding

support for the education sector in Sāmoa are related to the need for an adequate time frame to

implement a new approach; the need to recognise in-country capacity; the need for improved

communication about funding matters; and streamlining of data collection, monitoring and

reporting requirements.

The lessons learned can be summarised as follows:

• Take the time that is necessary at the outset of a major programme to plan well. Both

DPs and GoS agencies need to start planning early, allow time for GoS processes to be

completed, and be realistic about implementation time frames. International experience

suggests that implementation of a budget support modality needs a long-time frame of

at least five years before the benefits begin to become apparent.

• Start with small manageable steps and plan activities that can be phased in gradually in

order to achieve sustainability. Lack of attention to in-country capacity, adoption of

overly ambitious targets, and insufficient prior analysis of the country context are

recurring lessons in the Pacific.

• Align the budget planning and reporting requirements for any sector budget support

from development partners with those of the GoS Budget processes.

• Ensure there is effective and timely communication and transparency about the funding

available, and about the expectations with respect to performance of both IAs and MoF.

• Consult during the development of individual performance indicators, to ensure the data

required to report progress and achievement is available in the form required.

• Include clear guidelines for how performance-based funding will be distributed across

sector agencies when KPIs are achieved.

Evaluation of the Sāmoa Education Sector Support Programme – Final Report 35

4. EVALUATION CONCLUSIONS

The evaluation of the ESSP provided an opportunity to interview stakeholders and review the

supporting operational and policy documentation for the ESSP and ESP. Our analysis shows the

ESSP has a clear strategic framework aligned with Australian/New Zealand international

development policy and GoS development objectives, and the educational objectives of education

providers. While there were some serious initial challenges faced by the GoS in implementing the

programme, and the overall objectives were too ambitious, the IAs with responsibility for the

delivery of the ESP have made progress in many areas supported by the ESSP. There have been

gains in areas such as teacher development and appraisal, ECE and IE. The evaluation found the

additional activities made possible by the ESSP investment have addressed the priorities of target

groups such as schools and other education providers, and have targeted key issues facing the

education sector such as the need to improve literacy and numeracy.

The expressed commitment and hard work of staff has been a major factor enabling the

achievement of ESSP objectives. The limited capacity of staff to monitor, analyse and report on

data, the lack of timeliness in reporting, and staff turnover are factors that have constrained

further success. While there has been positive learning about management of the ESSP, the time

frame over which the ESSP has been operational has been too short to be able to point to hard

evidence of improved learning outcomes.

The challenges faced by the education sector in Sāmoa in implementing the ESSP relate

particularly to the development and implementation of new approaches to the sector budget

support modality, which the ESSP, and specifically the IVP, necessitated. Misalignment between

the ESSP processes and GoS Budget timelines created inefficiencies over the life of the ESSP

programme. Better alignment between these timelines and processes would make the operation

of the programme more efficient. The fixed tranche disbursement of the budget support

component of the funding modality (70 percent) is appropriate, although it is suggested that to

ensure predictability of funding and more effective planning, the fixed tranche amount should not

be reduced unless there are exceptional circumstances. The performance-based approach to the

other 30 percent of the ESSP funding, monitored by the IVP, should be continued, but with

modifications to ensure that there is wide consultation on any KPIs selected for a future

programme. The ring-fencing of funding for IE should continue, since the targeting made possible

by this arrangement is slowly bringing results.

Investment in a programme of staff capacity development is necessary in order to improve the

analysis of data and reporting against key performance objectives. Streamlining and simplifying

the ESSP monitoring tools such as the monitoring and evaluation framework and the medium-

term expenditure framework will be essential. If these adjustments are made, there is every

prospect that the planned outcomes of the ESSP related to improving student learning and

building the capacity of the education system can be achieved over time.

5. RECOMMENDATIONS

Continue to use a sector budget support modality to deliver any future ESSP-type

funding with the following conditions:

better alignment between the ESSP reporting and planning timelines and the

established timelines of the GoS Budget process;

expert guidance in the development of performance indicators for any future

ESP, and selection for inclusion in an ESSP, to ensure the KPIs are SMART

(specific, measurable, achievable, relevant and time-bound), and that the

required evidence is able to be collected and reported;

accountability requirements must be satisfied by regular reporting against

appropriate KPIs; and

the ‘fixed tranche’ component of the education sector budget support modality

should be delivered annually in total if a reasonable proportion of any fixed

process indicators are met.

Continue to use an Independent Verification Process for a small number of agreed

KPIs, with the following conditions:

include clear guidelines on how the individual KPIs for the sector relate to the

release of the performance funding component; and

specify the steps for the release of funding, including whether any IA is entitled

to a share of the funding disbursed.

Prepare a capacity development plan for staff of GoS agencies with involvement in all

levels of the education sector (e.g., MESC, SQA, NUS) to ensure those working in these

agencies have the necessary skills to support future planning and reporting. This plan

would sit alongside the National Teacher Development Framework, and include

capacity development plans for the ECE, PSET/TVET and IE sectors.

Improve the processes for engaging technical assistance to ensure timely recruitment

and engagement with a focus on capacity development in priority areas. For example,

resourcing a reading specialist in the Education Faculty of NUS to help ensure

teachers (participating in pre-service and in-service training) have the skills required

to teach reading.

Support current work to develop a Sāmoa Education Data Quality Improvement Plan,

and support implementation of the plan in order to strengthen data collection and

analysis, monitoring, and reporting.

Update, rationalise and simplify the M&E Framework to ensure that it can be used

effectively, and consider whether it gives the necessary focus to the range of activities

across the education sector (e.g., ECE, IE, primary, secondary, formal, and informal

PSET and TVET).

Improve the integration of MTEF update processes with other reporting to ensure the

necessary updates actually occur.

Evaluation of the Sāmoa Education Sector Support Programme – Final Report 37

REFERENCES

Adam Smith International [ASI]. (2015) Evaluation of the Sāmoa Country Programme. ASI.

—. (2017) Sāmoa Education Sector Plan 2013-2018: Mid-Term Review Report. ASI.

Australian Government (Department of Foreign Affairs and Trade), Government of Sāmoa & New

Zealand Ministry of Foreign Affairs and Trade. (August 2015) Joint Funding Arrangement

(Arrangement) Sāmoa Education Sector Support Program (ESSP). Ministry of Foreign

Affairs and Trade.

Author Unknown. (2015) Investment Design for Sāmoa Education Sector Support Program (ESSP),

July 2015 to June 2018. Ministry of Foreign Affairs and Trade.

—. (2016) Sāmoa Inclusive Education Demonstration Project: Project Evaluation Report, December

2016. Department of Foreign Affairs and Trade.

—. (2017) Sāmoa Early Grade Reading Assessment Report. World Bank.

Asian Development Bank [ADB]. (2016) Completion Report: Sāmoa: Education Sector Project II.

ADB.

Boyd, Shayne. (2017a) ‘Bilingual Policy in Sāmoa’, unpublished manuscript.

—. (2017b) ‘Introduction (Three Year 8 Examinations)’, unpublished manuscript.

—. (2018a) ‘Student Results across Sāmoa’, unpublished manuscript.

—. (2018b) ‘Sāmoa Secondary Leaving Certificate Prescription: A Comparison of the 2013

Prescription and the 2018 Prescription’, unpublished manuscript.

—. (2018c) ‘Curriculum Matters Discussion Paper’, unpublished manuscript.

—. (2018d) ‘Analysis of the 2017 SSC and SSLC examinations/examiners’ reports’, unpublished

manuscript.

—. (2018e) ‘Discussion paper Three – Literacy in the Secondary System in Sāmoa’, unpublished

manuscript.

Brooker, Fred, Pongi, Visesio, & Tuioti, Lili. (2016) Sāmoa Education Sector Independent

Verification Report.

Brooker, Fred, von Dadelszen, Jane, & Pongi, Visesio. (2017) Sāmoa Education Sector Independent

Verification Report.

Catherwood, Vince & Taylor, Lester. (2016) End of Programme Evaluation: Sāmoa (Primary)

School Fee Grant Scheme. Wellington: Vince Catherwood & Associates Ltd.

Education Sector Coordination Division [ESCD]. (2018a) Education Sector Quarterly Progress

Report: Quarter One 2017-18, July-August-September 2017. ESCD.

—. (2018b) Education Sector Quarterly Progress Report: Quarter Two 2017-18, October-November-

December 2017. ESCD.

Government of Sāmoa [GoS]. (2013) Sāmoa Education Sector Plan, July 2013 – June 2018: Improved

Focus on Access to Education and Training and Quality Learning Outcomes. GoS.

—. (2015) Quarterly Progress Report, Developmental Projects. GoS.

—. (2017) ‘Management Response to Recommendations from the Sāmoa Inclusive Education

Demonstration Program Evaluation’, unpublished manuscript.

—. (n.d.) Education Sector Fourth Annual Review FY 2016-2017. GoS.

Ministry of Commerce, Industry and Labour [MCIL]. (2016) Sāmoa Labour Market Survey of

Private Sector Employers Report 2016. MCIL.

Ministry of Education, Sports and Culture [MESC]. (2006) Strategic Policies and Plan, July 2006 –

June 2015: Sāmoa. MESC.

—. (2007) Education for All: Mid-Decade Assessment Report SĀMOA 2007. MESC.

—. (2014) Annual Report: 1 July 2013 – 30 June 2014. MESC.

—. (2015) Annual Report: 1 July 2014 – 30 June 2015. MESC.

—. 2016a) Minimum Service Standards for Primary and Secondary Schools in Sāmoa. MESC.

—. (2016b) Annual Report: 1 July 2015 – 30 June 2016. MESC.

—. (2017a) Annual Report: 1 July 2016 – 30 June 2017. MESC.

—. (2017b) Education Statistical Digest. MESC.

—. (2017c) ‘Sāmoa Early Grade Reading Assessment (SEGRA) 2017. Briefing Note for the

Honourable Minister of Education, Sports and Culture’, unpublished manuscript.

—. (n.d.) National Teacher Development Framework 2018-2028. MESC.

—. (n.d.) Corporate Plan: July 2015 – June 2018. MESC.

Ministry of Finance [MoF]. (2012) Strategy for the Development of Sāmoa, 2012-2016. MoF.

—. (2016) Strategy for the Development of Sāmoa, 2016/17-2019/20. MoF.

Ministry of Foreign Affairs and Trade. (n.d.) ‘New Zealand – Sāmoa Joint Commitment for

Development’, unpublished manuscript.

National University of Sāmoa [NUS]. (2018) Technical and Vocational Education (TVET) Strategy

for Sāmoa 2018-2022. NUS.

Pongi, Visesio and von Dadelszen, Jane. (2018a) Sāmoa Education Sector Independent Verification

Report.

—. (2018b) Sāmoa Education Sector IVP Recommendations Report Final Report.

Sāmoa Bureau of Statistics [SBS]. (2017a) Statistical Abstract 2017. SBS.

—. (2017b) 2016 Census Brief No. 1. Revised Version: Population Snapshot and Household

Highlights. SBS.

—. (2018a) 2016 Census Brief No. 2: Population Dynamics and Trends. SBS.

—. (2018b) 2016 Census Brief No. 3: Education and Economic Characteristics. SBS.

Sāmoa Bureau of Statistics, Ministry of Women, Community and Social Development, Pacific

Community & UNICEF Pacific (2018). 2018 Sāmoa Disability Report: An Analysis of 2016

Census of Population and Housing. Suva: UNICEF.

Evaluation of the Sāmoa Education Sector Support Programme – Final Report 39

Sāmoa Qualifications Authority [SQA]. (2010) Sāmoa Post School Education and Training (PSET):

Statistical Bulletin 2010. SQA.

—. (2012). Post School Education and Training (PSET): Statistical Bulletin 2011. SQA.

—. (2013) Post School Education and Training (PSET): Statistical Bulletin 2012. SQA.

—. (2014) The Sāmoa Qualifications Authority PSET Statistical Bulletin 2013. SQA.

—. (2016) The Sāmoa Qualifications Authority PSET Statistical Bulletin 2015. SQA.

—. (2017) Sāmoa Qualifications Authority PSET Statistical Bulletin 2016. SQA.

—. (2018a) Sāmoa Qualifications Authority PSET Statistical Bulletin 2017. SQA.

—. (2018b) Survey for Employer Satisfaction with 2012, 2013 & 2014 Graduates from Formal Post

School Education and Training (PSET) Providers in Sāmoa. SQA.

—. (2018c) National Technical and Vocational Education and Training (TVET) Policy 2018-2023.

SQA.

Sāmoa Qualifications Authority, Ministry of Education, Sports and Culture & National University

of Sāmoa [SQA, MESC & NUS]. (n.d.(a)) Sāmoa Education Sector Plan 2013/2014 Annual

Review Report. SQA, MESC & NUS.

—. (n.d.(b)) Education Sector Plan 2013-2018: Second Annual Review Report FY2014-2015. SQA,

MESC & NUS.

—. (n.d.(c)) Education Sector Third Annual Review Report FY2015-2016. SQA, MESC & NUS.

—. (n.d.(d)) Education Sector Fourth Annual Review FY 2016-2017. SQA, MESC & NUS.

UNESCO Institute of Statistics. (2018) Sāmoa Education Data Quality Assessment Report.

WizConsult. (2017) Situational Analysis for Creating Education Sector Management Information

System & Recommendations.

APPENDIX A: TERMS OF REFERENCE

External Evaluation for the Sāmoa Education Sector Program Terms of Reference

This Terms of Reference (TOR) is for the external evaluation for the Sāmoa Education

Sector Support Programme (ESSP) and is jointly undertaken by the Australian

Department of Foreign Affairs and Trade (DFAT), and the New Zealand Ministry of

Foreign Affairs and Trade (MFAT). The ESSP is delivered through budget support and

is jointly funded by MFAT and DFAT. It is led by the Government of Sāmoa targeting its

education priorities set in the Education Sector Plan July 2013-June 2018.

1. Overview and Evaluation Purpose

DFAT and MFAT wish to commission a joint external evaluation of the Sāmoa Education

Sector Program to assess its likely relevance, efficiency, effectiveness, sustainability

and short- and medium-term impact on the wider Sāmoan Education Sector.

The evaluation will inform any further commitments of Australia and New Zealand

and the funding modalities and assessment frameworks used.

2. Background

The Government of Sāmoa (GoS) recognises that the education and training of its

citizens is vital to the country’s development. In the Strategy for the Development of

Sāmoa (2012-2016) the GoS made it a national priority to improve access to education,

training and learning outcomes.

The education sector comprises government and non-government primary and

secondary schools; early childhood education (ECE); post-school education and

training (PSET); and the policy, planning and regulation bodies.

In accordance with Sāmoa’s commitment to the Millennium Development Goal (MDG)

of universal completion of primary education,

primary education from Years 1 to 8 is

compulsory for children between the ages of 5 and 14. Secondary education covers

Years 9 to 13. Students sit the Sāmoa School Certificate at the completion of Year 12,

after which successful students can attend a further year to sit the Sāmoa Secondary

Leaving Certificate (SSLC) in Year 13.

PSET encompasses a diversity of areas that include tertiary education at university;

pre- and in-service professional education; technical and vocational education and

training (TVET); theological and providers of religious instruction; apprenticeship;

non-formal and on the job training.

Prior to 2011 the Ministry of Education, Sports and Culture (MESC), the Sāmoa

Qualifications Authority (SQA) and the National University of Sāmoa (NUS) operated as

separate agencies with limited coordination. Recognising the need for increased

collaboration, the GoS began requiring all Ministries to develop a Sector Wide Approach

(SWAp) in order to more effectively coordinate financial and human resources and

improve service delivery.

Sāmoa’s Education Sector Plan still refers to Millennium Development Goals. The Sustainable Development

Goal of relevance is No. 4 – Ensure inclusive and quality education for all and promote lifelong learning.

Evaluation of the Sāmoa Education Sector Support Programme – Final Report 41

Education Sector Plan 2013-2018 (ESP)

Education authorities in Sāmoa with the assistance of development partners developed

the Education Sector Plan (ESP). The ESP sets out how the various education agencies

will work together as a sector, building on previous SWAp-like programmes and a

number of large donor funded projects. The current ESP is in place for five years from

July 2013 until June 2018.

It has five goals, with associated Sector Outcomes to be

achieved by 2018 as below:

Goals Sector Outcomes

1 Enhanced quality of education at all

levels.

Improved learning outcomes at all levels

2 Enhanced educational access and

opportunities at all levels.

At all levels, more students, including those with special

needs, have access to quality educational opportunities in

safe, climate- resistant learning environments

3 Enhanced relevance of education at all

levels.

Improved employability of school leavers as a result of

education and training responding to national economic,

social and cultural needs

4 Improved sector coordination of

research, policy and planning

development.

Analysis of research findings, evaluations and monitoring

evidence increasingly used to inform policy and planning

across the sector

A coordinated approach through effective partnerships

with key stakeholders ensures newly developed and

implemented policies contribute to improved quality

across the education sector

5 Established sustainable and efficient

management of all education resources.

Education resources are increasingly managed efficiently

and sustainably across the sector

Three Implementing Agencies (IAs) are responsible for implementing the ESP:

- The Ministry of Education, Sport and Culture (MESC) focuses on early

childhood, primary and secondary education.

- The Sāmoa Qualifications Authority (SQA) has responsibility for regulation,

accreditation and quality assurance in post-secondary education and training.

- The National University of Sāmoa (NUS) provides tertiary education including

teacher training; TVET; and, continuing professional learning through the

Oloamanu Centre for Professional Development (OCPD).

The ESP contains 22 programme activities. They include programme’s targeting

specific issues, or strengthening systems for planning, monitoring and financing the

sector, and for sector coordination.

The ESP functions through a coordinated governance structure designed to facilitate

joint decision-making amongst sector partners and the incorporation of input from a

range of stakeholders. The Education Sector Advisory Committee (ESAC) and the

GoS currently has plans to review and potentially revise the ESP.

Education Sector Working Group (ESWG) provide governance. Broader input is elicited

through a series of special interest Task Forces. The ESWG has responsibility for

coordination and working with the committees, IAs, Development Partners and other

stakeholders in order to meet requirements for monitoring, evaluation and annual

reporting. The ESAC is a decision-making body based on submissions from the ESWG.

The Medium-Term Expenditure Framework (MTEF) is intended to be updated annually

to guide budgeting for the sector. The MTEF covers recurrent expenditures, including

increases in these expenditures arising from ESP implementation e.g., from increased

teacher salaries and school and PSET grants. It also includes expenses for capacity

building activities.

Education Sector Support Program

The Education Sector Support Program (ESSP) is a 3-year activity between MFAT and

DFAT, designed to specifically support and finance implementation of elements of the

ESP. The ESSP commenced on October 2015 (Australia) and March 2016 (New

Zealand) and will be completed in December 2018. The total value of the investment is

SAT $35,865,685.

The goals of the ESSP are taken from the ESP. The Strategic Outcomes are:

Increased numbers of 3 – 5 year olds participating in Early Childhood Education.

Improved literacy and numeracy scores at years 4 and 6.

Increase in percentage of children commencing Year 1 Primary and completing

Year 8.

Increased numbers of children with special education needs in inclusive

classroom settings with Individual Education Plans.

Increase in participation rates and outcomes of Year 12 examinations.

Increase in transition rates to post-secondary education.

Increase in the employment rate of graduates of post-secondary education and

training.

In particular the ESSP investment aims to:

- Strengthen Sāmoa’s new approach to the development of the education sector as

a coherent whole.

- Support implementation of reform policies designed to improve learning

outcomes and skills development linked to realistic employment expectations.

- Strengthen government systems for sustainable achievement of improved

learning outcomes and skills development for employment.

- Support achievement of Sāmoa’s National Goals and Strategies, through a well-

educated and skilled Sāmoan society.

Funding Modalities

ESSP support to the education sector utilizes three funding modality types:

Evaluation of the Sāmoa Education Sector Support Programme – Final Report 43

- Sector budget support based on fixed-process indicators: general financing in

support of key operational areas identified in the Sāmoa Education Sector Plan

2013-2018 (70 per cent).

- Performance-linked contributions linked to the achievement of agreed key

performance indicators with financing (30%) released based on GoS’s

performance against key performance indicators (KPIs) assessed via an

Independent Verification Process (IVP).

- Ring-fenced funding for Inclusive Education (DFAT only): this is disbursed

through sector budget support and ring-fenced by the Ministry of Finance for

inclusive education providers.

Budget Support Allocations 2015-2018:

Budget Support % 2014/15 2015-16 2016-17 2017-18

Fixed process

indicators (12)

No IVP

9.5/12

55 7.5/12 44 No fixed

assessment

Performance linked

contribution (9 KPI)

1.5/9

5/9 17 5/9 60%

100%

60%

61%

60%

The table above shows the trend of the IVP since the ESSP began. The verification of the fixed and

performance indicators in FY2015-2016 showed the achievement of only 1.5/9 performance

indicators and 9.5/12 fixed process indicators. FY2016-2017 demonstrated strong improvement

with 5/9 performance indicators achieved with a slight drop in 7.5/12 fixed process indicators

achieved. FY2017-2018 did not assess the fixed process indicators as payment was made in the

previous financial year, the results for the performance indicators showed an achievement of 5/9

KPIs. Overall, there has been improvement in achieving the performance indicators while

achievement of the fixed has steadily declined.

Programme management and implementation arrangements

There are three groups of stakeholders critical to the governance and management of

the ESSP.

- The agencies responsible for implementation of the ESP: MESC, SQA and NUS

- The Ministry of Finance (MoF) and the Public Service Commission (PSC)

- Development partners: Australia and New Zealand

These bodies come together at head of agency senior levels, together with

representatives of private sector schooling and civil society (represented by the Sāmoa

Umbrella for NGOS) on the ESAC. The key task of the ESAC is to monitor and guide the

progress of the ESP at a strategic level.

A key monitoring mechanism is the annual Education Sector Review, which involves

representatives from all sector stakeholders and includes consultations with the wider

public. This provides an opportunity to review progress, identify challenges and realign

activities and targets.

Results measurement, monitoring and evaluation

The monitoring and evaluation framework of the ESP provides the basis for the results

measurement of the ESSP. It includes all of the measures identified under the education

outcomes sought by the ESSP. The Education Sector Coordination Division (ESCD) at

MESC is responsible for collating the overall monitoring information on the ESP from

MESC, SQA and NUS for the monthly ESWG meetings, the quarterly ESAC meetings and

the Annual Reviews.

The ESP sets out a range of sector and sub-sector level outcomes to be achieved by June

2018. It has undergone some modifications since the original design in 2012, including

some rewording and the revision of KPIs. Mechanisms for tracking progress against

outcomes have evolved over time in response to input from MoF and development

partners, through recommendations from the IVP and the four Sector Annual Reviews

(2013-2014, 2014-2015, 2015-2016, and 2016-2017).

Cross-cutting issues

The values and principles of inclusive education figure prominently in MESC policy

frameworks and related documents. In particular, meeting the needs of students with

disabilities is a very strong focus of existing and new education policy development for

inclusiveness in Sāmoa. The ESP aims to facilitate access to mainstream schools for

children with disabilities, and will support special education for children with

disabilities where this is the most appropriate setting.

The issue of gender imbalance is also targeted in ESSP with a number of key needs to

be addressed including: male drop-out rates in secondary education; low female

enrolment in technical TVET courses; and the low enrolment of males in higher

education.

Other issues are equitable access, participation and completion of good quality

education and training at all levels for children and youth from rural and remote parts

of Sāmoa, and from all socio-economic levels.

Mid-Term Review (MTR) of ESP

In 2016 the education sector conducted an independent MTR of the ESP to measure

progress against its goals. The review team, comprised of two education consultants

and one evaluation expert, completed an analysis of extensive documentary evidence

and conducted a field visit from 3 to 28 October 2016 to interview stakeholders. The

final MTR report was released in February 2017.

The MTR concluded that after three years of implementing the ESP, only gradual

progress has been made against the 34 indicators reviewed, including the 9 KPIs. 9

indicators had been achieved and 9 indicators partially achieved (but deemed

achievable over the subsequent three years). 13 indicators were not achieved and 3

were deemed very unlikely to be achieved by 2018.

The MTR found that robust data for reporting progress was lacking in a number of areas

and in some cases the indicators used did not align to international standards. Some

progress was being made in key areas (such as enrolment rates) but significant

challenges remained in addressing numeracy and the gap in education achievement

between females and males. The most intractable problem was found to be ineffective

implementation of new policies and processes.

Evaluation of the Sāmoa Education Sector Support Programme – Final Report 45

A number of issues were found to be limiting the effectiveness of the ESP. These

included weak school-based support, the limited impact of professional development

on teaching and learning in schools, and ongoing challenges with bilingual education.

The sector had demonstrated a willingness to coordinate in key areas but the tools for

effective, strategic coordination did not exist. The SWAp approach was deemed

complex and the ESCD was found to be struggling to effectively implement the

approach. Progress against Goal 4 was weakest and significant effort needed to be

expended to progress this goal.

It was impossible to determine the efficiency of the ESP as the financial data required

to assess cost-effectiveness did not exist. There were also concerns regarding the

sustainability of ESP outcomes due to limited financial and human resources and the

ineffective implementation of policies.

The MTR made a number of recommendations to the sector to implement between

February 2017 and 2018, these included the need for:

- provision of financial and human resource support; for the ESCD in the areas of

strategic management and coordination;

- development of new performance-based indicators for inclusive education;

- development of a centralized EMIS;

- development of new relevant and measurable indicators for Early Childhood

Education;

- definition of clear roles and responsibilities for Task Forces;

- review of the validity of outcome indicators where data constraints exist; and

- prioritization of sector wide gender analysis of access and performance and

development of a medium-term strategy for addressing gender issues.

Results of the MTR have been accepted by ESSP stakeholders. Thus, the current

evaluation does not need to revalidate findings. Rather, the evaluation should build on

MTR findings, alongside assessments from the intervening period, to respond to key

evaluation questions.

Sāmoan Qualifications and National Competency Standards have been developed and

applied in PSET. SQA conducts triennial tracer studies using the results to enhance the

relevance of PSET policy and practice, and to strengthen linkages between the PSET

system in Sāmoa and the labour force development needs.

SQA has invested substantial resources to build capacity within TVET subsector and

providers are supported to offer National Competency Standards and Sāmoa

Qualifications.

There is limited data to measure PSET graduates’ satisfaction and the relevance of their

knowledge and skills in the workplace. SQA has put in place a quality assurance system

which is in line with ESP monitoring and evaluation requirements and is reflected in

the organization’s management plan. SQA has also implemented a strategy for

recognition of Sāmoan qualifications against international benchmarks.

3. Evaluation Scope

The evaluation will cover all objectives of the ESSP since commencement of the

program in July 2015. It will include a specific review of management and governance

arrangements, progress in achieving the programme’s intended impact and outcomes,

based on key documents identified and any adjustments made to the design during the

duration of the programme.

The evaluation will also consider the achievements of the education and skills

development outcomes for 2014 and 2015, the 2 years preceding the ESSP – a period

when Australia and New Zealand provided project support.

4. Indicative Evaluation Criteria and Questions

The following table presents only draft evaluation questions. These can be refined in

the scoping phase of the project, and a clear set of key evaluation questions with sub-

questions should emerge in the Evaluation Plan.

Criteria

Education

Management and Systems

Relevance

• Does ESSP have a clear strategic framework, and is this aligned with the DFAT and

MFAT development policy and the GOS development objectives?

• How well has the programme articulated the objectives in terms of the theory of

change in the ESSP?

• Did the

programme address the key issues

facing the education (ECE, primary,

secondary, TVET, higher) sectors?

• Was the funding modality the most

appropriate/effective way to achieve the

intended outcomes?

Effectiveness

• Review the Monitoring and Evaluation Framework. Has this been an effective tool for

monitoring progress? Consider the OECD DAC budget support evaluation methodology.

• How effective is the Independent Verification Process (IVP) and is this something we

would wish to continue?

• How effective has the governance of the programme been?

• What outcomes (positive and negative and unintended) have occurred because of the

activities?

• What has constrained or enhanced the achievement of outcomes?

• Has the programme met its key

objectives

of improving learning?

• To what extent has the ESSP contributed

to the strengthening of:

• Literacy and numeracy

• Teacher development (appraisal and

PD)

• Classroom-based assessment

• School leadership (MSS)

• Employability including accreditation,

registration

• Inclusive Education

• Have the management structures

improved coordination among the

agencies?

•

To what extent has the development of IA

staff capacity helped the programme

meet set objectives?

•

Has data and an information management

system (EMIS) been utilized for evidence-

based planning and decision-making?

• What can be done to improve effectiveness?

Evaluation of the Sāmoa Education Sector Support Programme – Final Report 47

Criteria

Education

Management and Systems

Efficiency

• Has the programme been managed and delivered cost-effectively against planned

budgets, expenditure and results achieved?

• What has been the return on investments in terms of systems strengthening, service

delivery and Value for Money?

• To what extent has it added value (TA, Policy, governance, managing for results),

reduced transaction costs, and reduced duplication?

• To what extent has the program led to policy implementation?

• Are the agencies using efficient methods

in the delivery of key activities:

• Literacy and numeracy

• Teacher development (appraisal and

PD)

• Classroom-based assessment

• School leadership (Meeting the

Minimum School Standards)

• Employability including accreditation,

registration

• Inclusive Education

• How have the management

arrangements (including donor

coordination, donor TA and connections

back to Wellington and Canberra)

supported the achievement of the activity

outcomes? What has worked well? What

has slowed progress?

• How have the governance arrangements

(including the ESAC and ESWG, the sector

coordination unit and the IA leads)

supported the management of risks,

resolution of programme issues and

achievement of ESSP outcomes?

•

How well has MFAT/DFATs engagement to

support the education sector (policy

dialogue, participation in management

bodies) in Sāmoa

worked?

• What can be done to improve efficiency?

Sustainability

• Is it likely that the preferred modality will be sustainable, particularly in reducing

transaction costs and fragmentation, facilitating policy engagement and leveraging

donor investments?

• To what extent has the ESSP has supported the financing gap in education?

• To what extent has the programme laid a

foundation for future donor partner

engagement and what critical lessons

have been learnt?

• To what extent are partner government

and agencies likely to be able to sustain

the skills, management capacity and

funding for the ESSP?

• To what extent have the management

skills and knowledge across the IAs

improved and can any improvement

identified be sustained?

• What can be done to improve sustainability?

Impact

• Is the approach to sector budget support likely to achieve what it was expected to

achieve? If not, why not? What has worked well? What has not worked so well?

• What can be done to improve impact?

Criteria

Education

Management and Systems

Lessons

Learned and

Cross-Cutting

Issues

• What are the key lessons from the ESSP – what works and what does not?

• To what extent have cross cutting issues, in particular gender equality and disability

inclusive development been effectively addressed in planning, implementation and

monitoring and evaluation? How have they been integrated into the sector? What is

recommended to improve and to strengthen these?

• Where are the constraints/opportunities for enhanced (a) sector engagement; and

(b) donor engagement to further achievement of outcomes? How can these be

exploited in the future?

5. Evaluation Approach

The methodology will be refined in consultation with the selected consultants and

presented in the Evaluation Plan. The evaluation will consist of a desk review and in-

country consultations with key stakeholders. The consultant team’s evaluation plan can

revise and re- scope the evaluation questions and reduce the number of questions in

order to improve focus. The Evaluation Plan will require DFAT and MFAT approval.

The methodology should include:

the OECD- DAC budget support methodological approach;

a desk review of documentation relating to DFAT and MFAT’s education response

and partner documentation;

interviews with internal and external stakeholders involved in implementing the

education response (e.g. development partners, Government of Sāmoa Ministry

officials, school committees, principals and teachers, and key non-state actors

including private sector and civil society organisations of the partner country).

Fieldwork in Sāmoa, which will include stakeholder interviews and will guide a

detailed beneficiary analysis, possibly involving focus group discussions with

communities including at least one in a remote location.

Data analysis and synthesis of findings into an evaluation report suitable for

publication.

6. Outputs / Deliverables

Recommendations will be made against the criteria and specific questions of the

evaluation. These should include changes that could be made to facilitate a potential

design, implementation or management for any future investment.

The outputs / deliverables will include:

Evaluation Plan – The plan will define the scope of the evaluation, articulate key

evaluation questions, describe methodologies to collect and analyse data,

propose a timeline linked to key milestones, propose a schedule for in-country

field work, outline costs and a detailed breakdown of responsibilities of all team

members. The plan will be developed in close consultation with DFAT and MFAT.

See http://www.oecd.org/dac/evaluation/evaluatingbudgetsupport.htm

Evaluation of the Sāmoa Education Sector Support Programme – Final Report 49

Aide Memoire – The aide memoire will present emerging issues, seek

verification of facts and assumptions and discuss the feasibility of the initial

recommendations. This will be a working document (no more than 5 pages), and

the audience for this document would include all stakeholders.

Interim findings workshop – The interim findings workshop is an opportunity

to discuss the aide memoire and provide early feedback on the direction of the

evaluation.

Draft evaluation report – The draft evaluation report will be shared and

discussed with the Government of Sāmoa and the development partners.

Final Evaluation Report – The final Evaluation Report will incorporate any

agreed changes or amendments as requested by DFAT and MFAT. The final

Evaluation Report will include an executive summary (of no more than 2 pages),

a clear summary of findings and recommendations (no more than 20 pages) and

relevant attachments. This report will be published by DFAT on their website.

7. Indicative Evaluation Timeline

Activity Due date

Document review and introductory brief with DFAT, MFAT (via phone) TBC

Evaluation plan finalised based on Development Partners’ feedback, including

refining scope, results framework and key evaluation questions

TBC

Organise interviews and field work in Sāmoa TBC

Field work TBC

Document review TBC

Aide Memoire and interim findings workshop TBC

Analysis and report writing TBC

Report finalised based on DFAT and MFAT feedback TBC

Present key findings and conclusions to DFAT and MFAT and other stakeholders TBC

8. Evaluation Management Arrangements

DFAT and MFAT contract managers in Sāmoa are responsible for the day-to-day

management and administration of the evaluation. Their responsibilities include

contracting, coordinating approval of the evaluation TORs, chairing the selection

committee for the evaluation team, briefing the evaluators, reviewing and approving

all outputs, and liaising with the evaluation team to ensure the activity is being

undertaken as agreed.

A Steering Group for the evaluation will provide high-level strategic guidance to the

review process. Specifically, the group will review and provide feedback/comments at

key points in the exercise: the evaluation TORs, the draft review plan and the draft

report (including participation at the peer review). The Steering Group will be chaired

by Development partners.

The Steering Group is expected to be made up representatives from:

- DFAT Post in Sāmoa

- DFAT Canberra

- MFAT Post in Sāmoa

- MFAT Wellington

- Ministry of Finance (GoS)

- Education sector (GoS)

9. Evaluation Team Composition, Roles and Responsibilities

Applications and proposals from both individuals and a team will be considered. The

successful respondents will form a team with the below knowledge, skills and

experience to provide the services required. The following is an indicative composition

of the team:

a) Team Leader (also covers management and governance questions)

b) Education Specialist

c) Monitoring & Evaluation consultant

An organisation can propose either an individual for any of the above positions or can propose a

team with the above indicative composition. MFAT reserves the right to change the composition

of any team proposed by an organisation.

The attributes (knowledge, skills, experience) required of the evaluation team include:

demonstrated expertise in the independent evaluation of education sector

programs in a development context;

experience of DFAT and MFAT systems and monitoring and evaluation

standards;

sound knowledge and understanding of aid effectiveness and funding modalities;

excellent writing and analytical skills;

extensive knowledge and working experience in Sāmoa and/or the Pacific will be

highly desirable;

strong background in education in developing countries with expertise in teacher

development, school management and experience in managing education sector

programmes;

solid experience in evaluating aid programmes. Experience in education sector

programs is preferable; and

sound knowledge of monitoring and evaluation standards and principles.

The attributes (knowledge, skills, experience) required of the Team Leader include:

successfully delivering quality and efficient projects/ programmes on time;

sound knowledge and understanding of aid effectiveness and funding modalities;

excellent writing and analytical skills;

Evaluation of the Sāmoa Education Sector Support Programme – Final Report 51

extensive knowledge and working experience in Sāmoa and/or the Pacific will be

highly desirable;

working with partners to successfully deliver projects, employing innovation and

identifying and maximising opportunities to add value; and

effectively identifying, managing and mitigating risks.

The attributes (knowledge, skills, experience) required of the Education Specialist include:

strong background in education in developing countries with expertise in teacher

development, school management and experience in managing education sector

programmes,

extensive knowledge and working experience in Sāmoa and/or the Pacific will be

highly desirable;

broad understanding of Pacific Island education contexts; and

strategic thinking ability and research and analysis skills.

The attributes (knowledge, skills, experience) required of the Monitoring and Evaluation

consultant include:

solid experience in evaluating aid programmes. Experience in education sector

programs is preferable;

broad understanding of Pacific Island education contexts;

strategic thinking ability, research and analysis skills;

sound knowledge of monitoring and evaluation standards and principles;

Demonstrated expertise in the independent evaluation of education sector

programs in a development sector; and

Experience of DFAT and MFAT systems and monitoring and evaluation

standards.

10. Key Documents

DFAT and MFAT will make available to the team information, documents and

particulars relating to the ESSP. These will include, but not be confined to, the following

documents. The development partners shall make available to the evaluation team any

other reasonable requests for information and documentation relating to the

evaluation. The evaluation team is also expected to independently source other

relevant material. A list of the documentation (not exhaustive) is presented below:

Sāmoa Education Sector Plan 2013- 2018

Education Sector Support Program Design

Independent Verification Process Reports

Mid Term Review Reports

Annual Review Reports

Data Quality Assessment Framework

11. Evaluation Principles and Standards

The evaluation should meet the following standards:

- The purpose and objectives, and scope of the evaluation are clearly described in

the evaluation report.

- The evaluation is undertaken as specified in the evaluation plan. Any variation to

the agreed evaluation plan is stated in the evaluation report.

- The report clearly addresses the evaluation’s purpose, objectives and evaluation

questions.

- Findings answer the evaluation questions, are supported by evidence (with the

source of the evidence clear), are disaggregated where appropriate (e.g. by

variables such as gender, age etc.), and are separated from opinion and

judgements.

- There is a clear flow from evidence-supported findings to conclusions to

recommendations and to lessons learned.

- The recommendations are relevant and useful and directed to appropriate

people/organizations. Lessons learned are also relevant, useful and applicable.

- Report is readable, flows logically, and its style/tone and length is appropriate. Any

gaps in information are reported. The executive summary is well written, stands

alone and provides a good summary of the evaluation. The report does not contain

confidential information which would prevent public release of the report.

- Value for money of the activity is assessed using the most appropriate approach

and tools.

- Development Assistance Committee (DAC) or other evaluation criterion selected

for the evaluation is addressed appropriately.

- The integration of cross-cutting issues and the treatment of environmental and

social impacts are addressed appropriately under the relevant criteria.

- When conducting the evaluation, the evaluation team will comply with the

respective organizations’ Code of Conduct.

- It is DFAT and MFAT policy to make the evaluation report publicly available (e.g.

on public websites) unless there is prior agreement not to do so. Any information

that could prevent the release of an evaluation report under the Official

Information or Privacy Acts should not be included in the report.

- All the key deliverables and the data/information collected will become the joint

property of DFAT, MFAT and the GoS.

Evaluation of the Sāmoa Education Sector Support Programme – Final Report 53

APPENDIX B: STAKEHOLDER ENGAGEMENT LIST

DATE AGENCY / ORGANISATION STAKEHOLDERS INTERVIEWED

30 Aug 2018 IVP consultant (Wellington) Jane von Dadelszen

6 Sept 2018 MFAT and DFAT Pati Gagau, First Secretary Development MFAT

Felauai Tuaniu, Programme Manager, MFAT

Amanda Jewell, Counsellor Development, DFAT

Vicky Foalima, Programme Manager, DFAT

Nick Hurley, High Commissioner, New Zealand

6 Sept 2018 Ministry of Public Enterprise

ESAC Chairperson

Elita Tooala, CEO MPE

6 Sept 2018 Ministry of Education, Sports

and Culture

Afamasaga Karoline Fuatai, CEO

7 Sept 2018 ESCD Tapaautasi Kovi Aiolupotea

Hinorma Onesemo – procurement

Talia – Monitoring and evaluation

Aliitasi Su’a Afoa – policy and planning

Olivee Leilua – budget and finance

7 Sept 2018 Sāmoa Qualifications

Authority

Letuimanuasina Emma Vaai, CEO

Maposua Mose Asani, ACEO Corporate Services

7 Sept 2018 Public Service Commission

(ESAC member)

Salilo Margraff, ACEO Human Resource Management

Jolly Papalii – manages contract employees inc principals

and assistant principals

Sydney Sua – education employees inc teachers

Jason Hisatuke– ACEO senior executive services.

(Leadership and Management)

Osana Liki – ACEO public sector co-ordinator Assistant

Human Resource

Sarona Esera - ACEO human resource development

7 Sept 2018 World Bank (ESAC member) Maeva Natacha Betham-Vaai

10 Sept 2018 Mission School (ESAC) Fr Mikaele Tuimavave, Catholic Education Director

10 Sept 2018 Sāmoa Qualifications

Authority

Lealiifano Easter Manila, ACEO Policy

Maposua Mose Asani, ACEO Corporate Services

10 Sept 2018 Ministry of Sports, Education

and Culture

Salima Lasalo Salima, ACEO Policy and Planning

Alaifea Belford, ACEO Corporate Services

Aveolela Burgess – Policy and Planning, Project Officer

Sulamai Malo – Policy and Planning, Information analysis

officer

10 Sept 2018 Strategic Planning Advisor Julie Affleck, Technical Advisor

DATE AGENCY / ORGANISATION STAKEHOLDERS INTERVIEWED

10 Sept 2018 National University of Sāmoa Peseta Desmond Sua, DVC Academic research

Lineta Tamaniyakaroi

Sooalo Sydney Faasau

Mandria Sua

Lydal Visan

10 Sept 2018 Ministry of Sports, Education

and Culture

Salima Lasalo Salima, Principal Officer Policy

Alaifea Belford, ACEO Corporate Services

Aveolela Burgess, Project Officer

Sulamai Malo, Information Analysis Officer

10 Sept 2018 Palladium Group Julie Affleck, Strategic Planning Adviser

10 Sept 2018 Ministry of Finance Lita Lui, ACEO Aid Coordination

Abigail Lee Hang, ACEO Budget

11 Sept 2018 Ministry of Sports, Education

and Culture

Leota Valma Galuvao, ACEO Curriculum

Vau Peseta, ACEO Monitoring and Evaluation

Funealii Lumaava Sooaemalelagi, ACEO Assessments

Perenise Stowers, ACEO School Operation

Tauti Jenny Lauano, ACEO Teacher development

Salima Lasalo Salima, Principal Officer Policy

Aveolela Burgess, Policy and Planning Project Officer

11 Sept 2018 National University of Sāmoa

and Sāmoa Qualifications

Authority

Peseta Desmond Le Hang, DVC Academic research

Mandria Sua, Director Governance Policy and Planning

Sooalo Sydney Faasau, Manager In-Country Training

Lealiifano Easter Manila, ACEO Policy SQA

11 Sept 2018 Australia Pacific Training

Collaboration

Cheri Robinson Moors, Director

Andrew Colquhoun

11 Sept 2018 Loto Tamafai (Special School) Lagi Natanielu, Principal

Ata’a Dan Devoe, President

12 Sept 2018 Saoluafata Primary School Ae Lauese Moinge, School Inspector

Vaisugi Malio, Principal

12 Sept 2018 Anoamaa College Tuvale Leau-Mulinuu, Principal

12 Sept 2018 Sāmoa College Reupena Rimoni, Principal

13 Sept 2018 Vaivase Primary School Sa’o Tolai, Principal

Fa’atoia Malele, Vice Chair School Committee

13 Sept 2018 Sāmoa Ministry of Foreign

Affairs and Trade

Peseta Noumea Simi, CEO

13 Sept 2018 Sāmoa Ministry of Commerce,

Industry and Labour

Sa’u Taupisi, ACEO Employment

Faasuaga Soufale, Inspector of Apprentices

Evaluation of the Sāmoa Education Sector Support Programme – Final Report 55

DATE AGENCY / ORGANISATION STAKEHOLDERS INTERVIEWED

13 Sept 2018 SUNGO Tofua’iofoia Fuimaono Falefa Lima, CEO

13 Sept 2018 National Council of ECE Sāmoa

Kuinimeri Tamati, President

Lauiva Ah Kuoi, Secretary

Epenesa Esera, NUS

Kuinileti Lauina, NUS

13 Sept 2018 Faculty of Education, National

University of Sāmoa

Tofilau Dr Faguele Suaalii, Dean of Education

14 Sept 2018 Don Bosco Technical Centre Mane Su’a, Principal

Tuanai Onesimo Moli, Deputy

14 Sept 2018 National University of Sāmoa Fui Leapai Asofou Soo, Vice Chancellor

14 Sept 2018 St Mary’s College Sister Pelenatete, School Principal

Sina Fitu, Deputy Principal

14 Sept 2018 Ministry of Education, Sports

and Culture

Leota Valma Galuvao, ACEO Curriculum

Vau Peseta, ACEO Monitoring and Evaluation

Funealii Lumaava Sooaemalelagi, ACEO Assessment and

Examination

Perenise Stowers, ACEO School Operation

Jennifer Pemila – PEO IE Unit in Curriculum Division

Dawn Rogers – IE specialist (AVID) working in MESC

17 Sept 2018 Development Partners Felauai Tuaniu (NZHC)

Vicki Foalima (Australian HC)

18 Sept 2018 Debrief with Chair, Sector

Heads and ESCD on aide

memoire and findings

Afamasaga Karoline Fuatai, CEO MESC

Letuimanuasina Emma Vaai, CEO SQA

Fui Leapai Asofou Soo, Vice Chancellor NUS

Kovi Aiolupotea, ESCD

John Patch, Education Consultant

19 Sept 2018 Debrief with stakeholders on

aide memoire and findings

MESC, NUS, SQA, MCIL, MoF, St Mary’s College, SUNGO,

Loto Tamaufai, Sāmoa MFAT, APTC (35+ people).

19 Sept 2018 Final debrief with

Development Partners

Pati Gagau and Felauai Tuaniu (NZHC)

Amanda Jewell and Vicky Foalima (Australian HC)

John Patch, Education Consultant

Julie Affleck, Strategic Planning Adviser

1 Oct 2018 Inclusive Education Technical

Assistance consultant

Sally Baker

1 Nov 2018 Consultation with MFAT

(Development Partner)

Verity Smith, Former Development Manager, Sāmoa,

MFAT

Amy McAteer, Education Sector Leader, MFAT Wellington

DATE AGENCY / ORGANISATION STAKEHOLDERS INTERVIEWED

2 Nov 2018 Consultation with DFAT

(Development Partner)

Rosemary McKay, Director Education Section, DFAT,

Canberra (Former Deputy High Commissioner in Sāmoa)

6 Nov 2018 Consultation with DFAT

(Development Partner)

David Coleman, Senior Education Adviser, DFAT, Canberra

Evaluation of the Sāmoa Education Sector Support Programme – Final Report 57

APPENDIX C: AIDE MEMOIRE – EVALUATION OF THE SĀMOA EDUCATION

SECTOR SUPPORT PROGRAMME – INTERIM FINDINGS

In-country Evaluation team members: Vince Catherwood and Jacqui Haggland

NZ based Evaluation team members: Ned Hardie-Boys, Pounamu Aikman

Purpose

This Aide Memoire outlines the interim findings of the evaluation of the Sāmoa Education Sector

Support Programme (ESSP) based on a review of existing policy and practice. Operational

documents and published reports have informed the evaluation. Stakeholder engagement was

undertaken in-country during the period 6-18 September 2018 with:

• the New Zealand and Australian development partners (DPs)

• key administrators in Government of Sāmoa (GoS) agencies responsible for the

implementation of the Education Sector Plan (ESP) that the ESSP funds

• representatives from different types of education providers and the community.

Evaluation approach

The evaluation is being undertaken using the OECD Development Assistance Committee (DAC)

Criteria to underpin and focus the assessments being made.

DAC Criteria Key evaluation question for the Evaluation of the Sāmoa ESSP

Relevance

Whether and to what extent the ESSP has addressed the priorities of the target groups

(schools/other education providers) and is aligned with GoS policies and priorities.

Effectiveness

Whether and to what extent the ESSP has achieved the desired outcomes.

Efficiency

The extent to which the ESSP could have been implemented at less cost without

reducing the quality and quantity of the activities.

Impact

The positive and negative intended and unintended effects of the ESSP.

Sustainability

Whether and to what extent the benefits can be sustained after the end of the

development partners’ assistance.

The evaluation team also considered the extent to which cross cutting issues (gender equality and

inclusive education) have been identified and pursued in the implementation of the ESSP.

The initial findings from the evaluation are briefly outlined below with linkages to the evaluation

criteria to which they relate. This approach has been used (rather than linkage to individual Key

Evaluation Questions (KEQ) and sub-questions) as the focus for this Aide Memoire, specifically to:

• Present emerging issues

• Seek verification of facts and assumptions

• Discuss the feasibility of initial recommendations

This paper provides an overview of the issues that were raised, followed by a series of preliminary

conclusions and initial recommendations.

ESSP objectives, scope and design (Relevance)

• The ESSP has a clear strategic framework aligned with Aus/NZ policy and GoS

development objectives / educational objectives of education providers.

• The scope of ESSP encompasses the whole education sector and justifies the involvement

of the three Implementing Agencies (IAs).

• The outcomes sought in the Education Sector Plan (ESP) and the ESSP are very ambitious

within the relatively short timeframe.

Governance (Effectiveness)

• The Education Sector Advisory Committee (ESAC) and Education Sector Working Group

(ESWG) provide the primary system of governance – the structure is good, and

significant progress has been made recently to better utilise this structure.

• Revision of the ESSP could include consideration of the ESAC membership to ensure it

provides appropriate representation of those with an interest in employment skills

development, e.g., other TVET providers or Ministry of Commerce, Industry and Labour.

• The Education Sector Coordination Division (ESCD) is now better placed to support the

ESWG to provide the required analysis before reports are forwarded to the ESAC for

decision-making.

• Mission and Private schools are less engaged at the governance level – some

stakeholders were concerned with the quality of teaching in some Mission schools due

to lack of funding.

Monitoring and evaluation (Effectiveness, Efficiency)

• A Monitoring and Evaluation (M&E) Framework exists and is being used to report on a

broader range of indicators than are included in the ESSP (e.g. additional ESP indicators).

• The M&E framework needs to be updated, streamlined and simplified.

• ESAC and Ministry of Finance (MoF) expect quarterly reporting against the M&E

framework and updates to the Medium-Term Expenditure Framework (MTEF) as part

of the monitoring process – timeliness of reporting has been a problem to date.

Funding modality (Relevance, Effectiveness, Efficiency)

• Difficult to assess the efficiency of ESSP without detailed analysis to disaggregate the

impact of donor funding from ongoing GoS funding.

• Misalignment between the ESSP processes and GoS Budget timelines has provided

challenges over the life of the ESSP programme, including the delivery of reporting.

• Fixed tranche disbursement of budget support component (70%) of the funding

modality is appropriate.

• Some stakeholders questioned the fairness of the current practice of reducing the level

of the fixed tranche component, based on the GoS self-assessment of progress against the

12 fixed process indicators, and the DP’s assessment of the likelihood of an underspend

in the current financial year.

Ministry of Education, Sports and Culture (MESC), Sāmoa Qualifications Authority (SQA) and the National

University of Sāmoa (NUS).

Evaluation of the Sāmoa Education Sector Support Programme – Final Report 59

• One purpose of the 70% ‘fixed tranche’ is to provide the education sector with funding

predictability from year to year.

• Use of ring-fenced funding within the fixed tranche component to support Inclusive

Education (IE) was successful in ensuring progress was made in related activities.

• Mixed views about the effectiveness of the performance-based component (30%) – little

evidence that withholding funding provided incentives to IAs to improve performance.

• Performance-based component KPIs were not SMART (specific, measurable, achievable,

relevant and time-bound).

• In any new ESSP that is developed, consideration should be given to including process

KPIs that demonstrate measurable steps towards achieving outcomes, as well as

outcomes-focused KPIs.

• Poor initial communication about sector budget support modality, including IVP.

• Some evidence the three IAs are continuing to operate in silos and see the indicators as

‘belonging to’ individual agencies.

• Disagreement between IAs and MoF around the release of the 30% tranche of incentive

money – clarity needed about how the money will be released, e.g., to the individual IA

that reports on the KPI or to be shared across the three IAs.

Independent Verification Process (IVP) (Effectiveness)

• IAs and ESAC are generally supportive of the IVP. Many saw it as an opportunity to learn

from external consultants and develop staff capacity in monitoring and reporting.

• Insufficient consultation at the ESSP design stage meant IAs ill-prepared to report on Key

Performance Indicators (KPIs) at first IVP. The 2016 IVP was a ‘big wakeup call’, due to

a lack of understanding about the IVP and its implications for funding.

Communications and guidelines for schools, ECE and PSET/TVET providers (Effectiveness)

• Lack of provider awareness of the ESSP beyond ‘NZ and Australia provide funding for

education’. This is consistent with a sector budget support modality.

• Effective guidelines were produced by GoS for the Minimum Service Standards (MSS).

However, operationalisation of the MSS is labour intensive and could be rationalised.

• A Manual on school grants administration is available and will need to be reviewed as

the new integrated approach to school grant provision is implemented.

Progress in meeting objectives (Efficiency)

• ESCD was not initially successful in its coordination role, but this has significantly

improved in recent months. Lack of coordination is a weakness in implementation,

resulting in late reporting and missed deadlines.

• There is a need for stability of personnel in key roles across the IAs and clear articulation

of their responsibilities as they relate to the ESSP.

ESSP role in strengthening the education system (Sustainability)

• Two recent reviews commented on inefficiencies in the way information is managed

across the education sector. Progress has been made in data collection and analysis, but

further work is required (fairly urgently).

• The three IAs are currently collaborating on a joint TVET policy and strategy, including

TVET provision in schools. This approach should be encouraged.

ESSP role in strengthening the education sector via taskforces and technical assistance

(Sustainability)

• The taskforces have met with mixed success.

- The Teacher development taskforce had a role in supporting the development of a

National Teacher Development Framework.

- IE taskforce has been successful in developing policies and plans for the future.

This work should be further integrated into any new ESP and ESSP.

- Early childhood education (ECE) taskforce has made significant progress,

seemingly without much support from ESSP; the work should be further

integrated into any new ESP and ESSP.

• Technical assistance (TA) funding has mostly been used to supply short term capacity

for individual projects. Any future use of the TA facility could include a capacity

development component linked to a sector wide capacity development plan. This

development plan should include activities to improve the quality of analysis and

reporting across the education sector.

Alleviation of the funding gap (Impact)

• ESSP has provided IAs with a larger funding pool from which they can address big

initiatives (e.g., the professional development of teachers and the qualification upgrade

programme) or pilot new initiatives (e.g., an e-initiative that included training for

teachers to use e-learning, and PSET trainer development programme).

• Some of both the 70% fixed tranche and the 30% performance-based tranche have not

been made available, so this may have had a negative impact on some activity delivery.

Inclusive education (Cross-cutting issues)

• Ring fenced funding for IE has been successful. The sector is better prepared for future

mainstreaming of people with disabilities in education.

• The Inclusive Education Policy Implementation Plan 2017-2020 (June 2017) and

consideration of an MoU between MESC and the Inclusive Education Service Providers

and Special Schools (March 2018) are evidence of progress in this area.

• New KPIs on inclusive education have been developed for the future with the goal of

including these in a revised M&E Framework – this should be encouraged.

• Need to improve access to specialists (work with National Health Service) so those with

disabilities receive the health support needed to engage with mainstream education.

• Ongoing focus is needed to upskill teachers/PSET trainers and education provider

leadership on what is required to support successful inclusive education.

Gender (Cross-cutting issues)

• There are four KPIs within ESSP that report achievement results by gender – poorer

results for males than females in three of these (literacy, numeracy, transition to formal

PSET). Further analysis is required to identify reasons for these gender gaps.

Evaluation of the Sāmoa Education Sector Support Programme – Final Report 61

• More work needed to ensure sufficient appropriate education opportunities in place to

ensure that boys’ education needs are being met at primary, secondary and PSET levels.

Preliminary Conclusions

The IAs with responsibility for the delivery of the ESP have made progress in many areas

supported by the ESSP. The challenges they have faced are to be expected, specifically as they

relate to the development and implementation of new approaches to the sector budget support

modality which the ESSP, and specifically the IVP, necessitated.

While there has been positive learning about management of the ESSP, the time frame over which

the ESSP has been operational has been too short to be able to point to hard evidence of improved

learning outcomes. Furthermore, it is difficult to separate the effect of GoS budget funding from

the effect of additional funds delivered through the ESSP.

Recommendations

Continue to use a sector budget support modality to deliver any future ESSP-type

funding with the following conditions:

better alignment between the ESSP reporting and planning timelines and the

established timelines of the GoS Budget process,

consultation at the design stage on the selection of suitable KPIs with teams

across all three IAs who are to administer the system(s) and are expected to

provide the required evidence, to ensure the KPIs are SMART (specific,

measurable, achievable, relevant and time-bound),

accountability requirements must be satisfied by regular reporting against

appropriate KPIs, and

the ‘fixed tranche’ component of the education sector budget support modality

should be delivered annually in total if a reasonable proportion of any fixed

process indicators are met, and not be reduced except under exceptional

circumstances (e.g., demonstrated underspend).

Continue to use an Independent Verification Process for a small number of agreed KPIs,

with the following conditions:

include clear guidelines on how the individual KPIs for the sector relate to the

release of the performance funding component, and

specify the steps for the release of funding, including whether any IA is entitled

to a share of the funding disbursed.

Current work to develop a Sāmoa Education Data Quality Improvement Plan should be

supported, and implementation of the plan prioritised to strengthen data collection and

analysis, monitoring and reporting.

The M&E Framework should be updated, rationalised and simplified to ensure that it

can be used effectively. Consideration should be given to ensure the framework covers

the range of activities across the education sector (e.g., ECE, IE, primary, secondary,

formal and informal PSET and TVET).

MTEF update processes need to be better integrated with other reporting to ensure the

necessary updates actually occur.

APPENDIX D: RESPONSE TO ESP MID-TERM REVIEW

MFAT and DFAT have requested that the evaluation of the ESSP include consideration of the

extent to which the recommendations identified by the 2016 Mid-Term Review (MTR) of the

Sāmoa Education Sector Plan (ESP) have been addressed. The evaluators note that MTR had a

much wider scope than our evaluation of the ESSP.

The MTR noted that of the 34 (ESP) indicators reviewed, nine had been achieved at the time of

writing the MTR report (February 2017), nine had been partially achieved, and may be achieved

over the next three years, 13 had not been achieved, and three were very unlikely to be achieved

by 2018.

The most intractable problem was found to be ineffective implementation of new policies and

processes. Robust data for reporting on progress was lacking in a number of areas. While some

progress was being made, key challenges persisted in areas such as numeracy, gender differences

in student achievement, bilingual education, the limited impact of professional development on

teaching and learning in schools, and weak support for schools.

The MTR found the budget support approach was complex, and the ESCD struggled (in 2016) to

effectively implement the approach. Financial data to assess cost-effectiveness did not exist.

Weaknesses in human resource capacity and ineffective implementation of policies were ongoing

concerns. The October 2018 ESSP evaluation largely endorses these comments, and notes that the

main concerns raised in the February 2017 MTR are largely still areas of concern in the Sāmoa

Education Sector.

Table 4 outlines the 11 recommendations, the extent to which the ESSP has helped to address

these, and the ongoing relevance of the recommendations. Please note the recommendations are

worded exactly as they were in the MTR report provided to the evaluation team.

Table 4: MTR recommendations - have they been addressed and what is their ongoing relevance

MTR recommendation To what extent has the ESSP helped to

address this recommendation?

Ongoing relevance of the

recommendations based on

ESSP evaluation findings

1. Providing financial and

human resource support

should be provided forthwith

for the ESCD in the areas of

strategic management and

coordination.

Significant improvements have been made

in the ESCD coordination and

management role in recent months,

supported by the Strategic Planning

Adviser role

currently filled by Julie Affleck

through the use of Technical Assistance.

The Strategic Planning Adviser role has

included advice on development of cross-

agency reporting templates to support

efficient and effective reporting of outputs

and outcomes.

ESCD has only just begun to

deliver in the area of

strategic management and

coordination of the ESSP

implementation, and ongoing

support in this area is

required.

See section 2.2.3 of the

evaluation report.

2. Providing financial and

human resource support

should be provided for the

ESCD to develop better

financial data collection and

analysis including procuring

advice and support in the

There have been ongoing difficulties with

ESCD’s ability to meet the requirements of

efficient and effective expenditure review

reporting.

Recent appointments to ESCD have

included an adviser with responsibility for

budget and finance reporting.

It is too early to tell whether

the new appointment to the

ESCD will sufficiently address

this recommendation.

Further technical adviser

assistance in this space is

Evaluation of the Sāmoa Education Sector Support Programme – Final Report 63

MTR recommendation To what extent has the ESSP helped to

address this recommendation?

Ongoing relevance of the

recommendations based on

ESSP evaluation findings

conduct of Cost Effectiveness

Analysis.

likely to be beneficial for the

success of any future ESSP.

See section 2.2.3 of the

evaluation report.

3. Funding a rapid TA

assignment that examines how

to best provide direct support

for Principals.

There has been investment in developing

capacity such as Principal leadership and

other in-service training courses.

However, it is not clear to what extent this

investment has been supported by

technical advisers.

Further technical adviser

assistance in this space is

likely to be beneficial for the

success of any future ESSP.

See section 2.4.1 of the

evaluation report.

4. Providing financial and

human resource support to

agencies to develop

procurement plans.

Procurement was not a focus of the ESSP.

However, new procurement appointments

have been made across the IAs.

This issue was not a focus of

the ESSP, so has not been

directly addressed in the

ESSP evaluation.

5. Filling the two key positions

at MESC which have leadership

responsibility for literacy and

numeracy.

No evidence that these positions have

been filled. However, other initiatives are

in place e.g. minimum s

ervice standards in

literacy and numeracy.

Leadership is still required in

literacy and numeracy. See

section 2.5.1 of the

evaluation report.

6. Developing new

performance-based indicators

for inclusive education.

Progress has been made in IE, and some

draft performance indicators for IE were

sighted by the Evaluators. The draft

performance indicators, however, need

more work in order to accurately report

against progress towards achievement of

agreed inclusive education objectives.

Changes to existing KPIs,

including the development of

appropriate performance

indicators for inclusive

education, are included in

the ESSP evaluation

recommendations.

7. Developing a centralised

EMIS.

This objective has not yet been achieved. Included in ESSP evaluation

recommendations.

8. Developing new relevant and

measurable indicators should

be created for Early Childhood

Education.

While progress has been made in ECE, this

has not been reflected in the development

of indicators that are able to be accurately

reported against, or that will demonstrate

real progress towards quality early

childhood education delivery.

Changes to existing KPIs,

including the development o

appropriate performance

indicators for early childhood

education, are included in

the ESSP evaluation

recommendations.

9. Defining clear roles and

responsibilities for Task Forces.

It was reported that the taskforces have

met with mixed success. Three task forces

(IE, ECE and teacher development) have

had some success in developing and

implementing a work programme. It is

disappointing that more has not been

achieved by the literacy, numeracy and

assessment taskforces over this period.

Further work is required to

ensure the education sector

gets the full benefit out of all

six Task Forces. See section

2.5.1 of the evaluation

report.

MTR recommendation To what extent has the ESSP helped to

address this recommendation?

Ongoing relevance of the

recommendations based on

ESSP evaluation findings

10. Reviewing the validity of

outcome indicators where data

is missing and where it is likely

data will not be collected in the

next two years.

A number of indicators have been revised

based on the IVP findings, although

further work is required as many are still

unlikely to be achieved due to data

collection and analysis gaps.

Changes to KPIs are included

in the evaluation

recommendations.

Improvements in data

collection and analysis

capability are included in the

ESSO evaluation

recommendations.

11. The sector should prioritise

sector wide gender analysis

looking at a range of issues to

do with performance and

access, and it should develop a

medium-term strategy for

addressing ongoing gender

issues.

The gap male and female student

achievement remains, and it appears that

little (if any) focus was placed on

addressing gender issues over the period

of the ESSP.

Gender issues continue to be

areas of concern, and more

work needs to be done in this

area. This finding is included

in section

2.6.1 of the

evaluation report.

Evaluation of the Sāmoa Education Sector Support Programme – Final Report 65

APPENDIX E: KEY PERFORMANCE INDICATORS: EDUCATION SECTOR SUPPORT PROGRAMME

These Key Performance Indicators (KPIs) below are those originally attached to the Joint Funding Arrangement (August 2015 – December 2019) signed

by the Government of Sāmoa and Development Partners in 2015 and 2016. It is acknowledged that changes to the wording of some KPIs were made

by ESAC during the course of the programme.

Annual Fixed Process Indicators for 70% Tranche Disbursement

Disbursement Triggers Indicators Sources of Verification

The overall strategy policy and

governance arrangements are on

track as envisaged in the ESP or as

subsequently amended through the

annual review

• No concerns raised at the preceding annual review or ESAC about major

divergence from ESP strategy, policy or governance.

• ESWG is meeting at least quarterly

• At least two weeks before the January meeting, suggestions for priorities for

implementation in the coming year and any adjustments to the ESP arising

from the annual review are circulated to ESAC members

• ESAC is meeting on a quarterly basis, and the January meeting is providing the

guidance needed to enable the sector MTEF to be updated

• All approved task forces met at least quarterly each financial year (Not

applicable for the 2015/16 disbursements)

• Maintenance of mutual understandings between the GOS and DPs on pre-

sector budget support arrangements

• Annual review, meeting reports,

agendas and approved minutes;

• Annual budget estimates and

MTEF

ESP funds are not leading to a

reduction in the government’s own

financial commitment to the sector

• Fraction of the Estimated Payments to Education from the Treasury fund for

the current financial year exceeds 14% after deducting ESSP contributions

Budget Estimates for coming financial

year; Financial Management Reports;

audits of financial accounts

Financial allocation is on track, in

accordance with the sector plan as

subsequently amended through the

January meeting of ESAC

• Less than 10% divergence between the final estimates and the MTEF

approved at the January meeting of the ESAC for the previous financial year

Education sector estimates;

Minutes of January ESAC meeting;

Financial Management Reports;

Audits of financial accounts

Disbursement Triggers Indicators Sources of Verification

Implementation is on track

• No critical path outputs are more than a year behind schedule except for

those for which deferral has been approved by the ESAC

Annual progress reports

Monitoring and reporting are on

track

•

Dissemination of annual ESP progress reports sent to stakeholders at least two

weeks before the annual review, including an analysis of progress against the

KPIs and other indicators in the ESP results framework (and with a complete

set of baseline results included in 2015/16)

• Consolidated quarterly and annual Financial Management Reports (including

findings from the audit reports) provided on time by ESCD for consideration

by the ESAC and annual review

• Annual progress reports

• Consolidated reports from ESCD

and ESAC minutes

•

Final Report on the Annual Review

issued by the ESAC

• Finance sector and PFM annual

progress report

Risk management is on track

• Internal audit reports and ESCD follow-up on these reports show progress on

reducing the residual PFM and procurement risks within the sector

• Consolidated reports from ESCD

and ESAC minutes

• Final Report on the annual review

Evaluation of the Sāmoa Education Sector Support Programme – Final Report 67

Key Performance Indicators (KPIs for the Sector) (for the 30% tranche disbursement)

Responsible

IA(s)

KPI Key Outcome Indicators Subcategory

2012 baseline

2019

targets

MESC 1 Year 4 & 6 primary school children at risk in literacy (English and Sāmoan) Male 35% 23%

Female 18% 12%

MESC 2 Year 4 & 6 primary school children at risk in numeracy Male 26% 10%

Female 16% 5%

SQA, NUS, APTC

3 Percentage of PSET graduates finding employment within 6 months (gender disaggregated rates to

be determined in first year of ESSP)

31%

70%

MESC 4 Percentage of children commencing Year 1 Primary and completing Year 8 by gender (Primary

Completion Rate MDG2)

Male 86 (2014) 95%

Female 85 (2014) 95%

MESC, NUS,

SQA

5 Transition rate from Year 13 to formal PSET (target to be confirmed in Year 1 of ESSP) Male 37% 50%

Female 63% 70%

MESC 6 Percentage of children with disability enrolled in mainstream government schools 105 (in 2010) 150

MESC 7 Percentage of teachers meeting teacher performance standards

Appraisal process in

place

TBD

MESC 8 Number of schools meeting minimum service standards related to literacy and numeracy TBD 50%

SQA 9 Number of accredited courses provided by PSET providers 6 20

End targets may be adjusted based on mutual determination in the Annual Review Processes. KPI indicators reflect the most critical indicators identified by the

sector drawn from its sector plan and Monitoring and Evaluation Tracker as part of the annual review process in 2014.

This is based on only six PSET providers and on the 2006 Tracer Study. Tracer study undertaken in 2013 for TVET providers only. NUS will work on undertaking

annual tracer surveys. Scope exists to combine with those for TVET.

APPENDIX F: ESSP FUNDING TIMELINES

Figure 2: Alignment of GoS Budget and ESSP timelines

Figure 3: Timing of ESSP funding availability

Evaluation of the Sāmoa Education Sector Support Programme – Final Report 69

APPENDIX G: LINKAGES BETWEEN KEY EVALUATION SUB-QUESTIONS AND

SECTIONS OF THE DRAFT EVALUATION REPORT

Effectiveness

KEQ 2: Effectiveness – To what extent has the ESSP been effective in improving the quality of learning

and enhancing educational access and opportunities?

Key Evaluation Sub-Questions Source of Response in Draft Evaluation Report

2.1 To what extent is the ESSP meeting each of its

strategic outcomes? What activities are

progressing well, and which not so well?

See Improving Quality and Access in the draft

report under ‘Effectiveness’

2.2 What are the main factors enabling and

constraining the achievement of the ESSP’s

strategic outcomes?

See Improving Quality and Access in the draft

report under ‘Effectiveness’

Relevance

KEQ 1: Relevance – To what extent is the design of the Sāmoa ESSP relevant to the key issues facing

the education sector?

Key Evaluation Sub-Questions Source of Response in Draft Evaluation Report

1.1.

Does ESSP have a clear strategic framework, and

is this aligned with the DFAT and MFAT

development policy and the GoS development

objectives?

See ESSP objectives, scope and design in the draft

report under ‘Relevance’

1.2. Is the funding modality the most appropriate

way to achieve the intended outcomes?

See The Funding Modality

in the draft report under

‘Effectiveness’

1.3. To what extent is the ESSP aligned to the key

issues facing the education sector?

See ESSP objectives, scope and design in the draft

report under ‘Relevance’

1.4. How have the governance, planning, and

reporting arrangements contributed to the

relevance of the ESSP?

See Governance in the draft report under

‘Effectiveness’

1.5. To what extent does the design of the ESSP

assist education providers to meet their

educational objectives?

See ESSP objectives, scope and design in the draft

report under ‘Relevance’

1.6. To what extent are the objectives and scope of

the ESSP still relevant?

See ESSP objectives, scope and design in the draft

report under ‘Relevance’

1.7. To what extent has the implementation of the

ESSP been consistent with its design?

See Implementation in the draft report under

‘Efficiency’

Effectiveness

2.3 To what extent has the ESSP contributed to

strengthening the education system? (including

in teacher development, classroom-based

assessment, school leadership and inclusive

education)

See Capacity building in the education sector and

Strengthening the education system in the draft

report under ‘Sustainability’

2.4 Have management structures improved

coordination among agencies?

See Improving Quality and Access and Governance

in the draft report under ‘Effectiveness’

2.5 To what extent has the development of internal

assessment staff capacity helped the programme

meet set objectives?

See Implementation in the draft report under

‘Efficiency’

2.6 To what extent has the ESSP assisted education

providers with their development planning?

See

Communications and Guidelines for School, ECE

and PSET/TVET Providers in the draft report under

‘Effectiveness’

2.7 How has data and information management

been used for evidence-based planning and

decision-making?

See Data and Information Management in the

draft report under ‘Effectiveness’

2.8 Has the Monitoring and Evaluation Framework

been an effective tool for monitoring progress?

See Monitoring and Evaluation in the draft report

under ‘Effectiveness’

2.9 To what extent is the sector budget support

modality achieving benefits? (e.g. in terms of

reducing transaction costs and fragmentation,

facilitating policy engagement, and leveraging

donor investments)

See The Funding Modality

in the draft report under

‘Effectiveness’

2.10 How effective is the Independent

Verification Process (IVP) and should it be

continued?

See Independent Verification Process in the draft

report under ‘Effectiveness’

2.11 How effective has the ESSP’s governance

been in supporting the management of risks,

resolution of programme issues, and

achievement of ESSP outcomes?

See Governance in the draft report under

‘Effectiveness’

2.12 What training and guidelines (if any) have

been developed to assist schools in their

management of the ESSP, and how effective

have these been?

See

Communications and Guidelines for School, ECE

and PSET/TVET Providers in the draft report under

‘Effectiveness’

2.13 How well have communications about ESSP

been managed?

See

Communications and Guidelines for School, ECE

and PSET/TVET Providers in the draft report under

‘Effectiveness’

2.14 What can be done to improve the

effectiveness of the ESSP?

See Improvements to Effectiveness in the draft

report under ‘Effectiveness’

Evaluation of the Sāmoa Education Sector Support Programme – Final Report 71

Efficiency

KEQ 3: Efficiency – To what extent is the implementation of the ESSP being managed efficiently?

Key Evaluation Sub-Questions Source of Response in Draft Evaluation Report

3.1 To what extent has the implementation of the

ESSP’s activities been efficiently managed by

GoS and education institutions?

See Implementation in the draft report under

‘Efficiency’

3.2 To what extent have the development partner

management arrangements (MFAT

Wellington/MFAT Post/DFAT Canberra/DFAT

Post) supported efficient implementation?

See Development Partner Management Efficiency

in the draft report under ‘Efficiency’

3.3 Does GoS have the right systems in place to

efficiently manage MFAT/DFAT funding?

See Systems Efficiency in the draft report under

‘Efficiency’

3.4 Were funds delivered to education providers /

intended recipients in a timely way

See Implementation in the draft report under

‘Efficiency’

3.5 To what extent has ESSP added value (technical

assistance, governance, managing for results),

reduced transaction costs and reduced

duplication?

See Implementation in the draft report under

‘Efficiency’

3.6 How well has MFAT’s /DFAT’s engagement to

support the education sector (policy dialogue,

participation in management bodies) worked?

See Development Partner Management Efficiency

in the draft report under ‘Efficiency’

3.7 What can be done to improve the efficiency of

the ESSP?

See Systems Efficiency in the draft report under

‘Efficiency’

Impact

KEQ 4: Impact – What is the likely impact of the ESSP?

Key Evaluation Sub-Questions Source of Response in Draft Evaluation Report

4.1 Is the approach to sector budget support likely

to support the achievement of the ESSP goals

and sector outcomes?

See Impact of funding via budget sector support in

the draft report under ‘Impact’

4.2 What is the likely impact of ESSP on families? See Impact on communities, families and schools in

the draft report under ‘Impact’

4.3 What is the likely impact of ESSP on schools? See Impact on communities, families and schools in

the draft report under ‘Impact’

4.4 What can be done to improve the long-term

effect(s) of the ESSP impact?

See Capacity building in the education sector and

Strengthening the education system in the draft

report under ‘Sustainability’

Sustainability

KEQ 5: Sustainability – Are the benefits of the ESSP likely to be sustainable?

Key Evaluation Sub-Questions Source of Response in Draft Evaluation Report

5.1 To what extent has the ESSP supported the

financing gap in the education sector?

See Impact of funding via budget sector support

the draft report under ‘Impact’

5.2 To what extent would the implementation

partners likely be able to sustain the skills,

management capacity and funding for the ESSP

without continued assistance?

See Capacity building in the education sector and

Strengthening the education system in the draft

report under ‘Sustainability’

5.3 What challenges does the GoS face in

maintaining the benefits of the ESSP?

See Challenges and Opportunities in the draft

report under ‘Sustainability’

5.4 What can be done to improve the likelihood that

the ESSP’s benefits will be long-lasting?

See Capacity building in the education sector and

Strengthening the education system in the draft

report under ‘Sustainability’

Cross-cutting Issues

KEQ 6: Cross-cutting issues – To what extent have cross-cutting issues (gender equality and disability

inclusive development) been identified and pursued in the ESSP?

Key Evaluation Sub-Questions Source of Response in Draft Evaluation Report

6.1 To what extent have cross-cutting issues been

effectively addressed in planning,

implementation and monitoring and evaluation

of the ESSP?

See Inclusive Education and Gender equality in the

draft report under ‘Cross-cutting issues’

6.2 How have cross-cutting issues been integrated

into the education sector?

See Inclusive Education and Gender equality in the

draft report under ‘Cross-cutting issues’

6.3 What can be done to strengthen the ESSP’s

inclusive and equitable development outcomes?

See Capacity building in the education sector and

Strengthening the education system in the draft

report under ‘Sustainability’

Evaluation of the Sāmoa Education Sector Support Programme – Final Report 73

APPENDIX H: STUDENT ACHIEVEMENT IN LITERACY AND NUMERACY

A key reason for the investment of funds through the ESSP was to improve student achievement.

ESP Goal I is “enhanced quality of education at all levels”, ESSP outcome No 2 is “improved literacy

and numeracy scores at years 4 and 6”, and ESSP investment aim 2 is “support implementation of

reform policies designed to improve learning outcomes and skills development linked to realistic

employment expectations”. This appendix summarises at a high level some of the data relating to

student achievement available in recent MESC Education Statistical Digests and in other recent

reports. These reports have commented on student achievement in Sāmoa, in relation to literacy

(Sāmoan and English), numeracy, and science, technology engineering and mathematics (STEM)

subjects.

Sāmoa Early Grade Reading Assessment 2017

A Sāmoa Early Grade Reading Assessment (SEGRA) was conducted in Sāmoa from August to

September 2017. The assessment was supported by the Global Partnership for Education, the

World Bank, and Education Technology for Development, and was carried out in collaboration

with the Pacific Community (formerly the South Pacific Commission) and MESC. EGRA is a simple

instrument that measures foundational reading skills of early primary school students in Grades

(Years) 1 to 3. The overall purpose of SEGRA was to provide an initial measurement of how well

students are learning to read and write in their local language in the first three years of primary

schooling. A national sample of 1196 students (600 girls and 596 boys) who were enrolled in

Years 1, 2 and 3 were tested. The Early Grade Reading Assessment (EGRA) tool was adapted for

the Sāmoan context by staff of the MESC and selected early grade teachers in Sāmoa who attended

a workshop facilitated by language and education experts.

Several findings resulted from the assessment. First, early reading achievement in Sāmoa is low.

Overall, students in Sāmoa, even after 3 full years of schooling, are not yet able to read with fluency

and accuracy. This inability is preventing them from reading with comprehension. The

international benchmark for reading fluency is 45-60 correct words per minute, depending on the

complexity of the language. Overall, Oral Reading Fluency (ORF) scores for Sāmoa averaged 16

correct words per minute across the years. Year 3 students are reading at an average of 29 correct

words per minute and only 16% of them could comprehend 80% to 100% of the text.

Secondly, students show progression in word reading skills from Years 1 to 3. In order to assess

progression in reading skills across all three years, the same SEGRA instrument was applied to

students in Years 1-3. While the degree of progress varied per subtest, the SEGRA results showed

that there was progress between Years 1-3 in all subtests. In addition, it was noted that there was

significant learning progress between Year 1 and Year 2 and very little learning is happening in

Year 3 as shown by the graphs in the full report for SEGRA.

Third, students lack decoding skills. Students scored low across the three sub-tests that measure

decoding skills: letter sounds, initial sounds, and non-words. Overall, students correctly identified

an average of 23 letter sounds out of 100 letters, 3 initial sounds out of 10 words and 7 nonwords

out of 50.

Fourth, reading comprehension levels are well below the international benchmark. An analysis of

zero scores revealed that over 96% of students in Year 1, 72% in Year 2 and 50% in Year 3 could

not correctly answer a single comprehension question about the oral reading passage. The

majority of students in the sample across the three years are achieving well below the

internationally accepted reading comprehension benchmark of 80%. Only 6% of all students

tested met the benchmark (80%) and above.

Fifth, girls have higher reading fluency and comprehension. There were important differences

between boys and girls, with girls outperforming boys in all of the subtests except for the non-

word and listening comprehension subtests.

Finally, regional differences were significant. Students in the Savai’i region consistently

performed better than ‘Upolu Urban’ and ‘Upolu Other’ regions in 9 out of 10 subtests. ‘Apia Urban’

students scored the lowest in all subtests.

Student Results across Sāmoa

Shayne Boyd, a secondary curriculum adviser with MESC, produced a number of reports and

tables in 2017 and 2018 that examined student achievement in Sāmoa. These reports are noted

elsewhere in this report, in the list of references. The following text draws on Shayne Boyd’s work.

An important issue in a bilingual country like Sāmoa is the policy stance taken on bilingual

education. Shayne Boyd has prepared a useful background paper entitled Bilingual Policy in Sāmoa

that discusses some of the issues as they relate to student achievement. An important issue for the

education system in Sāmoa is how best to assist young students to be fluent and literate in Sāmoan

(for most, the language of the home), and how best to make the transition from Sāmoan to English.

The issue of bilingual policy is therefore significant for Sāmoa, but is beyond the terms of reference

of this evaluation.

The Sāmoa Primary Education Literacy Level Test 1 (SPELL 1) tested students at Year 4 level.

Table 5 below shows an increasing percentage of Year 4 students categorised as ‘at risk’ in the

years from 2013 to 2017. Note that in the 2017 MESC Education Statistics Digest, the ‘at risk’

categories for 2017 in SPELL 1 are labelled ‘Beginning- Critical’, ‘Beginning- Basic’ and

‘Developing’.

Table 5: SPELL 1 Percentage of Students at Risk - at Beginner level

2013 2014 2015 2016 2017

English 26 27.4 37 68 64

Sāmoan 16 19.5 12 44.5 69

Numeracy 29 41 59 68 76

The Sāmoa Primary Education Literacy Level Test 2 (SPELL 2) tested students at Year 6 level.

Table 6 below shows an irregular percentage of Year 6 students categorized as ‘at risk’ in the years

from 2013 to 2017. Note that in the 2017 MESC Education Statistics Digest, the ‘at risk’ categories

for 2017 in SPELL 2 are labelled ‘Beginning- Critical’, ‘Beginning- Basic’ and ‘Developing’.

Table 6: SPELL 2 Percentage of Students at Risk - at Beginner level

2013 2014 2015 2016 2017

English 42 55.9 44 54 72

Sāmoan 14 30 13 39 30

Numeracy 61 74.2 59 46 51

The Sāmoa Primary Education Certificate Assessment (SPECA) is a test that primary students in

Sāmoa sit at the end of primary schooling (Year 8). SPECA was introduced in 2013. It replaced the

Year 8 National Examination. The test was an aptitude test in 2014, 2015, and 2016. Reporting

Evaluation of the Sāmoa Education Sector Support Programme – Final Report 75

has changed, so that rather than report a raw score average, reporting now lists percentages of

students at ‘advanced’, ‘proficient’ and ‘beginning’ level. 2014, 2015, 2016 scores record students

at the beginning stage. In 2017 the examination changed to an achievement Test for all subjects,

designed to test learning outcomes. The scores in Table 7 below for 2017 are for students who are

not yet proficient (i.e. who are at Level 1). The change in both the nature of the test and reporting

makes comparisons from year to year difficult.

Table 7: SPECA: Percentage of students at the ‘beginning’ level/ level 1

2013 2014 2015 2016 2017

English 35.5 71.4 72 56 81

Sāmoan 27.6 19.5 21 21 82

Numeracy 31.4 61.3 69 76 98

Science 82

Social Science 98

Health/Phys Ed 98

Visual Arts 81

The data in Table 7 highlights that there has been a decline in student achievement over the years

2013 to 2017, with fewer students achieving ‘advanced’ level, and increasingly more students

testing at the ‘beginning’ level. Only in Sāmoan are the majority of students achieving proficiency.

However, in 2017 this score also declined.

The Sāmoa School Certificate (SSC) is sat by students at the end of Year 12. In 2014 the MESC

Educational Statistical Digest changed the manner of reporting Year 12 and Year 13 achievement,

replacing average percentage marks with the scale: ‘beginning’, ‘achieved’, ‘merit’ and ‘excellent’.

The majority of Year 12 students were at the ‘beginning’ stage in Mathematics (78 %) and English

(64%). The results for the other science subjects are not recorded in the Educational Statistical

Digest. Table 8 below shows the results for core subjects for the last ten years.

Table 8: Year 12 SSC Results: Pass Rates

2008 2009 2010 2011 2012 2013 2014* 2015 2016 2017

English 37 51 46 50 49 41 30.5 55.5 54.3

Sāmoan 55 53 50 50 46 61 62 61.5 80.6

Maths 36 51 46 53 49 19 11.5 7.5 8.3

Biology 61 57 53 57 54 37 30.5 19.5 31.9

Chemistry 68 59 60 58 57 35 14 20.5 37.8

Physics 87 61 60 59 59 32 19 53.5 86.8

The Sāmoa Secondary Leaving Certificate (SSLC) is sat by students at the end of Year 13. Between

2012 and 2016 there was a significant decline in student marks in science, technology,

engineering and mathematics (STEM) subjects. As shown below in Table 9, a change occurred in

the 2017 results, for a number of subjects, which show a dramatic turnaround. The increase in

results was more than one would expect statistically over a twelve-month period. The following

pass rates for 2017 were recorded.

• The physics pass rate improved from 27% to 77% - an improvement of 287%

• The chemistry pass rate improved from 12% to 35% - an improvement of 201%

• The biology pass rate improved from 12% to 50% - an improvement of over 420%

• The English pass rate improved from 44% to 67% - an improvement of 153.4%

Table 9: Year 13 SSLC Results

2014* 2015 2016 2017

English 36 45.5 44 67.5

Sāmoan 78.5 91 92.9

Maths 22 4 5 10.9

Biology 25 12 50.4

Chemistry 27.5 12 34.9

Physics 37 26.5 77.4

Geography 38.7

History 18

Computer Studies 51.6

Accounting 34.4

Economics 63.2

Food and Textiles 53.4

Design Tech 30.2

Visual Arts 44.6

HPE 65.5

Development Studies 100

The variability in student results in 2017 raises a number of questions. Statistically, student

results over time should be within a reasonably close range, with results either steadily

improving, showing consistency over time, or demonstrating slight declines. When numbers of

students are smaller, there is a greater chance of wider variations from the norm.

Conclusions

The overwhelming evidence from these results of student achievement in Sāmoa is clear: student

achievement across the board in Sāmoa primary and secondary schools is low, and needs to

improve.

Evaluation of the Sāmoa Education Sector Support Programme – Final Report 77

There are legitimate questions and caveats that can be raised about the comparability of data from

year to year, since, when baselines change, it is inadvisable to make direct comparisons from year

to year. It is therefore important for administrators to ensure that data is comparable from year

to year in order to be able to assess the ‘health’ of the education system. In spite of any caveats

about data comparability, the situation with respect to student achievement in Sāmoa is clear.

An important strategy that the education system in Sāmoa needs to consider is to appoint a

reading specialist to a position in the education faculty of NUS. Good teachers are the cornerstone

of quality education. Well-trained teachers of reading are essential to teach young people to read

at an early age. Students who do not learn to read in the early years of schooling will struggle with

their education throughout their school years. Poor readers who do not understand printed

information or who cannot follow written instructions risk falling further and further behind

those who can read well. A lack of skill in reading will affect peoples’ social and economic well-

being later in life.