Introduction

This guide aims to contribute to what we know about the relationship between educational technology (edtech) and educational outcomes by addressing the following overarching question:

It also offers recommendations to support advisors to strengthen the design, implementation and evaluation of programmes that use edtech.

We define edtech as the use of digital or electronic technologies and materials to support teaching and learning. Recognising that technology alone does not enhance learning, evaluations must also consider how programmes are designed and implemented, how teachers are supported, how communities are developed and how outcomes are measured  (see http://tel.ac.uk/about-3/, 2014).

Findings

Effective edtech programmes are characterized by:

• a clear and specific curriculum focus
• the use of relevant curriculum materials
• a focus on teacher development and pedagogy

• evaluation mechanisms that go beyond outputs.

These findings come from a wide range of technology use including:

• interactive radio instruction (IRI)
• classroom audio or video resources accessed via teachers’ mobile phones
• student tablets and eReaders
• computer-assisted learning (CAL) to supplement classroom teaching.

However, there are also examples of large-scale investment in edtech – particularly computers for student use – that produce limited educational outcomes. We need to know more about:

• how to support teachers to develop appropriate, relevant practices using edtech
• how such practices are enacted in schools, and what factors contribute to or mitigate against successful outcomes.

Recommendations

1. Edtech programmes should focus on enabling educational change, not delivering technology. In doing so, programmes should provide adequate support for teachers and aim to capture changes in teaching practice and learning outcomes in evaluation.
2. Advisors should support proposals that further develop successful practices or that address gaps in evidence and understanding.
3. Advisors should discourage proposals that have an emphasis on technology over education, weak programmatic support or poor evaluation.
4. In design and evaluation, value-for-money metrics and cost-effectiveness analyses should be carried out.

Introduction

Purposes of this guide

There is enormous interest and investment in the potential of educational technology (edtech) to improve the quality of teaching and learning in low and lower-middle income countries. The primary aim of this Topic Guide is to contribute to what we know about the relationship between edtech and educational outcomes. Taking evidence from over 80 studies, the guide addresses the overarching question: What is the evidence that the use of edtech, by teachers or students, impacts teaching and learning practices, or learning outcomes? It also offers recommendations to support advisors to strengthen the design, implementation and evaluation of programmes that use edtech.

Methodology

The research involved three distinct stages:

• an online literature search identifying 83 studies (45 research documents, 20 literature reviews and 18 grey literature reports) on edtech use in schools in select low and lower-middle income countries
• an appraisal process, against DFID’s agreed criteria (including transparency, rigour, validity), to identify key findings and rate the quality of the evidence

• a written analysis addressing the overarching and subsidiary research questions.

Definition: What is edtech?

Definitions of edtech are contested and changing, but for the purposes of this guide, and in line with emerging DFID policy, we define edtech as the use of digital or electronic technologies and materials to support teaching and learning. Implicit within this definition is the recognition that:

Findings

Use of edtech among reviewed studies

These findings come from a wide range of technology use including:

• interactive radio instruction (IRI)
• classroom audio or video resources accessed via teachers’ mobile phones
• student tablets and eReaders
• computer-assisted learning (CAL) to supplement classroom teaching
• computer suites
• one laptop per child.

For the purpose of this guide, we distinguish between edtech use by teachers (e.g. IRI, where the teacher uses technology in class) and edtech use by students (e.g. CAL or eReaders, where the students have hands-on access to technology).

Examples of effective edtech programmes

It is now widely acknowledged that while the Millennium Development Goals prompted improvements in access to education, quality remains a challenge. This issue is also reflected in edtech programmes. Reports of programmes that move beyond access to technology (both in programme design and evaluation) are emerging, but as yet relatively few programme evaluations focus on adequately capturing improvements in the teaching and learning process or measuring improvements in learning outcomes. The findings below are drawn from those that do.

The following uses of edtech by teachers were associated with positive changes in learning outcomes and classroom practice:

Interactive radio instruction (IRI). Several studies reported positive impacts on learning outcomes from IRI, particularly with early primary students. A World Bank review showed average effect sizes of +0.5 (World Bank, 2005, p. 16), while a subsequent review found effect sizes ranging from -0.16 to +2.19 (Ho & Thrukal, 2009). The variability in effectiveness was attributed to factors including quality of programme implementation, monitoring, and local human resources. The greatest effect sizes were seen at Grade 1, suggesting IRI is particularly effective for early primary years.

Improvements in classroom practice from IRI were evidenced by two studies in which IRI was used in the context of teacher professional development. Sous le Fromager in Guinea supplemented IRI with radio programmes for school staff and face-to-face professional development to instill respectful behaviour of teachers towards students. Qualitative classroom observations suggest teachers hit students less often and allowed more time for students to develop understanding (Burns, 2006, p. 9). Similarly, an IRI programme in Mali supplemented IRI with radio-based, in-service training. Systematic classroom observations showed year-on-year improvements in the percentage of observed lessons demonstrating select classroom practices (e.g. brainstorming, group work, total physical response) (Ho & Thrukal, 2009, p. 10).

Mobiles for classroom audio and teacher development videos. Several studies arising from one programme (English in Action [EIA], Bangladesh) reported positive impacts from mobile use on English language teaching (ELT) practices and student learning outcomes. EIA is primarily a teacher development (TD) programme. The approach has some similarities to IRI, in that mobile phones provide access to audio resources for classroom use, particularly for primary teachers. Mobiles are also used to provide access to TD materials, including videos of classroom practice, which underpin the programme. Materials are not broadcast, or downloaded, but provided as a library of digital resources, on a small memory card.

Several large-scale systematic observations of classroom practice (EIA, 2011b, 2012b, 2014b) showed significant increases in students’ talk time (including talk in pairs and groups), and students’ and teachers’ use of English (the target language), compared with baseline studies (EIA, 2009). Associated improvements in student learning outcomes were also evidenced, most recently with 35% more primary students achieving Grade 1 or above, and 20% more primary achieving Grade 2 or above, on recognized international frameworks of English language competence (Graded Examinations in Spoken English (GESE), Trinity College London, 2013, which map onto the Common European Framework of Reference, Trinity College London, 2007, EIA, 2014b).

Mobiles for classroom video. The BridgeIT programme (India and Tanzania) provided evidence of improved learning outcomes from teachers’ use of smartphones to play video lessons for their classes via flat-screen TVs or data-projectors. Teachers also had activity guides to support or extend the video lessons. In Tanzania, students showed average gains of 10–20% over control groups for maths and science. However, while some groups of students excelled, others showed modest gains if any (Enge, 2011). In India, there were average gains of 10% over control groups for science, but no gains for English (Wennerstan & Qureshy, 2012). BridgeIT also carried out systematic classroom observations pre- and post-intervention in India. These showed a 31% increase in the proportion of lessons identified as ‘high quality’, with a corresponding 24% drop in the proportion of (traditional) ‘direct instruction’ lessons (Wennerstan & Qureshy, 2012, p. 32).

Among the studies reviewed, the strongest evidence of changes in learning outcomes and classroom practice came from the use of mobile devices (such as eReaders) and CAL programmes to support improvement in mathematics:

eReaders and tablets to support early literacy. Several programmes presented evidence of improved learning outcomes (in terms of increased reading fluency in the mother tongue or English) that combined provision of eReaders and eBooks for students with TD programmes on phonics-based literacy instruction (Worldreader, 2012, 2013; Murz, 2011; Piper & Kwayumba, 2014).

Remedial CAL programmes in mathematics. Although CAL programmes in maths as a replacement for regular teaching were found to have limited impact (Banerjee et al., 2007, p. 1,240) or lower learning outcomes (Linden, 2008, p. 26), there is some evidence of improved learning outcomes from remedial CAL programmes as supplements for under-privileged students (Banerjee et al., 2007, p. 1,238) or under-performing students (Lai et al., 2011).

In addition, several studies presented evidence of students working more independently and collaboratively using online or offline digital resources to support project work. This was usually in the context of a teacher development programme, with clear curricular and pedagogic focus (for example: Light, 2009; Were et al., 2009; Leach et al., 2005). In two studies, students expressed a view that group work was better than individual work with a computer, as peer learning was valued. Young students suggested an optimum group size of three, four (Haßler,et al., 2011, p. 42) or five (Leach, 2008, p. 20).

Finally, while value for money (VFM) analysis is non-existent in most studies, the PRIMR study stands out for its strong focus on cost-effectiveness. The programme in Kenya implemented and compared the effectiveness of three different interventions – tablets for tutors (teacher educators), tablets for teachers, and eReaders for students. Similar gains in student learning outcomes were shown for all three treatments, with no statistically significant difference between treatment groups (Piper & Kwayumba, 2014, p. 2). Further analysis showed that while the outcomes were similar, the costs for teacher or tutor tablets were much lower than for class sets of eReaders, making the cost-effectiveness of teacher or tutor tablets approximately an order of magnitude greater (Piper & Kwayumba, 2014, p. 3).

Examples of less effective uses of edtech 

Among the studies reviewed, 21 reported on students’ use of computers, but very few of these presented evidence of measurable improvements in learning outcomes. Three that did were CAL programmes in maths, as discussed above.

Several studies showed that increasing students’ access to computers of itself has little discernable impact on teaching or learning practices. For example:

• ‘computers are often not used for teaching and learning purposes’ (EdQual, 2011, p. 3)
• while 98% of publicly supported schools in Chile have increased access, ‘ICT is not frequently used at school’ (Hinostroza et al., 2011, p. 1,360)
• although NEPAD provided schools with 20 suites of desktop computers, satellite connectivity, wireless networks, smartboards, and health software, ‘teachers are not in general using the Healthpoint software for purposes of health promotion’ (Rubagiza et al., 2011, p. 42)
• in Myanmar, 33,497 schools were provided with ICT facilities in 2009, but evaluation showed ‘the frequency of use of ICT for teaching and learning was considered to be very low’ (UNESCO, 2013a, p. 45)
• despite increasing student-computer ratios in Peru, the One Laptop per Child (OLPC) programme ‘did not seem to have affected the quality of instruction in the class…a substantial share of laptop use was directed to activities that might have little effect on educational outcomes’ (Cristia et al., 2012, p. 3).

These examples illustrate that even the most substantial investment in edtech can have limited impact. A common explanation for this is that:

“Provision of ICT in schools is only the first step. For ICTs to become a tool for improving teaching and learning…they need to be supplemented by teacher professional development.” (EdQual, 2011, p. 12)

There is scant evidence of this being done successfully (for example: Light, 2009; Were et al., 2009; Leach et al., 2005). More broadly, across the studies reviewed, detailed accounts of how edtech is used in classrooms are very rare. Relatively few studies examine classroom practice beyond teacher self-reporting.

Recommendations

Checklist for reviewing edtech programmes or proposals

Conceptual clarity

Programmes should have a design focused on enabling educational change, with an emphasis on curriculum, pedagogy, teaching and learning, not technology.

Programmatic support

Programmes should provide adequate support, to enable implementation of change in classrooms and schools.

Research, monitoring and evaluation

Systematic monitoring and evaluation procedures should capture changes in teaching and learning practices and learning outcomes, and participants’ experiences and perspectives.

Value for money (VFM)

Before the design stage, the costs of programme inputs should be estimated, key cost drivers analysed and forecast, and VFM metrics developed.

1.1 What is edtech?

Definitions of educational technology (edtech), or the use of technology for educational purposes, are contested, and have changed frequently over the last 40 years or more. Typically, current definitions go beyond merely defining what is meant by ‘technology’, but include reference to educational purposes and practices. For example:

For the purposes of this guide and in line with emerging DFID policy, we define edtech to be the use of digital or electronic technologies and materials to support teaching and learning. As such, edtech does not refer simply to hardware (radio, TV, mobile phones, computers, and tablets) or software (applications and learning resources). Rather, the definition encompasses an understanding of the technologies as tools, used by communities, in the social practices of teaching and learning, directed towards educational goals.

1.2 Research questions

These issues led the authors to focus their research on the following key questions:

• What is the evidence that the use of edtech byteachers impacts teaching practice or learning outcomes?
• What is the evidence that the use of edtech by students impacts teaching and learning practices or student learning outcomes?

Two other areas of evidence were also considered (but as these relate to an emerging field, there was not sufficient literature to include these as full research questions):

• Is there any evidence that the use of edtech in schools represents value for money?
• What are the enabling factors that inhibit or support integration of edtech?

1.3 How is edtech used?

There is a broad range of different ways that edtech can be used in schools. The studies reviewed included the following ways:

• interactive radio instruction (IRI)
• classroom audio or video resources accessed via teachers’ mobile phones
• student tablets and eReaders
• computer-assisted learning (CAL) to supplement classroom teaching
• computer suites
• one laptop per child.

For the purpose of this guide, we distinguish between edtech use by teachers (e.g. IRI, where the teacher uses technology in class) and edtech use by students (e.g. CAL or eReaders, where the students have hands-on access to technology).

1.4 Policy background: a shift from access to quality

Historically, programmes that incorporate edtech have focused on the distribution of hardware and programme evaluations have measured the number of devices in the hands of teachers or students. This approach is very limited. It fails to measure to the extent to which edtech has changed the process of teaching and learning, or the impact on learning outcomes, and thus prevents decision-makers evaluating the effectiveness and cost-effectiveness of edtech.

Programmes that move beyond access to hardware both in programme design and evaluation are still relatively rare. Equally, few programme evaluations focus on capturing improvements in the teaching and learning process or measuring improvements in learning outcomes.

This may be partly because measuring the impact of edtech is complicated by many contextual factors, such as access to appropriate infrastructure or local socio-cultural dynamics. It may also be because the role of edtech in many programmes is to supplement teaching and so isolating its impact can be a challenge. It is necessary to change the nature of learning activity, and the pedagogy, if you are to introduce edtech meaningfully.

While most of the studies appraised for this paper did not present evidence of impact of edtech on learning outcomes, those that did often painted a much broader picture of how edtech was integrated into the programme design and the success of doing so. It may well be that programmes that have clearly thought through how edtech will support and change teaching and learning will find it easiest to design evaluations that will show whether such changes are taking place.

2.1 Literature search

A clear methodology was planned for this Topic Guide, beginning with searching the literature using electronic databases, key journals, websites, reference lists and advice from key contacts. Sources included: 3ie Systematic Review Database, 3ie database of impact evaluations, Google Scholar, World Bank IEG, The Campbell Library, Centre for Education Innovations, R4D and documents provided by DFID advisers, members of HEART and academic leads.

A two-stage process followed: firstly a screening and coding of studies according to four thematic questions identified by the authors, which were then analysed and narrowed down to two key research questions, and two sub-questions; secondly a screening and rating of studies for critical appraisal of the quality of the evidence.

2.2 Screening and coding of studies

Inclusion criteria were first applied to screen out the studies that did not contribute sufficiently to answering the thematic questions. Papers meeting the criteria were read and stratified by question and, for each, titles, weblinks and abstracts were collated.

An extensive data extraction process was then followed to draw out the key data in each paper. Some documents were relevant to more than one thematic question; these were cross-referenced for ease of navigation through the document.

The authors then analysed these tables and once they had an overview of the evidence available, narrowed the focus to the key questions and sub-questions outlined in 1.2 Research questions.

2.3 Screening and rating of studies for critical appraisal

Research papers that met further inclusion criteria (relating to relevance and clarity of reporting) were then selected for critical appraisal by a team of experts and research officers working in the field of education. Both quantitative and qualitative research methods were considered. The studies were rated for the evidence they provided on a number of principles of quality: conceptual framing, openness and transparency, appropriateness and rigour, validity, reliability, cogency, overall key findings and relation to thematic questions.

As this guide is evidence focused, information from a number of source types was appraised, including programme evaluation reports, peer-reviewed published academic papers, internet reports and other grey literature. Information from literature reviews was included in the text of the guide, however these documents were not critically appraised, as this would have resulted in an appraisal of the reviews’ methodology rather than an appraisal of the programmes mentioned in the review. Additionally, some grey literature was not suitable for appraisal but the key findings were drawn out during the data extraction stage.

3.1 Use of edtech by teachers

 3.1.1 Introduction

This section looks at the evidence of changes in classroom practice or improved learning outcomes as a result of edtech use by teachers.

Among the 45 research studies reviewed, 15 referred to use of edtech by teachers. Among these, three forms predominated:

• interactive radio instruction (IRI)
• video materials (displayed on a large screen monitor, for viewing by the whole class), or audio materials (played on a rechargeable speaker, for listening by the whole class), played from the teacher’s mobile phone
• teacher development (TD) materials and activities (online and/or offline, print and/or audio-visual), related to curriculum and pedagogy, sometimes including how to use edtech for curriculum purposes.

Of these, IRI was by far the most common form of edtech use by teachers in the studies reviewed. This is in part because IRI is a well-established, low-cost, large-scale approach, with a much longer history than, for example, the use of mobile phones. Although IRI could be considered old technology, the simple fact of it being around for so long means that it has the greatest amount of evidence and evaluations available.

The video lessons evidence all comes from evaluations of BridgeIT projects, though in two different programme contexts (Tanzania and India).

The evidence around the use of edtech for teachers’ continuing professional development (CPD) is illustrated by studies from Sub-Saharan Africa (TESSA), and South Asia (EIA). In the TESSA study, the teachers’ CPD programme included use of open educational resource (OER) materials. In EIA, the teachers’ CPD programme included structured, peer-supported use of offline teacher development videos, and also use of offline audio resources in class (in a manner similar to IRI) via media players. In the eGranary programme, teacher educators in a primary teacher-training college in Uganda accessed offline copies of internet resources (either on a dedicated server or a memory stick) to use with student teachers.

3.1.2 Evidence of changes in classroom practice

When teachers use technology, they are well placed to positively impact classroom practice. Changes in teaching and learning practices are usually described by teachers’ testimonies from interviews or focus groups (although there are few detailed accounts of the nature of these changes). Evidence indicates that teachers use edtech for a number of purposes and with a variety of results.

Access to resources

In the eGranary report, teacher educators positively recount their experiences of being able to access new teaching resources from their offline repository, with the implication that these are used in practice:

Improving lesson delivery

A study performed by Egerton University in Kenya about the use of TESSA OERs to promote changes in teacher pedagogy found that headteachers perceived improving pedagogy, increased student motivation, and the creation of good learning environments, which they attributed to teachers’ use of the TESSA OERS in their CPD (Cullen et al., 2012).  Headteachers were asked an open question about whether learning had improved: all those surveyed reported improved lesson delivery by teachers and 8 out of 11 reported that students’ mean scores increased. However, while headteachers reported these changes – and valued them highly – no hard supporting evidence was given.

Using technology in a disability-specific programme

The evaluation of the USAID Tikwere IRI programme in Malawi reported positive impacts on classroom dynamics among blind children. The qualitative evidence reports that a small class of 17 blind or visually impaired students sat quietly waiting for the Tikwere lesson to begin; students then noticeably brightened and engaged in a very lively lesson, led by a capable teacher using the Tikwere radio broadcast and accompanying materials in Braille (Method et al., 2009).

Although the programme did not survey student perceptions of the programme, the results demonstrate that certain types of technology can be incorporated into a disability-specific programme without huge changes to the overall programme design, to bring significant changes in practice. This study also illustrates the value of direct field observations of practice, to validate teachers’ accounts.

Improve classroom management and more active learning strategies

Another IRI and TD programme, the Fundamental Quality in Education Level programme, includes a number of interventions – classroom-based IRI (Sous le Fromager), a radio programme aimed at teachers, principals and other staff (Pas à Pas) and face-to-face professional development sessions integrating audio-taped recordings of Pas à Pas. The IRI broadcasts were specifically designed to instil respect towards students in teachers and qualitative observations showed that teachers hit students less often for incorrect answers after their introduction. Following the programme, it was considered that teachers took a more developmental approach to learning, with field staff reporting more active learning strategies and a move away from transmissive approaches (Burns, 2006, p. 9).

Improved classroom practice and continuous professional development (CPD)

Looking at three programmes, quantitative observations of improvements in practice were reported, most notably through programmes that combine media for classroom practice, with teacher professional development. These include IRI in Mali, the BridgeIT video lessons in India, and classroom audio and teacher video in the English in Action (EIA) programme, Bangladesh.

First, the following was reported as evidence of impact when using IRI in Mali:

Second, BridgeIT was implemented in India as part of the Nokia Education Delivery system, aimed at improving both teaching practice and educational outcomes. Selected English and maths teachers were given a mobile telephone and a TV cable. The phones were connected to a TV or projector, and educational videos were broadcast over the 3G network. Activity guides suggesting teaching strategies accompanied the videos. Observed lessons were rated as ‘high quality’ or ‘direct instruction’ (didactic teacher-led practice). Before the intervention, almost three-quarters (74%) of lessons observed were categorised as ‘direct instruction’, whereas after the intervention almost three-quarters (74%) of lessons were rated as ‘high quality’ – an increase of 31% from the baseline (some lessons were rated as being both ‘high quality’ and ‘direct instruction’ in both sets of observations).

Third, EIA is a professional development programme for English language teachers in Bangladesh, which focuses on using audio-visual materials and peer support to improve teaching. The EIA pilot programme, which ran with 600 teachers from 300 schools, specifically assessed whether there were changes to students’ learning processes that could be attributed to the intervention, which consisted of professional development videos, activities for teachers and audio resources (all available offline), along with other learning materials to be used as part of interactive lessons.

The EIA programme pilot study (Walsh et al., 2013) reports teachers’ views on changes in practice, but – unusually among the studies reviewed – these are also triangulated against students’ views (EIA, 2011a). Further, the evaluations also report large-scale, systematic observation of objective features of practice (timed observations of student-talk time, teacher and student use of English, the target language) and instances of classroom organisation for pair and group work (EIA, 2012b).

Teachers indicated that the programme was very effective in changing the learning processes of the students. They considered that the students were more motivated and engaged in the lessons, and concentrated more on what was being taught:

3.1.3 Evidence of changes in learning outcomes

Although there is some hard evidence of improved learning outcomes from video lessons (Enge et al., 2011) and programmes where edtech is used for CPD and classroom use (EIA), there are relatively few programmes that specifically set out to gather this evidence and most of the evidence found comes from IRI programmes.

IRI programmes

Many IRI programmes that have measured student learning outcomes have compared the effectiveness of IRI between out-of-school learners using IRI to in-school learners without IRI: in other words, looking at IRI as an alternative to school, in contexts of chronic shortages of teachers or schools. Comparatively few programmes have measured learning outcomes associated with IRI for in-school learners. Additionally, IRI has often been implemented as part of a wider set of educational interventions, making independent assessment of its effectiveness difficult (World Bank, 2005).

Where IRI has been assessed, students in IRI classes outperform students in control groups with an average effect size of 0.5 standard deviations.

A more recent review, with a strong methodology, looked at 37 IRI programmes. The review found average effect sizes ranged from -0.16 to +2.19 across a variety of subject areas, projects and countries, suggesting that several factors (including availability of qualified local resources, quality of project implementation and monitoring, and extent to which students participate) affect the degree to which IRI can improve student achievement (Ho & Thrukal, 2009). Mathematics effect sizes were measured in Zambia, Sudan and Haiti, and local language literacy was measured in Zambia, Sudan, Somalia and Haiti; in both measurements the greatest effect size was seen at Grade 1, which was attributed to the effectiveness of IRI for this grade level.

Finally, there is some evidence that IRI can help to reach more disadvantaged students. Firstly, historical data indicated that IRI can be a mechanism for narrowing gender-based achievement gaps (World Bank, 2005), although more recent data (e.g. Ho & Thrukal, 2009) appeared mixed. Secondly, analyses of student assessment results revealed that rural IRI students enjoy approximately the same boost in achievement over their non-IRI peers as do urban IRI learners. This was particularly the case in Pakistan, where results for English Grades 1 and 2 showed significant large effect sizes for isolated students (Ho & Thrukal, 2009). Thirdly, there is also some limited evidence that IRI improves student learning outcomes in the context of fragile states, such as Sudan and Somalia: students who participated in IRI classes had a distinct advantage over their non-IRI peers, and this advantage was consistent across subjects (Ho & Thrukal, 2009).

Other forms of edtech

As well as IRI, there is also emerging quantitative evidence of impact on student learning outcomes from video lessons and mobile-enhanced CPD with classroom audio.

Test scores of students in BridgeIT and BridgeIT+LifeSkills in both maths and science showed significant gains during the 2010 academic year in comparison to those of students who did not benefit from BridgeIT. On average, test scores for students in BridgeIT schools were 10–20% higher. However, analysis shows that these increases were due to groups of students who excelled, while others showed more modest or no gains. Students in the BridgeIT schools with the added Life Skills curriculum performed the best academically (Enge, 2011).

BridgeIT in India gave teachers in the programme access to educational videos and an activity guide. The impact assessment found statistically significant improvements (10%) over control groups in science, although there was no statistically significant impact on English test scores (Wennerstan & Qureshy, 2012).

In Bangladesh, where teachers were also provided with SIM cards, classroom media, activity guides and regular follow-on support, EIA found improvements in students’ spoken English language competence. 50% of primary students achieved a Grade 1 or above in 2011 (compared to just 35% in 2010); for secondary students, 90% achieved a Grade 1 or above in 2011 (compared to 75% in 2010) and an improvement was found across all grades compared to those achieved in 2010. These improvements were found to be statistically significant for both groups (EIA, 2012c).

Finally, the Primary Math and Reading Initiative (PRIMR) in Kenya, explored the use of teacher tablets (preloaded with enhanced teachers’ guides, instructional materials such as flashcards, other literacy materials, and the Tangerine:Class continuous assessment software), in 20 schools. This was compared to other interventions (using tablets for tutors (teacher educators) only, and eReaders for students only). Baseline and endline Early Grade Reading Assessments (EGRAs) in Class 2 were carried out with samples of 20 students (10 boys and 10 girls) from each class, giving a sample size of over 1,500 students. The results showed statistically significant gains in learning outcomes over control students, of 0.47 standard deviations, for the use of teacher tablets (Piper & Kwayumba, 2014, p. 2).

3.1.4 Summary

In summary, among examples of edtech used by teachers, there is some evidence that edtech contributes to changes in classroom practice or improves learning outcomes.

Teachers perceive significant changes in practice that they attribute to edtech (eGranary, Tikwere, BridgeIT, EIA, TESSA). However, across studies reviewed, there is an overreliance on teacher self-reporting.

It is perhaps not coincidental that the programmes that have gathered the most systematic evidence of changes in practice (e,g. BridgeIT, EIA) are characterized by the use of audio or video media to support professional development and classroom practice. These programmes, evaluated with medium-to-large scale systematic classroom observations, suggest significant changes in classroom practice, for a majority of teachers observed.

Among the examples of edtech used by teachers, there are relatively few accounts of measured changes in student learning outcomes. However, for all three approaches to teachers’ use of edtech identified here, there are examples that provide statistically significant evidence of improvements in student learning outcomes (World Bank, 2005; Ho & Thrukal, 2009; Enge, 2011; EIA, 2012c & 2014c). Of these, EIA might also be seen as an example of statistically significant improvements in student learning outcomes from edtech-enhanced CPD (EIA).

3.2 Use of edtech by students

 3.2.1 Introduction

This section looks at the evidence of changes in classroom practice or improved learning outcomes as a result of edtech use by students.

In general, among the studies reviewed, there is little conclusive evidence of measurable changes in teaching and learning practices, through students’ use of edtech. Of 45 research documents, 17 (39%) measured learning outcomes in some way. However, changes are generally self-reported by teachers or students, rather than reported by observation or via objective indicators of practice.

There is some indication that where the focus is upon computers and ICT hardware, rather than subject curriculum and pedagogy, this may do little to improve students’ active learning (Rubagiza et al., 2011). When it is poorly introduced it may reinforce existing teacher-centred practices and when teachers are not supported in its use it may simply have no contribution to subject teaching (EdQual, 2011). An additional flaw with this approach is that current policy initiatives appear to be disadvantaging particular groups, such as girls and those living in rural communities.

3.2.2 Evidence of changes in classroom practice

Improved access but limited impact

The NEPAD e-schools Initiative in Rwanda aimed to equip both primary and secondary students with ICT skills. The provision of edtech was extensive and comprehensive, but the effect on teaching and learning practices was negligible. The study found that many teachers couldn’t access the software or had never used it (Rubagiza et al., 2011) and students felt they were not trained in practical use of the equipment.

Similarly, national survey data from Chile showed that the national ICT in education policy (Enlaces) had been very successful in reach (over 98% of the almost 11,000 publicly supported primary and secondary schools had participated by 2009), but found that ‘ICT is not frequently used at schools’ (Hinostroza et al, 2011).

One Laptop per Child (OLPC) seeks to provide a laptop to every student, preloaded with learning software and materials. The first large-scale study of the effects of OLPC, carried out in primary schools in Peru, found that despite increasing access to computers by an order of magnitude (from 0.12 computers per student, to 1.18 computers per student), and substantial increases in computer use at home and at school, there was little effect on motivation (as indicated by enrolment or attendance), time spent on homework, or student reading habits. Computer logs suggested most computer activities were not directed towards educational outcomes, and a qualitative observation suggested, at best, modest changes in pedagogical practices. This is attributed to a lack of curriculum-related materials and no clear pedagogic guidance for teachers (Cristia et al., 2012, p. 3).

In Rwanda, tens of thousands of laptops had been deployed by 2011, with plans to distribute half a million by 2017 (OLPC, 2011, p. 4). Despite the impressive scale of implementation, a case study from Rwanda provides scant evidence of changes in practice, illustrating just two examples of project-based learning carried outside of the classroom (OLPC, 2011, p. 6).

An absence of curriculum-related materials and inadequate technical and pedagogic support were also identified as key reasons for very low levels of use of computers provided to rural schools, from a survey of over 250 participating science teachers in Namibia (Ngololo et al., 2012, p. 11).

Similarly, the UNESCO review of ICT in Education in selected ASEAN countries highlighted Singapore as ‘a global leader in the use of technology in education’ but presented little evidence of the nature of changes in practice:

Strong pedagogy enables more positive outcomes

There is some evidence that positive changes in practice are more likely where there is a strong pedagogical – rather than technological – focus, supported by appropriate teacher development and subject-specific resources. For example, the evaluation of the Intel ‘Teach Essentials’ course in India, Turkey and Chile found that appropriate pedagogical context is key to successful ICT integration (Light, 2009).

Teachers and students spoke positively about three types of new learning activities:

• project-based work, which gave students a chance to collaborate, use multiple resources, and direct their own learning
• independent internet research, which gave students autonomy and a chance to develop and share their own perspectives
• schoolwork that was more relevant to students’ lives outside school, which made learning more meaningful.

Although only based on case studies from six schools, these findings show that it is the changes in lesson structures, learning materials and teaching practices that drove the observed improvement in learning outcomes. Access to the internet was a central part of this, but as an enabling tool or context for the new pedagogic practices.

Similarly, EdQual found some examples of changes in students’ learning activities where ICT was used in support of curriculum-focused activities (particularly in biology and maths). Students noted that they were no longer dependent upon what the teacher knew and told them, but they were able to do their own research and discover new areas beyond what the teacher taught; they also felt computers assisted understanding, accuracy and kept their interest better (Were et al., 2009).

In addition, a small but in-depth study in Zambia aimed to identify how mobile technologies could be used to embed more interactive teaching and learning activities within classroom practice. Much of the evaluation focused upon technical issues, but there were some interesting observations on the classroom use of the various technologies explored:

• “eBook readers…were not that successful at stimulating interactive activities in class…teachers found it difficult to bring e.g. the WikiReader into the lesson fairly dynamically…the devices can be quite idiosyncratic…the students did not find this easy…
• The webcams we had hoped to use as document cameras…the concept of using a document camera is fairly extraneous to the current Zambian teaching, so it was hard to introduce this idea…
• The battery powered [pico] projectors…are not bright enough for use in Zambian classrooms…
• When children go into the local community with cameras, through which interactive learning becomes visible to the community…the school standing is improved
• Generally speaking, there appears to be little awareness and experience of the benefit of mobile equipment in Zambian schools. When ICT in schools is discussed, for instance within government or within schools, there is generally a push towards putting equipment into computer rooms. Educationally speaking, we consider this to be outdated…

 (from Haßler et al., 2011)

In the context of a deliberate shift in pedagogy (“In this project, we were concerned with inquiry-based approaches…”), all six teachers were unanimously positive about the way the edtech had facilitated student collaboration and independent work (Haßler et al., 2011).

Finally, in Ghana, the Worldreader programme focused on introducing phonics-based literacy instruction; teachers were given training in this pedagogic approach and students were provided with eReaders. Subsequent teacher focus groups all reported that their students were reading more than they had previously, although they also thought it took a considerable amount of time for students to become proficient with the eReaders.

Another finding was that teachers who regularly used paper textbooks in their classrooms prior to the Worldreader intervention were generally more motivated to learn about and use the eReaders in their classrooms. Teachers who had not used textbooks frequently found it cumbersome to learn the new skills required to operate the eReaders and coach the students to use them.

Student focus groups uncovered additional project results, including increases in the reading of supplementary materials and storybooks, reading outside of school, and reading for pleasure among students at treatment schools. Nearly every Worldreader student interviewed indicated he/she was reading ‘a lot’ more and better at the time of the interview (compared to the start of the school year), whereas the majority of control school students felt they were reading about the same amount and slightly better (Worldreader, 2013, p. 18).

Quality without quantity

In the Digital Education Enhancement Project (DEEP) in Egypt and South Africa, teachers from schools serving disadvantaged communities were provided with a laptop and printer/scanner (shared between a pair of teachers), a handheld ‘pocket-computer’ each, and a digital camera and video camera (shared between several local schools). These were used in the context of professional development and participatory research, with supportive curriculum materials and ongoing peer-support (Leach et al., 2005). The combination of edtech, CPD and support led to significant changes to teaching practice and the nature of students’ learning activities. For example, a majority (77%) of DEEP participants reported that ICT had a ‘high’ impact on their ability to plan lessons. Within classrooms, ICT facilitated collaborative working (74%), presentation of material (54%), independent learning (31%) and new classroom practices (Leach, 2008).

Teachers also reported that having a very limited number of devices – while challenging in one way –actually promoted them to adopt new classroom practices: firstly, because they could not stand at the front and show the whole class something on the small screens of the mobile devices, they had to put the ICTs in the hands of the students; secondly, because only a few students could use the ICTs at a time, they had to adopt approaches where one group would use the mobile technology, but other groups (or the rest of the class) would do other, non-ICT activities, on rotation.

Students also recommended sharing in reasonably large groups, rather than providing edtech for individual or pairs of students:

3.2.3 Evidence of changes in learning outcomes

In general, there is sparse evidence to indicate measurable changes in learning outcomes. Among the 45 research studies reviewed, only a few found hard empirical evidence of learning outcomes from students’ use of edtech. The strongest of these related to students using mobile devices (TeacherMates or eReaders), in the context of support from teachers who had participated in teacher development programmes, with a strong and very specific pedagogic focus.

One Laptop per Child (OLPC)

The first large-scale RCT of OLPC, evaluated its implementation in Peru (Cristia et al., 2012). As discussed on page 20, despite the lack of evidence of large-scale impacts on learning practices or outcomes, investment in and planned roll-out of OLPC continues at enormous and accelerating pace, for example in Rwanda.

Computer suites

A comparative analysis conducted in Jordan, Oman, Palestine and Qatar found evidence of improvement in learning outcomes associated with use of computers was weak and mixed, and could possibly be attributed to variations in socio-economic status:

Similarly, a Jamaican study into the effectiveness of school improvement plans, in which computers in schools was one of nine interventions implemented in programme schools, found some positive association between computers and student learning outcomes, but again the authors suspected these improvements were due to correlation between computers and another variable (in this case, school leadership), rather than a causal relationship between computers and learning outcomes (Lockheed et al., 2010). In principle, RCTs should account for variations such as socio-economic status, provided enough is known about the variables that need to be controlled.

Using CAL programmes to improve mathematics

An RCT study was carried out on the effects of CAL for improving mathematics in India. The programme was implemented in two forms: as a replacement for regular teaching methods and also as an ‘out of class’ supplement to ‘in school’ lessons (Linden, 2008). The findings showed that when implemented in school, as a replacement for normal lessons, students learned significantly less than they otherwise would have:

However, when CAL was implemented outside of school as a supplementary programme, it did report positive gains in the ‘weakest’ students (0.4–0.69 standard deviations).

Strong gains in learning outcomes from supplementary CAL programmes in mathematics were shown from a Pratham-led project in India. In this RCT, Grade 4 students were offered two hours of shared computer time per week, during which they played games that involve solving maths problems.

Lai et al. (2011) also reported mixed findings from a large-scale RCT exploring the effects of CAL for mathematics, with 4,000 third-grade students from Bejing, from 43 schools predominantly serving poor migrant families. The results showed improvements of 0.14 standard deviations from standardized maths scores, but again reported that low-performing students, or those with less-educated parents, showed the biggest improvements.

Mobile devices

The TeacherMate Differentiated Instruction System was used in Rwanda by 620 students in Grades 2 and 3. It was developed with a strong phonics component. Students were intended to use TeacherMates for half of each of their regularly scheduled English periods each day (20 minutes). However due to the heavy demands placed on teachers to cover specific material from the national curriculum, students eventually ended up using the TeacherMates for an average of 40 minutes per week plus a total of 10 nights of home use (Murz, 2011, p. 9).

The results showed an average increase of 36% among standardized measures of verbal skills for students using it over the school year. This compared with average increases of 14% in each of two control groups (Murz, 2011, p. 12).

The Primary Math and Reading (PRIMR) Initiative in Kenya consists of an RCT of three ICT-based literacy initiatives. The three interventions are:

• using tablets to bolster the Teachers’ Advisory Centre (TAC) Tutor instructional support system
• using teacher tablets with classroom pedagogical support
• using eReaders to help pupils practise literacy.

All three initiatives are supported by PRIMR with provision of books, lesson plans, supplementary readers, and instructional aids alongside teacher training and supervision support.

In terms of oral reading fluency (ORF), evaluations of all three groups showed gains over the baseline, for both English and Kiswahili fluency, with the strongest gains in English:

The Worldreader programme in Ghana also reported learning outcomes for students using eReaders. Each Grade 1–3 student received a reader with approximately 140 books (15% textbooks, 85% age-appropriate storybooks). Students were also supported through extra-curricular activities that focused upon reading. Evaluation was carried out using a quasi-experimental design, with control and experimental groups, and random selection of students. Early Grade Reading Assessments (EGRA) were used as the basis of evaluation.

Improvements in reading scores of primary school students who received eReaders increased by 7.6% in a seven-month period, compared with control classrooms (Worldreader, 2012, p. 5). After one year, students showed substantial gains over control groups, as measured by the English subject examinations. The effects were particularly strong for girls.

The gains in mother tongue reading (Akuapem-Twi), in terms of correct words per minute (CWPM), provides a general indicator of progress, with Worldreader students learning to read almost twice as fast as students in the control group (see figure below). The authors attributed these gains to the programme’s focus on teacher training, and the emphasis on Twi phonics within this.

Using mobile devices to close achievement gaps

When disaggregated by gender, girls and boys in the Worldreader programme improved the same amount in terms of ORF in the local language, whereas girls in control schools improved only half as much as boys.

Similarly, the PRIMR study provided evidence that rural students show greater gains in learning from eReader access than their peri-urban counterparts. In control schools, there was a larger increase in literacy outcomes in peri-urban zones compared to rural zones between January and October 2013. But for the PRIMR Kisumu ICT intervention schools, an improved fluency of 9.7 CWPM was observed in rural schools as a result of treatment, compared to a 3.2 CWPM improvement that was observed for peri-urban schools (Piper & Kwayumba, 2014, p. 41).

The studies reviewed may suggest that mobile devices (such as eReaders and TeacherMates) may be better positioned to address issues of equity than suites of desktop computers. Perhaps surprisingly, none of the research studies evaluated provided evidence of students’ use of mobile phones for learning. This was the topic of a review focused on Asia, which found that while there was important evidence of mobile phones facilitating increased access to technology, there was much less evidence relating to whether or how this resulted in new learning (Valk et al., 2010, p. 1).

However, caution should be exercised when generalizing more broadly, from students’ use of mobile technologies such as eReaders and TeacherMates, to students’ use of other forms of edtech, such as computers. The EdQual programme found that home and community environments determine the quality of education, particularly for the most socio-economically disadvantaged groups. Attending a school where a substantial proportion of the pupils share one or more of these disadvantage indicators is also strongly and negatively associated with learning achievement, independent of the individual’s background (EdQual, 2011, p. 9).

3.2.4 Summary

In summary, among the examples of edtech used by students, there are relatively few accounts of whether changes in practice occur, and if they do, what the nature and extent of such changes are. As well as this, most accounts are also based on self-reporting rather than quantifiable metrics. Additionally, more focus needs to be on curriculum and pedagogy rather than simply supplying equipment, as there is some evidence that this leads to positive changes in practice.

Evidence of measurable impacts from students’ use of edtech on student learning outcomes, from the research studies reviewed, is sparse. Three of the research papers showed improvements in student learning outcomes, in terms of reading speeds, fluency, or literacy, in the mother tongue and the English language (Piper & Kwayumba, 2014; Worldreader, 2012; Murz, 2011).

Again, providing computers without thinking about pedagogy produces no measurable improvements (UNESCO, 2013a; Linden, 2008). There is some evidence that CAL in mathematics, as a supplement to, not replacement of, other teaching, may be of some benefit, especially for disadvantaged (Banerjee et al., 2007) or underperforming (Light, 2009; Lai et al., 2011) students.

3.3 Is there any evidence that the use of edtech in schools represents value for money?

Surveying the studies reviewed for this guide, there was a dearth of cost data and cost-effectiveness analysis for the edtech interventions assessed. Only a tiny number presented breakdown of costs, provided value for money (VFM) indicators, reported key cost drivers or conducted a cost-benefit analysis linking the input costs of the edtech interventions with their impact on pupil learning outcomes. None compared the cost of edtech interventions to non-edtech interventions, which is a key VFM comparison.

This is problematic, because without cost data it is difficult to assess the VFM of edtech interventions, or make comparisons between edtech and non-edtech interventions that could deliver the same outcomes.

Cost-effectiveness analysis determines how much impact (e.g. on teaching practice or learning outcomes) an intervention achieves relative to the inputs invested in it. Before the design stage of edtech programmes, the costs of programme inputs should be estimated, key cost drivers analysed and forecast, VFM metrics developed, and a cost-effectiveness analysis carried out. These cost indicators should be revisited throughout the project cycle, and be used when evaluating the effectiveness of interventions. The 3 Es (Economy, Efficiency and Effectiveness<ul><li><strong>Economy</strong>: Are inputs of the right quality at the right price? e.g. hardware, training, software.</li><li><strong>Efficiency</strong>: How well do the inputs convert into outputs? e.g. improved curriculum, number of people trained, improved facilities.</li><li><strong>Effectiveness</strong>: How well are the outputs from an intervention achieving the desired outcomes? e.g. improved student learning outcomes.</li></ul> of edtech interventions should be clearly analysed. A lack of clear information about key cost drivers or of a robust cost-effectiveness analysis indicates that issues may arise around the transparency and accountability of funds put towards edtech interventions, and the impact these interventions may have.

Case study: Analysing cost-effectiveness

One study of the Primary Math and Reading Initiative (PRIMR) programme in Kenya provided an excellent example of cost-related data and cost-effectiveness analysis for edtech interventions (Piper & Kwayumba, 2014). The main objectives of the PRIMR programme were to assess the impact on pupil outcomes of three ICT interventions implemented at different levels of the education system, and their cost-effectiveness against each other. The three interventions were:

• Teachers’ Advisory Centre (TAC) tutor tablets to bolster instructional support to students
• teacher tablets, with enhanced teachers’ guides and PRIMR materials
• eReaders for students, used during language lessons and after-school clubs.

To measure cost effectiveness, the unit costs per student (comprising the total of student books, teachers’ guides, teacher training, classroom observations and TAC tutor training) were analysed, in the context of the intervention and level it was applied at (student, teacher, school). The basic cost of the PRIMR model was $2.28 per student, slightly less than the control group costs. Per student, the eReader cost $40, the teacher tablet $3, and the TAC tutor tablet $0.10 (Piper & Kwayumba, 2014, p.34-5). Given that the impact of the three interventions on learning outcomes was similar, per-student cost-effectiveness analysis makes a significant difference. In the figure below, the ORF gains over the baseline results per dollar spent are shown. The TAC tutor tablet programme was nearly two times more cost-effective than the teacher tablet or control groups, and ten times more cost-effective than the student eReader. The teacher tablet programme was marginally more cost-effective than the control group. The student eReader group was six times less cost-effective than the control condition. Therefore, while PRIMR’s ICT programs were all effective, the per-student costs of each treatment were remarkably different.

The PRIMR study found no statistically significant difference in the improvements between the three treatment groups (tutor tablets, teacher tablets, and student eReaders), but given that the numbers requiring the edtech grew progressively larger from one group to another, PRIMR did find that the cost-effectiveness of teacher and tutor tablets to be orders of magnitude greater than that of student devices.

Unfortunately, these cost issues are rarely mentioned in the majority of studies.

3.4 What are the enabling factors that inhibit or support integration of edtech?

 

3.4.1 Inhibiting factors

In several of the appraised studies, authors identified issues that they felt had inhibited the effective implementation of edtech, or limited effects on teaching practice or learning outcomes. Such issues typically fall into one of the following categories:

• inadequate support for curriculum-related uses
• limited access to edtech resources for curriculum purposes
• difficulty in identifying appropriate curriculum content
• equity issues.

Inadequate support for curriculum-related uses

One issue is inadequate or inappropriate support for curriculum-related uses of ICT. For example, the Myanmar ICT Master-plan 2011–2015 includes several action steps relating to e-Education, including incorporation of ICT training into the school curriculum, teacher training in the use of ICT and the development of teaching materials for ICT training (UNESCO, 2013a, p. 44). But, in practice, the focus appears to have been on technology (LAN installation and internet connection for high schools) and satellite broadcast audio resources, rather than on the development of curriculum-related materials or teacher development; “the frequency of use for teaching and learning was considered to be very low” (UNESCO, 2013a, p. 45).

Similarly, one of the summary findings of the EdQual research programme was that “computers are not often used for teaching and learning purposes and that schools and teachers need to be supported in their use” (EdQual, 2011, p. 3). EdQual went on to draw out important policy implications:

In Chile, the long-running, large-scale Enlaces ICT policy has been highly successful in introducing computers and ICT infrastructure into huge numbers of schools, and has also provided training on computer use to 82% of teachers, yet “critical results are narrow in terms of classroom learning, and no additional competencies have been observed” (Sanchez & Salinas, 2008, p. 1,621). It has been found that that “school principals think that the integration of computers into classroom pedagogical practices is the most difficult task at the school level” (p. 1,629).

An over-emphasis on (and over-funding of) ICT equipment and infrastructure, at the expense of adequate provision (and funding) of effective support for implementation is central to this problem.

Gulbahar & Guven (2008) also point to the need for high-quality materials, developed in authentic classroom contexts, to support teacher professional development and classroom practice.

Limited access to edtech resources for curriculum purposes

After problems related to lack of teacher support and curriculum-related materials, one of the most frequently mentioned inhibiting factors is lack of access to IT facilities or internet services, particularly for subject teaching.

A field survey in Rwanda found that 52% of students had access to computers for a period of one hour per week and less than 30% of the respondents had accessibility of up to three hours a week (Rubagiza et al., 2011). The author suggested computers were prioritized for administration purposes.

While governments and donors fund ICT equipment, schools are often expected to maintain them out of their own funds. This can lead either to equipment falling into disrepair, or to use being restricted.

A study in Namibia found low usage of ICT in rural science classrooms due to poor ICT infrastructure, lack of technical and pedagogical support, and the shortage of digital learning materials (Ngololo et al., 2012).

While it is hoped that mobile internet use may offer an alternative to overcome some of the difficulties of wired access in rural areas, none of the studies appraised reported on this. It may be that this is because wireless data networks are only just becoming available in rural areas, and therefore there hasn’t yet been sufficient time to implement and report research studies. Alternatively, it may be that in many contexts, wireless connectivity currently remains too slow, unreliable or expensive for realistic use in schools. This is likely to be a rapidly changing landscape over the current decade.

Difficulty in identifying appropriate curriculum content

In some cases, curriculum materials are provided but are not felt to be appropriate by end-users. For example, in the case of the Rwandan NEPAD schools, EdQual reported that materials developed for a particular cultural context in South Africa went against the cultural values of teachers in Rwanda:

Equity issues

UNICEF (2013) caution on the use of ICTs, particularly with children, noting that ICTs often serve to exacerbate pre-existing areas of inequality for children, rather than reduce them. As such, inequality needs to be addressed intentionally in any project: technology will not be spread equitably by default.

3.4.2 Supportive factors

Across the appraised studies, a number of features were identified that seemed to be common to studies reporting implementation of edtech with positive effects on classroom practice or learning outcomes. These included:

• a focus on curriculum, teaching and learning throughout programme design
• mobile technologies with appropriate curriculum resources
• appropriate programmatic support for teachers
• sufficient time to develop new teaching and learning practices
• opportunities to harness ICT outside school.

Focus on curriculum, teaching and learning throughout programme design

Several studies showing impact on teaching practice or learning outcomes, have used relatively simple mobile technologies (for example eReaders, TeacherMate devices, teachers’ iPods or mobile phones) loaded with relevant curriculum materials. These technologies are typically deployed with a very clear pedagogic intent and curriculum purpose (such as teaching phonics, early literacy or engaging model language).

There is some evidence that teachers find task-orientated mobile technologies simpler to use than complex computers, both for professional development and for classroom use. For example, EIA in Bangladesh found “the iPod touch is playing a great role in this intervention for teachers’ professional development” (Shohel & Banks, 2010). Students also stated that they felt this equipment aided their pronunciation and teachers felt it helped keep their attention in class and made the lessons more enjoyable (Shohel & Banks, 2010, p. 5,490).

Mobile technologies with appropriate curriculum resources

Mobile technologies can potentially overcome barriers of access, particularly in rural areas, and also provide relevant, appropriate, contextualized learning materials:

• radios for IRI (Burns, 2006; Ho & Thrukal, 2009)
• mobile phones with offline audio or video resources (Enge et al., 2011; Walsh et al., 2013)
• eReaders with offline books (Piper & Kwayumba, 2014; Worldreader, 2012).

The programmes that use these technologies are among those that present the strongest evidence of impact on learning outcomes (Ho & Thrukal, 2009; EIA, 2012c; EIA, 2014c; Piper & Kwayumba, 2014; Worldreader, 2012), among studies reviewed.

Appropriate programmatic support for teachers.

A recent DFID-funded rigorous literature review identified four key findings that characterize teacher education programmes that improve teaching practice and learning outcomes:

• professional development aligned with classroom practices, with follow-up support and monitoring (including lesson modelling, constructive feedback on practice and discussion)
• peer support (formal and informal, in clusters or schools; focused on introduction of new classroom practices; joint observations, lesson planning and resource sharing)
• headteacher support (awareness of, and support for new methods of teaching, from headteacher and community)
• assessment (teaching development/student formative and summative) aligned with the curriculum.

The need for edtech programmes to offer such programmatic support is underlined by several of the appraised studies, either as a recommendation to bridge a gap where outcomes have been sparse or as a strength of current practice, where outcomes have been positive.

The need to engage the support of school leadership for implementation of edtech interventions has also been noted. Teachers surveyed in Kenya agreed that headteachers created a positive and supportive school culture towards web-based educational materials. “26 out of the 35 teachers describe their headteachers as what might be termed ‘lead learners’, role modelling the use of the [resources]” (Cullen et al., 2012).

A study of the use of the TeacherMate learning system in Rwanda found teachers needed monitoring and assistance to get into routine usage (Murz, 2011). EIA found teachers benefited from school visits and feedback, which kept their focus to practise their learning in classrooms (Shohel & Banks, 2010).

Sufficient time to develop new teaching and learning practices

Even with simple mobile technology, it can take time for teachers to become familiar in finding their way around resources (Shohel & Banks, 2010). Similarly, students may need a lot of time to become familiar with eReader operation (Worldreader, 2013, p. 7).

The Intel Teach Essentials Course includes 40 to 60 hours of training “to prepare teachers to integrate ICT across the curricula as a tool for learning and to design and implement inquiry-driven, project-based learning activities” (Light, 2009). Fieldwork looked at the changes in classrooms where this training had been used. Schools were found to have developed “new beliefs about learning and new practices, new ways to engage with content, changing relationships, and new ICT tools for learning”. EIA also provided ongoing support to teachers, over a period of a year or more, to enable them to introduce significant changes in practice (Walsh et al., 2013).

Opportunities to harness ICT outside school

Although, as previously noted, differentials in access to ICT outside school may raise several issues of equity, some students who may have very poor experiences of ICT in schools, may have rich ICT experiences outside schools (Rubagiza et al., 2011). Teachers also have been found to use internet cafes and other opportunities to access professional development OERs outside school (Cullen et al., 2012). There may be opportunity for considering how these external ICT opportunities, experiences and skills could be drawn upon to enrich in-school learning or in-school uses of ICT.

The overall picture that has emerged from this review is that the use of edtech in practice remains a complex challenge. With some notable exceptions (e.g. DEEP, EdQual and EIA), this guide has indicated the lack of evidence on how teaching practice changes as a result of edtech and found clear gaps in the research on the relationship between edtech use by students and improved learning outcomes. Thus, the review has highlighted the need for supportive and relevant curriculum materials, teacher CPD, clear understandings of the pedagogic context and the development of stronger evaluation strategies of ongoing changes in teachers’ practice.

Looking at the evidence explored in this guide through a more analytical lens, the following broad issues need to be considered before edtech implementation:

1. There is lots of research on teachers’ use of edtech

Much research has shown that teachers’ use of technology is contextual. Research has been undertaken to determine how teachers use technology on a day-to-day basis (e.g. preparing notes, using the internet for research purposes), the challenges they face (e.g. limited computer literacy, resource constraints) and understanding how the school setting impacts on their ability to undertake their teaching (e.g. power outages, limited infrastructure). Much of the work focuses on administrative issues such as the motivation to purchase equipment and resource constraints. While this research is worthwhile, it is not deeply related to the practice of teaching.

2. However, use is different to practice and we don’t know enough about practice

How do teachers use technology in the practice of teaching? Two key (and interrelated) points are:

• the pedagogy related to the subject being taught is often overlooked
• how the technology can support practice is not addressed.

Subject-specific pedagogy. There are good examples of how to investigate changes in classroom practice but these have yet to be directly linked to technology use. For example, in EIA there is a very clear focus on the importance of supporting subject-specific pedagogy, in this case communicative language teaching (CLT) in English language learning. The researchers were able to demonstrate very clearly the important changes in teaching practice, including the use of spoken English in the classroom (EIA, 2012b; EIA 2014b).

Teachers also used the project technology outside of the classroom to help them reflect on and improve their practice. This is a great first step as it highlights how technology use by teachers (often in their spare time) can impact upon changes in their practice. Moreover, many projects have shown how technology can support teachers to increase their subject knowledge (e.g. Leach et al., 2005, p. 61). The question is: can we go further?

The functionalities of the technology are important. One of the key motivations for governments to invest in technology in education is to provide students with new insights into particular phenomena. One example from maths is using dynamic geometry software to allow students to view the relationship between angles in a triangle visually, and manipulate the various parameters to get a better understanding of these relationships. This is an important alternative to using equations. The key point is that the functionalities of the technology linked to an appropriate pedagogy provide new ways of learning that can only be achieved with technology.

Mobile learning (Pachler et al., 2010) provides a good example of this. The functionalities of the technology provide numerous opportunities for learning, including geographical reach, communication, information storage and internet access. However, how these functionalities are used depends upon the pedagogical design and underpinning model of learning (Valk et al., 2010; Haßler et al., 2011). Where an intervention has been well designed, innovative uses have been documented. For example, as early as 2002, teachers on the DEEP project were using mobile devices for “creating images and movies for use as ongoing classroom resource; capturing classroom activity to document student progress; recording programme activity for self-assessment/portfolio” (Leach et al., 2005, p. 57).

Thus, change in practice derives from paying close attention to the interplay between pedagogical and technical innovation.

However, the challenge of designing and supporting this interplay should not be underestimated. From previous research, we know that teaching practice often doesn’t change with the introduction of technology. Also, our findings suggest that it can be difficult to analyse and review the specifics of practice with technology, as what software was used and how teachers used it in their practice is often not presented in sufficient enough detail. This makes the justification question, as outlined above, very difficult or even impossible to answer.

So more research is required to better understand the relationship between the design of edtech and teaching practice in low-resources settings. This goes to the heart of the question of what it means to embed technology in teaching practice.

While this is a complex issue, there are important starting points. The 2009 TALIS study on practice, although carried out mainly on OECD countries, showed that teachers engaged in teaching strategies in the following order: “structuring of lessons, followed by student-oriented practices and finally enhanced learning activities such as project work. This order applies in every country [in the project]” (OECD, 2009, p. 121). Interestingly, studies reviewed for this guide also found little evidence of student-oriented practices and project work, illustrating that the pedagogical challenges discussed are not resource-specific but are deeply related to the values that teachers (in whichever country they are working) place on classroom practice.We need to know more about teachers’ values and their perspectives on their preferred teaching strategies across low and middle-income countries. Only then can we think about developing evaluations of subject-specific pedagogies. Designing project evaluations in this way will allow for the development of deeper understandings of the specific nature of the change in teacher and student practice when using technology.

3. Programmes must be explicit about their conceptual framework

The conceptual framework underpinning many of the research projects was not clear.

This leads to defining the principles of the project, which in the case of EdQual were inclusivity, relevance and democracy. The implications for this are clear: including a focus on all learners, with learning outcomes determined through consensus.

The DEEP project was also clear about its theoretical principles, which primarily focused on teacher learning as a social practice, where developing teaching practice is a process of collaborative knowledge construction. This enabled the project to position technology as “an essential component of human agency…[helping] communities learn about, respond to, act on and manage their experience of the world” (Leach, 2008, p. 4).

While different projects may take different approaches to the design and use of edtech, a strong conceptual framing allows for a better understanding of the underpinning motivations of the researchers and practitioners. Furthermore, it makes their rationale for the role technology can play in education transparent. This is absolutely critical in development but is often not made explicit enough.

4. Teacher training is critical, but more attention must be paid to its design and focus

The importance of teacher training cannot be emphasised enough. Across the studies, this took a variety of forms, with many projects providing training via workshop models. However, these were often too focused on developing basic competencies regarding the use of technology. Instead, a key to the success of many of the larger and more longitudinal projects was the importance of ongoing support and training directly related to pedagogy. Both the DEEP and TESSA projects strongly promoted school-based teacher training, where partnerships and communication were seen as very important. Where there were few opportunities for this, it was seen as problematic. For example, in the Tikwere IRI project: “…there appears to be no systematic way for teachers to provide feedback to the Tikwere production unit on the errors they observe and the specific difficulties they experience” (Method et al. 2009, p. B1-4). Interestingly, more recently, the latest OECD TALIS Report (2014) emphasised the need for a stronger emphasis on teachers’ ongoing professional development and the DFID-funded research by Westbrook et al. (2013) stressed that professional development must be aligned with classroom practice, with follow-up support and monitoring (including peer and headteacher support) being of critical importance.

5. The challenges of generating evidence

The use of edtech in a development context is increasing, resulting in a growing interest in determining the role of technology in educational attainment. This raises a number of fundamental issues regarding what counts as evidence in this context.

There are no simple answers here. Every stage of the role of technology in education in developing countries is fraught with complexity. As such, education technology can be considered a “complex intervention”, i.e. one that has several interacting components and is very sensitive to features of the local context (Craig et al., 2008). While there is increasing recognition of the need for rigorous evaluation of complex interventions, this is sometimes framed in an overtly simplistic manner, where evidence-based practice – often in the form of an RCT – is sought to justify the promotion of technology in education. While we are not saying that there is not a role for RCTs, we emphasise caution in doing so. The complexities outlined in points 1–4 are not trivial and can seriously undermine any RCT, if not accounted for correctly.

6. More peer-reviewed evidence is needed

Of the 45 research documents including in this topic guide, only 17 were peer-reviewed journal articles. It is clear from this that the field, although a nascent one, needs to place more effort on opening its research up for peer review.

In the context of enormous global challenges to improve the quality of education, particularly in low to lower-middle income countries, governments, donors, schools and communities often seek to explore or exploit the potential of edtech. The studies reviewed for this guide provided some compelling examples of evidence that this potential can be realized, to produce educationally significant impacts on practice and outcomes. In particular, there is some evidence that mobile technologies (radios, mobile phones, and tablets) – used for curriculum-specific purposes in a context of appropriate support – can be particularly effective. There is also tentative evidence that such approaches may contribute to addressing issues of equity, in relation to gender and rurality.

But there are also many studies that either stopped at the point of identifying the difficulties and challenges experienced, or described what was done, but failed to provide adequate evidence of what difference was made to the educational experiences of the teachers, students and communities involved. There is therefore a pressing need, both to improve the quality of design and implementation of edtech programmes in order to raise their effectiveness in improving educational quality, and also to improve the evidence base on ‘what works, in which contexts, why and how’.