Cheryl Hodgkinson-Williams, ROER4D PI, reports back on her trip to Johannesburg, South Africa to attend the one-day Africa Evidence Network (AEN) Seminar.
The Africa Evidence Network (AEN), supported by the University of Johannesburg’s Building Capacity to Use Research Evidence (UJ-BCURE) and the University of the Witwatersrand’ s Regional Centers for Learning on Evaluation and Results (CLEAR) co-hosted the “Challenges of evidence synthesis” seminar on 3 June in Auckland Park, Johannesburg. The role of the AEN is to bring together researchers and policymakers intent on building, interpreting and using evidence to inform policy. Established in December 2012, the AEN now boasts over 300 policymakers, researchers and practitioners in South Africa and Malawi. AEN is funded by the UK’s Department for International Development (DFID) as are the ROER4D Impact Studies, so it seemed like a good opportunity to meet with our ‘peer-projects’ and with our London-based DFID Programme Manager, Harriet MacDonald -Walker.
Apart from useful conversations with Harriet, it was interesting to listen to the three panelists
Dr Philip Davis (3ie / Campbell Collaboration), Dr Taryn Young (Centre for Evidence-Based Health Care / Cochrane SA) and Dr Stephen Taylor (Department of Basic Education / Stellenbosch University) who provided their perspectives on addressing these key questions:
How can we integrate in a process of evidence synthesis as many studies and evaluations, without compromising the quality and validity of the overall findings of a meta-analysis?
How can results of different types of effectiveness studies (Random Control Tests, quasi-experimental, econometric and qualitative) be standardized to a common measure that can allow for comparative analysis and identification of best interventions to be chosen by policy-makers?
Dr Philip Davies argued strongly for undertaking systematic reviews in order to make sense of the sheer amount of available evidence and the necessity to organise and establish a ‘balance of evidence’ on a particular issue to inform policy makers. One key technique he illustrated was the use of a ‘gap map’ to visualise where evidence on a particular issue exists, its quality and where there are still evidence gaps. He also referred to Spencer et al’s. (2003) paper on Quality in Qualitative Evaluation and the tool of establishing the Certainty of Qualitative Evidence. I anticipate that these may be useful in informing the final ROER4D Report.
Dr Taryn Young also made a strong plea for undertaking systematic reviews, but emphasized that we also need to read systematic reviews to establish the extent of the problem, to reduce risk of bias, to reduce risk of being misled by play of chance and provide critical appraisal. This advice is applicable to the ROER4D adoption and impact studies too as we endeavour to locate our studies within the current literature. I haven’t yet been able to locate a systematic review of Open Educational Resources as yet, but the A Bibliometric Mapping of Open Educational Resources produced by Airton Zancanaro, José Leomar Todesco and Fernando Ramos is a good start!
Dr Stephen Taylor illustrated different types of evaluation (diagnostic, design, implementation and impact) in relationship to research he is undertaking on ‘Improving Early Grade Reading in South African schools’. He mapped the process (his Theory of Change) as follows:
> Textbooks are delivered > Textbooks are distributed by schools to learners > Textbooks are used in class and/or at home > Textbooks are of sufficient quality > Tests are able to measure improved learning > Improved test scores. He did point out that there there could be other unobservable factors that may biasing the results (e.g. teacher motivation) which is why he recommends the use of Randomised Control Tests (RCTs) which can account for both observable differences and unobservable differences). He does however acknowledge that there is value in slightly less robust quantitative techniques such as Propensities Score Matching (PSM) that one of our ROER4D projects is using where RCT was not practical.
Clearly there is a growing group of researchers and practitioners interested in really wanting to interrogate existing evidence and build upon this. The ROER4D project shares this sentiment and has therefore become a member of the Africa Evidence Network.