Discussions

The number of publications has been increasing exponentially, and as a result, so has the research field of evidence synthesis. Consequently, there is now a need to maintain the quality, currency and credibility of evidence synthesis approaches. Within this commentary, we provide a vision for evidence synthesis as a fundamental tool for generating and guiding decision-making. This paper is aimed at all stakeholders, including researchers, institutions, and the broader general community.

Transparency is vital for repeatability and verifiability of systematic reviews, as with all other research. Authors of quantitative systematic reviews (i.e. ones that include meta-analysis) can maximise transparency by publishing the data they collected and the analytic code describing and documenting their analyses. This project will conduct a map of evidence syntheses to assess the degree to which data and code used in meta-analyses are made public by authors and journals publishing systematic reviews in environmental science and healthcare.

The Evidence Synthesis Technology world can be complicated - tools can be hard to find, it can be difficult to know what skill level is needed, and they can involve a considerable learning curve. Here at ESH, we are trying to lower the costs associated with finding, learning and using new ESTech tools and frameworks. This blog series aims to introduce key tools that can help increase transparency, accessibility, efficiency and rigour in evidence syntheses. Each month, a member of the ESH family will introduce their ESTech, explaining its purpose, the background skills needed, and how it can be used to support evidence synthesis.

ESMARConf is a FREE, online annual conference series dedicated to evidence synthesis and meta-analysis in R. Our aim is to raise awareness of the utility of Open Source tools in R for conducting all aspects of evidence syntheses (systematic reviews/maps, meta-analysis, rapid reviews, scoping reviews, etc.), to build capacity for conducting rigorous evidence syntheses, to support the development of novel tools and frameworks for robust evidence synthesis, and to support a community of practice working in evidence synthesis tool development. ESMARConf began in 2021 and is coordinated by the Evidence Synthesis Hackathon.

As the appreciation and need for timely and rigorous evidence synthesis continue to grow, so too will the need for tools and frameworks to conduct reviews of expanding evidence bases in an efficient and time-sensitive manner. Efforts to future-proof evidence synthesis through such Evidence Synthesis Technology (ESTech) have so far been isolated across interested individuals or groups, with no concerted effort to collaborate or build communities of practice in technology production.

Training resources for evidence synthesis are disparate and often hard to find. This project aims to collate and catalogue training resources for evidence synthesis methods from across disciplines, sectors, sources and formats, making it easier to find the right training. The result will be a web-based user-support tool.

Reliable evidence synthesis requires access to a comprehensive, unbiased body of literature that can be searched for relevant information. Systematic reviewers typically search multiple (upwards of 10) bibliographic databases to identify sets of search results that might yield relevant results. Access to these databases is often restrictively expensive, hampering efforts to synthesise evidence by smaller organisations and groups from low- and middle- income countries, for example. When reviewers export references from these databases they must typically do so in small batches (this supposedly stops people from replicating commercial databases for profit): for Web of Science this must be done in batches of 500, which can add considerable time to a review with 20,000 search results or more! Finally, databases such as Web of Science exacerbate publication bias by selecting journals and publishers that are perceived to be of ‘high impact’, for example using citation indices. So, these resources may be expensive, hard to use, and offer a biased selection of evidence. In order to facilitate evidence synthesis and to reduce bias in how information is indexed and found, we call for the production of an Open Source, Open Access on-stop-shop database that catalogues all known academic research. Since tables of contents are freely available online, technology exists that can produce such an important and useful tool.

Evidence synthesis relies on primary research that is reliable, transparent, and where key information is readily accessible and useful for broader synthesis. We propose a succinct list of ideal attributes that primary research articles should report as standard so that they are more likely to be found and included in evidence syntheses. We discuss how implementing these changes to primary research reporting might be incentivised for authors, peer reviewers, editors, journals, and institutions such changes and this broad across medicine, environment, ecology, and social science disciplines. We focus on the area of prevention science, but our conclusions are applicable across disciplines and fields.

R is a widely-used, open source programming language and statistical environment. Users are able to contribute add-ons to R functionality in a standardised way by developing new software ‘packages’. However, identifying which packages are most useful for a specific task can be challenging, particularly for evidence synthesis (ES) projects which typically include a number of discrete tasks, many using packages that may have been designed for other purposes. Consequently, a valuable tool for future researchers (and hackathons) would be a ‘map’ of available software packages, showing how those packages apply to ES. This would help guide new users through effective workflows, as well as identifying parts of the evidence synthesis process that are currently well supported in R, or conversely, in need of further software development. This project is currently in the data collection phase, wherein participants systematically search for R packages of potential value to ES projects and catalogue their findings in a structured way. The intended output is an academic article describing our findings, linked to a live database of R packages, the functions they contain, and the specific ES tasks that they each solve.

Systematic reviews are complex, laborious tasks that produce vast amounts of data. However, the effort required to produce these data are typically lost once a review is completed: some information is reported in the review, but often information is missing or specific details are lacking.