Innovative Doctoral Training: Reproducibility and Open Science

Introduction

The Open Science Training Initiative (OSTI), is designed to address the problem of reproducibility in science. The programme, which is led by Sophie Kay of the Department of Computer Science, harnesses the principles and techniques of open science to develop students’ digital awareness. It uses a novel teaching approach called rotation-based learning to give them hands-on training in the knowledge and skills needed to produce high-utility, high-impact research.

OSTI is the first programme of its kind to offer fully integrated training in open science as part of a subject-specific taught course and is helping Oxford to lead the way in educational provision in this area. The course materials are being made available as Open Educational Resources with a Creative Commons Attribution licence (CC BY 3.0), for other institutions to use and develop according to their own requirements.

The Challenge

Reproducibility is the ability to rerun another researcher’s experiment by following the method as published and to obtain the same results. It is central to scientific research, not only to ensure methodological rigour but also to enable scientific advances, since researchers frequently build on each other’s work in order to make new discoveries. In other words, scientists are not only producers of research, they are also users of other people’s research. Young researchers therefore need to:

  1. understand that reproducibility is an issue, and
  2. learn how to make their own research reproducible through generating ‘open’ outputs: i.e. outputs that are freely accessible and licensed for others to reuse.

Reproducibility is thus a cornerstone of Open Science, and digital technologies and the Web now play a key role in making it possible to share methods, data, and outcomes on a global scale.

The Innovation

Photo of Sophie Kay teaching on the OSTI courseOSTI provides a series of lectures in open science, data management, licensing, and reproducibility for use with graduate students and postdoctoral researchers. At Oxford, the lectures have been initially incorporated into an existing three week face-to-face course on computational biology in the Doctoral Training Centre. During this course students develop mathematical and computational models of cancer and/or infectious diseases.

The course follows a rotation-based structure, with students working in small groups on separate scientific problems. In the first phase, each group reproduces the results reported in an existing published article and delivers a ‘coherent research story’: i.e. a written report, together with the accompanying data, code, and figures – all with an appropriate open licence permitting their reuse.

In the second phase, these materials are passed to another group via a version-controlled online repository. This successor group must then verify the results and use them as the basis of a new project – again producing a properly documented and licensed research story.

Throughout both phases of work, the groups are not allowed to interact with each other in any way. This incentivises the effective communication of their research through the outputs alone. In this way, pre-doctoral students experience the challenges of modern collaborative research and, thereby, understand the need to provide the scientific community with outputs that form a solid basis for further investigation.

Feedback

The OSTI-enhanced course was piloted in January 2013 with 43 students in the early stages of their doctoral research. At the outset, the majority had a minimal awareness of open practices; however, once they had been introduced to the main concepts they quickly absorbed and applied the methods. Indeed, 91% of respondents to the post-course survey said that they would be confident in implementing open science practices in the future, while all of the respondents felt that the course had contributed to their awareness of current practices in scientific research.

The course has subsequently attracted great interest from the Open community in the UK, USA, and other Commonwealth and European countries. Work is also under way to explore its applicability to other disciplines, including the social sciences, natural sciences and business.

Top Tips for Success

  1. A hands-on approach is instrumental in building students’ confidence in the use of Open techniques.
  2. Education in open practice is vital in developing student understanding of the changing research landscape they are about to enter. Integration of such training into existing courses allows students to develop these skills with subject-specific relevance.
  3. A rotation-based model of learning is instrumental in developing student perspectives on the utility of their own work, and has proved more successful in securing student engagement than single-project work.
  4. Give students ample opportunities to discuss the evolving research landscape and changing cultures of research.

Further Information

OSTI

  • Visit the OSTI website to obtain copies of the post-pilot report, links to course materials and videos of lectures from the pilot scheme.
  • For a peer-reviewed paper on the OSTI initiative, see Kershaw, S.K. (2013). Hybridised Open Educational Resources and Rotation Based Learning. Open Education 2030. JRC−IPTS Vision Papers. Part III: Higher Education (pp. 140-144). Link to the paper in Academia.edu

Open Science

  • Listen to the Open Science collection on Oxford iTunesU or on Oxford podcasts. The collection includes the Evolution of Science: Open Science and the Future of Publishing debate (Feb 2012) and presentations given at the Rigour and Openness in 21st Century Science conference (April 2013).
  • For information about Open Access publishing at Oxford, visit the OAO website.

This case study has been adapted from the OSTI website and the OSTI Post-Pilot Report, both of which are authored by Sophie Kay and are licensed under a Creative Commons Attribution 3.0 Unported Licence (CC BY).

oxtalent badgeWinner, OxTALENT 2013 award for open education initiatives

Posted in Mathematical, Physical and Life Sciences, OxTALENT Winner | Tagged , , | 5 Comments

5 Responses to “Innovative Doctoral Training: Reproducibility and Open Science”

  1. Excellent initiative! Over at Cambridge, a pilot project was started last year that integrates reproducibility and replication into social science teaching: the Cambridge Replication Workshop. In eight weeks, students replicate a published paper and thereby learn about reproducibility standards, data sharing, research methdos, and how to get publish at an early stage during doctoral studies.

    Here is a summary of the experiences of the grad students attending: http://politicalsciencereplication.wordpress.com/2013/05/14/nightmare-after-nightmare-students-trying-to-replicate-work/

    We also made handouts and literature of the course available: http://schreiberin.de/teaching/replication.html

    • Sophie Kay says:

      Thanks for the links! Fantastic to hear about your success with teaching replication & repro over in Cambridge – I see you’re running it as a full course again this coming year. Do you have a blog or other site detailing progress with the project? Would love to hear more.

      • Hi Sophie Kay,
        I’m commenting on the workshop and feedback from students and TAs, as well as things to improve, on my replication blog. I’d be very interested in chatting with you about our experiences – I will look through your materials in the mean time. Please get in touch via email (polscireplication@gmail.com).
        Cheers,
        Nicole

  2. Thanks for this very useful link. It’s good to have examples from different disciplines, especially as doubts are sometimes expressed about the relevance of open science to the social sciences (and humanities).

  3. […] University of Oxford recognised OSTI with an OxTalent Award in June 2013 (see their analysis on the LTG blog) and I hope to see the initiative go from strength to […]

Leave a Reply