Open science

Introduction to David Mayor and the Center for Open Science

Overview of Speaker and Organization

• David Mayor is introduced as the Director of Policy Initiatives at the Center for Open Science, located in Virginia, USA.

• His role focuses on enhancing transparency and reducing bias in scientific research.

• Prior to this position, he directed advising in the Division of Life Sciences at Rutgers University, collaborating with citizen scientists.

Mission and Activities of the Center for Open Science

Goals and Strategies

• The mission of the Center for Open Science is to increase trust, credibility, and reproducibility in scientific research.

• Their work involves meta-science to identify markers of credibility and barriers to reproducibility; advocacy efforts aim to address these issues.

• They develop tools and infrastructure that facilitate open science practices while striving to improve research culture.

Implementation Focus

• The organization emphasizes creating user-friendly tools that encourage researchers to adopt open science practices by normalizing them within their communities.

• Engagement with decision-makers across various sectors (journals, publishers, academic societies) aims to recognize and reward open science practices.

Challenges in Reproducibility

Barriers Identified

• Low rates of reproducibility are a significant concern; many barriers exist when attempting to replicate published studies.

• Issues include insufficiently detailed method sections, questionable research practices like selective reporting, and noise affecting results.

Addressing Reproducibility Issues

• Open science can mitigate these challenges through comprehensive reporting guidelines that clarify methods used in studies.

• By specifying methodologies upfront and allowing all results (positive or negative), researchers can minimize bad practices contributing to low reproducibility.

The Role of Open Science Practices

Key Takeaways

• A primary takeaway is that open science practices can eliminate poor reasons for low reproducibility while fostering productive discussions between older findings and new discoveries.

• Emphasizing environmental differences or variations among study groups can lead to meaningful insights rather than merely replicating past results.

Open Science Practices and Their Importance

Overview of Open Science at the Center for Open Science

• The focus at the Center for Open Science (COS) is on sharing underlying raw data, research materials (both physical and digital), and having clear research plans that specify study conduct in advance.

• Emphasis is placed on reporting results regardless of their nature, including null or statistically significant outcomes.

Rationale Behind Open Science

• The primary goal of open science practices is to enhance reproducibility in research, leading to better citation rates and increased funding opportunities.

• Practical recommendations will be provided for creating data management plans and informed consent documents as essential tools for researchers.

Benefits of Open Data

• Availability of open data can lead to higher citation rates; studies with openly available data are often more useful and citable within the literature.

• Observational studies indicate a citation advantage for comparable studies with open data, although establishing a causal relationship remains challenging.

Organizational Incentives in Open Science

• Implementing open science practices encourages researchers to organize their work effectively, making it easier for others to access and utilize their findings.

• A well-organized directory of datasets allows others to draw similar conclusions from the same dataset, enhancing argument quality in future projects.

Journal Policies on Data Transparency

• COS evaluates journal policies regarding data transparency through Topfactor.org, which categorizes journals based on their requirements for data availability statements.

• Journals are scored from zero to three based on how rigorously they enforce data transparency; higher scores indicate stricter mandates.

Funders' Expectations and Resources

• COS tracks funder expectations related to open science policies, providing resources for both funders and researchers about upcoming requirements.

• Examples of funders taking action towards implementing open science practices are shared as part of a curated resource available on COS's website.

Data Repositories

• The best repository listing can be found at 33data, which serves as a comprehensive registry of research data repositories available for researchers.

Data Sharing and Management in Research

Overview of Data Repositories

• Many data repositories are categorized by discipline or specific features, facilitating easier discovery of datasets.

• Ruby3 data is highlighted for its effective curation of information related to dataset sharing. A link to this resource will be provided later.

Protected Access and Ethical Considerations

• The concept of "protected access" is introduced as a solution to the challenge of sharing sensitive or identifiable information.

• Ethical checks are necessary before publicly posting datasets, ensuring that only legitimate researchers with ethics training can access sensitive data.

• A list of repositories offering protected access services is maintained, which includes staff available to vet data access requests.

Center for Open Science and OSF

• The speaker discusses their work at the Center for Open Science, which develops the open-source Open Science Framework (OSF). The OSF aims to manage research projects effectively while providing persistent repositories for data and materials.

• OSF supports various stages of research workflow: planning, managing collaborations, preserving project components, and disseminating findings through preprints.

Project Management Features in OSF

• Researchers can create new projects within OSF that allow them to choose appropriate storage options based on regional requirements (e.g., EU regulations). This ensures compliance with local data protection guidelines.

• Individual projects within OSF serve as central hubs for organizing research activities and adding collaborators with varying permission levels based on trustworthiness. Metadata enhances discoverability and preservation efforts.

Licensing and Data Curation

• Authors are encouraged to use openly available licenses such as CC0 or CC BY when sharing their datasets through OSF, promoting transparency in research practices. Guidelines exist for appropriately licensing projects.

• Projects are private by default until researchers decide they are ready to share them publicly; this approach reduces barriers associated with preparing datasets for public dissemination at the end of a project cycle.

Practical Tips for Data Management

• Researchers should avoid promising not to share or destroy data in consent forms; these outdated practices may hinder ethical sharing efforts today. Instead, focus on creating informed consents that build trust among participants regarding how their data will be used and shared responsibly.

Research Ethics and Data Management

Importance of Ethical Research Practices

• Researchers should avoid limiting their analysis to specific questions or topics, as this can restrict the applicability of their findings.

• Participants value that their data contributes to broader scientific benefits rather than focusing on citation counts; ethical communication is crucial.

• Obtaining informed consent and incorporating data retention clauses into ethics review templates are essential for ethical research practices.

• Researchers must be cautious about reidentification risks, especially with demographic information that could triangulate identities, particularly for underrepresented groups.

• Recommendations and templates for ethical documents are available to assist researchers in addressing these considerations.

Data Management Planning

• A well-defined data management plan includes details on file formats, access permissions, and preservation strategies.

• Resources like dmptool.org provide templates for creating effective data management plans; links will be shared at the end of the presentation.

• Best practices suggest organizing files in a way that is understandable years later, ensuring clarity in naming conventions and structure.

• Maintain raw data integrity by avoiding analyses directly on it; derived datasets can be modified without compromising original data quality.

• Readme files should include instructions and definitions for variables to aid future users in understanding the dataset.

Pre-registration of Research Studies

• Pre-registration involves submitting a study plan before conducting research, distinguishing between hypothesis testing and exploratory research.

• This process helps maintain rigor in confirmatory studies while allowing flexibility in exploratory contexts where unexpected findings may arise.

• The distinction between hypothesis testing (minimizing false positives) and exploratory research (minimizing false negatives) is critical for valid results.

• Pre-registration aims to clarify these distinctions amidst current incentives that often blur lines between exploratory results presented as confirmatory findings.

Incorporating Pre-registration in Research

Importance of Pre-registration

• Statistical tools can enhance the appearance of findings, making them seem more publishable but potentially compromising credibility.

• An ideal research workflow begins with a strong prediction and a confirmatory mindset, utilizing pre-registration to timestamp the study before data collection.

Workflow Examples for Pre-registration

• After collecting data, researchers should confirm results based on the pre-registered plan and explore additional moderators as exploratory findings for future studies.

• Alternatively, researchers may start with vague pre-registration indicating a discovery process, then split data into subsets for exploratory analysis while keeping part of it confidential.

Enhancing Credibility through Methodology

• Lock promising exploratory findings with a second pre-registration before conducting confirmatory tests on held-off datasets to maintain diagnostic value and credibility.

OSF Registry Network

• The OSF registry network assists in creating registrations applicable across various scientific disciplines, including clinical research and basic science.

• It provides examples and standards for different types of studies such as experimental research, observational studies, qualitative methods, and systematic reviews.

Broader Policy Initiatives in Science

Registered Reports Model

• The registered reports model integrates pre-registration into publishing workflows with two stages of peer review focused on hypothesis quality rather than outcomes.

• Authors submit proposed studies for initial evaluation; if approved based on rigorous methods and quality control steps, they receive "in-principal acceptance" regardless of final results.

Peer Review Process

• In the final stage of peer review, reviewers assess whether the study was conducted as specified without considering whether hypotheses were supported or impactful.

Resources for Authors and Reviewers

• Comprehensive resources are available to guide authors submitting registered reports and reviewers evaluating studies without results.

Addressing Novelty vs. Credibility

Top Factor Initiative

• Top Factor evaluates journals based on their policies promoting open science practices rather than flashy results or citation counts.

Importance of Transparency

• This initiative emphasizes reproducibility by assessing factors like data availability, pre-registration usage, and adherence to open science ideals among journals.

Open Science Practices and Their Importance

The Need for Improvement in Open Science

• The speaker emphasizes that journals can be sorted by various standards, particularly pre-registration, which is crucial for transparency in research.

• Most journals score poorly on open science practices, often receiving only 0 to 2 points out of a possible 29 in evaluations, indicating a significant lack of adherence to basic citation practices.

The Cyclical Nature of Scientific Advancement

• Science progresses through a cycle of idea creation, testing, and knowledge updating. Incorporating open science into these steps enhances the management and presentation of research findings.

• Regardless of the significance of results (positive or negative), open science practices can improve credibility and trustworthiness in scientific communication.

Introduction to Michael Asim

Background Information

• Michael Asim is introduced as the next speaker; he has nearly 200 attendees registered for his presentation.

• He is a professor at Berkeley specializing in genetics and genomics and advocates strongly for open science publishing.

Contributions to Open Science

• Asim co-founded the Public Library of Science (PLOS), known for its nonprofit model promoting open access to scientific literature.

• He serves as editor-in-chief for an open-access journal called Eli, furthering his commitment to accessible scientific communication.

The Evolution of Scientific Communication

Historical Context

• The internet was initially developed to facilitate communication among scientists at research institutions, aiming to enhance information sharing about research outcomes.

Current State of Scientific Publishing

• Despite advancements in technology over the past 50 years, traditional scientific publishing remains largely unchanged from its historical roots.

• Papers still resemble their predecessors from centuries ago; this stagnation raises concerns about the effectiveness of current publishing models.

Challenges Facing Open Access Publishing

Limitations within Current Systems

• The existing system fails to meet its goals effectively; it restrictively limits access rather than promoting widespread dissemination of knowledge.

Access to Scientific Literature: A Critical Examination

The Accessibility Crisis in Scientific Literature

• Only a small fraction of the global population has access to online scientific and medical literature, despite widespread internet connectivity. This includes not just the public but also researchers, teachers, healthcare workers, and journalists who could benefit from this information.

Delays in Publication

• The average time for a scientific paper to go from submission to publication is approximately nine months. During this period, the majority of papers remain inaccessible to both the public and other researchers. These delays hinder the application of scientific knowledge.

• The comparison between publishing timelines and space exploration highlights inefficiencies; it takes less time to send a robot millions of miles away than to publish a paper on Earth. This illustrates how absurdly slow current publishing processes are.

Cost Inefficiency in Publishing

• If all published science papers were sent to Mars and back as photographs, it would be cheaper than using the current publication system. This inefficiency results in significant financial waste within the scientific community, amounting to about $10 billion annually on an ineffective publishing system.

Flaws in Peer Review and Publication System

• The peer review process is criticized for being ineffective at determining which research is reliable or important. It often introduces biases based on author identity or field of study, making it harder for many researchers to publish their work fairly.

• The existing infrastructure idolizes certain journals over others without justification, distorting perceptions of scientific validity and influencing career trajectories negatively within academia. This distortion affects how science is conducted overall.

Failure of Science Communication via Internet

• Despite its potential for improving communication among scientists, the internet has been underutilized in enhancing access to scientific literature. Currently, publishers oversee a process that does not involve them creating content but still charge substantial fees for access that should be free or more accessible given that scientists do most of the critical work involved in research publication.

• There’s an urgent need for reform within the scientific establishment regarding these systemic issues that have persisted unaddressed while other fields like physics have already adopted better practices through preprint servers long before modern internet capabilities emerged.

What Are the Challenges in Biomedical Research Publishing?

The State of Biomedical Research and Science Publishing

• The speaker questions why the biomedical research community, along with chemistry and other scientific fields, struggles to solve problems as effectively as physicists do.

• The modern internet's inception in 1996 marked a pivotal moment for science publishing, coinciding with the speaker's postdoctoral work and efforts to reform this field.

• With the transition of journals online around 1996, there was an opportunity to utilize electronic copies for broader access and innovative uses of information.

Early Proposals for Reforming Science Publishing

• The speaker and collaborator Pat Brown recognized that existing technologies were not being fully leveraged in science publishing.

• In 1999, NIH proposed Ebiomed, a system aimed at providing free access to all published biomedical literature but ultimately failed due to opposition from publishers.

• Ebiomed would have allowed immediate access to research literature while empowering authors over publishers regarding paper sharing.

Reasons Behind the Failure of Ebiomed

• Despite its potential benefits, Ebiomed was killed by lobbying from both commercial and non-profit publishers who feared loss of revenue streams.

• Congress decided against funding Ebiomed after pressure from these publishers, leaving a gap in effective publishing reform.

Emergence of Open Access Publishing

• Following the failure of government-led reforms like Ebiomed, open access emerged as a grassroots response aiming for individual action towards change.

• PubMed Central evolved from Ebiomed as a platform where journals could voluntarily post content without obligation.

Principles Underlying Open Access Models

• Open access aims to make scientific works freely available rather than relying on subscription models that create paywalls.

• This model suggests upfront payment for publication costs instead of subscriptions which can limit accessibility due to financial barriers.

• The analogy used compares publishers' roles in delivering scientific manuscripts to obstetricians assisting births; they facilitate but do not create the content.

The Challenges of Open Access in Science Publishing

The Concept of Ownership in Scientific Publishing

• The speaker discusses a hypothetical scenario where ownership of a baby is transferred to an obstetrician, drawing a parallel to the current state of science publishing. This analogy highlights the absurdity of charging for access to scientific work.

• Open access aims to change the dynamics by ensuring that while services like peer review and formatting are compensated, ownership of content should remain with the authors.

Growth and Limitations of Open Access

• Despite growth in open access publications, which now approach 50%, it has not transformed science publishing as anticipated.

• The high costs associated with publishing (e.g., over $10,000 per paper at prestigious journals) reflect ongoing issues within the system rather than resolving them.

Missteps in Addressing Publishing Economics

• Initial hopes that open access would fundamentally alter science publishing have not materialized; instead, it has perpetuated existing economic challenges.

• A critical mistake was focusing on funding models without addressing the underlying reliance on journals themselves as gatekeepers of scientific knowledge.

Psychological Impact of Journals

• The existence and prestige associated with journals create a psychological barrier that hinders meaningful change in how research is published and valued.

• Changing journal funding structures alone does not suffice; there needs to be a broader reevaluation of their role within the scientific community.

Career Implications for Scientists

• Many scientists feel trapped by the necessity to publish in established journals for career advancement, despite recognizing flaws within this system.

• There is widespread agreement among scientists about the need for reform; however, fear and institutional pressures prevent them from pursuing alternative publication methods.

Profit Margins in Scholarly Publishing

• Scholarly publishers often enjoy profit margins comparable or superior to major tech companies like Apple and Google, raising questions about sustainability and fairness within academic publishing practices.

The Future of Scientific Publishing: Do We Need Journals?

The Problem with Current Scientific Publishing

• The scientific publishing industry is described as "extortionist," exploiting its control over science careers and perpetuating a flawed economic system.

• There is a growing sentiment that if journals hinder the improvement of science publishing, we may need to consider eliminating them altogether.

Rethinking the Concept of Publishing

• Clay Shery's perspective highlights that traditional publishing processes are outdated; in the internet age, publishing should be as simple as pressing a button to make work publicly available.

• The emergence of bioRxiv marks a significant shift in biomedical research, allowing authors to share their work freely and openly, similar to how physicists have done since 1992.

The Role of Peer Review in Modern Publishing

• A common argument against removing journals is the necessity for peer review; however, it is proposed that peer review does not require journals at all.

• If starting from scratch today, one would not create thousands of journals to categorize papers—this structure is an outdated legacy from pre-internet times.

Alternative Approaches to Peer Review

• An alternative model suggests reversing the traditional process: publish first, then review and curate works based on their merit rather than journal approval. This approach emphasizes transparency and accessibility in scientific discourse.

• BioRxiv allows for open posting of reviews alongside preprints, enabling readers to access both original papers and critiques directly, fostering informed judgments about research quality.

Continuous Engagement with Research

• The envisioned system promotes ongoing reviews throughout a paper's lifespan rather than limiting feedback to a single journal's decision point; this encourages broader participation from scientists in evaluating research.

• Curation remains an evolving aspect but could involve organizing literature by various criteria beyond just journal titles, enhancing discoverability and relevance for researchers.

Reimagining Scientific Publishing

The Future of Journal Curation

• Instead of relying on individual journals, researchers should directly inform readers about the most suitable journal for their paper, simplifying the curation process.

• Elife is exploring innovative technologies to enhance literature organization and curation, particularly in a landscape where papers are published as preprints followed by reviews.

• The concept of social networks around reviewers is emerging, aiming to improve how researchers access and engage with peer reviews.

Addressing Inefficiencies in Science Publishing

• A significant portion of the $10 billion spent annually on publishing is wasted due to inefficiencies like excessive journal numbers and unnecessary formatting.

• While subscription models and article processing charges (APCs) are problematic, they create different barriers for researchers with limited budgets.

Rethinking Funding Models for Publishing

• A proposed solution involves treating publishing as a utility funded by research consortia rather than charging authors or users directly.

• This model would reduce costs while ensuring equitable access to scientific knowledge without financial burdens on authors or readers.

Ensuring Fairness in Science Publishing

• Any new system must avoid creating barriers that could exacerbate existing inequalities within the scientific community.

• Emphasis is placed on addressing broader issues of fairness in science beyond just publishing practices.