Road Map to Open Science

Download the Road Map in PDF

• Michela Bertero, Centre for Genomic Regulation (CRG)
• Ignasi Labastida, Learning and Research Resource Center of the University of Barcelona (CRAI UB)​
• Rosina Malagrida, Living Lab for Health, IrsiCaixa AIDS Research Institute 
• Pastora Martínez, Open University of Catalonia (UOC)
• Nadja Gmelch, Open University of Catalonia (UOC)
• María Jesús Pinazo, Barcelona Institute for Global Health (ISGlobal)
• Anne-Sophie Gresle, Barcelona Institute for Global Health (ISGlobal)

1. Open access

1.1 What has happened in the session?

The parallel session on Open Access was divided in two parts. The first part of the morning was dedicated to identify what the problems are to make the transition to Open Access. We divided the attendees in four groups, each one led by the four experts, representing four stakeholders: researchers, institutions, publishers and funders. All the attendees participated in two of the four groups and they agreed on identifying the following problems:

• There is still a lack of understanding about the meaning of Open Access and all the terms used.

• There is still a problem on the sustainability of Open Access models because current article processing charges (APC) are high. Publishers want profit.
• It is not easy to introduce Open Access in reward evaluation systems.
• The Green Road (self-archiving in repositories) implies more work for researchers.
• There is a lack of awareness of the publishing system.
• Funders have problems in enforcing the non Open Access compliant grantee.

The second workshop was dedicated to find some solutions to overcome the problems found in the first part of the session. Attendees choose to group again in the four stakeholders to work in the solutions. The list of the solutions includes some recommendations that were included in the final part of the session.

• Sign DORA and apply new models to evaluate science: institutions should use different indicators, and measure quality instead of quantity.
• Create Open Access units in each research institution.
• Create coalitions or networks of Open Science. Create a Seal of Open Science.
• Control the costs of APC, caps on APC, divest from subscriptions and pay APC, create funds for promoting Open Access. Support development of different funding allocations.
• Monitor Open Access and Open Data. Measure the performance of institutions on Open Access. Publish the results of measurements and flag no compliance.
• Create more policies to force Open Access. Draft a policy if it does not exist yet. Create a mandatory deposit and systems to help compliance
• Raise awareness of the current situation. Communicate better and get to the press, create materials for general public. Look for evidences.
• Pressure higher institutions like CRUE, EUA or even the UN.
• Foster the creation and development of alternative publishing platforms.
• Provide clarity and support: guidance, tools, information on benefits of Open Access.
• Train everyone in the institution on Open Science Practices. Brief administrators on Open Access.
• Train funders and institutions on how to evaluate, what to measure, how to report.
• Recognize good practices.
• Form consumer/producer association to challenge in court the current publication system.

In the last part of the second workshop everyone wrote some recommendations. Each recommendation had to be addressed to one of the stakeholders we previously worked with. Therefore the list of recommendations is grouped by targeted stakeholders.

1.2  What are the recommendations?

Recommendations addressed to researchers:

• Deposit all your works in a public repository.
• Support and sign declarations advocating for changing the evaluation system like DORA.
• Be activists: associate with like minded colleagues, crowdact by discipline and organize events to raise awareness, for instance screen documentaries like Paywall.

Recommendations addressed to research institutions:

• Dedicate, at least, 2,5% of your library budget to fund Open Access initiatives.
• Look at your current publication data, analyze the value of the subscriptions, the financial implications of Open Access publications and prepare for big deal cancellations.
• Cancel subscriptions and refuse packages and Big Deals from Publishers.
• Start and pilot your own publishing platform.
• Participate in International initiatives for Open Access like the OA2020 Initiative and support and sign declarations advocating for changing the evaluation system like DORA.
• Create a new and fair evaluation system including open access for promotion and do not use the impact factor for evaluation.
• Empower researchers to work Open Access and engage with them.
• Train students and researchers in Open Access.
• Invest in Open Access units and in general support for researchers regarding Open Access, Open Science, and Open Data.
• Support tools to practice Open Access and leverage other archives like Arxiv
• Create a local policy.

Recommendations addressed to publishers:

• Practice transparency in business models, cost structure.
• Introduce workflows to facilitate OA.

Recommendations addressed to governments:

• Pass laws defending Open Access and enforce existing mandates.
• Override unfair theft of copyright in laws like it has been done in some countries as in the Netherlands, Germany or France.
• Develop tools for Open Access and Open Data.
• Fund Open Access journals.
• Develop Open Science Plans with implementation and funds and aligned with the European Commission.
• Don’t use impact factor for evaluation.
• Support and encourage the implementation and expansion of initiatives like Plan S.
• Open an inquiry on the business practices of the publishing industry.
• Prioritise Open Science in the research agenda in international negotiations.

Recommendations addressed to funders:

• Adopt new models of evaluation: value Open Access in grant evaluations and do not use impact factor for evaluation.

2. Ethics, integrity & reproducibility

2.1 What has happened in the session?

• Funders should promote reproducibility through different mechanisms. Examples could include making costs for reproducibility eligible in grants, providing additional funding to reproduce relevant results, including a design phase in granted projects.

• Publishers should have an active role in promoting reproducibility. For example they could provide reward or incentive mechanisms for papers containing reproduced experiments. We could have a new indicators for papers showing the reproducibility of the experiments they describe.

• Funders, research institutions, and principal investigators (PIs) should implement mechanisms to reproduce results: e.g. performing internal replication (PIs), implementing replication of some random experiments before publication (research institutions) or commissioning replication of a random set of experiments by an external actor (funders).
• Funders, publishers and research institutions should promote the development of new tools, technologies for automatic screening for issues on research data. There is still a shortage of such tools to detect potential problems, most of the tools are used manually without automation.
• Researchers should consult statisticians very early on when designing projects and experiments.
• Data management should be promoted and implemented at different levels and by different stakeholders. Funders should all include a mandate for data management plans for their grantees, including mandates for data sharing, archiving, open methods, etc. Institutions should provide data stewardship. Research institutions should provide data management tools and resources. Technicians should support researchers in different data management tasks, including archiving, electronic lab notebooks, etc. Junior researchers should all have a data management plan before starting their project.
• Infrastructure is key to ensure research integrity and reproducibility. Research institutions should be responsible for maintaining supportive infrastructure, including required software, documentation, etc. Funders should provide support for infrastructure (development when required and long term maintenance)
• Funders, publishers, and researchers should promote a culture that supports sharing negative, intermediate or contradictory results
• Research institutions should have in place an ombudsman (independent advice).
• More training, more specialized and more engaging, is required. Funders should provide certified training material (e.g. the Health Department should provide certified training in biomedicine). Research institutions and PIs should create spaces for discussions on research integrity and share research integrity as common value. Research institutions should provide discipline-specific training in new engaging ways. Principal investigators should also be trained on research integrity and data management.
• Researchers should engage with society. For example, in clinical research patients should be involved in research design, methodology and outcomes. In basic research, representatives of society could be involved in board or committees.
• The practice of authorship could be revised. For example, publishers could promote authorship of single experiments or observations.

3.New Model for Research Evaluation

3.1What has happened in the session?

The session started with some presentations on the following topics:

• The work of the European Commission Expert Group on Open Science Indicators with an emphasis on the understanding of Open Science as Open Knowledge Practices.
• Examples of new ways of evaluating research:
  • Presentation of the ISRIA statement (International School of Research Impact Assessment) - Ten point guidelines for an effective process of research impact assessment
  • Insights from a multiple sclerosis patient participating in in the British Medical Journal’s Patient Panel.
• Some criticisms of Open Science:
  • Discussion of usual assumptions of the Open Science movement in the light of current understanding in Science Studies.
  • Comparison between diagnosis and remedies on present science usually attributed to Open Science with approaches in the history of sociology of science dealing with similar issues.
• Presentation of a practical case: CAREnet - UOC research group: how does the current and conventional way of evaluating the research group hinder the way forward to Open Science?

After these presentations, discussion was organised in three groups to address the following question: what changes are needed when evaluating people (research careers), projects and organisations?

3.2  What are the recommendations?

• There has been a broad consensus among all participants that, if we want to move towards Open Science there is a big and urgent need for a change in the evaluation model. This change does not only pass through the modification or adding of more and new indicators, but requires a profound change in the way we perceive and organise research evaluation. More than including Open Science indicators / checklists in the evaluation process, there is a need to include the philosophy of Open Science.

• Altmetrics are no alternative in this context.

• New research evaluation models have to be contextualised and use contextualised indicator frameworks (not one size fits all indicators).
• The purpose of evaluation has to be extended, moving beyond the current focus on auditing. Evaluation has always to be a formative process.
• For an effective impact assessment, follow the steps proposed in the ISRIA Statement:
  
  • understand the context
  • define and understand the purpose of the assessment
  • identify the key stakeholders and their needs and engage with them
  • use only the conceptual frameworks that are useful
  • use mixed evaluation methods
  • select and use metrics responsibly
  • take into consideration ethics and possible conflicts of interest
  • communicate your results effectively.
• Promote the participation of key stakeholders in the evaluation process, for example patients.
• Open Science promoters often use arguments of more efficiency in science. We need to be careful to promote still a more accelerated scientific process, fostering on the contrary Slow Science.
• To advocate and work on all these issues, the group proposed to create a community of practice (at least at Catalan level) to further discuss in incentives changes in the evaluation models.

4. Stakeholder and Citizen Engagement

4.1 What has happened in the session?

In addition to the speaker who gave a plenary session on the first day, two additional speakers were invited to participate in the workshop. During the first workshop, they started with a presentation of their work: the first one presented methodologies to analyse with different stakeholders the risks and benefits of the products that could be developed with one line of research and the second one focused on citizen science processes.
After the presentations, participants were divided into three different groups in which the three speakers acted as facilitators. The speaker who had given a plenary session on the first day facilitated a table on methodologies to engage stakeholders in research processes, focusing mainly on how to co-define a research proposal together with different actors. Finally, all the attendees were invited to think about problems or challenges to implement such methodologies.

The second workshop, focused on finding solutions to the challenges identified in the previous session. Those challenges were first completed with those coming from a tool developed by the EC funded project PROSO. The main clusters of challenges identified were:

• lack of relevance, i.e. perception of stakeholders as if engaging actors was not relevant to their interests, concerns and goals
• lack of impact, i.e. perception of stakeholder of engagement process having little or non impact
• lack of trust and critical view of others, i.e. lack of trust towards the organisers, funders or other participants
• lack of knowledge and skills to engage in research and innovation processes or in research and innovation decision making and/or policy making
• lack of time and financial resources to invest in engagement processes
• lack of legitimacy of the engagement process

Those clusters were splitted among the 3 groups of participants who worked on solutions. Finally solutions were shared and clustered by the facilitator of the session. The resulting clusters have been summarised within the recommendations listed below.

4.2  What are the recommendations?

• Policy makers should establishing international, national, regional and city level strategies and/or guidelines on multi-stakeholder engagement

• Research Funding Organizations should provide funding for programs based around societal challenges or needs and for implementation of results after the projects finish.

• Research Performing Organisations and other organisations such as Civil Society Organisations and companies should implement structural changes committing to engagement, starting from the mission and affecting all levels of the organization, including the development of strategies or internal roadmaps and the inclusion of different stakeholders in decision making bodies.
• Research Performing Organizations and Administrations should provide support for all stakeholders at research group level, institutional level and city, regional, national and translational level through the creation of Open and City Labs.
• Research Funding and Performing Organizations should embed engagement indicators into the new evaluation paradigm of research.
• All stakeholders promoting engagement should raise awareness of the impact of multi-stakeholder engagement by disseminating case studies, acknowledging the results of the engagement in publications, giving evidence on how closed is the R&I system currently and encouraging opinion leaders to become ambassadors of Open Science.
• Research Founders and Researchers Organizations should ensure incentives and awards for outstanding projects and policies developed by the different stakeholders and the former should recognize financial compensation for the efforts of stakeholders and citizens in engagement processes.
• Funders should assure the quality and relevance of the engagement processes by funding research on methodologies and establishing infrastructures for exchanges of good impact practices
• Education Institutions should promote scientific literacy and education on engagement and Open Science in formal and informal education.