The Open Metric

https://drive.google.com/drive/folders/1-HNz4qqVG0TiwarHmm2TVAyv16u7qGH7?usp=sharing

https://drive.google.com/drive/folders/1vmmH3_80YEXlbm6mWGb9s6QyQYnOREef?usp=sharing

What is Successful Science?

As was recently demonstrated during the COVID-19 global pandemic, the scientific community has been responsible for some of the most important turning points in our history. Science has enabled humanity to advance and change the environment around them throughout recorded history. However, traditionally, scientific knowledge and procedures have been kept apart from society by using complex and technical language or by withholding information or data. In recent years, there has been a significant trend toward open science, breaching the walls between science and society. This has improved science's accessibility, diversity, and inclusivity and has encouraged several cross-disciplinary scientific field collaborations. The objective of this trend is to positively influence society and involve people in the scientific process by incorporating new concepts such as “human practices," which consists of studying how science impacts society and how society impacts science (iGEM, 2003).

Current scientific research metrics and limitations

One of the main challenges of scientific research has always been the analysis of its impact and how to measure it. Historically, the metrics used for this aim have been merely focused on quantitative measurements based on data such as article citations or the number of publications. While these measurements might have been considered a good way to measure the outcome of research, they are not effective for measuring the social impact of this research and its outreach (Pourret et al., 2022). This emphasizes the need for new and better metrics able to measure not only the quantitative side of scientific research but also the quality of the different scientific activities that open science has normalized among the scientific community.

What is the Open Metric?

The Open Metric is a new measurement system that allows scientists to assess the effectiveness of their open science activities, such as journal articles, social media or news appearances, and outreach activities. It also allows independent researchers, especially those from developing countries, to identify the reproducibility of a specific research project and plan for future research. The factors included in this metric encompass the quantitative and traditionally measured values for evaluating scientific outcomes while incorporating social media interactions or online reader behavior altimetric to assess the quality of these outcomes.

How does the Open Metric work?

The Open Metric takes different input factors to compute a final value that effectively identifies the openness, engagement, and impact of any journal article, data repository, or scientific outreach activity.

The input factors and their different weights are the following:

• The credibility is measured using the citation index, which takes into account the number of times this specific content has been cited in scientific articles or scientific media. This value has a 50% of the weight, since it is highly effective to measure the impact on new scientific projects and outcomes. 
• The accessibility of the content considers if it has been published under a pay-per-view format or it is open access and free. This factor has a weight of 25%, which highlights the importance of the parameter when measuring the potentially social impact of scientific work, especially towards developing countries.
• The societal engagement factor, which is divided between social media interaction and online reader behavior, incorporates the social impact of the content. This factor has a weight of 25%, divided into 15% for social media interaction, considering both the post numbers and the reactions, and a 10% for online reader behavior (currently not yet included in the prototype as we could not find the right data), which is characterized by the time spent by readers on the content.

All these factors play a key role in the social impact and effectiveness of open science activities, and are therefore included in the Open Metric.

Looking to the future.

There are still a lot of things that could be improved in the Open Metric, such as:

For some of the factors, new approaches could be considered. 

• For the accessibility factor, it will be more holistic to include the proportion of referenced content that is open and free aside from only scoring between paid and free content. This will make it more comprehensive since it will allow independent researchers to check whether resources to reproduce the work are accessible. Since paywalls can potentially hinder reproducibility, especially by researchers from developing countries (Buehling et al., 2022; Fontúrbel & Vizentin-Bugoni, 2021), we would also want to create a feature where they can see whether the publication’s references are open or not. And if they are not open, they can click on the publication and see the corresponding author. This way they might be able to contact the corresponding author and ask for access to the publication. 
• Another thing to be considered is the inclusion of citations in patents for the credibility factor. This will show how a research work is solid enough to encourage technological transfer and scientifically support inventions.
• Lastly, for societal engagement, we were not able to find the right data for online reader behavior. Initially, we thought the most ideal input for this would be the time spent by readers on the content but we will need to further look into this.

From the individual Open Metric, which is given to each content type as described above, a normalized metric could be used for both scientific journals, conferences, and also for individual scientists or research groups. 

Overall, the Open Metric is the first step towards the measurement and improvement of the effectiveness and social impact of scientific research. 

Taking Inspiration from Space Agencies

While formulating our solution, we utilized the resources provided in the challenge, especially the resources of NASA and ESA on Open Science (ESA, n.d.; NASA, n.d.). We also learned a lot about the importance of open science and the different ways to implement it. In particular, Ramachandran et al. (2021) emphasized the importance of enabling collaborations. This inspired us to develop The Open Metric in a way that not only considered the openness of a work but also of the works before it. This measures not only the accessibility of a work but also, its reproducibility. This way, independent researchers interested in reproducing, or developing future research from a specific work can assess the accessibility of a research’s scientific background. This also inspired us to think about how we can help researchers, especially those from developing countries, access paid research (as explained in the future work). Lastly, the European Commission's (n.d.) resources introduced us to Altmetrics and Next-Generation Metrics, which are concepts we have not heard of before. This inspired us to include societal engagement to the Open Metric, which included social media interactions. This way, we we can measure how a specific research is 

Using Data 

To initially develop the input factors of the Open Metric, we utilized the following:

• For credibility and accessibility, we utilized OurResearch.org's OpenAlex (2022)
• For societal engagement, we utilized Altmetric's Free Tools (2015)

The Space Apps Hackathon 2022 has been a great experience that has allowed us to grow and learn about many different aspects of scientific research that normally go unnoticed or are ignored. One of these aspects is the measurement of the effectiveness or social impact we can generate as scientists. While we are usually focused on how our results might be used or misused, there is a big component to our work that is based on society’s feedback. Since measuring this impact is a complex problem, we decided to work on it from multiple perspectives, analyzing first what we and the community around us consider successful science is, and then what are the factors that we internally use to conclude this. This was a big challenge since most of these factors are entirely based on qualitative properties rather than quantitative ones, and they are therefore difficult to measure but also prone to subjectivity. To overcome this challenge, we did a lot of research on the current state-of-the-art altmetrics and computational capabilities but also tried to envision the future of qualitative analysis and how this could benefit the Open Metric system. 

Altmetric. (2015, July 9). Free tools. Altmetric. https://www.altmetric.com/products/free-tools/

Buehling, K., Geissler, M., & Strecker, D. (2022). Free access to scientific literature and its influence on the publishing activity in developing countries: The effect of Sci-Hub in the field of mathematics. Journal of the Association for Information Science and Technology, 73(9), 1336–1355. https://doi.org/10.1002/asi.24636

ESA. (n.d.). Open Science. Retrieved October 2, 2022, from https://www.esa.int/About_Us/Digital_Agenda/Open_Science

European Commission. (n.d.). MLE on Open Science—Altmetrics and Rewards | Research and Innovation. Retrieved October 2, 2022, from https://ec.europa.eu/research-and-innovation/en/statistics/policy-support-facility/mle-open-science-altmetrics-and-rewards

Fontúrbel, F. E., & Vizentin-Bugoni, J. (2021). A Paywall Coming Down, Another Being Erected: Open Access Article Processing Charges (APC) may Prevent Some Researchers from Publishing in Leading Journals. The Bulletin of the Ecological Society of America, 102(1), e01791. https://doi.org/10.1002/bes2.1791

iGEM. (2003). IGEM Human Practices. igem.org/Human_Practices

NASA. (n.d.). Open-Source Science Initiative | Science Mission Directorate. Retrieved October 2, 2022, from https://science.nasa.gov/open-science-overview

OurResearch.org. (2022, January 3). OpenAlex: The open catalog to the global research system. https://openalex.org/

Pourret, O., Irawan, D. E., Shaghaei, N., Rijsingen, E. M. van, & Besançon, L. (2022). Toward More Inclusive Metrics and Open Science to Measure Research Assessment in Earth and Natural Sciences. Frontiers in Research Metrics and Analytics, 7. https://www.frontiersin.org/articles/10.3389/frma.2022.850333

Ramachandran, R., Bugbee, K., & Murphy, K. (2021). From Open Data to Open Science. Earth and Space Science, 8(5), e2020EA001562. https://doi.org/10.1029/2020EA001562

#OpenScience #Accessibility #Inclusivity #SocialImpact #ResearchAssessment