Loominaries

https://loominaries.herokuapp.com/

https://youtu.be/qlPm13LLAHM

BACKGROUND

Open Science is defined as the collaborative culture that is permitted by technology that promotes sharing of ideas, data, and knowledge among scientific communities (Ramachandran et al., 2021). Open Science activities are measured on how they have an impact on society. But measuring it will encounter difficulties since "impact" is an impalpable thing, on a large body, which is society.

fig.1 : conceptual framework of Open Science

Fig.1 illustrates the conceptual framework of Open Science and it shows here the current metrics for measuring the effects on society of research/journals. Open peer review is the possible revisions of the research/journals by the traditional peer review process. Altmetrics, a portmanteau of "alternative metrics", is a traditional measure that determines its uses in scholarly and non-scholarly channels. Bibliometrics, statistical analysis of articles, books, and other publications. Semantometrics, which assesses the research/journal contribution. Lastly, Webometrics is measured by the ranking of the universities where it was published. Even though various metrics currently exist, it is lacking in evaluating their societal impacts. The researchers proposed new metrics to measure the effects on society by utilizing reproducibility, and how it will have an advantage over conventional metrics.

REPRODUCIBILITY INDEX FLOWCHART

Public Opinion is done by using sentiment analysis on various blogs and posts in social media platforms. Transparency and Openness Factor and Altmetrics use an API to get the corresponding scores. All of the collected scores from various sources and made sources such as Public Opinion are then merged to make the final score called "Reproducibility Metric" which is a weighted score using Public Opinion and Altmetrics as the weights while public opinion is the sole metric that is being weighted by these two factors since Transparency and the number of expert reviews(provided by Altmetrics) are necessary to evaluate the expected credibility of an article. Other open science activities such as software/tools primarily use public opinion in metrics since public opinion extremely matters when using tools and other metrics. Web scraping is primarily done on Twitter and Reddit using their respective APIs.

Getting Mentions on social media is done by scraping social media websites using their respective web scraping APIs, while also abiding to their terms of use. Article mentions are counted by searching for important keywords while also considering context through applying Sentiment Analysis in order to get the Positivity Score. Transparency and Openness Factor or TOP is a journal rating system that takes into account various criteria such as Data Citation, Data Transparency, Materials Transparency, etc. While Expert Reviews indicate the number of peer reviews that a said article had gone through and its respective assessment score. These two are used in calculating the Credibility Score. By analyzing and incorporating both the Positivity Score and Credibility Score, we can calculate the Reproducibility Index of an article which measures the ease and possibility of its methods and results being reproduced while making a significant societal impact.

VADER SENTIMENT ANALYSIS FLOWCHART

The specific Implementation of the project’s sentiment analysis is the VADER (Valence Aware Dictionary for Sentiment Reasoning) sentiment analysis or the Valence Aware Dictionary for Sentiment Reasoning. The main flow that the VADER sentiment analysis incorporates is assigning each word with a dictionary sentiment for example {‘like’:0.9, ‘hate’: -0.5} and adding all of these to a text corpus then applied to a sigmoid function. There are some added heuristics to make VADER sentiment analysis such as punctuation capitalization causing words in the sentence to have a higher value. Other contextual heuristics are the use of degree modifiers which make words it affects have an amplified sentiment value. All the sentiment values are added up and applied to a sigmoid function to get the final sentiment of the sentence. Sentiment analysis is further segregated to ones that are slightly subjective using the TextBlob library to detect.

The exact method used to retrieve social media mentions on Twitter and Reddit is as follows. Initializing the respective APIs for Twitter and Reddit by providing the proper auth tokens. The respective Digital Object Identifier (DOI) or Keywords (Software/Tools) towards the search query of the provided API. All the collected data with cleaning of blank strings are then fed in the VADER sentiment analysis and a percentage score of the positive sentiment is generated for the text data.

REPRODUCIBILITY INDEX PROCEDURE

Sigmoid function, a common machine learning function that is used to naturally clamp values between 0 and 1. Used in this context to clamp the values of positive description sentiment and negative description sentiment between 0 and 1.

Positive Description Sentiment and Negative Description Sentiment is just a weighted summation of words evaluated to be either positive or negative multiplied by the respective word modifiers for each positive or negative word. These values are naturally restrained to 0 and 1 by the sigmoid function. After the calculation these values are used to compute the Total Positive Sentiment.

Total Positive Sentiment which details the ratio between the total amount of mentions and the number of positive mentions. This is one of the important derived values from the calculations of sentiments since it tells an estimate on the public opinion of the study used in the search query. Furthermore this value is weighted against the credibility score to normalize it against values that would skew it away from potential attackers that would automate social media tweets to game for a higher reproducibility index. Total Positive Sentiment tells the whole picture on the views of users of social media of the research in question since it uses the VADER sentiment analysis that is further fine tuned to provide the best values.

Credibility Score is the metric that allows users and data analysts to gauge the amount of transparency and peer reviews an article has. Credibility score weighs the value of the Transparency and Openness Score and the number of expert reviews (clamped to the max value of transparency and openness score) to provide the credibility score. By Combining Transparency and Openness and the number of expert reviews and clamping it to 0 and 1 it captures credibility since with high transparency and high number of expert reviews an article can be viewed as a credible one. Credibility Score is multiplied by Total Positive Sentiment to gain the final value sought out by this project, the Reproducibility Index.

Reproducibility Index is the final value computed to gauge the reproducibility of a study/article. The component of Credibility Score which is defined by Transparency and Number of Expert Reviews will make Reproducibility scale with these values and the more credible a study is the more reproducible it might be since studies that can’t be reproduced are frowned upon in the scientific community. The amount of data available also is a part of the credibility score with less data available, the credibility score is low. Less data availability means a low likelihood of the study ever being reproduced. Finally Positive Sentiments are used to gauge the current level of impact the said study has in society since it shows it has already made impressions throughout social media platforms. 

USE CASE SCENARIOS

Interpreting the Reproducibility Index

A High value in the reproducibility index in an academic paper or an academic result indicates that the research in question can be reproduced efficiently, and the results largely impact society. A High value reproducibility index is in between the range of 75% and 100% are usually highly prestige articles and articles that had a high impact on their field in question, this category belongs to articles like Deep Residual Learning for Image Recognition which has made a large contribution towards the field of Machine Learning and Computer Science and has made the field significantly progress. Articles between 50% and 75% have two types, first with high credibility score and fair positivity score while the other has fair credibility score and high positivity score. Next, articles with scores between 25 and 50% is what readers should be wary of. Articles within this scope also has two types, the first has fair credibility scoreand low positivity score and the second has low credibility score and fair positivity score. Most of the articles within this range might be of the latter type, having high remarks in social media however after thorough peer reviewing was found to be lacking in transparency or results. The last category of articles is those with reproducibility less than 25% which indicates low credibility scoreand low positivity score. Articles within this category struggle to garner positive remarks on social media and are rated poorly by expert peer reviews.

*CSA - Article under Canadian Space Agency

Article 1 (CSA): Fine guidance sensor/near-infrared imager and slitless spectrograph on James Webb Space Telescope: pupil alignment methodology and metrology

Positivity Score: 100

Credibility Score: 92.33

Reproducibility Index: 92.33

HIGH REPRODUCIBILITY

This indicates that the above article can be very easily reproduced. Its methods and results are transparent and accessible which garners positive reviews from both social media and expert peer reviews. Therefore, article 1 from the Canadian Space Agency is very likely to have a significant societal impact.

Article 2 (CSA): Optimal feedback linearization control of brushless motors

Positivity Score: 80

Credibility Score: 90.66

Reproducibility Index: 72.53

SATISFACTORY REPRODUCIBILITY

This indicates that the above article can likely be reproduced, however either its methods and results or social media and peer reviews are only satisfactory. The above mentioned article can likely have a societal impact.

Article 3 (Not CSA): A Causal Framework for Cross-Cultural Generalizability

Positivity Score: 93.33

Credibility Score: 27.77

Reproducibility Index: 25.92

FAIR REPRODUCIBILITY

This indicates that the above article can somehow be reproduced, although readers should be wary. Although the article has a very high positivity score, it has a very poor credibility score. This indicates that while the article had positive reviews by social media users, its methods and results were rated poorly by expert peer reviewers.

Article 4 (Not CSA): Interpersonal and intrapersonal approach and avoidance motives after social rejection

Positivity Score: 50

Credibility Score: 34.44

Reproducibility Index: 17.22

LOW REPRODUCIBILITY

The above article shows a low positivity score and low credibility score, which indicates that it can less likely be reproduced as it shows very low ratings and negligible interaction in social media, cannot be integrated among other studies and activities, and therefore even more so less likely to have a societal impact.

Example of the Application Interface

The application interface of the project provides a simple interface towards the calculation of the reproducibility index and other metrics such as credibility score and total positive sentiment shall also be displayed to provide further useful information to the user. Proper arguments such as the use case (results for research articles) and the proper field must be used to have the best results when using the application. Digital Object Identifier (DOI) must be inputted for research articles to get the accurate scores. The ArXiv checkbox must be selected if the article will use an ArXiv id instead of a DOI.

After all necessary info are inputted in the form and the metric button has been sent. A web page is shown where all of the metrics is shown with accompanying graphs for users to further visualize data. Other information such as authors and a link to actual article(hyperlink in the title) is also present in the resulting web page.

Coding Languages

• Python

Tools/Software:

• Vader Sentiment Analysis
• Text Blob
• requests
• Dash
• Dash-Bootstrap
• pandas
• Tweepy
• praw
•  bs4
• Selenium
• Google Collab
• Chocolatey

EXTRACTING SPACE AGENCY ARTICLES

The API data available in the resources tab of the Measuring Open Science challenges were used to assess the effectiveness of the newly formed metric. The used API data is accessible in Canadian Space Agency, with an URL of https://donnees-data.asc-csa.gc.ca/dataset/dd18f931-a23e-4126-874f-68e1bc73851b.

The data that will be utilized for data analysis; Article title (enclosed by yellow box), abstract (red box) and DOI (orange box)

The database will be web-scrape and the code screens and only acquires an abstract, article title, and DOI of the article. The acquired data will be subjected to sentiment analysis to determine the context of the article. The results will be used to determine if the article/journal is reproducible thus proving its societal impact. The APIs – especially the data from the Canadian Space Agency, which has a database of research/articles that has an abstract that was proven useful for evaluating the advantage of the proposed metric.

           The team has fun doing challenges and surpassing them together. Even though we have different fields of expertise, we have become coherent in resolving problems. Everyone on the team is transparent in giving constructive criticism and able to comply to give the best result that they can give. There is no “carry” on our team, everyone carries each other and grows progressively.

The brother of one of the developers saw an interesting topic on social media, and the best way to learn more about the topic in terms of practicality is to turn to the search engine. However, he stumbled upon suspicious research while browsing the internet. Not knowing about the topic, he was fooled by the unreliable and irrelevant research. He believed it for days until he opened up the said topic to his brother - who is one of the developers, who has more knowledge about the topic. Fortunately, he was corrected. If there is no one to correct someone, they will retain the wrong information without knowing that they are fooled and they can pass on the wrong information to others. The team was formed to mitigate this situation and screen unnecessary and unreliable research/journal topics. We have used the Waterfall methodology – Waterfall model to approach the problem. A sequential development process to resolve the existing phase before transitioning to the next phase. If we have encountered difficulties resolving the problems, we further break down the phase to troubleshoot it in its simplest form and review the basics of our knowledge. And if the problem persists, we go back to the previous phase to detect the abnormality.

 We would like to give our thanks to various agencies who have given us access to valuable resources and data, to the staff and coordinators behind this event that allowed us to showcase our skills in engineering, problem-solving and critical thinking, to mentors and organizers who correct our mistakes and guided us on the right path, and lastly, we give the utmost thanks and appreciation the help from family and friends who have given us motivation and dedication after many sleepless nights of brainstorming and coding. Thank you very much. Salamat!

Altmetric. (2015). Free tools. Altmetric. https://www.altmetric.com/products/free-tools/

Asher, J. (2022, June 20). Dash & Bootstrap — Bring Your Projects To Life In A Beautiful Way. Analytics Vidhya. https://medium.com/analytics-vidhya/dash-bootstrap-bring-your-projects-to-life-in-a-beautiful-way-6b1c3bd7cfcf

Blumenfeld, J. (2021). Data Chat: Kaylin Bugbee. Earthdata. https://www.earthdata.nasa.gov/learn/data- chats/data-chat-kaylin-bugbee

Brazilian Space Agency. (2022). IMATEC. IMATEC Lite. https://imatec.aeb.gov.br/#/home

Calderon, P. (2018). VADER Sentiment Analysis Explained. Medium. https://medium.com/@piocalderon/vader-sentiment-analysis-explained-f1c4f9101cd9#:~:text=VADER%20(Valence%20Aware%20Dictionary%20for

Canadian Space Agency. (2019). Report on Funded Open Access Scientific Publications - CKAN. Government of Canada. https://donnees-data.asc-csa.gc.ca/dataset/ca983b71-df22-431a-9d5f-94e77b835dd5

Canadian Space Agency. (2021a). 2021-2024 Open Science Action Plan. Government of Canada. https://www.asc-csa.gc.ca/eng/publications/open-science-action-plan-2021-2024.asp

Canadian Space Agency. (2021b). Open data and information. Government of Canada. https://www.asc-csa.gc.ca/eng/open-data/

Canadian Space Agency. (2022a). Canadian Space Agency (CSA) peer-reviewed publications list - CKAN. Government of Canada. https://donnees-data.asc-csa.gc.ca/dataset/dd18f931-a23e-4126-874f-68e1bc73851b

Canadian Space Agency. (2022b). Data on the Number of Scientific Publications and Presentations - CKAN. Government of Canada. https://donnees-data.asc-csa.gc.ca/dataset/35103995-4m2o-7a0i-964fusc106cj97

Castillo, D. (n.d.). Develop Data Visualization Interfaces in Python With Dash – Real Python. Realpython.com. Retrieved October 2, 2022, from https://realpython.com/python-dash/#:~:text=Dash%20is%20an%20open%20source

Cinergix Pty Ltd. (2019). Diagram Maker | Online Diagram Software | Creately. Creately.com. https://creately.com/

Earth Science Data Systems. (2022a). Earthdata. Earthdata. https://www.earthdata.nasa.gov/

Earth Science Data Systems. (2022b). Earthdata. Earthdata. https://www.earthdata.nasa.gov/eosdis/daacs

Euro Data Cube. (2022). Earth Observing Dashboard. Earth Observing Dashboard. https://eodashboard.org/

European Commission. (2022a). MLE on Open Science - Altmetrics and Rewards | Research and Innovation. European Commission. https://ec.europa.eu/research-and-innovation/en/statistics/policy-support-facility/mle-open-science-altmetrics-and-rewards

European Commission. (2022b). Next-generation metrics: Responsible metrics and evaluation for open science. In European Commission. https://scholarlypublications.universiteitleiden.nl/access/item%3A2945683/view

European Space Agency. (2022). Open Science. European Space Agency. https://www.esa.int/About_Us/Digital_Agenda/Open_Science

Google LLC. (2019). Google Colaboratory. Google.com. https://colab.research.google.com/

Muthukadan, B. (2018). Selenium with Python — Selenium Python Bindings 2 documentation. Readthedocs.io. https://selenium-python.readthedocs.io/

Mexican Space Agency. (n.d.). Datos Abiertos. Gobierno de México. Retrieved October 2, 2022, from https://datos.gob.mx/busca/dataset?theme=Geoespacial

National Aeronautics and Space Administration. (2022a). Open-Source Science Initiative | Science Mission Directorate. Science.nasa.gov. https://science.nasa.gov/open-science-overview

National Aeronautics and Space Administration. (2022b). Transform to Open Science (TOPS) | Science Mission Directorate. Science.nasa.gov. https://science.nasa.gov/open-science/transform-to-open-science

PyPI. (2022). beautifulsoup4. PyPI. https://pypi.org/project/beautifulsoup4/

Python Software Foundation. (2019). Welcome to Python.org. Python.org; Python.org. https://www.python.org

Python Software Foundation. (2022). Python 3.11.0-rc2. Chocolatey Software. https://community.chocolatey.org/packages/python

Ramachandran, R., Bugbee, K., & Murphy, K. (2021). From Open Data to Open Science. Earth and Space Science, 8(5). https://doi.org/10.1029/2020ea001562

TextBlob. (2018). TextBlob: Simplified Text Processing — TextBlob 0.15.2 documentation. Readthedocs.io. https://textblob.readthedocs.io/en/dev/

TOP Factor. (n.d.). Www.topfactor.org. Retrieved October 2, 2022, from https://www.topfactor.org/

Treasury Board of Canada. (n.d.). Open Government Portal. Government of Canada. Retrieved October 1, 2022, from https://search.open.canada.ca/grants/?sort=agreement_start_date+desc&page=1&search_text=&gc-search-orgs=Canadian+Space+Agency&owner_org=csa-asc

#OpenScience #MeasuringOpenScience #software #code #TOP #API