High-Level Project Summary
We have develop a web page where you can make searches through a chatbot implemented with Watson Assistant and Watson Discovery services. The chatbot will be the manager of the search that a researcher may want, and Discovery will implement searches through the NTRS (NASA Technical Reports Server) documents. Watson Discovery is a useful tool which can help researchers to create cognitive applications that can extract great value from a huge amount of structured and unstructured documents. Our project is already useful but it is also scalable since we can continue training the Watson AI to obtain more appropriate responses, with the appropriate vocabulary and teaching new scientific concepts.
Link to Final Project
Link to Project "Demo"
Detailed Project Description
Our solution is fully developed in the IBM Cloud because we are using Code Engine to deploy our web page, Watson Assistant to manage the search and Watson Discovery to do the search through documents. Watson Discovery functionally is capable of tracking, converting, enriching and normalize data to bring better answers. It helps decrease search times and improve researchers productivity. We have uploaded 38 local documents about the subject category "Lunar and Planetary exploration" from the NASA Technical Reports Server (NTRS) to Watson Discovery, they can be from different data sources such as Salesforce, Box, IBM Cloud Object Storage, Sharepoint or a web crawl. The service ingest the documents with Natural Language Processing (NLP) that allows to convert and enrich through metadata to make more navegable and easy to explore. Then the NLP are normalized and cleaned through an automatic process to improve the data quality. This will allow the service to understand more quickly and give better answers.
We have trained our AI with the Smart Document Understanding (SDU) tool that allows us to classify in different fields how the documents ingested are organized, thus we can train convertible models.
Space Agency Data
We have used the NASA Technical Reports Server (NTRS) to download the Subject Category "Lunar and planetary exploration" documents and then upload them to the Watson Discovery instance. We have analyzed them to make an optimal classification to train our service.
Hackathon Journey
For us, it was a very good experience, with many challenges involved, which were overcome through teamwork. We believe that it is very important to maintain all the years of research, and the fact of finding a way to facilitate the search in them was what led us to choose this challenge.
To solve the problems, we divided the team into 2, one half tried to solve the problem and the other half concentrated on advancing with other aspects of the solution.
References
Code repository, here you can find the code of our solution, as well as, the diagram of the solution and the next steps we would like to take with this project.
Search results with the filter used at the moment of downloading the documents for training our Watson Discovery instance.