High-Level Project Summary
Website: http://www.sunwatcher.earthWe processed datas from WIND and DSCOVER satelites, and compared them. We ran DTW but only on very small windows (few hours) . We couldn't go as far as developing an AI, but had great plans to do so :)We developped a website which displays our results, as well as other infos on ou project !
Link to Final Project
Link to Project "Demo"
Detailed Project Description
This project has two main parts:
-Front end, where our two motivated engineers learned the django framework in two days, and managed to create a web page that displays plots, available for the default date only :). You will also find other infos about the project on this webpage !
-Data processing, where we retrieved data from WING and DSCOVER satelites, cleaned it and plotted it.
we use linear interpolation to size our data samples to the same frequency, for further analysis.
We then tried to perform DTW on our data to show correspondances between the two satelites data but couldn't run it efficiently(10min/day of data)...
In the end we haven't been able to create the dataset. If we had the idea was to develop the following:
- use the pytorch lightning framework to train several networks on the dataset (inputs from the sensor of DSCOVR and targets from WIND) using the Gaussian Negative Log Likelihood to account for the sensor noise. The idea is to predict a distribution for the resulting value in order to have an uncertainty estimate
- Measure the quality of the results using a classic loss such as the RMSE (to be minimized) or the L1 Loss. Measure the quality of the uncertainty estimation using calibration plots (actual quantiles vs. expected quantiles)
- If necessary, use deep ensembles (Lakshminarayan et al. 2017) to improve uncertainty estimates. It will reduce bias and provide better estimates: the networks will agree on points with a lot of data evidence, but will differ on points where doubt is possible, and therefore improve uncertainty estimates
Space Agency Data
DSCOVR Mission's Magnetic Field Data Sets
BGSE datas (magnetic field)
Wind Mission's Magnetic Field Data Sets
B1GSE datas (magnetic field)
Wind Mission's ion Parameters
Proton_Np_nonlin (Proton number density Np (n/cc))
Proton_W_nonlin (Scalar [isotropic] proton thermal speed)
Proton_VX_nonlin (Proton velocity component Vx (GSE, km/s))
Proton_VY_nonlin (Proton velocity component Vx (GSE, km/s))
Proton_VZ_nonlin (Proton velocity component Vx (GSE, km/s))
Hackathon Journey
We've been inspired by the technical challenge !
We were 5 people working on this project, and we all had assigned tasks. We all learned new skills (from front end development, to data analysis and cleaning)
References
DSCOVR Mission's Magnetic Field Data Sets
Wind Mission's Magnetic Field Data Sets
Wind Mission's ion Parameters
Spaceapp challenge logo
djangoproject.com
godaddy.com (domain)
heroku.com (hosting website)
Lakshminarayan 2017 (for AI) (https://arxiv.org/abs/1612.01474)
python libs:
pytorch
torch
torchmetrics
pathlib
typing
argparse
wget
tqdm
pickle
cdflib
numpy
os
xarray
scipy
matplotlib
Tags
#DataEngineering #Discover #SolarFlare