Aashish Sood

Analysis and Visualization of UFO sighting dataset

Why the study?

To analyze -

• Unintended consequences of merging multiple datasets
• Impact of joined datasets on improving value and veracity of data

What I did?

To analyze -

• Developed python scripts to scrape websites and fetch data fron REST API's to capture four datasets :-
• - Airports Dataset ( http://ourairports.com/data/airports.csv)
• - Meteorite Landings Dataset: Mime Type – Application/JSON
    Source - https://data.nasa.gov/resource/y77d-th95.json?$limit=50000
• - Census Data: MIME Type – text/CSV
     Source - https://github.com/grammakov/USA-cities-and-states?files=1
• - Sci-Fi Movies: Mime Type – text/HTML
    Source - https://en.wikipedia.org/wiki/Lists_of_science_fiction_films
• Developed python scripts to capture geolocation of nearest airport to the sightings using geocode APIs. Used Memcached to limit the number of queries fired to geocode API.
• Used Jaccard Similarity and Cosine Similarity to identify hidden clusters in the joined dataset.

What I found?

• 90% of the UFO sightings occurred where the nearest airport was less than 25 miles away.
• Most sightings happened in rural areas.
• Highly sighted UFO locations are sparsely populated.

Technologies Used:

• Python
• Apache Tika
• JAVA 8
• D3.js
• K-Means Clustering

---