Consuming NEXRAD (Doppler Radar) using Containers on Amazon Elastic Beanstalk
One of the growing number of public data sets available on Amazon Web Services is the Next Generation Weather Radar (NEXRAD). This is data collected from a network of 160 high-resolution Doppler radar sites that detect precipitation and atmospheric movement at approximately 5 minute intervals from each radar site. NEXRAD enables severe storm prediction and is used by researchers and commercial enterprises to study and address the impact of weather across multiple sectors. The real-time feed and full historical archive of original resolution (Level II) NEXRAD data, from June 1991 to present, is now freely available on Amazon Simple Storage Service (S3) for anyone to use. This is the first time the full NEXRAD Level II archive has been accessible to the public via HTTP in bulk.
Using NEXRAD as an example, I will demonstrate a simple architecture that starts with a subscription to the NEXRAD SNS topic. You can subscribe to the SNS topic using
Amazon Simple Queue Service (SQS) or Lambda. I will demonstrate the simple steps required to subscribe using SQS and how to consume the subsequent message queue using code packaged as Docker containers running on Amazon Elastic Beanstalk (in worker mode). This is a pattern that can be applied to many kinds of big data to support both batch and real-time processing workflows. For more info on this data see: https://aws.amazon.com/noaa-big-data/nexrad/