Everything's WINDIER in TEXAS

Mainly because I wanted a project to learn some data streaming technologies, and dive into Scala programming.

But also... the Texas electricity market (ERCOT - Electricity Reliability Council of Texas) produces more wind-generated power than any other state. This massive wind capacity, combined with the effects of the Federal Production Tax Credit, make for an interesting (and often dramatic) dynamic between realtime wholesale power prices and wind resources. On several particularly windy occasions in the past, localized power prices have actually gone negative. This visualization is meant to monitor the wind - power price relationship, spatially and in realtime.

• Python scripts pull weather data from here and pricing data from here, and push the data to a local single-node two-topic Kafka broker.
• A Scala app uses Spark Streaming to connect to the Kafka broker, and cleans/filters/processes all data according to topic
• Scala app publishes the DStream to a realtime messaging cloud service, PubNub, on a singular channel
• A simple frontend client subscribes to the channel and receives all messages in realtime;

• Scala 2.11
• sbt
• Spark 2.x
• pip install kafka-python
• pip install satori-rtm-sdk

• If you don't want to run it locally, but want to see the visualization, contact me to "turn it on".

• Otherwise, clone this repo.

• (tab #1) - start a local single-node zookeeper instance

  kafka_2.11-0.11.0.0/bin/zookeeper-server-start.sh kafka_2.11-0.11.0.0/config/zookeeper.properties

• (tab #2) - create the kafka topics on the broker

  kafka_2.11-0.11.0.0/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic price

  kafka_2.11-0.11.0.0/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic weather

• (still tab #2) - start the single-node kafka broker

  kafka_2.11-0.11.0.0/bin/kafka-server-start.sh kafka_2.11-0.11.0.0/config/server.properties

• (tab #3) - publish weather data to kafka

  python data_generators/satori_to_kafka.py

• (tab #4) - publish price data to kafka

  python data_generators/price_to_kafka.py

• (tab #5) - compile and run

  sbt compile

  sbt run

• navigate to https://ian-wright.github.io/everythings-windier-in-texas/ and watch some data pour in!