Kafka Summit 2019 San Francisco: a review 2 day summit review : welcome to the world of streaming data! Kafka Summit San Francisco (October 2019) was a two day conference aimed at data architects, engineers and developers who are keen to learn about streaming data and appreciate the real-world use cases of others. The event is organised by Confluent, the commercial company founded by the original creators of Apache Kafka.

Continue Reading

Confluent Kafka Kubernetes Operator — Setup and Playing! Confluent released their Kafka Kubernetes Operator this week (as part of their CP 5.3 release). The TL;DR summary is this gives you a straight forward way to deploy your own clustered & secured Apache Kafka on Kubernetes (ZooKeeper, Apache Kafka) along with the cool bits (Kafka Connect, KSQL, Schema Registry, REST Proxy, Control Center). This is pretty cool as it opens the way for Kafka deployments to on premises (eg.

Continue Reading

Did I beat Ben? Race tracking with Kafka KSQL, MQTT & Kibana Live race tracking with MQTT and Kafka streaming. Race statistics using KSQL and visualisations with Kibana. Can I run a half marathon faster than Ben? Kibana dashboard — showing live race analytics Design Overview Runners are tracked with GPS based apps installed on their phones. Location updates are sent (via MQTT) to a central server. Kafka acts as a streaming platform with KSQL performing the live race computations (speed, rank, distance to finish line).

Continue Reading

Machine learning & Kafka KSQL stream processing — bug me when I’ve left the heater on Continuous household power monitoring with a trained model to alert on unexpected power usage for time of day. Let my Google Home and Phone notify me when I’ve used more electricity than expected. What on earth is this? Household power consumption fluctuates throughout the day. However, the *pattern *of electricity use follows a typical curve and becomes predictable after sufficient observations.

Continue Reading

Home Power Monitoring using a Raspberry Pi. Home electricity usage monitoring using a Raspberry Pi. Plus some shaky maths to find out when we left the heater on. As we approached the colder months I wanted a way of monitoring our household power consumption. Our electric heaters are fairly ancient, and I had no accurate way of monitoring their cost (either financially or environmentally). I want to know this number!

Continue Reading

Using KSQL, Apache Kafka, a Raspberry Pi and a software defined radio to find the plane that wakes Snowy the cat Using open-source streaming solutions to map the correlation between aircraft traffic and cat behaviour Aircraft determine their position using GPS; and periodically transmit that position along with identity string, altitude, speed etc as ADS-B signals. Cats behave erratically, but generally display their displeasure by jumping on your face. The post describes how we can use open source streaming solutions (Apache Kafka), KSQL (streaming SQL engine) and a Raspberry Pi to process aircraft movements in real-time to determine which plane is upsetting my cat.

Continue Reading