Jul 25, 2021

Cleaning messy sensor data in Kafka with ksqlDB

Cleaning messy sensor data in Kafka with ksqlDB Collecting streaming data is easy — but understanding it is much harder! With months of 🐱 weight data captured I discovered sensor data can be very messy⚡. Let me share some of the real world data problems I encountered — and how I solved my stream processing & cat dining challenges with ksqlDB Snowy the cat — an expert in streaming data

Jul 10, 2021

Snowy’s eating — tweeting my cats weight & dining habits with a Raspberry Pi

Snowy’s eating — tweeting my cats weight & dining habits with a Raspberry Pi Our cat Snowy 😸 has been enjoying her meals over winter. I wanted to start collecting data on her eating habits, and analyse her weight over time. This data is collected with food and cat weight measurements; alongside phots taken by two cameras. Data is collected and images processed locally using a Raspberry Pi. At the risk of body-shaming our cat, updates are posted regularly to Twitter #snowydata.

Feb 18, 2021

Oracle to Kafka — Playing with Confluent’s new Oracle CDC Source Connector in Docker

Oracle to Kafka — Playing with Confluent’s new Oracle CDC Source Connector in Docker It’s really exciting to have a new option for streaming Oracle data into Kafka. With Confluent releasing their “Oracle CDC Source Premium Connector” there’s a new way to capture data that has been added to, updated, or deleted from Oracle RDBMS with Change Data Capture (CDC). Think of this as a low touch way to stream both the Oracle data and schema changes into Kafka and Schema Registry.

Feb 4, 2021

Cat tracking with Amazon Web Services— AWS IoT, DynamoDB and SageMaker

Cat tracking with Amazon Web Services— AWS IoT, DynamoDB and SageMaker If you read 1 Cat, 3 Clouds — Where is Snowy sleeping (Part 1) you’ll know I’m trying to build a real-time location tracker for my cat. Following on from that initial blog (which described the hardware and networking), this “Part 2” article describes data processing with Amazon Web Services. I’ll be using native AWS services for data ingest, data processing and mapping visualisation to build a low cost pet tracking device to show Snowy’s location in real-time.

Jan 10, 2021

1 Cat, 3 Clouds — Part 1 - Where is Snowy sleeping?

1 Cat, 3 Clouds — Where is Snowy sleeping? Our cat Snowy is an outdoor cat — but we are beginning to wonder where she travels once she’s left the house. Let me share how I built a low cost pet tracking device to show Snowy’s location in real-time. Plus, what’s it like to build the same IoT data project in AWS, Azure & Google Cloud in 2021? This is article 1 in a 4 part series.

Sep 21, 2020

Pi Peripheral - Controlling computer actions with a $10 Raspberry Pi Zero

Pi Peripheral Landing gear down and strobe lights active. How to add physical buttons, switches and dials to control computer actions with a $10 Raspberry Pi Zero. Why would you want to do this? Well you might want to add some additional buttons or switches to your favorite game. Or maybe you can streamline your video production but switching cameras with a foot pedal. For me, I wanted to add tactile controls to Microsoft Flight Simulator 2020.

Jun 9, 2020

Home office ergonomics — with TensorFlow

Home office ergonomics — with TensorFlow Monitor good posture — with machine learning. Ensure proper office ergonomics with TensorFlow, a webcam and OpenCV Python real-time computer vision libraries. Code available at https://github.com/saubury/posture-watch Real-time posture monitoring — as demonstrated with a teddy bear If you sit behind a desk for hours at a time, you’re possibly going to slump. This may result in neck and back pain. This can be avoided by teaching a machine to recognise “good” posture from “bad” posture.

May 18, 2020

🍹 Cocktail Pi

🍹 Cocktail Pi Cocktails based on your mood created by a Raspberry Pi bartender. Drinks are selected based on your emotion and multilingual voice prompts let you know when your drink is available. How to build a fully automated home build cocktail maker. Code available at **https://github.com/saubury/cocktail-pi**🍸 Pumps I used 4 peristaltic pumps to provide a “food safe” way to pump the liquids from the drink bottles. These pumps provide a steady rate of flow of liquids when powered.

May 2, 2020

Kafka with AVRO vs., Kafka with Protobuf vs., Kafka with JSON Schema

Kafka with AVRO vs., Kafka with Protobuf vs., Kafka with JSON Schema Kafka serialisation schemes — playing with AVRO, Protobuf, JSON Schema in Confluent Streaming Platform. The code for these examples available at https://github.com/saubury/kafka-serialization Apache Avro was has been the default Kafka serialisation mechanism for a long time. Confluent just updated their Kafka streaming platform with additional support for serialising data with Protocol buffers (or protobuf) and JSON Schema serialisation.

Feb 23, 2020

Kafka stream processing: sorting socks with ksqlDB - Part 2

Kafka stream processing: sorting socks with ksqlDB Sorting socks with a streaming solution? Pair socks with ksqlDB, Kafka and Kafka Connect. Socks — sorted! ksqlDB is a pretty cool event streaming platform. A few commands allow you to build a detailed real-time stream processing application. Capture, transform and perform continuous transformations on Kafka with a simplified SQL dialect. In addition, ksqlDB allows you to capture events from an external system using Kafka connect.