Kafka/KSQL Streams Lost When Producing With Golang

Written by Archyblog - - Aggregated on Thursday March 14, 2019

Odd one this, and one that took me a little while to debug. I recently set up a Confluent/Kafka data pipeline with transformations being handled by KSQL and data being produced by an application written in Go. As part of the test process I persisted data using a MongoDB Sink connector.

The command line producers had no problems and producing a large file would persist the expected data to MongoDB.

However, I ran into issues when producing from Golang, I would notice that somewhere between 7% and 12% of the messages were being persisted to MongoDB, the others were lost somewhere in the processing.

« Take care of non-technical skills - Stefan Koopmanschap

Symfony blog - New in Symfony 4.3: Routing improvements »