Writing Custom Sink Connectors for Pulsar I/O
A presentation at Pulsar Summit Europe 2021 by Ricardo Ferreira
Writing Custom Sink Connectors for Pulsar I/O Ricardo Ferreira Principal Developer Advocate, Elastic Pulsar Virtual Summit Europe 2021
Who am I? • Principal Developer 🥑 at Elastic • Community, Developer Relations • Before Elastic: Confluent, Oracle, Red Hat • Observability, Streaming Systems, Databases, 🧑💻 Java and Golang • Pulsar Virtual Summit Europe 2021 @riferrei https://riferrei.com
If you Need a Pulsar I/O Crash Course Pulsar Virtual Summit Europe 2021
Getting Started: https://github.com/riferrei/pulsario-custom-sink-connector Pulsar Virtual Summit Europe 2021
Slides and Code from this Presentation Pulsar Virtual Summit Europe 2021
Thank You Pulsar Virtual Summit Europe 2021
You currently have massive amounts of data sitting on Apache Pulsar and likely — even more data coming in every second. But part of your project requires this data to be sent to some system to explore the dataset further. You look into the Pulsar website for built-in connectors to that particular system and realize that there are none. What a bummer.
But you don’t have to give up on Pulsar because of this. You can write custom sink connectors for Pulsar I/O that work just like the built-in ones, and all you will need is a bit of Java development experience and creativity. The rest you can leave to this talk. This talk will explain how to start developing your custom connector, show how the code looks like, demonstrate how to deploy, and discuss some of the design decisions that your custom connector may need to address.
Video
Code
The following code examples from the presentation can be tried out live.
Buzz and feedback
Here’s what was said about this presentation on social media.
