Is Using KoP (Kafka-On-Pulsar) a Good Idea?

A presentation at Pulsar Summit San Francisco 2022 in August 2022 in San Francisco, CA, USA by Ricardo Ferreira

Pulsar Summit San Francisco Hotel Nikko August 18 2022 Ricardo Ferreira Senior Developer Advocate • AWS Ecosystem Is Using KoP (KafkaOn-Pulsar) a Good Idea?

Ricardo Ferreira Senior Developer Advocate Amazon Web Services Ricardo is Senior Developer Advocate at AWS, working in the DevRel team for North America. With more than 20 years of experience, he may have learned a thing or two about distributed systems, messaging, fast data analytics, databases, and observability. Before AWS, he worked for software vendors like Elastic, Confluent, and Oracle.

How to Lose a Guy in 10 Days ● Kate Hudson ● Matthew McConaughey

How to Lose a Guy KoP in 10 Days ● KoP (Kafka-On-Pulsar) ● Ricardo Ferreira

Code for this presentation: https://github.com/riferrei/is-using-kop-a-good-idea

Ricardo Ferreira
@riferrei

1 / 7

Building microservices around Apache Kafka is your job, and life is great. One day, you hear community members talking about some neat Apache Pulsar features, and get you intrigued. I mean, we all love Kafka, but you can’t avoid wondering if migrate one of your projects to Pulsar is a good idea. Then it happens. You find Pulsar supports Kafka clients natively via a protocol handler called KoP: Kafka-On-Pulsar.

This gets you pumped. Is that it? Can I go ahead and simply point my microservices to Pulsar and be a hero with this migration? But you must be responsible; and history says you shouldn’t believe migrations like this are refactoring free. Reason you may get interested in this session.

We will revisit the architecture behind protocol handlers to understand what it means having one enabled on Pulsar. Plus, we will discuss the internals of KoP. Finally, we will use a show-and-tell approach to detail the effort to migrate a microservice written for Kafka to Pulsar, and whether the code need to change for this.

Video

Code

The following code examples from the presentation can be tried out live.

https://github.com/riferrei/is-using-kop-a-good-idea

Is Using KoP (Kafka-On-Pulsar) a Good Idea?

Buzz and feedback

Here’s what was said about this presentation on social media.

Already LOVING @riferrei ‘s talk. Major props for the combo of being hilarious right off the bat, for caveats on gender inclusivity, & for a great demo 👏🏻👏🏻👏🏻 #PulsarSummitSF pic.twitter.com/H7M6w1BMot
— CAITOs (@caito_200_OK) August 18, 2022
Ok this was a super cool “everything is talking to each other” moment. very mic-drop demo ending 🎤👏🏻@riferrei ‘s talk on @apachekafka on @apache_pulsar at #PulsarSummitSF pic.twitter.com/tRlUeddxT5
— CAITOs (@caito_200_OK) August 18, 2022
What a helpful session from @awscloud’s @riferrei! In his demos, he tests #Kafka on #Pulsar (KoP) in three scenarios: #Microservices built for Kafka, CDC using @debezium for @MySQL, and stream processing using @ksqlDB. #PulsarSummitSF pic.twitter.com/FGWQR2KvzY
— Pulsar Summit (@PulsarSummit) August 18, 2022
the downside with sitting at the back after presenting is you don’t get as good of shots of the awesome talk that comes after yours.

So, this is your PSA to watch @riferrei’s talk when #PulsarSummitSF publishes it. Super engaging & helpful demo on @apachekafka & @apache_pulsar pic.twitter.com/BvGf8qmbYE
— CAITOs (@caito_200_OK) August 18, 2022
@riferrei is crushing it with #KoP #ApachePulsar with an all-demo talk. This is some seriously powerful examples of KSQLDB interacting with a Pulsar cluster. https://t.co/Cpb5B8mqMQ #PulsarSummitSF
— 🇺🇦 Tim Spann (@PaaSDev) August 18, 2022
I just realized two of my favorite talks this WHOLE summer (@riferrei at #PulsarSummitSF & @ajunaky @DevOpsDaysSEA) have been comparing software to romComs. I guess I should give romComs another chance?! Because this is fantastic 🎉#romanticComedy #SoftwareEngineer #devRel pic.twitter.com/ysbg5NMEtX
— CAITOs (@caito_200_OK) August 18, 2022

Is Using KoP (Kafka-On-Pulsar) a Good Idea?

Link for this presentation:

HTML code for embedding:

Share on social media:

Video

Code

Is Using KoP (Kafka-On-Pulsar) a Good Idea?

Buzz and feedback