How to Kafka and Java ☕️ — Kafka as a Service on the cloud ☁️ for Java hardcore devs 👩‍💻

7 min readJun 29, 2018

If you are a Java Developer and need to use Kafka service — this blogpost is for you — just slide over the below talk to Introduction.

June was a fun month. I dove into Docker and Kubernetes with DockerCon2018 visiting San Francisco.

I started to imagine things in the containers. And my 11yo son introduced me to Fortnite. In order to listen to him I need to either follow his videoblog (mincraft, and tricks), or chat with him on Fortnite (and I always ask him “please revive me!”). I did some gigs at AI & IoT Dev Conference — with robots using AI powered Voice UI, chatbot tech, and tone analysis. Since I like to engage in the Apple ecosystem I gave a workshop at the AltConf — the side conference to WWDC2018 in San Jose — on building a chatbot iOS app in less than 60 minutes — see the lab here: https://github.com/blumareks/2018-chatbot-101/tree/master/lab-ios

I even survived Slack and Wrike outages!!! Was that really happening or what?

In this post you will get introduced to building a basic CRUD-like service that is using Kafka on IBM Cloud ⛅️ (with the “free of charge” lite account). This is a part of the series of posts — the first and previous one is on using a cloud based “Compose for MongoDB” with Java. All my posts in this series instruct you on how to check the response time 🏃‍♀️ of Java based microservices with datasources hosted on IBM Cloud ⛅️ — IBM Platform as a Service (or abbreviated PaaS). This is a very simple tool that would also show you how to do the basic CRUD operations in your Java Liberty 🗽application / microservice on strategic datasources like Cloudant, IBM Message Hub, MongoDB, Elastic Search, etc.

Introduction

One of my customers asked me to help them with a simple dashboard concept that would display series of availability and response times of the CRUD operations (Create, Read, Update, Delete) of the various data services exposed on the IBM Dedicated Cloud. By coincidence it also has become a good example on how one can use CRUD operations on those data services accessed from a Java Open Liberty 🗽 server on a local machine, as well as bound in the cloud through VCAP configuration file.

This particular how-to post is an example on using IBM Message Hub, or Kafka as a Service on IBM Cloud ⛅️ , that is hosted and managed in the above mentioned IBM Cloud ⛅️ . The approach shown here is to generate a JSON response file with the report from accessing basic operations (Message Produced and Message Received) and provide the time needed to access each and every operation, in addition to the total operation time. The example of the response file you will try to create is the following:

{service: ‘message-hub’, operations: [
{type: ‘create topic’, response_time: 121, response_code: 200, desc: {‘Admin REST response’: ‘no response — the topic has been already created’}},
{type: ‘produce message’, response_time: 1, response_code: 200, desc: {‘Message created’: ‘Message produced, offset: 523’}},
{type: ‘consume message’, response_time: 2064, response_code: 200, desc: {‘Message consumed’: ‘Message consumed: ConsumerRecord(topic = test-java-messagehub-topic, partition = 0, offset = 523, CreateTime = 1499458409907, serialized key size = 3, serialized value size = 25, headers = RecordHeaders(headers = [], isReadOnly = false), key = key, value = This is a test message #2)’}}
], response_code: 200, desc:’operations implemented CrPCo/CrPCoD’}

If you undertook the previous How-Tos in this series, you can go ahead to step 2. Otherwise please continue reading here.

You will be able to learn how to run Java Open Liberty 🗽 server in the cloud ☁️ , and how to connect to the exposed service.

Prerequisites

Before you start you will need the following software on your machine:

- Git,
- Cloud Foundry CLI,
- Maven,
- and a Dedicated Bluemix account or Public Bluemix account (it needs to be a Cloud regular account, since the services aren’t Cloud lite account enabled),
- I am also using Eclipse with Bluemix pluggin to deploy the application to the cloud — however you can easily use any IDE for Java, and run Maven with Cloudfoundry CLI (command line interface) in the terminal to achieve the same results.

Step 1. Clone the sample app.

Let us startwith getting your test application. You can simply clone my github repository : https://github.com/blumareks/BluemixTestDashboard and change the directory to where the sample app for IBM Message Hub is located. In order to do it use the instructions below:

git clone https://github.com/blumareks/BluemixTestDashboard
cd BluemixTestDashboard/GetStartedJavaMessageHub

Step 2. Create the cloud services

In this step you need login to the IBM Cloud console. If you are using IBM Cloud Public — through http://bluemix.net URL you might search for IBM Message Hub and Java Liberty 🗽 .

Create the Java Liberty 🗽 App,

then create the IBM Message Hub service

and bind it with the Java Liberty 🗽 App.

After creating the services you need to bind IBM Message Hub service to Open Liberty 🗽 service. In order to do it you need to show Connection tab of the Open Liberty 🗽 service and click on the connect button, then get the IBM Message Hub service, and select Connect button again.

This action will initiate restaging process of the Open Liberty service. The following screenshots show the process:

The VCAP file could be obtained by selecting the connection tab, and from the three dot pull down menu selecting the View Credentials as shown below:

The connection credentials from the VCAP JSON file obtained as shown above will look like this:

{“messagehub”: [{“credentials”: {
“instance_id”: “<your service instance id>”,
“mqlight_lookup_url”: “https://mqlight-lookup-prod02.messagehub.services.eu-gb.bluemix.net/Lookup?serviceId=<your service instance id>”,
“api_key”: “<your api key>”,
“kafka_admin_url”: “https://kafka-admin-prod02.messagehub.services.eu-gb.bluemix.net:443",
“kafka_rest_url”: “https://kafka-rest-prod02.messagehub.services.eu-gb.bluemix.net:443",
“kafka_brokers_sasl”: [
“kafka02-prod02.messagehub.services.eu-gb.bluemix.net:9093”,
“kafka03-prod02.messagehub.services.eu-gb.bluemix.net:9093”,
“kafka05-prod02.messagehub.services.eu-gb.bluemix.net:9093”,
“kafka04-prod02.messagehub.services.eu-gb.bluemix.net:9093”,
“kafka01-prod02.messagehub.services.eu-gb.bluemix.net:9093”],
“user”: “<your service user ID>”,
“password”: “<your service user password>”},
“syslog_drain_url”: null,“volume_mounts”: [],“label”: “messagehub”,“provider”: null,“plan”: “standard”,“name”: “Message Hub-9u”,
“tags”: [
“ibm_dedicated_public”,“web_and_app”,“ibm_created”]}]}

Step 3. Running on Open Liberty 🗽 with IBM Message Hubin the IBM Cloud

There are at least two ways you can deploy and run this service in the cloud. You can either manually adjust the Cloudfoundry files and run it with Bluemix/CF CLI, or use the IBM Cloud connector for Eclipse.

You will learn how to do it manually with Bluemix Command Line Interface bx CLIfor short.

In the previous blog post you were able to follow the process of installing the Command Line Interface, and configuring an access to the IBM Cloud. The process is the same. At the last stage you would use the manifest file as shown and the command: $ bx app push; as shown below:

If you bound an IBM Message Hub to the server as it was shown in the beginning you are able to run the query for the provided URL address with the following call:/test/messagehub/all :

After running the API call for the test: https://<link to your app on bluemix.mybluemix.net>/test/messagehub/allyou should see the topic being created and the following responses:

the response without the topic created earlier:

{service: ‘message-hub’, operations: [{type: ‘create topic’, response_time: 1693, response_code: 200, desc: {‘Admin REST response’: ‘{}‘}},

{type: ‘produce message’, response_time: 2, response_code: 200, desc: {‘Message created’: ‘Message produced, offset: 2’}},

{type: ‘consume message’, response_time: 2039, response_code: 200, desc: {‘Message consumed’: ‘Message consumed: ConsumerRecord(topic = test-java-messagehub-topic, partition = 0, offset = 2, CreateTime = 1526338535165, serialized key size = 3, serialized value size = 25, headers = RecordHeaders(headers = [], isReadOnly = false), key = key, value = This is a test message #2)’}}], response_code: 200, desc:’operations implemented CrPCo/CrPCoD’}

the response with the topic created earlier:

{service: ‘message-hub’, operations: [{type: ‘create topic’, response_time: 346, response_code: 200, desc: {‘Admin REST response’: ‘no response — the topic has been already created’}},{type: ‘produce message’, response_time: 1, response_code: 200, desc: {‘Message created’: ‘Message produced, offset: 5’}},{type: ‘consume message’, response_time: 2040, response_code: 200, desc: {‘Message consumed’: ‘Message consumed: ConsumerRecord(topic = test-java-messagehub-topic, partition = 0, offset = 5, CreateTime = 1526342672837, serialized key size = 3, serialized value size = 25, headers = RecordHeaders(headers = [], isReadOnly = false), key = key, value = This is a test message #2)’}}], response_code: 200, desc:’operations implemented CrPCo/CrPCoD’}

You might want to look at the logs of the document both on the dashboard of the service (tab logs), or just use the CLI in the terminal for that — the command to get the logs trail is the following:

$ bx applogs <your service name>

The next How-to blog post will be on adding an app to read, parse and store the results. Stay tuned and please subscribe to this blog post. Please follow me on Twitter at blumareks or at Medium/@blumareks

How to Kafka and Java ☕️ — Kafka as a Service on the cloud ☁️ for Java hardcore devs 👩‍💻

Introduction

Prerequisites

Step 1. Clone the sample app.

Step 2. Create the cloud services

Step 3. Running on Open Liberty 🗽 with IBM Message Hubin the IBM Cloud

References

Written by Blumareks

No responses yet