How to connect your Spring Cloud Application to a local Google Cloud Pub/Sub Emulator

Twitter | LinkedIn | YouTube | Instagram

Google Cloud Pub/Sub is a powerful message broker, but developing for it might be a bit of a pain in the neck when you have multiple developers working on the same project and all pointing to the same topics and subscriptions in the cloud. Depending on the size of your organization, your long-term plans, and also how much you want to be worrying about infrastructure for development puraposes in the cloud, having a topic and subscription for each developer may also be excessive.

Luckily, Google provides a Google Cloud Pub/Sub emulator that we can run locally and emulate your Pub/Sub infrastructure locally. Let’s dive in on how to set up it locally and connect it to a Spring Cloud application.

Setting Up the PubSub Emulator

Prerequisites to use the emulator

You must have the following to use the Pub/Sub emulator:

• Properly set up python development environment. Refer to this guide for details.
• Java JRE (version 7 or higher) installed.
• Google Cloud CLI installed.

Installing the emulator

Install the emulator from a command prompt:

gcloud components install pubsub-emulator
gcloud components update

Starting the emulator

Start the emulator by invoking pubsub start from a command prompt. Before running the command, replace PUBSUB_PROJECT_ID with a valid Google Cloud project ID. string. The string does not need to represent a real Google Cloud project because the Pub/Sub emulator runs locally.

gcloud beta emulators pubsub start --project=PUBSUB_PROJECT_ID

After you start the emulator, you see a message that resembles the following:

...
[pubsub] This is the Pub/Sub fake.
[pubsub] Implementation may be incomplete or differ from the real system.
...
[pubsub] INFO: Server started, listening on 8085

Cool, it means your emulator is running at localhost:8085.

Join Medium with my referral link - Raphael De Lio

Setting up the Spring Cloud Application

We will build a Spring Cloud Application that will be responsible for two things:

• Supplying messages with two numbers to a specific topic and subscription
• Read the message from this specific topic and subscription and sum up the numbers

We will use the Spring Cloud Stream dependency to connect the application to PubSub, which will also require the stream binder of PubSub provided by Google.

Since I’m using Intellij, I will start a new project using the Spring Initializr. I will be using Kotlin and Gradle, but you can achieve the same results by using Java.

These are the dependencies we will need for our application:

implementation("org.springframework.cloud:spring-cloud-stream")
implementation("com.google.cloud:spring-cloud-gcp-starter-pubsub")
implementation("com.google.cloud:spring-cloud-gcp-pubsub-stream-binder")

And this is what our build.gradle will look like:

Let’s create a Configuration class that will hold our two Beans:

• A supplier to generate the messages
• A consumer to read the messages

The generateNumbers function generates two random numbers and sends them separated by a comma as the payload of the message.

The sumNumbers will read the message, decode the payload from bytes, split the String, and sum them by using the fold function.

Now, let’s bind our functions to Spring Cloud Stream by setting the following properties to our application.yml:

spring:
  cloud:
    stream:
      function:
        definition: sumNumbers;generateNumbers
      bindings:
        generateNumbers-out-0:
          destination: sum_numbers
          group: sum_numbers-sub
          binder: pubsub

        sumNumbers-in-0:
          destination: sum_numbers
          group: sum_numbers-sub
          binder: pubsub

And tell our application to connect to our emulator at the localhost:8085 host and port:

spring:
  cloud:
    gcp:
      pubsub:
        emulator-host: "localhost:8085" 

And, just to avoid producing millions of messages, let’s limit our supplier to run every five seconds:

spring:
  cloud:
    stream:
      poller:
        fixed-delay: 5000

And this is what your application.yml should be looking like:

Let’s test it!

Testing the application

If you run the application, you should see the following being printed to the console:

Intellij Console

If you do, that’s great, it’s working!

Making sure we’re connecting to our local emulator

Spring Cloud Stream will automatically generate the topic and subscription for you. An easy way to make sure we’re not connecting to our cloud infrastructure is by manually going to Google Cloud Platform and verifying if this topic and subscription haven’t been created there.

Another way is by using the Python Client for Google Cloud PubSub which you can find here.

To use it, you must set the environment variables so that the Python Client connects to the emulator instead of PubSub in GCP. Set these environment variables on the same machine that you use to run your application by running:

$(gcloud beta emulators pubsub env-init)

Clone the Python Client project from GitHub to your local machine, and before we continue, let’s remove the Bean that is supposed to consume the messages and the respective properties from our application and restart it.

We should see the random numbers being printed to the console, but not the sum of them.

Once you have done it, go to python-pubsub-main/samples/snippets and run:

python3 subscriber.py my-project receive sum_numbers.sum_numbers-sub

And you should see a response similar to:

Listening for messages on projects/my-project/subscriptions/sum_numbers.sum_numbers-sub..
Received Message {
  data: b'35,20'
  ordering_key: ''
  attributes: {
    "contentType": "application/json"
  }
}

And when running:

python3 subscriber.py my-project list-in-project

You should see only the sum_numbers.sum_numbers-sub resource being printed back. None of the resources you have available in your cloud environment should show up here.

Once you’re done, don’t forget to unset the environment variables after you stopped the emulator by running:

unset PUBSUB_EMULATOR_HOST

And that’s it!

Conclusion

Today you learned how to start a Google PubSub Emulator locally, connect to it by using a Spring Cloud application, generate and consume messages by using Spring Cloud Stream and also how to connect to the emulator by using the Google PubSub Python Client. Congratulations!

I’m very happy you got to this part of the story, I‘m truly thankful for this.
Support my work: follow me and clap to this story.

Check this code in my GitHub Repository.

See you around!

Contribute

Writing takes time and effort. I love writing and sharing knowledge, but I also have bills to pay. If you like my work, please, consider donating through Buy Me a Coffee: https://www.buymeacoffee.com/RaphaelDeLio

Or by sending me BitCoin: 1HjG7pmghg3Z8RATH4aiUWr156BGafJ6Zw

Follow Me on Social Media

Stay connected and dive deeper into the world of Spring with me! Follow my journey across all major social platforms for exclusive content, tips, and discussions.

Twitter | LinkedIn | YouTube | Instagram