Google Cloud Pub/Sub is a service for event-driven computing system. It allows you to publish thousands of messages per second that will be consumed by subscribers. It's one of the alternatives to Amazon's SQS (Simple Queue Service).
Pub/Sub ensures at-least-once delivery for each messages. Each received message must be acknowledged after successfully processed, otherwise the server will sent the same messages multiple times even it has been processed.
The advantage of using Pub/Sub is easy integration with other Google products. It's quite simple as it does not have shards or partitions. It's also has global scope by default, which means messages can be published and consumed anywhere. In addition, libraries are also provided for popular programming languges, makes it easy for developers to integrate Pub/Sub on applications.
In this tutorial, I'm going to show you the basic usage examples of Google Cloud Pub/Sub in Node.js
Preparation
1. Create or select a Google Cloud project
A Google Cloud project is required to use this service. Open Google Cloud console, then create a new project or select existing project
2. Enable billing for the project
Like other cloud platforms, Google requires you to enable billing for your project. If you haven't set up billing, open billing page.
3. Enable Google Pub/Sub API
To use an API, you must enable it first. Open this page to enable Pub/Sub API.
4. Set up service account for authentication
As for authentication, you need to create a new service account. Create a new one on the service account management page and download the credentials, or you can use your already created service account.
In your .env file, you have to add a new variable
GOOGLE_APPLICATION_CREDENTIALS=/path/to/the/credentialsThe .env file should be loaded of course, so you need to use a module for reading .env such as dotenv.
Dependencies
This tutorial uses @google-cloud/pubsub. Add the following dependencies to your package.json and run npm install
"@google-cloud/pubsub": "~0.19.0"
"dotenv": "~4.0.0"1. Topic
Every messages must be sent within a topic. It means you need to have at least one Pub/Sub topic in your account.
To create a topic, open Pub/Sub console and click on "Create Topic". You'll need to enter a name for the topic.
You can also create a topic programatically, as examplified below.
require('dotenv').config();
const PubSub = require(`@google-cloud/pubsub`);
const pubsub = new PubSub();
pubsub
.createTopic('topic1')
.catch(err => {
console.error('ERROR:', err);
});2. Publisher
A publisher sends a message to a topic. In order to publish a message, the publisher sends a request to the Cloud Pub/Sub server containing the message. A unique identifier will be given to each successfully published message. Below is the code example for sending message to a topic
require('dotenv').config();
const PubSub = require(`@google-cloud/pubsub`);
const pubsub = new PubSub();
// In this example, the message is current time
const data = new Date().toString();
const dataBuffer = Buffer.from(data);
const topicName = 'topic1';
pubsub
.topic(topicName)
.publisher()
.publish(dataBuffer)
.then(messageId => {
console.log(`Message ${messageId} published.`);
})
.catch(err => {
console.error('ERROR:', err);
});3. Subscription
A subscription is required to retrieve messages on a topic. Therefore you must create at least a subscription on the topic. A subscription can only read messages on a topic, while a topic can have multiple subscriptions.
In order to create a subscription, on the Pub/Sub console, choose your topic, then click on "Create Subscription". In addition to entering the name for the subscription you want to create, you also have to choose delivery type. There are two delivery types: pull and push.
Pull
In pull subscription, the subcriber sends a request to the Cloud Pub/Sub server in order to retrieve messages
- The subscriber sends a request to pull messages
- The Cloud Pub/Sub server sends a response containing messages if any
- The subscriber acks each messages
Here is the example.
require('dotenv').config();
const PubSub = require(`@google-cloud/pubsub`);
const pubsub = new PubSub();
const subscriptionName = 'sub1';
const timeout = 60;
const subscription = pubsub.subscription(subscriptionName);
let messageCount = 0;
/**
* Handler for received message.
* @param {Object} message
*/
const messageHandler = message => {
console.log(`Received message ${message.id}:`);
console.log(`Data: ${message.data}`);
console.log(`tAttributes: ${message.attributes}`);
messageCount += 1;
// Ack the messae
message.ack();
};
// Listen for new messages until timeout is hit
subscription.on(`message`, messageHandler);
setTimeout(() => {
subscription.removeListener('message', messageHandler);
console.log(`${messageCount} message(s) received.`);
}, timeout * 1000);
Push
In push subscription, data will be sent by Cloud Pub/Sub server to the specified endpoint. When you create a new subscription using push method, you need to specify an endpoint where the messages will be sent.
- The Cloud Pub/Sub server sends a request containg messages
- The subscriber receives the request and acks each messages
In case you get INVALID_ARGUMENT error when creating a new subscriber, do the following:
- Verify your site on Google Search Console.
- Add your domain on Google Cloud Console Domain Verification.
Now you should be able to create a new subscriber. If you need to create another subscriber whose endpoint is under different domain, you need to do the same.
A message sent to an endpoint is on the message property of req.body. The data is in base64 form, so you have to decode it first. Sendng a response with a status code of 204 implicitly acknowledges the message. Here is the code example.
router.post('/google-cloud-pubsub-subscriber', (req, res) => {
const message = req.body ? req.body.message : null;
if (message) {
const buffer = Buffer.from(message.data, 'base64');
const data = buffer ? buffer.toString() : null;
console.log(`Received message ${message.messageId}:`);
console.log(`Data: ${data}`);
}
return res.send(204);
});
After creating both publisher and subscriber, it's time to test if it works as we expect. Run the subscriber and the publisher. If successful, you should see the messages printed on your console.</p
