Node.js - Google Cloud Storage Object Listing with Pagination

Do you use Google Cloud Storage and need to get the list of objects on your bucket using your Node.js application? This tutorial will show you how to do so. I assume you've already have set up your Google Cloud account, create a project, know how to upload files to your buckets and of course you've got the credentials needed for authentication. If you haven't, you can read this tutorial.

First we create a helper with a prototype method to get the list of objects

helpers/google-cloud-storage.js

  const GoogleCloudStorage = require('@google-cloud/storage');

  function CloudStorageClient() {
    if (!(this instanceof CloudStorageClient)) {
      return new CloudStorageClient();
    }

    this.storage = GoogleCloudStorage({
      projectId: 'gcs-demo-123456', // Replace with your project ID
      keyFilename: 'path-to-the-private-key', // Replace with the path to the downloaded private key
    });
  }

  /**
   * Get list of objects in a bucket.
   * @param {string} bucketName - Name of the bucket.
   * @param {Object} options - Query object for listing files.
   * @returns {Promise.<Object>

Then we use the code above in another file

example.js

  const CloudStorageClient = require('./helpers/google-cloud-storage');

  const client = new CloudStorageClient();

  const options = {};
  client.getFiles('your-bucket-name', options)
    .then(console.log);

If the options is empty object {}, you'll get all objects on your bucket. You mau have some question such as how to get a list of objects in a specific directory without its child directories and how to use pagination. To make it clear, I'll give some examples.

For example we have the following directory structure

  a.png
  b.png
  folder-1
    c.png
    d.png
    folder-2
      e.png
      f.png

Below are several cases of getting object listing

1. All objects in a bucket

To get the list of objects in a bucket, you don't need any filter

[
  'a.png',
  'b.png',
  'folder-1/',
  'folder-1/c.png',
  'folder-1/d.png',
  'folder-1/folder-2/',
  'folder-1/folder-2/e.png',
  'folder-1/folder-2/f.png'
]

2. All file objects in the root of bucket

If you specify a delimiter, you'll get the list of objects whose name doesn't contain delimiter. Therefore you can set '/' as delimiter to get the list of objects in the root of your bucket.

{
  delimiter: '/',
}

[ 
  'a.png', 
  'b.png'
]

3. All file and directory objects in the root of bucket

How about listing all objects in the root of your bucket including the directories. It's very similar to the previous case, but you also need to use includeTrailingDelimiter: true, so that if the delimiter is at the end of the name, it will be included in the result.

{
  delimiter: '/',
  includeTrailingDelimiter: true,
}

[
  'a.png',
  'b.png',
  'folder-1/'
]

4. All objects under a directory

To get all oll objects under a directory, just provide the directory name.

{
  directory: 'folder-1',
}

[
  'folder-1/',
  'folder-1/c.png',
  'folder-1/d.png',
  'folder-1/folder-2/',
  'folder-1/folder-2/e.png',
  'folder-1/folder-2/f.png'
]

5. All file objects in the root of a directory

If you need to list all files (non-directory) objects in the root of a directory, you can also use delimter with value of '/'. Note that the directory is also included in the result.

{
  directory: 'folder-1',
  delimiter: '/',
  includeTrailingDelimiter: false, // default
}

[
  'folder-1/',
  'folder-1/c.png',
  'folder-1/d.png'
]

6. All file and directory objects in the root of a directory

If you want to list all objects, either file or child direcotry, in the root of a directory, you can use includeTrailingDelimiter: true. Note that the directory is also included in the result.

{
  directory: 'folder-1',
  delimiter: '/',
  includeTrailingDelimiter: true,
}

[
  'folder-1/',
  'folder-1/c.png',
  'folder-1/d.png',
  'folder-1/folder-2/'
]

7. All objects under a sub directory

To get all objects under a sub directory, use the path the the sub directory as the value of directory.

{
  directory: 'folder-1/folder-2',
}

[
  'folder-1/folder-2/',
  'folder-1/folder-2/e.png',
  'folder-1/folder-2/f.png'
]

8. Pagination (first page)

To use custom pagination, you only need to provide maximum number of returned objects on maxResults.

{
  maxResults: 3,
}

[
  'a.png',
  'b.png',
  'folder-1/'
]
nextPageToken: Cglmb2xkZXItMS8=

9. Pagination (next page)

To get the list of objects on next page, use the token obtained from the previous request as pageToken. The value of maxResults doesn't have to be the same as the previous request.

{
  maxResults: 3,
  pageToken: 'Cglmb2xkZXItMS8='
}

[
  'folder-1/c.png',
  'folder-1/d.png',
  'folder-1/folder-2/'
]

That's all about how to get list of objects on Google Cloud Storage.