PubSub Subscriptions Should Have Set Expiration For Messages

More Info:

Ensure that PubSub Subscriptions have set expiration

Risk Level

Low

Address

Cost Optimization, Operational Excellence

Compliance Standards

• Cloudanix Best Practice

Triage and Remediation

Remediation

Remediation

Using Console

To remediate the misconfiguration of PubSub Subscriptions not having set expiration for messages in GCP using GCP console, you can follow the below steps:

1. Open the GCP console and navigate to the Pub/Sub section.
2. Select the Subscription that you want to remediate.
3. Click on the "Edit" button for the selected Subscription.
4. In the "Edit Subscription" window, scroll down to the "Message retention duration" section.
5. Set the message retention duration to the desired value. This value should be based on how frequently the Subscription is polled and how long it takes to process the messages.
6. Click on the "Save" button to save the changes.

By following these steps, you will remediate the misconfiguration of PubSub Subscriptions not having set expiration for messages in GCP using GCP console.

Using CLI

To remediate the issue of PubSub subscriptions not having set expiration for messages in GCP using GCP CLI, you can follow the below steps:

1. Open the Cloud Shell in your GCP console.

2. Run the following command to list all the subscriptions in your project:

   gcloud pubsub subscriptions list

3. Choose the subscription for which you want to set the expiration time for messages.

4. Run the following command to set the expiration time for messages in the chosen subscription:

   gcloud pubsub subscriptions update <subscription-name> --expiration-period=<duration>

   Replace <subscription-name> with the name of the chosen subscription and <duration> with the duration for which you want to set the expiration time for messages. The duration should be in the format of n[d|h|m|s] where n is the number of days, hours, minutes or seconds.

   For example, to set the expiration time for messages in a subscription named my-subscription to 7 days, run the following command:

   gcloud pubsub subscriptions update my-subscription --expiration-period=7d

5. Verify that the expiration time for messages has been set for the chosen subscription by running the following command:

   gcloud pubsub subscriptions describe <subscription-name>

   Replace <subscription-name> with the name of the chosen subscription. The output should display the expiration time for messages in the subscription.

By following these steps, you can remediate the issue of PubSub subscriptions not having set expiration for messages in GCP using GCP CLI.

Using Python

To remediate the issue of PubSub subscriptions not having set expiration for messages in GCP using Python, you can follow the below steps:

1. First, you need to identify the subscription(s) that do not have set expiration for messages. You can use the following command to list all subscriptions in a project:

   gcloud pubsub subscriptions list --format="value(name)"

   This will give you a list of all subscriptions in your project.

2. Once you have identified the subscription(s) that need to be remediated, you can use the Google Cloud Pub/Sub client library for Python to set the message expiration time for each subscription. Here is a sample Python script that sets the message expiration time for a subscription:

   from google.cloud import pubsub_v1
   import datetime
   
   # Set the expiration time for messages to 7 days from now
   expiration_time = datetime.timedelta(days=7)
   
   # Set the subscription name
   subscription_name = 'projects/<PROJECT_ID>/subscriptions/<SUBSCRIPTION_NAME>'
   
   # Create a Pub/Sub subscriber client
   subscriber = pubsub_v1.SubscriberClient()
   
   # Set the subscription message retention duration
   subscription_path = subscriber.subscription_path(
       '<PROJECT_ID>', '<SUBSCRIPTION_NAME>')
   subscription = subscriber.modify_push_config(
       subscription_path,
       push_config=pubsub_v1.types.PushConfig(
           expiration_policy=expiration_time))
   
   # Print the updated subscription information
   print('Subscription {} updated with expiration time: {}'.format(
       subscription_name, expiration_time))

   Replace <PROJECT_ID> and <SUBSCRIPTION_NAME> with your project ID and subscription name respectively.

3. Run the Python script to set the message expiration time for the subscription(s) that need to be remediated.

   python set_subscription_expiration.py

   This will update the subscription(s) with the message expiration time and print the updated subscription information.

By following these steps, you can remediate the issue of PubSub subscriptions not having set expiration for messages in GCP using Python.

Using Terraform

resource "google_pubsub_subscription" "PUBSUB_SUBSCRIPTION_NAME" {
  name  = "PUBSUB_SUBSCRIPTION_NAME"              # replace with your subscription name
  topic = google_pubsub_topic.TOPIC_RESOURCE.name # or "projects/PROJECT_ID/topics/TOPIC_NAME"

  # Ensure messages expire after a bounded retention period
  # Set to your desired value, e.g. "604800s" for 7 days
  message_retention_duration = "MESSAGE_RETENTION_DURATION_IN_SECONDS_S"

  # Optional: if you retain acknowledged messages as well, they will also expire
  # retain_acked_messages = true
}

Replace:

• PUBSUB_SUBSCRIPTION_NAME with your subscription’s name.
• TOPIC_RESOURCE with the name of the google_pubsub_topic resource (or hardcode the full topic name).
• MESSAGE_RETENTION_DURATION_IN_SECONDS_S with an appropriate duration string (e.g. "86400s" for 1 day, "604800s" for 7 days), matching your policy threshold.

No resource replacement is required; updating message_retention_duration is an in-place change.

Verification: terraform plan should show message_retention_duration changing from null (or its previous value) to your specified duration on the google_pubsub_subscription resource.

Additional Reading:

• https://cloud.google.com/pubsub/docs/subscriber