Skip to main content

Dataflow Worker Ip Should Not Be Public

More Info:​

Ensure Dataflow jobs worker ip is not public

Risk Level​

Critical

Address​

Security

Compliance Standards​

  • CIS GCP
  • Cloudanix Best Practice
  • GDPR
  • HITRUST CSF
  • NIST CSF
  • PCI
  • SOC2
  • Securities and Exchange Board of India (SEBI) - Cloud Security Adoption Framework

Triage and Remediation​

Remediation​

Using Console

To remediate the misconfiguration "Dataflow Worker IP should not be public" in GCP, you can follow the below steps using GCP console:

  1. Go to the GCP console and navigate to the Dataflow Workers page.
  2. Select the worker pool that you want to update.
  3. Click on the "Edit" button.
  4. In the "Network settings" section, select the "Private" option for the "Worker IP" setting.
  5. Click on the "Save" button to save the changes.

By selecting the "Private" option, the Dataflow worker IP will not be exposed to the public internet. This will help to ensure the security of your Dataflow jobs and prevent unauthorized access.

Using CLI

To remediate the "Dataflow Worker IP Should Not Be Public" misconfiguration in GCP using GCP CLI, you can follow these step-by-step instructions:

  1. Open the Cloud Shell in your GCP console.

  2. Run the following command to get the list of all Dataflow jobs:

gcloud dataflow jobs list

  1. Identify the job for which you want to remediate the misconfiguration.

  2. Run the following command to update the job with the --no-use-public-ips flag:

gcloud dataflow jobs update <JOB_ID> --no-use-public-ips

Replace <JOB_ID> with the actual ID of the job you want to update.

  1. Verify that the job has been updated by running the following command:
gcloud dataflow jobs describe <JOB_ID> | grep "usePublicIps"

This command should return usePublicIps: false.

By following these steps, you should be able to remediate the "Dataflow Worker IP Should Not Be Public" misconfiguration in GCP using GCP CLI.

Using Python

To remediate the misconfiguration "Dataflow Worker IP Should Not Be Public" in GCP using Python, follow these steps:

  1. Open the Google Cloud Console and select the project that you want to work on.
  2. Go to the Dataflow section of the console.
  3. Select the Dataflow job that you want to remediate.
  4. Click on the "Edit" button to edit the job configuration.
  5. In the "Networking" section, select "Custom" for the "Worker IP Configuration" option.
  6. In the "Custom" section, select "Private" for the "Worker IP" option.
  7. Click on the "Save" button to save the changes to the job configuration.

Here is the Python code to remediate the misconfiguration:

from googleapiclient.discovery import build
from google.oauth2 import service_account

# Set the project ID and Dataflow job ID
project_id = 'your-project-id'
job_id = 'your-job-id'

# Set the worker IP configuration to private
body = {
    'environment': {
        'workerIPAddressConfiguration': 'WORKER_IP_PRIVATE'
    }
}

# Authenticate with GCP using a service account key file
credentials = service_account.Credentials.from_service_account_file('path/to/keyfile.json')
dataflow = build('dataflow', 'v1b3', credentials=credentials)

# Update the Dataflow job configuration
request = dataflow.projects().locations().jobs().update(
    projectId=project_id,
    location='us-central1',
    jobId=job_id,
    body=body
)
response = request.execute()

print(response)

Note: Replace "your-project-id", "your-job-id", and "path/to/keyfile.json" with your actual project ID, Dataflow job ID, and the path to your service account key file, respectively.

Using Terraform
resource "google_dataflow_job" "DATAFLOW_JOB_NAME" {
  name              = "DATAFLOW_JOB_NAME"                     # replace with your Dataflow job name
  template_gcs_path = "gs://TEMPLATE_BUCKET/TEMPLATE_PATH"    # replace with your template path
  region            = "GCP_REGION"                            # replace with your region, e.g. "us-central1"

  # Ensure workers do NOT get public IPs
  ip_configuration = "WORKER_IP_PRIVATE"

  # Workers must be on a VPC/subnet to use private IPs
  network    = "projects/PROJECT_ID/global/networks/NETWORK_NAME"          # replace with your VPC
  subnetwork = "regions/GCP_REGION/subnetworks/SUBNET_NAME"               # replace with your subnet

  # other required arguments for your job...
}

Changing ip_configuration from a public setting to "WORKER_IP_PRIVATE" will cause Terraform to replace the Dataflow job (a new job run), which may interrupt any in-flight processing.

After the change, terraform plan should show ip_configuration changing to "WORKER_IP_PRIVATE" on the google_dataflow_job.DATAFLOW_JOB_NAME resource, with the job marked for replacement.