> ## Documentation Index
> Fetch the complete documentation index at: https://cloudanix.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Monitor system problem state remediation

### Triage and Remediation

<Tabs>
  <Tab title="Remediation">
    ### Remediation

    <AccordionGroup>
      <Accordion title="Using Console" defaultOpen="true">
        If the misconfiguration is related to monitoring system problem state in GCP, here are the step-by-step instructions to remediate it using the GCP console:

        1. Login to your GCP console and select the project where the misconfiguration exists.
        2. Go to the "Monitoring" section from the left-hand menu.
        3. Click on the "Uptime Checks" tab, which is located under the "Alerting" section.
        4. Check if an uptime check is already created. If not, click on the "Create Uptime Check" button.
        5. In the "Create Uptime Check" form, provide the necessary details such as the name of the uptime check, the resource type, and the resource to be monitored.
        6. Select the protocol that should be used for monitoring (HTTP or HTTPS) and provide the URL to be monitored.
        7. Configure the check frequency and the number of consecutive failures needed to trigger an alert.
        8. Once all the details are provided, click on the "Create" button to create the uptime check.

        By following these steps, you can remediate the misconfiguration related to monitoring system problem state in GCP using the GCP console.

        #
      </Accordion>

      <Accordion title="Using CLI">
        The misconfiguration "Cloud Monitoring Should Monitor System Problem State" means that the monitoring system is not correctly configured to monitor the system problem state. To remediate this issue in GCP using GCP CLI, follow these steps:

        1. Open the GCP Cloud Console and navigate to the Cloud Monitoring page.

        2. Click on the "Alerting" tab and then click on "Create Policy".

        3. In the "Create Policy" window, give a name and description for the policy.

        4. Under "Add Condition", select "Metric Threshold".

        5. Select the resource type for which you want to monitor the system problem state.

        6. Choose the metric for which you want to set the threshold. For example, you can choose "System Uptime" or "CPU Utilization".

        7. Set the threshold values for the metric. You can set the threshold based on the percentage of usage or a specific value.

        8. Under "Add Notification Channel", select the notification channel where you want to receive alerts when the threshold is breached.

        9. Click on "Save" to create the policy.

        10. Once the policy is created, it will start monitoring the selected metric and send alerts to the selected notification channel when the threshold is breached.

        By following these steps, you can remediate the misconfiguration "Cloud Monitoring Should Monitor System Problem State" in GCP using GCP CLI.
      </Accordion>

      <Accordion title="Using Python">
        To remediate the misconfiguration "Cloud Monitoring Should Monitor System Problem State" for GCP using Python, you can follow the below steps:

        1. Import the necessary libraries:

        ```
        from google.cloud import monitoring_v3
        from google.oauth2 import service_account
        ```

        2. Set up the credentials for the service account:

        ```
        credentials = service_account.Credentials.from_service_account_file('<path_to_service_account_file>')
        ```

        3. Create a client for the Monitoring API:

        ```
        client = monitoring_v3.MetricServiceClient(credentials=credentials)
        ```

        4. Define the resource for which you want to enable system monitoring:

        ```
        resource_name = 'projects/<project_id>/instances/<instance_id>'
        ```

        5. Define the metric descriptor for the system problem state:

        ```
        metric_descriptor = monitoring_v3.types.MetricDescriptor()
        metric_descriptor.type = 'agent.googleapis.com/system/problem_state'
        metric_descriptor.metric_kind = monitoring_v3.enums.MetricDescriptor.MetricKind.GAUGE
        metric_descriptor.value_type = monitoring_v3.enums.MetricDescriptor.ValueType.BOOL
        metric_descriptor.description = 'The system problem state of the instance'
        metric_descriptor.unit = '1'
        ```

        6. Create the metric descriptor using the client:

        ```
        client.create_metric_descriptor(parent='projects/<project_id>', metric_descriptor=metric_descriptor)
        ```

        7. Verify that the metric descriptor has been created by listing all the metric descriptors:

        ```
        for descriptor in client.list_metric_descriptors(parent='projects/<project_id>'):
            print(descriptor)
        ```

        8. Enable the monitoring of the system problem state for the resource:

        ```
        client.create_time_series(
            name=client.project_path('<project_id>'),
            time_series=[
                monitoring_v3.types.TimeSeries(
                    metric=monitoring_v3.types.Metric(
                        type='agent.googleapis.com/system/problem_state',
                        labels={
                            'instance_id': '<instance_id>',
                            'zone': '<zone>'
                        }
                    ),
                    resource=monitoring_v3.types.Resource(
                        type='gce_instance',
                        labels={
                            'project_id': '<project_id>',
                            'instance_id': '<instance_id>',
                            'zone': '<zone>'
                        }
                    ),
                    points=[
                        monitoring_v3.types.Point(
                            interval=monitoring_v3.types.TimeInterval(
                                end_time=timestamp_pb2.Timestamp.GetCurrentTime()
                            ),
                            value=monitoring_v3.types.TypedValue(
                                bool_value=True
                            )
                        )
                    ]
                )
            ]
        )
        ```

        9. Verify that the monitoring is enabled by listing all the time series for the metric:

        ```
        for series in client.list_time_series(
                name=client.project_path('<project_id>'),
                filter='metric.type="agent.googleapis.com/system/problem_state" AND resource.type="gce_instance" AND resource.labels.instance_id="<instance_id>"'
            ):
            print(series)
        ```

        These steps will remediate the misconfiguration "Cloud Monitoring Should Monitor System Problem State" for GCP using Python.
      </Accordion>

      <Accordion title="Using Terraform">
        ```hcl theme={null}
        resource "google_monitoring_alert_policy" "system_problem_state" {
          project      = "YOUR_PROJECT_ID"          # Replace with your GCP project ID
          display_name = "System problem state alert"

          combiner = "OR"

          conditions {
            display_name = "System problem state > 0"

            condition_threshold {
              # Monitors the built‑in system problem state metric from Ops Agent.
              # 0 = OK, 1 = Warning, 2 = Problem. Alert on any non‑zero.
              filter = "metric.type=\"agent.googleapis.com/system/problem_state\""

              comparison      = "COMPARISON_GT"
              threshold_value = 0

              # How long the condition must hold before firing
              duration = "300s" # 5 minutes; adjust if you have a specific threshold duration

              trigger {
                count = 1
              }

              aggregations {
                alignment_period   = "60s"
                per_series_aligner = "ALIGN_MAX"
              }
            }
          }

          notification_channels = [
            "projects/YOUR_PROJECT_ID/notificationChannels/YOUR_NOTIFICATION_CHANNEL_ID"
          ]

          documentation {
            content  = "System problem state reported by Ops Agent is non-zero (warning or problem). Investigate the affected system."
            mime_type = "text/markdown"
          }

          user_labels = {
            environment = "YOUR_ENVIRONMENT" # e.g. "prod"
          }
        }
        ```

        This change does not force replacement of any existing monitored resource; it only creates/updates an alert policy in Cloud Monitoring.

        Verification: `terraform plan` should show a new `google_monitoring_alert_policy.system_problem_state` resource (or updates to it if it already exists) with the `filter` on `metric.type="agent.googleapis.com/system/problem_state"` and the `threshold_value` set to `0`.
      </Accordion>
    </AccordionGroup>
  </Tab>
</Tabs>
