By Sujay Maheshwari in devops — Jul 12, 2020

CloudTrail, CloudWatch, Splunk, Custom logs solution. How do you make sense of these?

Introduction

To understand what is going on in the AWS environment, your Cloud Operations team member (or your engineering team member) needs a defined and robust system for ingesting logs and then analyzing & reacting. This is not an easy task with today’s strict compliance rules, security, and law standards. With Amazon Web Services’ native or hybrid environments, torrents of data are moving in and out of their network at all times. Hence, by hook or by crook, a DevOps developer will need AWS logs.

The Ops team can face the challenges of mining this data. On the other hand, the operations team can also transform the logging data into tremendous golden opportunities to improve application ops support and reduce costs.

CloudTrail and CloudWatch, both help a developer in analyzing data collected from their AWS environment. Splunk is basically an application that makes machine data accessible across an enterprise by identifying data patterns, providing metrics, diagnosing problems, and also providing intelligence for business operations. It is mainly used for business and web analytics for logging. Some might use Amazon CloudWatch, some use AWS CloudTrail, and some prefer custom logging solutions, in which each has equal importance for the productive and efficient management of logs.

What is CloudTrail?

AWS CloudTrail is an application that gathers all pertinent information about API calls made within a DevOps developer’s AWS environment. It also reveals the caller’s identity, IP address, call request, and other related data. CloudTrail logs contain information critical for audits and intrusion response, and it is a service that enables governance, compliance, operational auditing, and risk auditing of their account.

CloudTrail can easily log, continuously monitor, and store account activity-related actions across an account throughout the AWS infrastructure. It even provides event history of their AWS account activity, including actions taken through the AWS management console, AWS SDK’s, command-line tools, and other services. Hence, developers can use CloudTrail to detect unusual activity in their AWS accounts.

All these capabilities help simplify operational analysis and troubleshooting. CloudTrail provides excellent visibility into a user’s activity by recording AWS console actions and API calls that were made, including who made that call and from which IP address that call was made. It even shows when the call was made. AWS CloudTrail logs very high volume activity events on other services, such as AWS Lambda, S3, and EC2. It could be turned on from the moment developers create and activate their AWS account.

An example of now AWS CloudTrail looks like

{"version":"0","id":"5f4648fa-5be4-cdec-e7fc-114539d13474","detail-type":"AWS Console Sign In via CloudTrail","source":"aws.signin","account":"xxxxxxxxxxx","time":"2020-06-30T08:53:15Z","region":"us-east-1","resources":[],"detail":{"eventVersion":"1.05","userIdentity":{"type":"Root","principalId":"017653914175","arn":"arn:aws:iam::017653914175:root","accountId":"017653914175"},"eventTime":"2020-06-30T08:53:15Z","eventSource":"signin.amazonaws.com","eventName":"ConsoleLogin","awsRegion":"global","sourceIPAddress":"102.112.10.2","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36","requestParameters":null,"responseElements":{"ConsoleLogin":"Success"},"additionalEventData":{"LoginTo":"https://console.aws.amazon.com/console/home?nc2=h_ct&src=header-signin&state=hashArgs%23&isauthcode=true","MobileVersion":"No","MFAUsed":"Yes"},"eventID":"d8d8c991-e46f-4992-b4fc-892ec50217fe","eventType":"AwsConsoleSignIn"}}

What is CloudWatch?

CloudWatch is an application that monitors and manages an enterprise’s data and provides actionable insights to monitor their applications. It also understands and responds to system-wide performance changes while optimizing the enterprise’s resource utilization and getting a unified view of operational health. CloudWatch also collects monitoring and operational data in the form of logs, metrics, and events. It can also set high-resolution alarms and automate actions. AWS CloudWatch is mainly used to understand what is happening in the AWS environment and for logging all the events for a particular service or application.

CloudWatch can collect logs from far more resources and native logs and also from services. It can also collect optional published logs from over 38 services and custom logs from other applications or on-premise resources. It also provides operators with the ability to go deeper into the metrics and pull out only those relevant data.

It covers over 70 services and also provides a variety of built-in metrics to make them understand how well their resources are and for running of services, including latency errors or any other changes in state. Going further into their analysis of the process, CloudWatch provides up to 15 months of metric data and CloudWatch metrics math usage. Developers can also perform calculations across multiple metrics to understand and find utilization performance and the operational health of their enterprise. CloudWatch logs metrics and alarms work, and it helps developers find, diagnose, and rectify issues to increase the efficiency of the cloud environment.

This way, it provides a highly efficient and reliable cloud environment, and it is more of a real-time function that looks after their resources. It also records historical logs and keep their infrastructure healthy and secure, lays its values in extensive integration with other AWS services, and also ongoing live, actionable insights. CloudWatch Event rule can recognize the change in logs and then trigger a Lambda function to open a ticket for investigation for the same. By default, the Lambda function blueprint logs the decoded data batch from CloudWatch logs. Developers can get started with Amazon CloudWatch at no extra charge, and most AWS services like EC2, S3, Kinesis, etc., vend metrics automatically for free to CloudWatch.

What is Splunk?

Splunk is a third-party tool that helps in fields like big data and machine-generated data. Splunk makes machine data accessible across an enterprise by identifying data patterns and also by providing metrics. It diagnoses problems and also provides intelligence for business operations. It is a horizontal technology developers use for application management, security, and compliance and also in business and web analytics. It is a powerful alternative for the applications, tools, and systems that developers use every day to build, test, and ship products that can help run DevOps practices like continuous integration and continuous deployment.

It empowers developers to quickly trace and identify errors anywhere in the code base with real-time search engines and monitoring of applications. It delivers application intelligence from logs and provides real-time insights from the system and processes that drive the application life cycle. Bugs and errors that take days to track and fix, take minutes to track using Splunk. With the Splunk platform, developers can collect, index, and correlate data from various sources. Once data is inside, you can quickly search, explore, and visualize the data to provide insight into data in any environment like testing, staging, and production.

Custom logging solution

Developers and CloudOps teams can transform the right approach with big data partners and challenges of mining all data from the enterprise to improve user performance and reduce costs.

Three of such best practices to complete control for developers AWS log data are:

Know your logging responsibilities

Secure Your Logging Environment

In AWS, Always Be Watching

I will briefly explain each point, respectively

Know Your Logging Responsibilities

A developer must understand one’s role in log data management. Public cloud usage works on the shared responsibility principle. It means that AWS protects developers/enterprises against intruders and any other threats. Still, it’s the developers and operators who are always responsible for the code, data, and credentials of users who they allow into their environment. They must always be sure to analyze their native or hybrid cloud structure. AWS customers (developers + operators) must also identify the applications and data hand-off points that might eventually lead to vulnerabilities and attacks.

Secure Your Logging Environment

As a developer, you must always protect yourself and keep your log data clean by maintaining some security practices such as

Restrictive access permissions
Multi-factor user authentication
Update security certificates
Audit your AWS logs

I will briefly describe each point, respectively.

Restrictive access permissions

Access restrictions in essential transactions must have minimal access to resources, and frequent audits and update access control lists must occur.

Multi-factor user authentication

To ensure that intruders can not sneak through any vulnerability on security gaps, developers must use multi-factor authentication.

Update security certificates

According to the latest requirements outlined by PCI Security Standards Organization, developers must keep the Secure Socket Layer (SSL) and Transport Layer Security (TLS) certificates in a network up to date all the time to most recent and secured versions. Most logged security breaches are because of weaknesses in this compliance area.

Audit your AWS logs

The PCI Security Standards Organisation also stipulates annual audits to be performed internally, at least one audit per year by any third-party security firm. These tests help to identify gaps that will prepare enterprises for any audit eventually and give the operators practice in dealing with critical incidents. With these steps at an early stage, in the log management approach, as a developer, you will ensure that the data you are logging in and using to keep things secure, will be secure.

In AWS, Always Be Watching

As a DevOps developer in the Amazon Web Services platform, you must always watch log data and users in the system.

Amazon CloudWatch tracks your AWS resources and applications. It also collects and tracks metrics, monitors log files, and deploy automated responses to everyday events in your AWS environment.

AWS CloudTrail gathers information about API calls within your AWS environment. It also reveals the caller’s identity, IP address, call request, and other data.

AWS Inspector is also a great automated tool that probes your AWS environment for vulnerabilities and provides a complete log report with the most common fixes and also with improvements for better security. Familiarity with all these tools and other key data sources and applications gives any developer a head start developing comprehensive logging practices in AWS.

Conclusion

Hence I can conclude by saying that CloudTrail, CloudWatch, and Splunk or any other custom logging solution and even AWS Inspector are all needed for a DevOps developer. Not just one of them is enough for the completion of any cloud environment. Together, all of these provide metrics, logging data, and security to the cloud environment of an enterprise and help the DevOps developer to gain experience, also making their life more comfortable.

Bonus

If you are a busy and small DevOps team, a product like Cloudanix can help you make sense of these logs in a much efficient way. We have to build Cloudanix, which abstracts some of these nuances and fosters productivity for your busy team. Give it a spin with its free trial and see how it can improve your security posture.

Start Your Free Trial Now!