Logging .NET to AWS CloudWatch: Understanding the basics using AWS SDK
Learn to log .NET applications to AWS CloudWatch using the AWS SDK in this step-by-step guide. It covers setting up the SDK, creating log groups and streams, and sending log messages programmatically.
AWS CloudWatch is a management tool in AWS that allows us to monitor resources in AWS and hosted elsewhere. One form of monitoring that AWS CloudWatch offers is ingesting logs from different services so that they can be browsed and queried from a single interface.
Many services in the AWS ecosystem feed their logs directly into CloudWatch by default without any configuration needed. This is convenient when debugging AWS services, but you might find yourself figuring out how to send your own application's logs to AWS (when AWS doesn't automatically forward them to AWS CloudWatch like it does with some services). This could happen if you are running your application on an EC2 instance or in a container.
In this article, we are going to look at how we can log messages to AWS CloudWatch in .NET. We will be using the AWS SDK to get a low-level, first principles understanding of how it's done before having a look at logging to AWS CloudWatch using popular logging libraries such as Serilog and NLog in future posts.
Logging .NET to AWS CloudWatch Series
- Understanding the basics using AWS SDK
- Using Serilog
- Using NLog
Video
Prerequisites
AWS Account
You will need an active AWS account that we can use to push logs from our .NET application to AWS CloudWatch. Our volumes will be minimal so we shouldn't exceed the free tier for AWS CloudWatch.
If you don't have an AWS account yet, you can sign up for one here.
AWS Profile
Since this article is just focusing on AWS CloudWatch, we won't be configuring any AWS credentials in our .NET application. Rather, we will let the AWS SDK automatically use our AWS profile that is set up when running the AWS CLI. This simplifies the demonstrations as we don't have to fumble around with permissions and credentials.
However, please note that you should always make sure to give applications their own IAM roles/users in production with minimal permissions according to AWS best practices.
Let's Go!
With that out the way, let's get started!
1. Create a new Console application
We will be using a .NET 6 Console application in this demonstration. You can create this project using either Visual Studio 2022 or the .NET 6 CLI.
2. Add the AWS SDK Nuget package
Before we can do anything with AWS CloudWatch from .NET, we need to install the AWSSDK.CloudWatchLogs Nuget package. This will allow us to make calls to the AWS CloudWatch service from .NET programmatically using an SDK developed by AWS.
3. Create a Logging class
We will be encapsulating our logging logic in a single Logging class (called CloudWatchLogger). This will neaten up our logging logic and keep it out of our application code.
While we are creating our own CloudWatchLogger, it will be a heavily simplified logger that aims to simply demonstrate how most logging libraries log to AWS CloudWatch. In production, it is advisable to use an existing logging library to log to AWS CloudWatch rather than try to roll your own.
4. Asynchronously construct our CloudWatchLogger class
When interacting with AWS services using the AWS SDK, all of our operations occur asynchronously. This is important to remember because we are going to do some initialization in AWS CloudWatch when creating our logger. To do this initialization, we need to be able to construct our CloudWatchLogger asynchronously.
You can see how we can do this using a private constructor and asynchronous static method:
CloudWatchLogger.cs
namespace AWSCloudWatchLoggingWithAWSSDK;
public class CloudWatchLogger
{
private CloudWatchLogger()
{
}
public static async Task<CloudWatchLogger> GetLoggerAsync()
{
var logger = new CloudWatchLogger();
// Do more initialization work here
return logger;
}
}
The pattern you're seeing is similar to the factory method of constructing objects where we have a method to create instances of an object.
5. Create the AmazonCloudWatchLogsClient
We first need to create an AmazonCloudWatchLogsClient that we will then use to call AWS CloudWatch methods. We will create a new instance in the constructor and assign it to a private field for later use:
CloudWatchLogger.cs
using Amazon.CloudWatchLogs;
namespace AWSCloudWatchLoggingWithAWSSDK;
public class CloudWatchLogger
{
private IAmazonCloudWatchLogs _client;
private CloudWatchLogger()
{
// If don't have an AWS Profile on your machine and application is hosted outside
// of AWS infrastructure (where IAM roles cannot be assigned to infrastructure),
// rather use:
// _client = new AmazonCloudWatchLogsClient(awsAccessKeyId, awsSecretAccessKey, RegionEndpoint.AFSouth1);
_client = new AmazonCloudWatchLogsClient();
}
public static async Task<CloudWatchLogger> GetLoggerAsync()
{
var logger = new CloudWatchLogger();
return logger;
}
}
6. Create a Log Group
In AWS CloudWatch, logs are grouped at a top-level into Log Groups. Each Log Group can belong to a different application or service in AWS. For example, a Log Group for an AWS Lambda function called first_python_function might look like this:
/aws/lambda/first_python_function
When we construct our new logger, we will need to create a new Log Group that we can log to:
CloudWatchLogger.cs
using Amazon.CloudWatchLogs;
using Amazon.CloudWatchLogs.Model;
namespace AWSCloudWatchLoggingWithAWSSDK;
public class CloudWatchLogger
{
private IAmazonCloudWatchLogs _client;
private string _logGroup;
private CloudWatchLogger(string logGroup)
{
// If don't have an AWS Profile on your machine and application is hosted outside
// of AWS infrastructure (where IAM roles cannot be assigned to infrastructure),
// rather use:
// _client = new AmazonCloudWatchLogsClient(awsAccessKeyId, awsSecretAccessKey, RegionEndpoint.AFSouth1);
_client = new AmazonCloudWatchLogsClient();
_logGroup = logGroup;
}
public static async Task<CloudWatchLogger> GetLoggerAsync(string logGroup)
{
var logger = new CloudWatchLogger(logGroup);
// Create a log group for our logger
await logger.CreateLogGroupAsync();
return logger;
}
private async Task CreateLogGroupAsync()
{
_ = await _client.CreateLogGroupAsync(new CreateLogGroupRequest()
{
LogGroupName = _logGroup
});
}
}
While this might look like it should work, after calling our constructor a second time, you will get the following exception:
As you've probably already guessed, this exception is being thrown because we're trying to create a Log Group that already exists! We need some way of making sure we don't crash the application because the log group already exists. There are a few techniques for doing this. We will take a rather simple and naive approach and update our CreateLogGroupAsync method as follows:
CloudWatchLogger.cs
using Amazon.CloudWatchLogs;
using Amazon.CloudWatchLogs.Model;
namespace AWSCloudWatchLoggingWithAWSSDK;
public class CloudWatchLogger
{
// ...
private async Task CreateLogGroupAsync()
{
var existingLogGroups = await _client.DescribeLogGroupsAsync();
if (existingLogGroups.LogGroups.Any(x => x.LogGroupName == _logGroup))
return;
_ = await _client.CreateLogGroupAsync(new CreateLogGroupRequest()
{
LogGroupName = _logGroup
});
}
}
This will now fetch a list of all the Log Groups in AWS and check if our Log Group already exists. If it does, we return before creating a new Log Group.
Let's do a quick test to see what happens when we create a new instance of our logger:
Program.cs
using AWSCloudWatchLoggingWithAWSSDK;
var logger = await CloudWatchLogger.GetLoggerAsync("/dotnet/logging-demo/awssdk");
AWS CloudWatch Console
You can see that we have a new Log Group created in the AWS CloudWatch Console now!
7. Create a Log Stream
Previously we created a Log Group to contain logs for our application. In AWS CloudWatch, our logs are grouped again into Log Streams. Ideally, these Log Streams contain log events from a single invocation or request in an application. For example, if you had a web application, you would group a single request sent to your web application in a Log Stream.
In our example, we are going to create a Log Stream in the constructor and reuse that Log Stream throughout the lifetime of that instance of the CloudWatchLogger class:
CloudWatchLogger.cs
using Amazon.CloudWatchLogs;
using Amazon.CloudWatchLogs.Model;
namespace AWSCloudWatchLoggingWithAWSSDK;
public class CloudWatchLogger
{
// ...
private string _logStream;
// ...
public static async Task<CloudWatchLogger> GetLoggerAsync(string logGroup)
{
var logger = new CloudWatchLogger(logGroup);
// Create a log group for our logger
await logger.CreateLogGroupAsync();
// Create a log stream
await logger.CreateLogStreamAsync();
return logger;
}
// ...
private async Task CreateLogStreamAsync()
{
_logStream = DateTime.UtcNow.ToString("yyyyMMddHHmmssfff");
_ = await _client.CreateLogStreamAsync(new CreateLogStreamRequest()
{
LogGroupName = _logGroup,
LogStreamName = _logStream
});
}
}
If we now run our program again and look in AWS CloudWatch, we should see a newly created Log Stream in our Log Group:
Awesome!
8. "Put" log messages into AWS CloudWatch
Now that we've set up a Log Group and Log Stream, we're finally ready to start sending log messages to AWS CloudWatch! The AWS SDK provides a method called PutLogEventsAsync which allows us to upload a batch of log events to AWS CloudWatch.
For simplicity's sake, we are going to call this method every time we log a message, however, we can exceed our maximum requests per second very quickly this way. Hence, it is best practice to batch these log events when sending them through to AWS CloudWatch.
Let's create a LogMessageAsync method which will write our log message to AWS CloudWatch when called:
CloudWatchLogger.cs
using Amazon.CloudWatchLogs;
using Amazon.CloudWatchLogs.Model;
namespace AWSCloudWatchLoggingWithAWSSDK;
public class CloudWatchLogger
{
// ...
public async Task LogMessageAsync(string message)
{
_ = await _client.PutLogEventsAsync(new PutLogEventsRequest()
{
LogGroupName = _logGroup,
LogStreamName = _logStream,
LogEvents = new List<InputLogEvent>()
{
new InputLogEvent()
{
Message = message,
Timestamp = DateTime.UtcNow
}
}
});
}
// ...
}
Let's run the following program and see what happens:
Program.cs
using AWSCloudWatchLoggingWithAWSSDK;
var logger = await CloudWatchLogger.GetLoggerAsync("/dotnet/logging-demo/awssdk");
await logger.LogMessageAsync("This is my first message!");
await logger.LogMessageAsync("this is another message much like the first!");
You will see that we get the following exception when trying to call LogMessageAsync a second time:
This is caused by not providing a Sequence Token when putting log events into the Log Stream. The first time we call PutLogEventsAsync on an empty Log Stream, the Sequence Token is not required because the Log Stream is empty. But, once we've written messages to the Log Stream, subsequent calls need to provide a Sequence Token so that they can be written in the correct order.
Fortunately, this isn't too hard to fix in our simple demonstration:
CloudWatchLogger.cs
using Amazon.CloudWatchLogs;
using Amazon.CloudWatchLogs.Model;
namespace AWSCloudWatchLoggingWithAWSSDK;
public class CloudWatchLogger
{
//...
private string _nextSequenceToken = null;
// ...
public async Task LogMessageAsync(string message)
{
var response = await _client.PutLogEventsAsync(new PutLogEventsRequest()
{
LogGroupName = _logGroup,
LogStreamName = _logStream,
SequenceToken = _nextSequenceToken,
LogEvents = new List<InputLogEvent>()
{
new InputLogEvent()
{
Message = message,
Timestamp = DateTime.UtcNow
}
}
});
_nextSequenceToken = response.NextSequenceToken;
}
// ...
}
We've now created a private field to store the next Sequence Token that should be used the next time we insert log events into the Log Stream. Initially, the private field is set to null, but that's not a problem for our implementation because we know that our first request to log a message to the Log Stream will always be to an empty Log Stream. Thereafter, we get the NextSequenceToken value from the response object and store it in the private field for next time.
And that solves the issue! You should now see our two, beautiful logged messages in AWS CloudWatch:
Closing
That's all for this first post in the Logging .NET to AWS CloudWatch series. The goal of this post was to give you a first-principles, hands-on demonstration of how we use the AWS SDK to log to AWS CloudWatch.
It is a very simplified demonstration, but it paves the way for understanding how existing logging libraries implement this functionality such as Serilog or NLog. We will focus on implementing logging to AWS CloudWatch using these libraries in the next few posts in this series.