Serilog and AWS CloudWatch Logging for .NET 6: A Step-by-Step Tutorial

Logging is an essential part of any web application development. It helps you to track errors, debug issues, monitor performance, and improve user experience. However, logging can also be challenging, especially when you have to deal with large volumes of data, multiple sources, and different formats.

That's why in this blog post, I will show you how to use Serilog and AWS CloudWatch Logging for .NET 6 web applications. Serilog is a popular and powerful library supporting structured logging and various sinks. AWS CloudWatch Logging is a cloud-based service that allows you to collect, store, analyze, and visualize your logs in a centralized place.

By following this step-by-step tutorial, you will learn how to:

• Set up Serilog and AWS CloudWatch Logging for your .NET 6 web application
• Configure Serilog to send logs to AWS CloudWatch Logging using AWS.Logger.Serilog sink
• Use Serilog enrichers to add contextual information to your logs
• Query and filter your logs using AWS CloudWatch Insights
• Create dashboards and alarms using AWS CloudWatch Metrics and Alarms

Ready to get started? Let's dive in!

Overview of Serilog and AWS CloudWatch

Before we dive into the tutorial, let's briefly review what Serilog and AWS CloudWatch are and why they are useful for logging.

Serilog

Serilog is a user-friendly logging library that provides structured logging and powerful support for structured diagnostic data. Serilog can be used to log information about application events, errors, and performance metrics. It is designed for .NET frameworks. Some of the features of Serilog are:

• Built-in support for structured logging, enabling logs to be treated as datasets rather than text.
• Seamless compatibility with asynchronous applications and systems.
• Flexible logging targets, including files, console, email, and other customizable outputs.
• Convenient message templates that simplify object serialization using the "@" operator.

Serilog uses message templates, a simple DSL that extends .NET format strings with named as well as positional parameters. Instead of formatting events immediately into text, Serilog captures the values associated with each named parameter.

Serilog’s support for structured event data opens up a huge range of diagnostic possibilities not available when using traditional loggers. You can query and filter your logs based on the properties, create dashboards and metrics from them, and leverage the power and flexibility of the cloud for your logging needs.

AWS CloudWatch

AWS CloudWatch is a comprehensive logging and monitoring service by Amazon Web Services. It provides a wide range of features for monitoring and troubleshooting resources within an AWS environment, including logging capabilities. It allows you to collect, store, analyze, and visualize your logs in a centralized place. Some of the features of AWS CloudWatch are:

Log Groups: Log groups are collections of log streams that have the same retention settings, access control, and monitoring parameters. Log groups organize and manage logs by providing a logical grouping for logs created by various AWS services or applications.

Log Streams: Log streams are sequences of log events that have the same source. Each log stream represents a distinct source of logs, such as an EC2 instance, a Lambda function, or an application running in an ECS container.

Retention Policies: Logs are stored in Cloudwatch forever by default. You may further customize this to your needs by changing the retention duration from 1 day to up to 10 years. Properly configuring the retention period also helps you save money on your monthly AWS bill, particularly for Production applications.

Log Data Collection: CloudWatch offers several methods for collecting log data. It collects logs directly from Amazon resources such as EC2 instances, Lambda functions, and CloudTrail. It also provides APIs and SDKs for sending custom logs from EC2 instances or on-premises systems.

Log Search and Analysis: CloudWatch has robust log search and analysis features. CloudWatch Logs Insights is a built-in query language that allows you to search and analyze log data using extensive filtering and aggregation tools. It also supports real-time monitoring of logs, making it easier to troubleshoot issues and gain insights into the behavior of your resources.

Monitoring and Alerting: CloudWatch allows you to configure metric filters and alerts on logs, which helps in monitoring and alerting certain log events or patterns. For example, you may configure an alert to warn you when the number of failures in your application logs surpasses a specific level. This provides proactive monitoring and alerting based on log data, assisting you in swiftly identifying and resolving issues.

Cloudwatch can also respond to events. For instance, you can create an alarm 🔔for a particular event which will then notify your team.

AWS CloudWatch Logging integrates well with Serilog using AWS.Logger.Serilog sink. This sink allows you to send your Serilog events to AWS CloudWatch Logging with minimal configuration. You can then use AWS CloudWatch Logging to view, search, filter, query, and visualize your Serilog events in the cloud.

By combining Serilog and AWS CloudWatch Logging, you can create a powerful and flexible logging solution for your .NET web application.

Setting Up Serilog with .NET 6

Serilog is a popular open-source logging framework for .NET applications that lets you capture, store, and query log events in a flexible and extensible manner. It enables you to set logging configuration in code, making it extremely versatile and adaptable to various logging needs.

Let's install and configure Serilog in our .NET 6 web API project. Open up the package manager console and run the following command.

dotnet add package Serilog.AspNetCore

Now, let's open the Program.cs file and configure Serilog in the web host.

using Serilog;

Log.Logger = new LoggerConfiguration()
    .WriteTo.Console()
    .CreateBootstrapLogger();

try
{
    Log.Information("Starting web application");

    var builder = WebApplication.CreateBuilder(args);

    builder.Host.UseSerilog((ctx, services, config) => config
    .ReadFrom.Configuration(ctx.Configuration));

    builder.Services.AddControllers();
    builder.Services.AddEndpointsApiExplorer();
    builder.Services.AddSwaggerGen();
    
    var app = builder.Build();
    app.UseSerilogRequestLogging();
    
    app.UseHttpsRedirection();
    app.UseAuthorization();
    app.MapControllers();
    app.Run();
}
catch (Exception ex)
{
    Log.Fatal(ex, "Application terminated unexpectedly");
}
finally
{
    Log.CloseAndFlush();
}

There are two ways to configure Serilog. The first is via the code in the startup method (in the Program.cs file) and the second is via app settings. We will use the app settings way. Before configuring the appsettings.json file, first let's understand the above code snippets.

Line #3-5: set up a static Log.Logger instance with two-stage initialization and write output to the console. An initial "bootstrap" logger is configured immediately when the program starts, and this is replaced by the fully configured logger once the host has loaded.

Line #13-14: We are instructing the Serilog Middleware to read the configurations from the appsettings.

Line #21: The built-in request logging is noisy with multiple events emitted per request. To enable this middleware, first, we need to override the default log level for Microsoft.AspNetCore to Warning in our appsettings.json file.

They are condensed by the provided middleware into a single event that has more manageable information. When we run the API now, we can clearly see the difference in the events emitted.

We have also added a try / catch block that will ensure any configuration-related issues are properly logged.

Next, let's add the Serilog configuration in the appsettings.json.

{
  "Serilog": {
    "Using": [ "Serilog.Sinks.Console"],
    "MinimumLevel": {
      "Default": "Information",
      "Override": {
        "Microsoft": "Warning",
        "Microsoft.AspNetCore": "Warning",
        "Microsoft.Hosting.Lifetime": "Warning"
      }
    },
    "WriteTo": [
      {
        "Name": "Console",
        "Args": {
          "outputTemplate": "[{Timestamp:G} [{Level:u3}] {Message}{NewLine}{Exception}"
        }
      }
    ],
    "Enrich": [ "FromLogContext", "WithMachineName", "WithThreadId" ]
  },
  "AllowedHosts": "*"
}

That's it, we now have Serilog configured in our web API project.

Setting Up AWS CloudWatch Logging with Serilog

AWS CloudWatch is a centralized and highly scalable logging service that collects, stores, and analyzes logs from multiple AWS services and custom applications.

There are different ways to integrate AWS CloudWatch logging for your .NET 6 application.

You can use the AWSSDK.CloudWatchLogs NuGet package that provides AmazonCloudWatchLogsClient to interact with CloudWatch logs. This approach requires you to write a little bit more code. Why repeat yourself if it's already done for you 🤔.

You may also use the Amazon logging provider for .NET. The GitHub project AWS Logging Dotnet provides a plugin for it that also utilizes AmazonCloudWatchLogsClient, which abstracts all of the complexities and mechanics of communicating with CloudWatch.

Serilog offers flexible configuration options for sinks, allowing you to receive log messages via a configuration file or programmatically through code. We will go through each method but first, let's install the following required NuGet package.

dotnet add package AWS.Logger.SeriLog

Configuring Serilog Sinks with AWS CloudWatch via a Configuration File

Up to this point, we have already configured Serilog in our application via the configuration file. Now let's open the appsettings.json file and modify it to write logs to CloudWatch.

{
  "Serilog": {
    "Using": [ "Serilog.Sinks.Console", "AWS.Logger.SeriLog" ],
    "LogGroup": "JokeFetcher",
    "Region": "eu-west-1",
    "MinimumLevel": {
      "Default": "Information",
      "Override": {
        "Microsoft": "Warning",
        "Microsoft.AspNetCore": "Warning",
        "Microsoft.Hosting.Lifetime": "Warning"
      }
    },
    "WriteTo": [
      {
        "Name": "Console",
        "Args": {
          "outputTemplate": "[{Timestamp:G} [{Level:u3}] {Message}{NewLine}{Exception}"
        }
      },
      {
        "Name": "AWSSeriLog",
        "Args": {
          "textFormatter": "Serilog.Formatting.Json.JsonFormatter, Serilog"
        }
      }
    ],
    "Enrich": [ "FromLogContext", "WithMachineName", "WithThreadId" ]
  },
  "AllowedHosts": "*"
}

Let's go through the changes that we have made to this file.

Line #3: We added AWS.Logger.Serilog in the using array.

Line #4: Added log group name that will be used to segregate the logs. We want to separate the log group for each application. In this case, we have supplied JokeFetcher which is the name I came up with for this demo project.

Line #5:  This is the AWS region where you want to store the logs. This is optional if you have configured an AWS profile on your machine because it is going to fetch it from there.

Line #22: We added AWSSerilog sink to the WriteTo Section. WriteTo Section accepts a list of sinks where you would want it to write logs. In our case, we have two sinks, the first one is the console and the second one is AWS CloudWatch.

Line #23-24: We have added a text formatter that will prettify the logs into clean JSON messages which is useful for structured logging. We will see its benefits in the filtering section.

Serilog automatically searches for AWS credentials using the standard .NET credentials search path, which includes checking for a profile named "default", environment variables, or an instance profile on an EC2 instance. To use a different profile other than "default", simply add a "Profile" under the "Serilog" node in your configuration. This allows you to easily customize the AWS credentials used by Serilog for authenticating with AWS services, ensuring secure and seamless logging in your .NET 6 application.

{
  "Serilog": {
    "Profile": {
      "Region": "your-aws-region",
      "AccessKey": "your-access-key",
      "SecretKey": "your-secret-key"
    }
  }
}

Programmatic Configuration of Serilog Sinks with AWS CloudWatch

Next, let's configure the AWS sink for Serilog using code. Simply use the "WriteTo" method to add the AWS sink to the logger. This provides you with the flexibility to programmatically configure the AWS logging sink for Serilog in your .NET 6 application, allowing you to easily customize your logging setup according to your requirements.

    AWSLoggerConfig configuration = new("JokeFetcher")
    {
        Region = "eu-west-1"
    };

    builder.Host.UseSerilog((ctx, services, config) => config
    .ReadFrom.Configuration(ctx.Configuration)
      .WriteTo.AWSSeriLog(configuration, textFormatter: new RenderedCompactJsonFormatter())
    );

Writing Log Events with Serilog and AWS CloudWatch

Once you've completed these steps, your application will be ready to write logs to AWS CloudWatch. Next, let's add a controller to test it.

using JokeFetcherAPI.Models;
using Microsoft.AspNetCore.Mvc;
using System.Text.Json;

namespace JokeFetcherAPI.Controllers
{
    [Route("api/[controller]")]
    [ApiController]
    public class JokesController : ControllerBase
    {
        private readonly ILogger<JokesController> _logger;
        private readonly IHttpClientFactory _httpClientFactory;

        public JokesController(IHttpClientFactory httpClientFactory, ILogger<JokesController> logger)
        {
            _httpClientFactory = httpClientFactory;
            _logger = logger;
        }

        [HttpGet]
        public async Task<IActionResult> GetJoke()
        {
            JokeApiResponse jokeApiResponse;
            try
            {
                var httpClient = _httpClientFactory.CreateClient("Jokes");

                _logger.LogInformation("Getting programming jokes.");

                var httpResponseMessage = await httpClient.GetAsync("Programming");

                if (!httpResponseMessage.IsSuccessStatusCode)
                {
                    _logger.LogError("Failed to retrieve joke from external API with status code:{code}", StatusCode((int)httpResponseMessage.StatusCode));
                    return StatusCode((int)httpResponseMessage.StatusCode);
                }

                var joke = await httpResponseMessage.Content.ReadAsStringAsync();
                jokeApiResponse = JsonSerializer.Deserialize<JokeApiResponse>(joke)!;

                _logger.LogInformation("Got programming joke with ID:{jokeId} and category:{jokeCategory}", jokeApiResponse!.Id, jokeApiResponse.Category);

            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "An error occurred while retrieving joke");
                return StatusCode(StatusCodes.Status500InternalServerError, "An error occurred while retrieving joke.");
            }
            return Ok(jokeApiResponse);
        }
    }
}

Here, we have injected the Logger instance in the constructor via DI. Next, we have added a GET endpoint that will fetch a joke from an external API and would log some messages.

Now, let's test the endpoint from the postman.

Now that we have a 200 success status code from our API endpoint, we should have:

• A log group named JokeFetcher was created on AWS CloudWatch.
• Logs would have been logged to this log group.

Let's go to CloudWatch in the AWS Management Console to verify this.

You can see that the new log group has been successfully created. By clicking on the log group name, you can view the logs it contains, organized into log streams based on timestamps.

We have one log stream created inside the JokeFetcher log group. Click on the log stream to view the application logs.