How to: AWS OAM Sink and Link

How and Why to Setup Cross-Account Logging in AWS

I recently implemented cross-account logging using AWS Observability Access Manager (OAM) for a comprehensive monitoring application my team and I are building. In this, we are looking to create one UI to understand performance and costs across the products in our domain. When setting up cross-account logging, I struggled to find clear documentation on how to do so in Terraform and made a few mistakes with permissions until I understood how OAM policies work. When troubleshooting, ChatGPT made up some silliness that did not make sense, and so this article aims to help you solve a problem that took me a bit longer to solve than I think it should have.

Cross-account logging is great if you want to colocate logs from multiple accounts. This approach can help you understand portfolio-level questions like: How are all the API gateways in your domain performing? What is the breakdown of response types per endpoint? What is causing latency in your application? Do all applications in your domain experience similar metric patterns?

AWS OAM

Observability Access Manager is a game-changer for organizations managing observability data across multiple AWS accounts and regions. Instead of manually setting up complex IAM roles, policies, and log subscriptions, OAM provides a seamless, scalable, and secure way to unify metrics, logs, and traces into a centralized monitoring account.

Prior to OAM’s release in 2020, you might have had to ferry your data between accounts using Kinesis Data Firehose, Kinesis Data Streams, or a slightly more complex architecture involving a lambda, kafka, and some complex permissioning. Fortunately, times have changed.

With OAM, you can:

• Effortlessly share CloudWatch metrics, logs, and X-Ray traces across accounts and regions.
• Eliminate the need for manual log forwarding — no more subscription filters or Kinesis streams!
• Use a single-pane-of-glass dashboard in CloudWatch to monitor applications spanning multiple AWS environments.
• Improve security and governance by managing access with fine-grained resource-based policies.
• Reduce costs by consolidating observability data instead of duplicating logs and metrics. This approach does not yield any additional costs over the initial generation of the data.

As organizations scale, their AWS footprint often spans multiple accounts. Factors such as organizational dynamics, poor governance, security concerns, compliance requirements, and cost management drivers can create account sprawl. AWS OAM simplifies observability by connecting all this data — ensuring DevOps, SREs, and security teams can troubleshoot faster, optimize resources, and detect anomalies in real time.

Sinking

A sink is a central resource within AWS OAM for receiving observability data that needs to be created in the central logging account. These pieces of code below need to be run first before the link is established so that the policy document can be attached to the sink, thus confiring permissions for any linking accounts. If this is not setup, linking attempts will fail.

You can use the code below and change the resource names according to your needs, but you’ll need to setup the account IDs for the accounts you plan to link through your locals.

While we only needed to share CloudWatch metrics and log groups, additional conditions can be set depending on your application architecture. XRay and ApplicationInsights are two other services that might be relevant here and have their own set of resource types you can add to the list in the policy document below.

resource "aws_oam_sink" "monitoring_sink" {
  name = "monitoring-sink"
}

data "aws_iam_policy_document" "cross_account_policy_doc" {
  statement {
    actions = [
            "oam:CreateLink",
            "oam:UpdateLink"
            ]
    principals {
       type = "AWS"
       identifiers = ["${local.account_one}","${local.account_two}"]
    }
    effect = "Allow"
    resources = ["*"]
    condition {
      test = "ForAllValues:StringEquals"
      variable = "oam:ResourceTypes"
      values = ["AWS::CloudWatch::Metric", "AWS::Logs::LogGroup"]
    }
  }
}

resource "aws_oam_sink_policy" "monitoring_sink_policy" {
  sink_identifier = aws_oam_sink.monitoring_sink.id
  policy = data.aws_iam_policy_document.cross_account_policy_doc.json
}

Linking

Any accounts sharing logs with the sink will need to have the following adding to their Terraform repositories.

The sink_identifier should match the region, account, and sink_id shown in the Terraform output for the sink creation.

resource "aws_oam_link" "account_one_source_link" {
  label_template = "account_one_source_link"
  sink_identifier = "arn:aws:oam:${local.region}:${local.monitoring_account_id}:sink/${local.sink_id}"
  resource_types = ["AWS::CloudWatch::Metric", "AWS::Logs::LogGroup"]
}

Finally, you can also add the link_configuration parameter to aws_oam_link to filter out different log groups that arent relevant to your cross-account monitoring efforts.

You should be able to validate these steps worked because not only should your Terraform succeed, but you should see CloudWatch Log Groups from other accounts in your monitoring account amongst its CloudWatch Log Groups.

Architecture

The above diagram shows a lean view of this application architecture. Of course, there are many networking, security, authentication, and authorization components not included in this image such as Azure AD, AWS WAF, Route53, and CloudFront.

Next Steps

There are a few options for creating cross-account dashboards that we tinkered with. If you are looking for an internal developer dashboard, a CloudWatch dashboard may very well suffice. This worked well for us in the beginning when we were trying to show the usefulness of the concept of cross-account log sharing and that the same metrics could be viewed for different accounts in an often directly comparable way.

However, we ended up building a more comprehensive dashboard in QuickSight. There are a few reasons this worked better for our product:

1. The CloudWatch dashboard approach did not have simple permissions management for embedding in a webapp for our various users. The options for this seemed to be using an <iframe>, but the user has to have IAM permissions. This was not exactly useful for our leadership users.
2. QuickSight dashboards are more customizable. For example, we could integrate error messages with tables to paint a comprehensive picture of the application performance at each moment in time.
3. QuickSight dashboards have a cleaner UI that lets us easily toggle between different timeframes and products.
4. QuickSight permissions are configurable at the row level, which let us filter dashboards to specific user groups.
5. We were already familiar with embedding using the QuickSight Embedding SDK and understood the paradigm for using Active Directory permissions over the top of that to provide more granular permissions.

Conclusion

Cross-account logging was one piece of a larger project where we setup monitoring of various processes and workflows in a centralized application dashboard. In addition to the approach and code I’ve shared above, we built an accompanying workflow using cross-account eventing to understand performance of specific workflows.

Here at New Math Data, we recommend looking into creating something similar to understand performance and monitoring of applications within a specific domain of your business. Now that you have some ready-to-use Terraform you can setup some cross-account CloudWatch dashboards in just a few hours! Please reach out or comment below if you would like to discuss more!