Lessons Learned: Integrating AWS MAC Instance in the Splunk Attack Range

note this blog was written by Teoderick and myself 🥷🕵️‍♂️

In November 2020, when AWS announced the general availability of macOS instances in AWS EC2 Cloud, the Splunk Threat Research Team took this opportunity to test and implement the instance as an additional endpoint in the Splunk Attack Range. Adding the ability to build MacOS for detection development and testing is a feature we have wanted to include since the project’s inception. Having a MacOS endpoint would allow us to not only simulate attacks against this new OS, but also do it programmatically and replicably using the Attack Range framework.

In this blog, we will discuss the approaches, challenges and lessons learned trying to add this instance with Terraform. However, due to the current AWS service limitations that are needed to be compatible with this project, we cannot implement the MacOS feature within Splunk Attack Range.

In this blog, we will discuss the approaches, challenges and lessons learned trying to add a MacOS instance with Terraform. However, due to the current AWS service limitations that are needed to be compatible with this project, we cannot implement the MacOS feature within Splunk Attack Range.

Building a MacOS instance in AWS Using Terraform

Under the hood, the Splunk Attack Range uses Terraform to build the environment (machine, networking, routing, etc.) and Ansible to configure the machines and applications like logging agents and OS parameters. To be able to build an AWS MacOS instance using Terraform you need two things:

• Instance Type: mac1.metal
• Region: MAC OS instances are not available in all regions, so make sure it is available to the region where you are planning to build it.

Note that MacOS instances need to be placed onto a “Dedicated Host”. The minimum lease time for a dedicated host is 24 hours.

For our implementation in the Attack Range, we leverage a Terraform community module created by Daniel Diaz that creates a dedicated host with a mac1.metal instance. The Terraform resource.tf file we ended building as a proof of concept in the Splunk Attack Range is shown below.

data "aws_ami" "macos" {
  count       = var.config.mac_os_machine == "1" ? 1 : 0
  most_recent = true
  owners      = ["amazon"]

  filter {
    name   = "name"
    values = ["amzn-ec2-macos-10.15*"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }
}

module "dedicated-host" {
  source            = "DanielRDias/dedicated-host/aws"
  version           = "0.3.0"
  instance_type     = "mac1.metal"
  availability_zone = "eu-west-1a"
  cf_stack_name     = "mac-stack"

    tags = {
    Name = "Terraform Mac"
  }
}

resource "aws_instance" "mac_os_machine" {
  count                  = var.config.mac_os_machine == "1" ? 1 : 0  
  ami                    = data.aws_ami.macos[count.index].id
  instance_type          = var.config.mac_os_instance_type
  key_name               = var.config.key_name
  subnet_id              = var.ec2_subnet_id
  vpc_security_group_ids = [var.vpc_security_group_ids]
  host_id                = module.dedicated-host.dedicated_host_id
  private_ip             = var.config.mac_os_private_ip
  tags = {
    Name = "ar-macos-${var.config.range_name}-${var.config.key_name}"
  }


  provisioner "remote-exec" {
    inline = ["echo booted"]

    connection {
      type        = "ssh"
      user        = "ec2-user"
      host        = aws_instance.mac_os_machine[count.index].public_ip
      private_key = file(var.config.private_key_path)
      timeout    = "60m"
    }
  }

}

output "dedicated_host_id" {
  description = "Dedicated Host ID"
  value = module.dedicated-host.dedicated_host_id
}

resource "aws_eip" "macos_ip" {
  count    = var.config.mac_os_machine == "1" && var.config.use_elastic_ips == "1" ? 1 : 0
  instance = aws_instance.mac_os_machine[0].id

}

With this proof-of-concept Terraform script, we were able to successfully build a MacOS instance and connect to it via ssh port as we can see in the screenshot below.

From here our plan was to build the necessary Ansible tasks to configure the MacOS instance for data collection, install Splunk Universal Forwarder and configure logging tools like OSquery, etc., but we quickly ran into a handful of challenges.

Challenges and Lesson Learned

Timeout

As mentioned in this article, the creation and deletion time of AWS “mac1.metal” instance is much longer than the Linux EC2 instance. We also encountered some ssh timeouts during the building of the mac os machine. We can see in the screenshot that 34.245.2.165 (MAC OS ip address) has a timeout in port 22.

But this issue can be easily solved by increasing the timeout in our remote-exec provisioner function on the Terraform script.

provisioner "remote-exec" {
  inline = ["echo booted"]

  connection {
    type        = "ssh"
    user        = "ec2-user"
    host        = aws_instance.mac_os_machine[count.index].public_ip
    private_key = file(var.config.private_key_path)
    timeout    = "60m"
  }
}

Dedicated Host Leasing Time

As noted earlier, the minimum lease for a dedicated host is 24 hours in AWS. A user needs to wait for a day before they can release or terminate the dedicated host, and essentially have a new “clean” MacOS instance for testing again. This goes against the operational premise of the Attack Range, that all machines are ephemeral and when you run python attack_range.py destroy, all instances are successfully deleted/cleaned. In this case, not being able to “clean” MacOS instances for 24 hours will increase the cost of usage to any Attack Range user, as well as break one of the core functions of the Attack Range (destroy).

Finally, not all AWS regions support the mac1.metal instance type, which is also another consideration that needs to be considered when supporting the feature in the Attack Range.

Conclusion

Kudos to the AWS Cloud team for adding support for MacOS instance types as part of their EC2 services. We are excited to see cloud providers support endpoint platforms like these that are common in enterprise environments. This is a great start, and also helps application developers and the security industry research using MacOS. Unfortunately, until a dedicated host can be destroyed without a minimum lease time (instantly via API), we likely won’t be able to support mac1.metal instances in the Splunk Attack Range. One of the attack range goals is to have a platform where you can build and destroy lab infrastructure as quickly as possible to simulate an attack and to develop detections.

In this scenario, the Splunk Threat Research Team decided to implement the MacOS instance in Attack Range in the near future as soon as we find solutions to the challenges and resource compatibility issues of the mac1.metal instance.