Infrastructure as Code without Infrastructure
Introduction
Infrastructure as Code (IaC) is a powerful process - replacing manual, error prone and expensive operations with automated, consistent and quick provisioning of resources. In many cases, IaC is dependent on existing infrastructure, typically including a configuration management system. Chef, Puppet and SaltStack are all commonly referenced players in this market, each requiring resources to be in place and having their own difficulties in setup and maintenance. As we move to microservices and container orchestration, our need for resource-intensive and complex tooling to provision infrastructure and application dependencies diminishes. So how do you solve the chicken-and-egg problem of standing up IaC without relying on other infrastructure?
Our solution in Amazon Web Services (AWS) was Terraform, cloud-init, Minimal Ubuntu and Ansible. Terraform was an easy choice given our existing use and expertise with the product for provisioning in AWS. We were building Amazon Machine Images (AMIs) using Packer with a minimal set of software packages to bootstrap systems for dynamic configuration based on their role by our configuration management system. However, every change, no matter how subtle it was, required building a new AMI. It also didn't save much on boot time since an agent would configure the system dynamically at first boot-up. We were also spending a lot of time maintaining a configuration management system and scripts, as well as keeping up on Domain Specific Languages (DSLs).
Minimal Ubuntu
Enter Minimal Ubuntu - images designed for automating deployment at scale with an optimized kernel and boot process. Needing only to install a small set of packages and most of our tooling at the orchestration layer, we are still able to provision a system that is ready for production traffic in under four minutes. The simplicity of these images also provide greater security and ease of administration.
Cloud-init is installed on Minimal Ubuntu, which allows further configuration of the system using user data. Given the lack of documentation and more sophisticated features of other configuration management systems, we were still looking for something else. Ansible became an attractive option for several reasons: simplistic yet powerful approach to automation, readable configuration and templating using YAML and Jinja2 versus a DSL, and the community contributions and industry embracement.
Ansible
Most of the documentation for Ansible, though, focuses on the use of a master server that pushes configuration to clients. This doesn't solve the problem of IaC without relying on infrastructure. Also, maintaining dynamic inventories of clients and pushing configurations to systems in auto scaling groups that need to be ready for production traffic as soon as possible did not make sense. Ansible has a concept of local playbooks, but there isn't much light shed on the power and simplicity of it. This blog post will walk you through combining these tools to build a bastion host configured with Duo Multi-Factor Authentication (MFA) for SSH and a framework to easily add additional host roles. For brevity, other configuration of our bastion hosts is left out. You will want to perform further tuning and hardening depending on your environment.
Terraform
Starting with Terraform (note all examples are using version 0.12.x) at the account/IAM level, you will need a EC2 instance profile with access to an S3 bucket where the Ansible playbook tarball will be stored. Terraform for creating the S3 bucket is left to the reader - it is straightforward, and many examples exist for it. It is recommended to enable encryption at rest on the S3 bucket as sensitive information may be required to bootstrap a host:
data "aws_iam_policy_document" "ansible" {
statement {
actions = [
"s3:ListBucket",
"s3:GetObject",
]
resources = ["${aws_s3_bucket.ansible.arn}/*"]
}
}
resource "aws_iam_policy" "ansible" {
name = "ansible"
description = "Access to the Ansible S3 bucket"
policy = data.aws_iam_policy_document.ansible.json
}
data "aws_iam_policy_document" "bastion" {
statement {
actions = ["sts:AssumeRole"]
principals {
type = "Service"
identifiers = ["ec2.amazonaws.com"]
}
}
}
resource "aws_iam_role" "bastion" {
name = "bastion"
assume_role_policy = data.aws_iam_policy_document.main.json
}
resource "aws_iam_role_policy_attachment" "bastion" {
role = aws_iam_role.bastion.name
policy_arn = aws_iam_policy.ansible.arn
}
resource "aws_iam_instance_profile" "bastion" {
name = aws_iam_role.bastion.name
role = aws_iam_role.bastion.name
}}
With a policy to read the S3 bucket and an instance profile the bastion host can assume, define the bastion host EC2 instance:
resource "aws_instance" "main" {
ami = var.ami
instance_type = var.instance_type
user_data = data.template_cloudinit_config.main.rendered
key_name = var.ssh_key
iam_instance_profile = "bastion"
subnet_id = var.subnet_id
vpc_security_group_ids = [var.vpc_security_group_ids]
associate_public_ip_address = true
}
Most variables are self-explanatory. For this exercise, we will bring attention to the ami and user_data values. The ami value can be found by selecting the version of Ubuntu and the Amazon region for your instance here: https://wiki.ubuntu.com/Minimal.
The user_data value defines the cloud-init configuration:
data "aws_region" "current" {}
data "template_cloudinit_config" "main" {
gzip = true
base64_encode = true
part {
filename = "init.cfg"
content_type = "text/cloud-config"
content = templatefile("${path.module}/cloud-init.cfg", {})
}
part {
content_type = "text/x-shellscript"
content = templatefile(
"${path.module}/cloud-init.sh.tmpl",
{
ROLE = var.role
ENV = var.environment
VPC = var.vpc
REGION = data.aws_region.current.name
}
)
}
}
The cloud-init.cfg specifies a minimal configuration - installing the AWS CLI tool and Ansible to handle the rest of the process:
# Package configuration
apt:
primary:
- arches: [default]
apt_update: true
package_upgrade: true
packages:
- ansible
- awscli
write_files:
- path: /etc/apt/apt.conf.d/00InstallRecommends
owner: root:root
permissions: '0644'
content: |
APT::Install-Recommends "false";
The shell script following the cloud-init template downloads the Ansible playbook tarball and executes it. Variables for the environment (dev, stage, prod), VPC name and AWS region are passed to customize the configuration based on those settings. The role variable is passed as a tag to define what role the host will play, somewhat correlating to Ansible roles (explained later):
#!/bin/sh
# HOME is not defined for cloud-init
# Ansible, and likely others, don't like that
HOME=/root
export HOME
cd /opt
aws s3 cp s3://s3-bucket-name/ansible.tar.gz .
if [ $? != 0 ]; then
echo "Error: Cannot download from S3, check instance profile."
exit 1
fi
tar zxf ansible.tar.gz && rm -f ansible.tar.gz
ansible-playbook --connection local --inventory 127.0.0.1, \
--extra-vars env=${ENV} --extra-vars vpc=${VPC} --extra-vars region=${REGION} \
--tags ${ROLE} ansible/site.yml
The Ansible tarball is created from another Git repository with the Ansible playbook and uploaded to the secure S3 bucket. The directory layout is as follows:
ansible/
roles/ # Ansible roles, see https://docs.ansible.com/ansible/latest/user_guide/playbooks_reuse_roles.html
common/
tasks/
main.yml # Applied to all systems
bastion/
tasks/
main.yml # Bastion host "role"
duo/
files/
common_auth # /etc/pam.d/common_auth
sshd # /etc/pam.d/sshd
sshd_config # /etc/ssh/sshd_config
tasks/
main.yml
site.yml # Master playbook
vars/ # Variable configuration
[environment]/ # i.e. dev, stage, prod
main.yml # Variables specific to an environment
[vpc]/ # VPC name, i.e. dev-ops
main.yml # Variables specific to an environment and VPC
[region]/ # i.e. us-west-2
main.yml # Variables specific to the environment, VPC and region
main.yml # Global variables
Ansible roles provide convention over configuration to simplify units of work. We break out each package into a role so they can be reused. We leverage Ansible tags to associate Ansible roles with our concept of a host "role," i.e., bastion. This keeps site.yml simple and clear:
- hosts: localhost
connection: local
roles:
- { role: common, tags: ["always"] }
- { role: bastion, tags: ["bastion"] }
always is a special tag, specifying to always run a task regardless of the tag specified at execution. It provides the mechanism to run common tasks regardless of the host "role." For this example, we will only use roles/common/tasks/main.yml to load our variable hierarchy but could include tasks for creating admin users, installing default packages, etc.:
---
- name: Include site variables
include_vars: vars/main.yml
- name: Include environment variables
include_vars: vars/{{ env }}/main.yml
- name: Include VPC variables
include_vars: vars/{{ env }}/{{ vpc }}/main.yml
- name: Include region variables
include_vars: vars/{{ env }}/{{ vpc }}/{{ region }}/main.yml
This provides a powerful and flexible framework for defining variables at different levels. Site level variables apply to all hosts. Variables that might differ between dev and prod (i.e., logging host) can be defined at the environment level in vars/dev/main.yml and vars/prod/main.yml. main.yml must exist for each environment, VPC and AWS region, if only just "—" for its content. In this example, we will define one site level variable in vars/main.yml:
---
aws:
secrets: s3-bucket-name/secrets
This defines the variable aws.secrets, an S3 bucket and path for downloading files that need to be secured outside of the Ansible playbook Git repository. This value can be customized per environment, VPC and/or region by moving it down the variable hierarchy. Moving onto bastion, roles/bastion/tasks/main.yml disables selective TCP ACKs and installs Ansible roles for software, which for this example, is limited to duo:
---
- name: Disable selective acks (CVE-2019-11477)
sysctl:
name: net.ipv4.tcp_sack
value: '0'
state: present
- include_role:
name: "{{ item }}"
with_items:
- duo
Lastly, we have duo in roles/duo/tasks.yml:
---
- name: Add key
apt_key:
data: |
-----BEGIN PGP PUBLIC KEY BLOCK-----
mQINBF25pcQBEADBIWPx6DJ+EItyXif/zgDZjsuwi/4pbd5NBHVpdsK2piteY1h4
QG0CtfmCrwPRz/q5RlCNKLZ8HJiMrURGGwbts9BM57aVmn7C/OsPo3oOiOOpiiUA
qFNhuTTQQ812uO+2sULt3/UdRiKUquUgNpdp6SNkNjg5lvKCOWIhKp8l3JbvI572
0DnSuLGP9pSyQulz7B6vsCQHq1Ib7AArxk88+9QeUmhVKbXHf0K3vaQkmm7KaveK
fgyxJfNh6ilFBTZq8yxY362vP18goEdOl+2pK0If2r4w1gjEXVLyGaYHKqr7vVC7
tGYDP6ibzXNDhTNbvN+XZOlk85ttu77TRiKglcuOz3rAY6OybxUo12MYGv3vntgl
OD6XaL9+dYPVW8R5886Nq0W88wRNUa0jpY1tvO1h7j4OFvSk2xDQml8ugvbvBTZC
XuzCx//m8UyF617nlUxY4gMs/GiWs7PlJ/Bjd8bNTaATMCdD3s3RX9XUEAUMo+LM
k4hM+EaWoG++Pym/009fgdI0AAZa7igNTPcLdvAZTGVJ1K7V/QlKIz3RwTfozUtR
a3/1XfS2Zllj/Nzmx7FI1aWyTScyfl44jfjpnPc1BvUfCmuV/28pKCYsJ3yPtN4i
ccQKERQF9vUEnCZ67DmksFsrKrn9n38jd02or8ZzRRDx7NJOILhhhlzKTwARAQAB
tDJEdW8gU2VjdXJpdHkgUGFja2FnZSBTaWduaW5nIDxkZXZAZHVvc2VjdXJpdHku
Y29tPokCVAQTAQgAPhYhBN8aYLVu/i3IyoqaYQHvmOkQRI/bBQJduaXEAhsDBQkJ
ZgGABQsJCAcCBhUKCQgLAgQWAgMBAh4BAheAAAoJEAHvmOkQRI/bDjkP/0dUsJgx
qTqPDgKimmLUM6xL3cuWcoIWe7kh0GVqBEY9wPJzL9aaggkURjbwtQcHiNBV2Mk5
M6IaoIVQiSGHSob7il1sCSDRb7zWcZPqZqB9QtRBOgZcOLGxW62+UIGXPCQVvaCy
FDxsmwMnPRYz4rS3X5zK1c2Fo9D59rQmTjj71UGVliNNq8GMH64I/goa4pEryk2t
Jeby82la6gP+BPHFNc5hi/em3xxdgO16WKfe8uN0NmRZvOUnpThHbUzDjfj3uEf9
/W6XcJfIyqdGMLpzrjkdaWr1CZg4XrF5q0c0hDzxrshNV04iFreg4ds5HsboNnPX
M5HN60R1zeqBu+tVADSKYLpGOCczZTOUzYhlJfNOVSd83DE9vaeVTZQYgOK/1oVY
WYPnr0spZ06FWb/1+irSWXhdDU8tAzRO3IFq4M8eBEkCrOt17dWDfgOcXN/I34IU
I6RiAUVNc3W1aEB6r9WDPCDr9WBxrwMlceNrwFSJl9InrVIfJG54E6iYR/vBxyzS
hjFd5PJxNQIhTTOfA4YplDoviSHw/3Ci64OOPmh06Z5zfd+HOH7E+I9SRk2GunPU
odvnpWELquFzA9OwDLYUlUQC7cYHnGCzzHKHcxFuQKmH1hnAuBvq5H6OzlAgjuA7
UX82Bl6FLsm7gJUmHq7xCM3zRG3ZKus/JeUJuQINBF25pcQBEADp3Z1ovqhzfFM6
Oe/0zme9ynaGGcpxktncuvcpirsI5CYjqHWi11g1dG0HXANGDn2+kHrrJOwO6fVQ
c4d1iImKoTR6ZmYd/Ae7TthsmjZXe3P/s15JpEMhsvwkSH6FOkrCkhgaNArZr6yn
kb0s2zcJ69h7gz1rmnjmCsDjM9C/Pa99th4CBb2yo8Xq9mSjQKVCHcfFdrdGOMJc
YtZCJz6Uno4CQSRPAq0l5lxM+HXhkUdPNdoxSUV4IIwZnxxHhXA+WSMC0Px04nVi
XVDlJ+Vb5Nhf8bbJaiQXoHGFJY7u8+6QruoPQNmKkD7QwVdoEkd8Pb/6Q7ih5lIn
1ksjIG1G+N8AhkOZCm0aBz/uzMBZV8lswjNW1JEcXafe3QOnS3MxqHUXUzLN3tMB
bG6me8ENbOMlBGCQa22NVf/C1KGL9nZts0Ljz9eNQTT1mxRvuU4twScomFVXh5ZF
0LWQdlxVueaebeXQBAtdROyd2wKGO+KMuJXD6Brqh2fCx+kK+zh7cFHLeS4rKLGt
7h9yI+lbmFArzVIEuiTYx5pspzYrclbiHOGYBKhV/b7iJ66zSxy2FryPzfeKBWzX
C3kpVQ3RrhMyykUvfMfyx9+gbrCvwz7BoYD0EguPfnYYB2V7A5kM/ljwVRKY2mDL
UKQ8v12pgegp/TeWvkKJ4AGr06lzNwARAQABiQI8BBgBCAAmFiEE3xpgtW7+LcjK
ipphAe+Y6RBEj9sFAl25pcQCGwwFCQlmAYAACgkQAe+Y6RBEj9so2w//TF2rdbgD
boKM2odifrEWv11HQzVpu/xU7gN0vvha11P5qj7V9yGgy31kRCtPZ9Xp/UfLAIaw
vP85MydLY5/eUa7pRf207Hle5jl4L6g/Uuv41v9NRyOdldXzFmk0XvJfJ9ptXPTR
0E3m5t0IK2XzVhQhgCgyMb27Eh+kPbegnQV8hRNk8PVpFQNjDh6lDv7aFxmjt76x
kPUTsFriC3NRDMdun5es+74NMfTuNLF8EPcVfByR+tQuKPCXaSzux02arYFEkVdT
w9EOrNNagWX9wbI5tB80XNd1BcHCV1QOA9XCeQmcvBN5ww7nOTOwDjAMIqyoJX9D
l9l/AFJa0PH3xENICpHmapS+LJfgKD1MNfQNKl8nTLRINOXnH/7L4q7LFn024nZa
B4MvOMq7P3Hs2/iZlfIumk2AeeMemR2G72erPa6zx6I2dyp16rdi0mYHS0m4T+ud
Ye7pnNwU7EqkuUYcd1oj9txfKFYj0nlOhEzSnLnshr3LsBVtzJi42RZc10rIWZbZ
bXkcoJgoBo0P+QACduNVZ072OqDquv/OpU3UVszwotMV+IANJ9cX3bXKBCjevfTP
VsXFL+WQaiGz2OcyD2uFLtLeHCDuZ6oL3Rw9pgT4E5ZXKYj4xd1qXhQea0sQu+8I
5oRM/JeaPuYz7lH+PhzcqVqpKaWDL0Q9ixs=
=EHJ5
-----END PGP PUBLIC KEY BLOCK-----
- name: Add repository
apt_repository:
repo: deb [arch=amd64] https://pkg.duosecurity.com/Ubuntu bionic main
state: present
filename: duo
- name: Install
apt:
name: duo-unix
state: present
update_cache: yes
- name: Download configuration
command: "aws s3 cp s3://{{ aws.secrets }}/{{ role_name }}/pam_duo.conf /etc/duo/pam_duo.conf"
- name: Secure configuration
file:
path: /etc/duo/pam_duo.conf
owner: root
group: root
mode: 0600
- name: Configure PAM common
copy:
src: common_auth
dest: /etc/pam.d/common_auth
owner: root
group: root
mode: 0644
- name: Configure PAM sshd
copy:
src: sshd
dest: /etc/pam.d/sshd
owner: root
group: root
mode: 0644
- name: Configure sshd
copy:
src: sshd_config
dest: /etc/ssh/sshd_config
owner: root
group: root
mode: 0644
- name: Restart sshd
service:
name: sshd
state: restarted
daemon_reload: yes
The duo configuration file contains secrets, so it is downloaded from the encrypted S3 bucket in the secrets/bastion path:
; This file is managed by Ansible, do not modify locally
[duo]
ikey = [redacted]
skey = [redacted]
host = [redacted]
failmode = safe
; Send command for Duo Push authentication
pushinfo = yes
autopush = yes
The remaining files are kept in version control for auditing:
# This file is managed by Ansible, do not modify locally
# /etc/pam.d/common-auth - authentication settings common to all services
#
# This file is included from other service-specific PAM config files,
# and should contain a list of the authentication modules that define
# the central authentication scheme for use on the system
# (e.g., /etc/shadow, LDAP, Kerberos, etc.). The default is to use the
# traditional Unix authentication mechanisms.
#
# As of pam 1.0.1-6, this file is managed by pam-auth-update by default.
# To take advantage of this, it is recommended that you configure any
# local modules either before or after the default block, and use
# pam-auth-update to manage selection of other modules. See
# pam-auth-update(8) for details.
# here are the per-package modules (the "Primary" block)
#auth[success=1 default=ignore]pam_unix.so nullok_secure
auth requisite pam_unix.so nullok_secure
auth [success=1 default=ignore] /lib64/security/pam_duo.so
# here's the fallback if no module succeeds
authrequisitepam_deny.so
# prime the stack with a positive return value if there isn't one already;
# this avoids us returning an error just because nothing sets a success code
# since the modules above will each just jump around
authrequiredpam_permit.so
# and here are more per-package modules (the "Additional" block)
authoptionalpam_cap.so
# end of pam-auth-update config
# This file is managed by Ansible, do not modify locally
# PAM configuration for the Secure Shell service
# Standard Un*x authentication.
#@include common-auth
# Disallow non-root logins when /etc/nologin exists.
account required pam_nologin.so
# Uncomment and edit /etc/security/access.conf if you need to set complex
# access limits that are hard to express in sshd_config.
# account required pam_access.so
# Standard Un*x authorization.
@include common-account
# SELinux needs to be the first session rule. This ensures that any
# lingering context has been cleared. Without this it is possible that a
# module could execute code in the wrong domain.
session [success=ok ignore=ignore module_unknown=ignore default=bad] pam_selinux.so close
# Set the loginuid process attribute.
session required pam_loginuid.so
# Create a new session keyring.
session optional pam_keyinit.so force revoke
# Standard Un*x session setup and teardown.
@include common-session
# Set up user limits from /etc/security/limits.conf.
session required pam_limits.so
# Read environment variables from /etc/environment and
# /etc/security/pam_env.conf.
session required pam_env.so # [1]
# In Debian 4.0 (etch), locale-related environment variables were moved to
# /etc/default/locale, so read that as well.
session required pam_env.so user_readenv=1 envfile=/etc/default/locale
# SELinux needs to intervene at login time to ensure that the process starts
# in the proper default security context. Only sessions which are intended
# to run in the user's context should be run after this.
session [success=ok ignore=ignore module_unknown=ignore default=bad] pam_selinux.so open
# Standard Un*x password updating.
@include common-password
# Duo MFA authentication
auth [success=1 default=ignore] /lib64/security/pam_duo.so
auth requisite pam_deny.so
auth required pam_permit.so
# This file is managed by Ansible, do not modify locally
# This is the sshd server system-wide configuration file. See
# sshd_config(5) for more information.
# This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin
Protocol 2
StrictModes yes
AuthenticationMethods publickey,keyboard-interactive
PubkeyAuthentication yes
ChallengeResponseAuthentication yes
PasswordAuthentication no
X11Forwarding yes
AcceptEnv LANG LC_*
Subsystem sftp /usr/lib/openssh/sftp-server
UsePAM yes
UseDNS no
Create the Ansible playbook tarball that extracts to ansible/ and upload it to the S3 bucket specified in Terraform. Apply the Terraform for IAM first, and then continue to the EC2 instances. Minutes later, you will be able to login to your bastion hosts with Duo MFA.
You now have a framework that is easy to extend – add software packages to existing host roles, customizing configuration, and adding new host roles that consume software packages. A special thanks to @_p0pr0ck5_ for his work on the variable hierarchy loading in Ansible.