At Lab Zero we ran into a use case a while back where we had some files that needed to be created and shared across Docker containers in the same EC2 Container Service (ECS) cluster. We typically use S3 for solving most types of shared storage situations, but in this case, we needed the storage to be mounted locally on each container instance. Thankfully Amazon Elastic File System (EFS) fit the bill. It provides elastic storage that can be mounted as an NFS drive with just a few simple steps.

For the sake of this article, I’ll assume you are at least a little familiar with configuring Amazon Web Services, and that you already have an ECS cluster launching docker images (Tasks) inside a Virtual Private Cloud (VPC). Before we get going, make note of your VPC id, we’ll need it soon.

For those who just need a TLDR; of the steps:

1. Create an EFS drive
2. Configure security groups to allow access to NFS
3. Attach the EFS drive as an NFS volume to your instances
4. Add a VOLUME command to your Dockerfile & publish a new image
5. Add volumes and corresponding mountPoints to your ECS Task Definition
6. Update running tasks to use your new Task Definition version

Create a new EFS drive

In your browser, open up the EFS console and create a new file system. Select the same VPC id that your ECS cluster is running in, then select your desired availability zones - typically this should be all zones. Select Next Step, enter a value for the Name tag, and keep the IO setting on “General Purpose” unless you have a massive cluster. Select Next Step, then Create File System. You now have an elastic virtual drive that will expand in size depending on your usage. Make note of the file system id of this new drive, it should look something like fs-1a2b3c4d

Set up security groups

Next set up some security groups to make sure that your EC2 instances can communicate with this new network drive. You need to configure two security groups, one for the instances, and one for the drive. Navigate over to the VPC dashboard and find the security group assigned to your EC2 instances. Make note of the security group ID, and doublecheck that there is an inbound rule for TCP port 22 with a network source of 0.0.0.0/0

If this security group doesn’t already open TCP port 22 for inbound traffic, either add a new inbound rule, or create a new security group with the rule. If you created a new security group for this, be sure to assign it to all of the EC2 nodes in your ECS cluster.

Next, create a new security group for the new EFS drive. Create a new inbound rule for NFS, TCP port 2049. For the source, enter the id of the security group that was configured for the EC2 instances. This will allow all instances to be able to communicate with the drive using the NFS protocol. Make note of this new NFS security group id, you need to assign this to the EFS drive that you created.

To assign this new security group, navigate back over to the EFS dashboard, select your file system to expand the row, then click ‘Manage file system access’ in the ‘File system access’ section. For each availability zone listed, add the id of the NFS security group to the list of security groups, then click Save.
Attach the EFS drive to each EC2 instance.

NOTE: I will list out each individual step of this part below as if you were doing it by hand, but note that you must script this in your instance User Data section in order to retain these settings whenever an instance is restarted. For an example of how we do this, please see my gist here:

https://gist.github.com/brienw/b16460190b9df25c981e04cfed5c939e

Now that you’ve created the EFS drive and have configured the security groups, it’s time to mount it to the EC2 instances. I’ll walk you through how to do it manually, if you automate the creation of your infrastructure, simply adapt these steps to suit your specific orchestration. Please note that these manual steps must be repeated on ALL instances. For the sake of this article, I will assume you are running the Amazon Linux AMI.

# install NFS client
$ sudo yum -y install nfs-utils

# create mount point
$ sudo mkdir -p /mnt/efs_drive

The docker daemon only knows about drives that were mounted at the time the deamon was started. You can add this new drive to /etc/fstab` to make sure the drive is mounted when your EC2 instance boots up.

Edit /etc/fstab as root and add a line for the new drive. Follow the following pattern to create the DNS name for your drive. You will need to create the DNS url for each instance, assuming your instances are spread among different availability zones.

<ec2 instance availability zone>.<file system id>.efs.<aws region>.amazonaws.com

For example, file system id fs-1a2b3c4d being attached to an instance in the us-west-2b availability zone would be:

us-west-2b.fs-1a2b3c4d.efs.us-west-2.amazonaws.com

# /etc/fstab
LABEL=/     /           ext4    defaults,noatime  1   1
tmpfs       /dev/shm    tmpfs   defaults        0   0
devpts      /dev/pts    devpts  gid=5,mode=620  0   0
sysfs       /sys        sysfs   defaults        0   0
proc        /proc       proc    defaults        0   0

# EFS drive
us-west-2b.fs-1a2b3c4d.efs.us-west-2.amazonaws.com:/  /mnt/efs_drive  nfs  defaults,vers=4.1  0 0

After editing /etc/fstab mount the drive manually to make sure everything is configured correctly

sudo mount /etc/efs_drive

With any luck, you will be almost immediately returned to a command prompt. Verify that the drive was mounted

mount | grep efs_drive

If the mount command stalls and does not return you to a command prompt after a handful of seconds, it most likely means something was not set up correctly with your security group. Go back and verify the security group configurations and verify that the NFS security group is assigned to the EFS drive.

Once you have verified that the drive is mounted on each instance, go ahead and restart each one. This will mount the drive on boot and before the Docker daemon is started back up. (You could also send a SIGHUB signal to restart the Docker daemon)

Configure your Dockerfile for the new mount point

Now that you’ve set up all the groundwork, you need to configure your Docker container to prepare for the mounted volume. By default with all the steps completed above, the drive will be mounted in the container, but it will remain owned by root when it is started up. Typically your application will be running as a different user, and if you want that user to be able write to the drive, you need to change ownership. Add a few configuration lines to your Dockerfile to allow the container to prepare for this new volume. Note that the VOLUME command must be added after any commands that will affect the volume. This example creates the mount point at /mnt/efs_drive, and changes the ownership to the application user, app_user:

RUN mkdir -p /mnt/efs_data
RUN chown -R app_user:app_user /mnt/efs_data
VOLUME /mnt/efs_data

A note if your user is not part of your base image:
Since this volume will be shared across containers, you need to make certain that the permissions remain consistent for the user account that will write to it. If you create your user inside your Dockerfile, you need to make sure you are setting the uid value when you create the user, as a rule, keep the userid above 9000 to be safe.

# If you bind mount a volume from the host or a data container, ensure you use the same uid
​RUN useradd -u 1000 -m -s /bin/bash app_user

Build your new Docker image and publish the new version up to AWS.

Configure the ECS Task Definition

The EFS drive is now connected to the host instance, and your Dockerfile is set to look for a new drive, but Docker containers won’t automatically be able to see this new volume. To allow the container to mount the drive, you have to make some modifications to your task definition. In the ECS dashboard, navigate to your Task Definition and then then create new revision. Look for the Volumes section and add a new volume by selecting ‘Add volume’. Enter a name and source path, for our example, we set these to name: efs_volume, source path: /mnt/efs
Now edit the Task Definition JSON data. Inside the list of containerDefinitions look for a mountPoints key, or add one if it doesn’t already exist. Add an entry for the volume you just added. This will tell Docker where to mount the volume when it runs your container.

mountPoints: [
  {
    "containerPath": "/mnt/efs_data",
    "sourceVolume": "efs_volume",
    "readOnly": false
  }
]

Save your Task Definition, and make note of the revision number.

Update ECS Service to run the new Task Definition revision.

Everything is ready, the last thing to do is to configure the service that runs you Task Definition to use the new version. From the ECS Clusters dashboard, select the desired cluster, then select the service running the Task Definition you just configured. Click “Update”, then edit the Task Definition identifier and bump the version number to the latest revision that you just created above. Click “Update Service” and you’re done. Within a few moments you should see ECS spinning up new services with this new Task Definition (and then spinning down the old ones).
You now have access to this shared drive across all tasks running this service. This is a quick way to get basic file sharing across Docker containers thanks to EFS.

There are obviously some caveats to this, since race conditions can arise depending on how this shared storage is used, but for basic usage it may go a long way.