by Adrian Mlodzianowski, Sr. Systems Engineer, Logicworks

In the new family of AWS instances (the C5 and M5), EBS volumes are exposed as NVMe block devices. If you’re already using some form of automation to build instances, identifying the EBS device name can be a problem.

Logicworks manages tens of thousands of EBS volumes, and so we use Puppet to programmatically manage volumes. In a single configuration run, you want to create the EBS volume and mount it, and to do that you need to know the device name. While issuing the API call to attach an EBS volume to an EC2 instance, we pass in the device name as a parameter. With older instance types, the block device driver would simply use that name to expose the new volume within the OS. But if we try mounting the same EBS volume on a C5 or M5, the device is automatically enumerated and presented as /dev/nvme[0-6]n1, losing its relation to the device name from the API call. Not good.

When I began to look for a way to fix this problem, my first thought was that someone had probably already encountered this problem. So I hunted around, and found the answer buried in the latest Amazon Linux AMI.

The Solution: udev rules and symlinks

The latest Amazon Linux AMI has a set of udev rules to solve this problem. They are invoked by the udevd daemon, which runs in the background and listens for all user space related hardware events, such as the attachment of a new block device. The most interesting rule in our case is present in /etc/udev/rules.d/70-ec2-nvme-devices.rules and has the following content:

# ebs nvme devices

KERNEL=="nvme[0-9]*n[0-9]*", ENV{DEVTYPE}=="disk", ATTRS{model}=="Amazon Elastic Block Store", PROGRAM="/sbin/ebsnvme-id -u /dev/%k", SYMLINK+="%c"

Translating udev syntax into more human readable code – For any devices that match the “nvme[0-9]*n[0-9]*” naming scheme, are a “disk” type and have “Amazon Elastic Block Store” as the model, invoke the /sbin/ebsnvme-id program with the device name and create a symlink using the output value.

/sbin/ebsnvme-id is a Python script written by Amazon and available under the Apache license. It takes the NVMe device name as the argument and uses the sys module to pull information about the block device. For example:

[root@m5-test ec2-user]# /sbin/ebsnvme-id -u /dev/nvme1n1

xvdf

In case of EBS volumes, there is a lot useful metadata available about the volume itself:

ipdb> self.id_ctrl.mn #vendor

'Amazon Elastic Block Store'

ipdb> self.id_ctrl.sn #volume id

'vol0f7b93d7bf9c1ce8d'

ipdb> self.id_ctrl.vs.bdev #device name, AWS side

'xvdf'

When I noticed the symlinks on the OS and started reading through the udev rules and Python code, I realized that everything here is OS agnostic and should work on any modern Linux distro. I used Puppet to create the same udev rule and place the ebsnvme-id script in /sbin regardless whether it is a Ubuntu, RHEL or CentOS system.

Results

For Logicworks, this gives us the ability to mount volumes with ease again. From Puppet’s perspective, no changes were necessary to the custom provider and defined type used by us for EBS volume management, it works the same as non-NVMe block devices.

I hope this script is useful for anyone that creates EBS volumes in an automated fashion. This script bridges the gap between creating the EBS volume and getting the device name to perform further operations on the block device (create a filesystem, mount it etc). The benefit of this solution is that it’s seamless and works right away. The very second that the EBS volume is attached it triggers the udev rule, creating the needed symlink.