Extracting secrets from AWS-Vault

Recently I was in the situation in which I needed to do some automation involving AWS-Vault, which is a tool for securely managing AWS access keys on developer workstations. Until now I had been using the file backend, which relies on encrypted-at-rest files to store the secrets.

Since there is obviously no caching involved here, every time I wanted to use a secret I would need to type in the passphrase to unlock it, with no cached copy of the passphrase involved. Coupled with a bug that my team faces when applying Terraform (sessions cause remote state S3 bucket reads to fail, worked around with the --no-session option), there was no way that I could create a loop for it.

The solution for this was to move to a different backend. Since I run Linux on my workstation, that meant moving to the secret-service backend, which in my case is backed by gnome-keyring-daemon.

Researching this yielded no solution for how to migrate from one backend to another. There was nothing in a contrib/ directory or any blog posts, so instead I had a dig into the source code. This showed that the AWS-Vault key files (located in $HOME/.awsvault/keys were a base64-encoded blob, that when decoded revealed some JSON content from a standard called JOSE (Javascript Object Signing and Encryption). Specifically, I learned that they contained JWK (Javascript Web Key) attributes, although just trying to decoded it yielded encoding errors:

$ cat .awsvault/keys/mykey |base64 -d
{"alg":"PBES2-HS256+A128KW","created":"2019-12-04 09:33:34.461409668 -0800 PST m=+9.768812157","enc":"A256GCM","p2c":8192,"p2s":"FFFFFFFFFFF"}base64: invalid input

“Perfect, from here on out I can just find a good JOSE library in my language of choice, load it up, and print the payload” was what I thought. This didn’t end up working though. I wasn’t able to figure out a workable method using python-jose, try as I might. My next effort was simply replicate the libraries and methods from the original file in AWS-Vault’s keyring library. This actually ended up working. The final code for this is as follows:

package main

import (
    jose ""

type Item struct {
    Key         string
    Data        []byte
    Label       string
    Description string

    // Backend specific config
    KeychainNotTrustApplication bool
    KeychainNotSynchronizable   bool

type Key struct {
    AccessKeyID         string
    SecretAccessKey     string
    SessionToken        string

func main() {
    if len(os.Args) != 3 {
       fmt.Println("Wrong input format, expected: ./aws-vault-decrypt <key_file> <password>")

    bytes, err := ioutil.ReadFile(os.Args[1])
    if os.IsNotExist(err) {
        fmt.Println("File not found")
    if err != nil {
        fmt.Printf("Error: %s", err)

    payload, _, err := jose.Decode(string(bytes), os.Args[2])
    var decoded = Item{}
    err = json.Unmarshal([]byte(payload), &decoded)
    var data = Key{}
    err = json.Unmarshal([]byte(decoded.Data), &data)
    fmt.Printf("AWS_ACCESS_KEY_ID: %s\n", string(data.AccessKeyID))
    fmt.Printf("AWS_SECRET_ACCESS_KEY: %s\n", string(data.SecretAccessKey))

This resulted in good output:

$ ./aws-vault-decrypt /home/bkero/.awsvault/keys/mykey 'mypassword'

From here it was simply a matter of setting my new backend in AWS-Vault and running aws-vault add:

$ export AWS_VAULT_BACKEND=secret-service
$ aws-vault add mykey
Added credentials to profile "mykey" in vault

Note that you’ll want to add that export AWS_VAULT_BACKEND=secret-service line to your $HOME/.bashrc if you want the change to be permanent.


Hackerbeach Days 1-3

Sunset from the malecon in El Rodadero

For the last few days I’ve been spending time in El Rodadero, Colombia before setting off to Palomino for Hackerbeach. Another participant arrived today, and joined me in relaxing before the primary Hackerbeach work in Palomino. It’s peak season here in El Rodadero, which means all the restaurants, the malecón, and the beach are filled to the brim with people, mostly traveling Colombians who come here for what is justifiably one of the most gorgeous beaches in the country.


Hackerbeach Day 0

The view from my hostel’s lobby, featuring Memphis the cat

It’s been hell, doing a 4 flight stint for the better part of a day with no off-plane sleep. I read recently that airplane humidity is typically around 20%, which is drier than the Sahara desert. Lesson learned: always bring and fill your water bottle. When you’re dehydrated, the mucus membranes have trouble keeping a layer of mucus, and can let more dangerous external matter through. That results in a sensation, that, when you swallow, can be unpleasant.

But though the trial, I’m finally here on the Caribbean coast to kick off the Nth annual Hackerbeach in beautiful Santa Marta, Colombia.

The weather is a muggy 31C, not a fluffy cloud in the firmament, and high season is in full swing. I’m the first to arrive, and spending the 29th through the new year in the city before settling down in (hopefully) more peaceful Palomino, further east on the coast.

The Centro Historico where I’m staying is rife with touts. Taxi drivers trying to drum up business, pre-packaged adventure outfits promising the very best ziplines, and plenty of folks standing around with convincing smiles holding a restaurant menu.

Vague electronica fills the muggy air with super hip beats. It would be inappropriate to dance, but we do feel a bit more in tune with the forced vibe.

The night continued on. I took my stroll of the malecon and beach, found a handicraft market, and my guilty pleasure: street meats. Sausage fresh off a charcoal grill, covered in just-squeezed lime and roughly chopped chilis on a bed of yuca in a small styrofoam container for $1. There were a few touts there, but nobody bothered me. It must either be the default expression of a determined scowl, or having low value as a single young foreign male walking by.

While waiting for a hyped up blues-and-rock dive bar to open, I took another stroll through the neighborhood. I changed upon a shrill noise from the super(sub?)sonics of some music hurting my ears while walking through an alleyway. Investigating the source yielded a cultural celebration. It was a delightful cultural dance competition. That occupied a very pleasant half hour of my time until the acts petered out.

Onto my quarry! It’s a small dive bar in the Centro Historico called Crab’s Bar, proudly slinging blues and rock since 1999. The owner is a foreigner and doing an excellent job not paying top 80s rock, which is often the trap of places like this. When I arrive it is empty. I must have come too soon. An hour later and most of the tables are full and the atmosphere of the place is much more convivial. The cocktails are pricey, but all are double-sided, so I give it a pass.

Besides the boilerplate 60s Rock kitsch, the place is run by some excellent bar staff, who are sporting faded tattoos of the place’s logo. It’s always a good sign when folks are (hopefully not coerced into being) that invested and passionate. As the night progresses I am increasingly comfortable. Eventually I head back to my hostel and try to acclimate to the EST timezone.

That’s about it. A simple night as others have yet to arrive. I’ll be on my own for a few more nights wandering and enjoying the city.

Bonus: The exchange rate here is 3,000ish Colombian Pesos (COP) per USD, which is hard to keep in my head. Using the ATM does make me a millionaire though.


Working WWAN on the X1 Carbon Gen 6

When I initially received my new X1 Carbon I was very excited at the new hardware. More pixels, more lumens, more cores.

After I got over my initial euphoria, I noticed that some things didn’t work, and confirmed such by a very helpful Archwiki page.

Some of the things that didn’t work include ACPI suspend-to-RAM sleep (since corrected in a BIOS update), the the fingerprint sensor (still outstanding), and the WWAN LTE Cat9 card that Lenovo seems so proud about. It has all the LTE channels you could ever want and can achieve speeds of up to 150Mbit. It’s a pity that the Linux drivers are not up to snuff.

It was doubly disappointing because I had gotten used to the always-connected nature of having a WWAN card in my primary machine.

The WWAN card included in USA-bound X1 Carbons is a Fibocom L850-GL. The card itself is primarily comprised of an Intel modem and chipset, which does have support in the Linux kernel (it is used in several Android phones). The reason it is unsupported on Linux is because of the interface they chose to use. Normally WWAN cards electrically use a USB interface, even if they are primarily found in a M.2 form factor inside a laptop.

To support the higher data rate Lenovo decided to use the card’s PCI Express interface. This interface does not have support in the Linux kernel. Although the card itself enumerates (via lspci) as a device on the PCI bus, no driver will bind to it. Hopefully someday someone will boost the Linux support.

In an attempt to get it to work, some enterprising owners even taped over the PCI Express pins on the card so that it would force connection over USB. Unfortunately this reveals another problem: Lenovo’s card whitelist. Lenovo disallows any card that is not in a list embedded in their signed system firmware.

My online research had revealed that some folks had luck using some Sierra Wireless cards that had been used in previous ThinkPad models as well as other Dell laptops.

So I did it. I went on eBay and purchased a Sierra Wireless 7455 LTE WWAN card. I removed the original Fibocom part and put it in. Unfortunately I was greeted with the very same whitelist screen.

Typical whitelist error message

I knew from having previous Sierra Wireless devices that when running correctly it produced several serial console devices (/dev/ttyUSB[0-2]). Details in a Reddit post showed which commands I would have to run to make it work. The commands would alter the USB vendor and product IDs and set “fastboot” behavior for the card.

The trick to bypassing the whitelist protection turned out to be a timeout. The card spend so long booting that the whitelist protection in the laptop’s system firmware times out.

I spent the better part of a jetlagged morning trying to find any device in my house with a M.2 Key B slot. It turns out I had none other than the X1 Carbon discussed in this article. Frustrated, I went back to sleep and had a think.

At some point in my rest I had an epiphany: if the card itself simply interfaces via USB, couldn’t I hotplug it after the laptop had booted?

I immediately got up and gave it a shot: I removed the Fibocom card, booted the laptop, ran journalctl -xf to see log output, and closed the lid to put it to sleep. Then I took the bottom panel off and slotted in the new Sierra Wireless card, attaching both of the antennas.

I closed it back up and opened my laptop. I watched delightfully as the card enumerated itself and was recognized by my system.

I proceeded to flash the card via instructions that a very helpful person wrote, including links to native Linux binaries! Afterwards I reloaded the card, connected to the card’s serial console, and ran the AT commands from the earlier-mentioned reddit post. This disabled the card’s fastboot from cold system boot (but enabled for a warm-boot), enabled LTE (disabled low-power mode), and set the USB IDs.

I crossed my fingers and rebooted my system. To my relief it Just Worked (TM). My system booted with nary an error and I was soon back in my regular desktop environment. I was able to add the card to NetworkManager, and was online. I did a SpeedTest and found the result to be perfectly sufficient: 60/30Mbit. Not as good as the original Fibocom card, but I’ll take it.

Given that we got this card to work by simply relying on slow card bootup to bypass the whitelist, I wonder if the same can be done with the original Fibocom card. If we simply taped over the PCI-Express pins to force USB, and inserted the card after boot, could we also find a setting to slow boot enumeration so that it would cause the whitelist checking to timeout? Further research is needed.


IndieWebCamp 2018

Attending bkero

I’m looking forward to attending the 2018 IndieWebCamp. It’s a small 2-day event happening in Portland and is exploring the topics of independent web hosting and technologies to knit them together.

If you’re in Portland, you should attend too!


Introduction to Linux Containers presentation materials

Here is a link to the presentation materials for my talk, Introduction to Linux Containers.

Press ‘c’ to see the presenter console for the slides.


The Dark Arts of SSH presentation materials

Here is the presentation material for my talk entitled The Dark Arts of SSH. Please note this is a single HTML rendering that incldues presenter’s notes.

Linux Uncategorized

Linux Kernel Compilation presentation materials

As promised to my audience, here are the slides from my presentation titled Building your First Linux Kernel.


Installing RemixOS to an internal drive

Your old pal syslinux is there to greet you
Your old pal syslinux is there to greet you

After initially running RemixOS, the new Android build for PCs, I decided that I would rather play with booting it natively from my SSD instead of from a USB device. Performance should be better, it would free my USB thumb drive up for other duties, and it would make booting more convenient.

This turned out to be a relatively simple operation. What follows is my methodology for doing that. Please note that these instructions assume you are running Linux.