Working WWAN on the X1 Carbon Gen 6

When I initially received my new X1 Carbon I was very excited at the new hardware. More pixels, more lumens, more cores.

After I got over my initial euphoria, I noticed that some things didn’t work, and confirmed such by a very helpful Archwiki page.

Some of the things that didn’t work include ACPI suspend-to-RAM sleep (since corrected in a BIOS update), the the fingerprint sensor (still outstanding), and the WWAN LTE Cat9 card that Lenovo seems so proud about. It has all the LTE channels you could ever want and can achieve speeds of up to 150Mbit. It’s a pity that the Linux drivers are not up to snuff.

It was doubly disappointing because I had gotten used to the always-connected nature of having a WWAN card in my primary machine.

The WWAN card included in USA-bound X1 Carbons is a Fibocom L850-GL. The card itself is primarily comprised of an Intel modem and chipset, which does have support in the Linux kernel (it is used in several Android phones). The reason it is unsupported on Linux is because of the interface they chose to use. Normally WWAN cards electrically use a USB interface, even if they are primarily found in a M.2 form factor inside a laptop.

To support the higher data rate Lenovo decided to use the card’s PCI Express interface. This interface does not have support in the Linux kernel. Although the card itself enumerates (via lspci) as a device on the PCI bus, no driver will bind to it. Hopefully someday someone will boost the Linux support.

In an attempt to get it to work, some enterprising owners even taped over the PCI Express pins on the card so that it would force connection over USB. Unfortunately this reveals another problem: Lenovo’s card whitelist. Lenovo disallows any card that is not in a list embedded in their signed system firmware.

My online research had revealed that some folks had luck using some Sierra Wireless cards that had been used in previous ThinkPad models as well as other Dell laptops.

So I did it. I went on eBay and purchased a Sierra Wireless 7455 LTE WWAN card. I removed the original Fibocom part and put it in. Unfortunately I was greeted with the very same whitelist screen.

Typical whitelist error message

I knew from having previous Sierra Wireless devices that when running correctly it produced several serial console devices (/dev/ttyUSB[0-2]). Details in a Reddit post showed which commands I would have to run to make it work. The commands would alter the USB vendor and product IDs and set “fastboot” behavior for the card.

The trick to bypassing the whitelist protection turned out to be a timeout. The card spend so long booting that the whitelist protection in the laptop’s system firmware times out.

I spent the better part of a jetlagged morning trying to find any device in my house with a M.2 Key B slot. It turns out I had none other than the X1 Carbon discussed in this article. Frustrated, I went back to sleep and had a think.

At some point in my rest I had an epiphany: if the card itself simply interfaces via USB, couldn’t I hotplug it after the laptop had booted?

I immediately got up and gave it a shot: I removed the Fibocom card, booted the laptop, ran journalctl -xf to see log output, and closed the lid to put it to sleep. Then I took the bottom panel off and slotted in the new Sierra Wireless card, attaching both of the antennas.

I closed it back up and opened my laptop. I watched delightfully as the card enumerated itself and was recognized by my system.

I proceeded to flash the card via instructions that a very helpful person wrote, including links to native Linux binaries! Afterwards I reloaded the card, connected to the card’s serial console, and ran the AT commands from the earlier-mentioned reddit post. This disabled the card’s fastboot from cold system boot (but enabled for a warm-boot), enabled LTE (disabled low-power mode), and set the USB IDs.

I crossed my fingers and rebooted my system. To my relief it Just Worked (TM). My system booted with nary an error and I was soon back in my regular desktop environment. I was able to add the card to NetworkManager, and was online. I did a SpeedTest and found the result to be perfectly sufficient: 60/30Mbit. Not as good as the original Fibocom card, but I’ll take it.

Given that we got this card to work by simply relying on slow card bootup to bypass the whitelist, I wonder if the same can be done with the original Fibocom card. If we simply taped over the PCI-Express pins to force USB, and inserted the card after boot, could we also find a setting to slow boot enumeration so that it would cause the whitelist checking to timeout? Further research is needed.


The Dark Arts of SSH presentation materials

Here is the presentation material for my talk entitled The Dark Arts of SSH. Please note this is a single HTML rendering that incldues presenter’s notes.

Linux Uncategorized

Linux Kernel Compilation presentation materials

As promised to my audience, here are the slides from my presentation titled Building your First Linux Kernel.


Installing RemixOS to an internal drive

Your old pal syslinux is there to greet you
Your old pal syslinux is there to greet you

After initially running RemixOS, the new Android build for PCs, I decided that I would rather play with booting it natively from my SSD instead of from a USB device. Performance should be better, it would free my USB thumb drive up for other duties, and it would make booting more convenient.

This turned out to be a relatively simple operation. What follows is my methodology for doing that. Please note that these instructions assume you are running Linux.

android Linux

Trying out RemixOS

The RemixOS boot logo
The RemixOS boot logo

I’ve always been one for trying out new operating systems, so when I heard news about the latest desktop-conversion effort from Jide I wanted to give it a try.

RemixOS is a proprietary offering based on the work of android-x86, which aims to bring the stock Android experience to commodity PCs. RemixOS adds on interface and convenience changes to make the operating system more usable on PC hardware. This includes UI changes such as multi-windows and a classic ‘desktop’.

The Alpha for PC was released this morning, and can be downloaded here. There was also a leaked version that landed a couple days earlier. If you’ve seen reviews online, most of them came from this. What follows are my impressions of the experience.


The ‘Try’ repository and its evolution

As the primary maintainers of the back end of the Try repository (in addition to the rest of Mercurial infrastructure) we are responsible for its care and feeding, making sure it is available, and a safe place to put your code before integration into trees.

Recently (the past few years actually) we’ve been experiencing that Mercurial has problems scaling to it’s activity. Here are some statistics for example:

  • 24550 Mercurial heads (this is reset every few months)
  • Head count correlated with the degraded performance
  • 4.3 GB in size, 203509 files without a working copy

One of the methods we’re attempting is to modify try so that each push is not a head, but is instead a bundle that can be applied cleanly to any mozilla-central tree.

By default when issuing a ‘hg clone’ from and to a local disk, it will create a hardlinked clone:

$ hg clone --time --debug --noupdate mozilla-central/ mozilla-central2/
linked 164701 files
listing keys for "bookmarks"
time: real 3.030 secs (user 1.080+0.000 sys 1.470+0.000)

$ du --apparent-size -hsxc mozilla-central*
1.3G   mozilla-central
49M    mozilla-central2
1.3G   total

These lightweight clones are the perfect environment to apply try heads to, because they will all be based off existing revs on mozilla-central anyway. To do that we can apply a try head bundle on top:

$ cd mozilla-central2/
$ hg unbundle $HOME/fffe1fc3a4eea40b47b45480b5c683fea737b00f.bundle
adding changesets
adding manifests
adding file changes
added 14 changesets with 55 changes to 52 files (+1 heads)
(run 'hg heads .' to see heads, 'hg merge' to merge)
$ hg heads|grep fffe1fc
changeset:   213681:fffe1fc3a4ee

This is great. Imagine if instead of ‘mozilla-central2’ this were named fffe1*, and could be stored in portable bundle format, and used to create repositories whenever they were needed. There is one problem though:

$ du -hsx --apparent-size mozilla-central*
1.3G	mozilla-central
536M	mozilla-central2

Our lightweight 49MB copy has turned into 536MB. This would be fine for just a few repositories, but we have tens of thousands. That means we’ll need to keep them in bundle format and turn them into repositories on demand. Thankfully this operation only takes about three seconds.

I’ve written a little bash script to go through the backlog of 24,553 try heads and generate bundles for each of them. Here is the script and some stats for the bundles:

$ ls *bundle|wc -l

$ du -hsx
39G    .

$ cat

/usr/bin/parallel --gnu --jobs 16 \
    "test ! -f {}.bundle && \
    /usr/bin/hg -R /repo/hg/mozilla/try bundle --rev {} {}.bundle ::: \
    $(/usr/bin/hg -R /repo/hg/mozilla/try heads --template "{node} ")

If this tooling works well I’d like to start using this as the future method of submitting requests to try. Additionally if developers wanted, I could create a Mercurial extension to automate the bundling process and create a bundle submission engine for try.


Mozilla’s “try” repository

At Mozilla we use Mercurial for Firefox development. We have several repositories/trees that are used depending on where the code should be. If a developer wishes to test the code they have been developing, they can submit it to a Mercurial repository called ‘try‘, since running our entire test suite is not feasible on developer machines.

We have quite a bit of infrastructure around this including Tinderbox Pushlog (TBPL) and  more. This post deals with the infrastructure and problem we face while trying to scale the ‘try’ repository.

A few statistics:

  • The try repository currently has 17943 heads. These heads are never removed.
  • The try repository is about 3.6 GB in size.
  • Due to Mercurial’s on-wire HTTP protocol, this number of heads causes HTTP cloning to fail
  • There are roughly 81000 HTTP requests to try per day
  • To fix problems (mentioned below), the try repository is deleted and re-cloned from mozilla-central every few months

There are a number of problems associated with such a repository. One particularly nasty one has been present through several years of Mercurial development, and has been tricky in that it is seemingly unreproducible. The scenario is something like:

  • User ‘hg push’es some changes to a new head onto try
  • The push process takes a long time (sometimes between 10 minutes and hours)
  • A developer could issue an interrupt signal (ctrl+C) which causes the client to gracefully hang up and exit (his typically has no effect on the server
  • Subsequent pushes will hang with something similar to ‘remote: waiting for lock on repository /repo/hg/mozilla/try/ held by ‘’
  • When this happens a hg process is running on the server has the following characteristics:
    • A ‘hg serve’ process runs single-threaded using 100% CPU
    • strace-ing and ltrace-ing reveal that the process is not making any system calls or external library calls
    • perf reveals that the process is spending all of its time inside some ambiguous python function
    • pdb yields that the process is spending all of its time in a function that (along some point in the stack trace) is going through ancestor calculations
    • The process will eventually exit cleanly
  • As operators there is nothing we can do that to alleviate the situation once the repository gets in this state. We simply inform developers and monitor the situation.

There have been several ideas on ways to alleviate the problem:

  • Periodically reset ‘try’. This is considered bad because 1) it loses history, and 2) it is disruptive to developers, who might have to re-submit try jobs again
  • Reset try on the SSH servers, but keep old try repositories on the HTTP servers. This has the potential to create unforeseen problems of growing these repositories even further on the HTTP servers. If reset (staggered from SSH server resets) this will remove unforeseen problem potential, but still lose history.
  • Creating bundle files out of pushes to ‘try’, then hosting these in an accessible location (S3, http webroot, etc). I will detail this method in a future blog post.

As of now though, try will periodically need to be reset as a countermeasure to the hangs mentioned in this post. Getting a reproducible test case might allow us to track down a bug or inefficiency in Mercurial to fix this problem after all. If you’d like to help us with this, please ping fubar or me (bkero) on