Using Nix to write rust on the esp32

In the interest of repeatable builds we must create a nix derivation for a rust-esp32 toolchain

Rust smells like an ideal language for developing embedded systems. There has been recent traction in getting it to target the esp32, especially work by mabezdev and this blog post by Yoshinari Nomura

TLDR:

$ git clone https://github.com/sdobz/esp32-hello
$ cd esp32-hello
$ nix-shell shell.nix
<wait for a long long time while we build the world>
[shell] $ ./all.sh /dev/ttyUSB0
...
 (2317) Rust: I, live, again!.

Installing rust

Following the manual vanilla rust is installed.

Additionally pkgs.cargo-generate was installed

Building a sample project using docker container

Following the readme the following command was used to make a demo project:

cargo generate --git https://github.com/MabezDev/xtensa-rust-quickstart

There were fixes required to the included flash command, namely:

Use docker image to flash chip
Chown the target directory to the current user as docker uses root
Use esptool.py rather than esptool due to packaging differences
Switch the interpreter to use env
Finally change the LED pin from 2 to 5 since that is what is on my hardware

The final flash script:

#!/usr/bin/env zsh

set -e

docker run -v $PWD:/code mtnmts/rust-esp32 xargo build
sudo chown -R $(id -u ${USER}):$(id -g ${USER}) target Cargo.lock

# change this for release flashes
BIN_PATH=target/xtensa-esp32-none-elf/debug/esp32

test -f $BIN_PATH.bin && rm $BIN_PATH.bin

# convert to bin
esptool.py --chip esp32 elf2image --flash_mode="dio" --flash_freq "40m" --flash_size "4MB" -o $BIN_PATH.bin $BIN_PATH

# flash
esptool.py --chip esp32 --port /dev/ttyUSB0 --baud 115200 --before default_reset --after hard_reset write_flash -z --flash_mode dio --flash_freq 40m --flash_size detect 0x1000 $BIN_PATH.bin

time ./flash
...
./flash  0.19s user 0.04s system 0% cpu 59.027 total

Well, a minute to rebuild the project isn't exactly okay. We will have to do something about that!

Building esp-idf project

That sample uses pure rust,

Next let's try again using the esp32-hello sample project, and the cton container see their writeup

Aaand this results in a boot loop. (see Fixing flash offset) for how that could have been fixed.

Let's get this up on the bench.

NixOS

The nix build system forces you to define your entire build system.

Following the quickhack instructions and translating to nix we can build llvm like so:

llvm-xtensa.nix

{ stdenv, fetchFromGitHub, pkgs }:

stdenv.mkDerivation rec {
  name = "llvm-xtensa";
  version = "33d79cce656c8c85c38832c8f52810875a3fbddf";

  src = fetchFromGitHub {
    owner = "espressif";
    repo = "llvm-project";
    rev  = "${version}";
    fetchSubmodules = true;
    sha256 = "1a433q374in781l7sjavdlajrhbd568jdr540n2qlgzvkas44g4v";
  };

   buildInputs = [
     pkgs.python3
     pkgs.cmake
     pkgs.ninja
   ];

  phases = [ "unpackPhase" "buildPhase" "installPhase" "fixupPhase" ];

  # http://quickhack.net/nom/blog/2019-05-14-build-rust-environment-for-esp32.html
  buildPhase = ''
    mkdir llvm_build
    cd llvm_build
    cmake ../llvm -DLLVM_ENABLE_PROJECTS="clang;libc;libclc;libcxx;libcxxabi;libunwind;lld;parallel-libs" -DLLVM_INSTALL_UTILS=ON -DLLVM_EXPERIMENTAL_TARGETS_TO_BUILD="Xtensa" -DCMAKE_BUILD_TYPE=Release -G "Ninja"
    cmake --build .
  '';

  installPhase = ''
    mkdir -p $out
    cmake -DCMAKE_INSTALL_PREFIX=$out -P cmake_install.cmake
  '';

  meta = with stdenv.lib; {
    description = "LLVM xtensa";
    homepage = https://github.com/espressif/llvm-project;
    license = licenses.asl20;
  };
}

The exact cmake configuration was developed iteratively as rust failed to compile. Important notes are to enable the clang project, enable installing utils, and to add xtensa to experimental targets

Packaging rustc

Packaging rustc was a nightmare. 4 full days of bashing on it, learning the intricacies of rust compilation, bootstrapping, nix conventions, and so forth.

The primary resource used was the rust nixpkg along with innumerable github issues describing various ways compiling rust has gone wrong. In addition the bootstrap.py rust source file was useful to know how the default system built rust.

At a high level the packaging process works like so:

Load the rust source from the MabezDev repo
Use binary.nix to download and patch for nix the exact toolchain required
Use makeRustPlatform to create a nix rust platform from the binary
Override the existing rustc packaging process to include llvm-xtensa, some custom config flags and src
Use fetchCargoTarball to pre-download the cargo deps
Unpack the vendor directory
Continue with the "standard" rustc packaging

Specific issues overcome:

Injecting llvm required very specific configuration flags
Nix packaging runs in a sandbox and can only download files with predefined hashes.
Rust uses rust to compile and will use curl to fetch binaries if they are not present
Choreographing fetching different binaries and injecting them into the rust build process was tricky
Injecting the cargo dependencies required a thorough understanding of the rust build support

rust-xtensa.nix

{pkgs}:
let
    llvm-xtensa = (pkgs.callPackage ./llvm-xtensa.nix {});
    lib = pkgs.lib;
    lists = lib.lists;
    fetchCargoTarball = pkgs.callPackage <unstable/pkgs/build-support/rust/fetchCargoTarball.nix> {};

    toRustTarget = platform: with platform.parsed; let
        cpu_ = {
            "armv7a" = "armv7";
            "armv7l" = "armv7";
            "armv6l" = "arm";
        }.${cpu.name} or platform.rustc.arch or cpu.name;
        in platform.rustc.config
        or "${cpu_}-${vendor.name}-${kernel.name}${lib.optionalString (abi.name != "unknown") "-${abi.name}"}";

    # bootstrap
    date = "2020-03-12";
    # from rust-xtensa github
    version = "1.44.0";

    rustBinary = pkgs.callPackage <unstable/pkgs/development/compilers/rust/binary.nix> rec {
        # Noted while installing out of band
        # https://static.rust-lang.org/dist/2020-03-12/rust-std-beta-x86_64-unknown-linux-gnu.tar.xz
        # https://static.rust-lang.org/dist/2020-03-12/rustc-beta-x86_64-unknown-linux-gnu.tar.xz
        # https://static.rust-lang.org/dist/2020-03-12/cargo-beta-x86_64-unknown-linux-gnu.tar.xz
        # https://static.rust-lang.org/dist/2020-01-31/rustfmt-nightly-x86_64-unknown-linux-gnu.tar.xz

        version = "beta";

        platform = toRustTarget pkgs.stdenv.hostPlatform;
        versionType = "bootstrap";

        src = pkgs.fetchurl {
            url = "https://static.rust-lang.org/dist/${date}/rust-${version}-${platform}.tar.gz";
            sha256 = "1cv402wp9dx6dqd9slc8wqsqkrb7kc66n0bkkmvgjx01n1jhv7n5"; # beta
        };
    };
    bootstrapPlatform = pkgs.makeRustPlatform rustBinary;

    src = pkgs.fetchFromGitHub {
        owner = "MabezDev";
        repo = "rust-xtensa";
        rev  = "25ae59a82487b8249b05a78f00a3cc35d9ac9959";
        fetchSubmodules = true;
        sha256 = "1xr8rayvvinf1vahzfchlkpspa5f2nxic1j2y4dgdnnzb3rkvkg5";
    };
in
rec {
    rust-src = src;

    rustc = (pkgs.rustc.override {
        rustPlatform = bootstrapPlatform;
    # override the rustc result attrs before calling
    }).overrideAttrs ( old: rec {
        pname = "rustc-xtensa";
        inherit version src;

        llvmSharedForBuild = llvm-xtensa;
        llvmSharedForHost = llvm-xtensa;
        llvmSharedForTarget = llvm-xtensa;
        llvmShared = llvm-xtensa;
        patches = [];

        configureFlags =
            (lists.remove "--enable-llvm-link-shared"
            (lists.remove "--release-channel=stable" old.configureFlags)) ++ [
            "--set=build.rustfmt=${pkgs.rustfmt}/bin/rustfmt"
            "--llvm-root=${llvm-xtensa}"
            "--experimental-targets=Xtensa"
            # Nightly because xbuild requires it
            "--release-channel=nightly"
        ];

        cargoDeps = fetchCargoTarball {
            inherit pname;
            inherit src;
            sourceRoot = null;
            srcs = null;
            patches = [];
            sha256 = "0z4mb33f72ik8a1k3ckbg3rf6p0403knx5mlagib0fs2gdswg9w5";
        };

        postConfigure = ''
            ${old.postConfigure}
            unpackFile "$cargoDeps"
            mv $(stripHash $cargoDeps) vendor
            # export VERBOSE=1
        '';
    });

    cargo = # see below
}

Packaging cargo

There was considerable difficulty in getting a version of cargo that worked with the other build tooling. Research said to use rustup to configure a custom toolchain, but rustup does not play well with nix packaging.

The most idiomatic solution was to use the existing cargo package and pass in our custom bootstrap platform and xtensa src.

    (pkgs.callPackage <unstable/pkgs/development/compilers/rust/cargo.nix> {
        rustPlatform = bootstrapPlatform;
        inherit (pkgs.darwin.apple_sdk.frameworks) Security CoreFoundation;
        inherit rustc;
    }).overrideAttrs(old: rec {
        name = "cargo-xtensa-${version}";
        inherit version src;
        cargoDeps = fetchCargoTarball {
            inherit name;
            inherit src;
            sourceRoot = null;
            srcs = null;
            patches = [];
            sha256 = "1w5fz966vf09p87xbxc5pm9xq4f1gx8a2vj7fskx30skkwb97d13";
        };

        # cargoVendorDir = builtins.trace "${cargoDeps}" null;
        postConfigure = ''
            unpackFile "$cargoDeps"
            mv $(stripHash $cargoDeps) vendor
            # export VERBOSE=1
        '';
    });

Installing bindgen, xbuild

After extensive experimentation I have yet to be able to package bindgen and xbuild in a manner that cargo is happy with, so I am using the more direct and less deterministic

$ cargo install bindgen
$ cargo install xbuild

Building the project

The static esp-idf installed by nix in a previous post does not work for the make process as used here because it is not cloned from git. This feels against nix philosophy, part of the packaging should be to remove that need.

Bindgen libraries

Getting bindgen to be happy required tweaking llvm-xtensa build flags, specifically enabling the clang project. Unfortunately it was on the far side of a 45m compilation to compile both llvm and rust.

Linking

Getting the linker appeased was difficult, I ended up finding a repository from mattcaron that used esp-idf v4.0

.cargo/config requires very specific linking instructions which differ significantly between idf versions. When it was incorrect the build completed but the resulting binary was a couple hundred bytes rather than several kilobytes

Unfortunately this required downgrading my idf tooling from 4.2-dev to 4.0, which thanks to nix was not too difficult. Here is a snippet from esp32-toolchain.nix, see an earlier post for the rest:

  toolHashes = {
    "xtensa-esp32-elf" = "06b6hw4m1jy79yw1mkj3kgibssrw4d4c5kbipbnckrivw107acw0";
    # "xtensa-esp32s2-elf"
    "esp32ulp-elf" = "02rnzkha3fvzx631y27l9nkzls2qky0v645d4pw888lxkx8p5il9";
    # "esp32s2ulp-elf"
    "openocd-esp32" = "00529xj2pmzy49w3j0wzxlw0phcbmx4vpkqbi0la88smwnqv0nqd";
  };
  version = "0a03a55c1eb44a354c9ad5d91d91da371fe23f84";

  tools = let
    toolInfoFile = fetchurl {
      url = "https://raw.githubusercontent.com/espressif/esp-idf/${version}/tools/tools.json";
      sha256 = "19dlp282mb6lpnwxc7l5i50cnqdj1qlqm5y9k98pr7wyixgj409g";
    };

Overriding python library version

Due to the wonders of version ranges idf 4.0 requires pyparsing < 2.4.0

Nix makes it possible to override a package version globally using a snippet like this (source):

let
  pythonPackageOverrides = self: super: {
    pyparsing = super.buildPythonPackage rec {
      pname = "pyparsing";
      version = "2.3.1";
      doCheck = false;
      src = super.fetchPypi {
        inherit pname version;
        sha256 = "0yk6xl885b91dmlhlsap7x78hk2rdr879fln9anbq6k4ca42djb6";
      };
    };
  };

  idf-package-overlay = self: super: {
    python2 = super.python2.override {
      packageOverrides = pythonPackageOverrides;
    };
  };

  pkgs = import <unstable> {
    overlays = [
      idf-package-overlay
    ];
  };
in
...

This seems to result in compiling python from source, a very long process that worked first time. Is there a way to substitute a library more rapidly? There are alternative constructions that might avoid it.

Fixing flash offset

After finally getting rust to compile, tooling installed, headers generating, and the code linking I ended up with a boot loop outputting something like this over serial:

rst:0x10 (RTCWDT_RTC_RESET),boot:0x17 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3f400020,len:30944
ets Jun  8 2016 00:22:57

rst:0x10 (RTCWDT_RTC_RESET),boot:0x17 (SPI_FAST_FLASH_BOOT)
flash read err, 1000
ets_main.c 371
ets Jun  8 2016 00:22:57

After reading a bunch of documentation and more importantly looking over the build artifacts of the standard c project I realized that the 0x1000 I had been using in my build command was an offset and I was overwriting part of the boot loader.

The fix was to flash an existing project onto the chip in order to write the partition-table and bootloader files, then use the 0x10000 as defined in the flasher args from the standard process.

build/flasher_args.json

{
  "write_flash_args": [
    "--flash_mode",
    "dio",
    "--flash_size",
    "detect",
    "--flash_freq",
    "40m"
  ],
  "flash_settings": {
    "flash_mode": "dio",
    "flash_size": "detect",
    "flash_freq": "40m"
  },
  "flash_files": {
    "0x8000": "partition_table/partition-table.bin",
    "0x1000": "bootloader/bootloader.bin",
    "0x10000": "esp32.bin"
  },
  "partition_table": {
    "offset": "0x8000",
    "file": "partition_table/partition-table.bin"
  },
  "bootloader": { "offset": "0x1000", "file": "bootloader/bootloader.bin" },
  "app": { "offset": "0x10000", "file": "esp32.bin" },
  "extra_esptool_args": {
    "after": "hard_reset",
    "before": "default_reset",
    "stub": "TRUE",
    "chip": "esp32"
  }
}

Results

a tty console displaying a message from rust from the microcontroller — It lives!

# First time
$ time bash -c "./build.sh && ./flash.sh /dev/ttyUSB0"
109.78s user 26.08s system 313% cpu 43.327 total
# Incremental, after editing main.rs
$ time bash -c "./build.sh && ./flash.sh /dev/ttyUSB0"
4.20s user 0.47s system 63% cpu 7.378 total

Next steps

Packaging rust in an overlay overwriting the existing rustc implementation might make compiling additional tools more sane

bindgen and xbuild should be packaged using the buildRustPackage convention.

The rust build should trigger the makefile, and the makefile should emit the bootloader and partition table.

The flash command should use the flasher args as output by the idf tooling.