New linux kernels for Oyster

Oyster
1

Enclave images for Oyster use Amazon Linux 4.14 kernels provided as blobs by nitro-cli. Some use cases like tuna images and serverless require custom and/or more modern kernels.

The bootstrap repo now has new kernels which can be built reproducibly.

There are some weird compatibility issues w.r.t nitro-cli and aws-nitro-util which need to be resolved for production use.

The following artifacts were built from the above bootstrap repo:
Kernel - https://artifacts.marlin.org/oyster/kernels/vanilla_7614f199_amd64/bzImage
Kernel config - https://artifacts.marlin.org/oyster/kernels/vanilla_7614f199_amd64/bzImage.config
NSM - https://artifacts.marlin.org/oyster/kernels/vanilla_7614f199_amd64/nsm.ko
Init - https://artifacts.marlin.org/oyster/kernels/vanilla_7614f199_amd64/init
Linuxkit - https://artifacts.marlin.org/oyster/kernels/vanilla_7614f199_amd64/linuxkit

nitro-cli

  • (!!) not compatible with the linuxkit built above
  • compatible with the new kernels and NSM
  • works with both old and new init
  • works with default cmdline

aws-nitro-util

  • does not use linuxkit
  • compatible with the new kernels and NSM
  • (!!) works only with old init, not new
  • (!!) works only with specific cmdline from aws-nitro-util, not default nitro-cli cmdline
2

Update on init.

Tried building init explicitly with the flake using the below snippet which works. The commit is the last one which still has a Makefile.

				init = pkgs.stdenv.mkDerivation {
					name = "eif-init";
					src = (pkgs.fetchFromGitHub {
						owner = "aws";
						repo = "aws-nitro-enclaves-sdk-bootstrap";
						rev = "62dd55c";
						sha256 = "sha256-OzLTHLX7pnrmldJJHx9xrb48ZOWawpMwrVL4gsGDlN8=";
					}) + "/init"; # we just need the subfolder of this repo

					nativeBuildInputs = [ pkgs.gcc pkgs.glibc.static ];
					buildPhase = "make";
					installPhase = "cp -r ./init $out";
				};

Built using gcc explicitly instead of make which also works.

				init = pkgs.stdenv.mkDerivation {
					name = "eif-init";
					src = (pkgs.fetchFromGitHub {
						owner = "aws";
						repo = "aws-nitro-enclaves-sdk-bootstrap";
						rev = "62dd55c";
						sha256 = "sha256-OzLTHLX7pnrmldJJHx9xrb48ZOWawpMwrVL4gsGDlN8=";
					}) + "/init"; # we just need the subfolder of this repo

					nativeBuildInputs = [ pkgs.gcc pkgs.glibc.static ];
					buildPhase = "gcc -Wall -Wextra -Werror -O2 -o init init.c -static -static-libgcc -flto && strip --strip-all init";
					installPhase = "cp -r ./init $out";
				};

Both of the above methods produce the same PCRs so they should be identical. Which makes sense since the gcc command is from the Makefile.

Switching to the latest commit, which also works!!

				init = pkgs.stdenv.mkDerivation {
					name = "eif-init";
					src = (pkgs.fetchFromGitHub {
						owner = "aws";
						repo = "aws-nitro-enclaves-sdk-bootstrap";
						rev = "7614f19";
						sha256 = "sha256-jcdxssY3m/YAMIZSscZtnbPeAdFT5on2evD58YNwyxE=";
					}) + "/init"; # we just need the subfolder of this repo

					nativeBuildInputs = [ pkgs.gcc pkgs.glibc.static ];
					buildPhase = "gcc -Wall -Wextra -Werror -O2 -o init init.c -static -static-libgcc -flto && strip --strip-all init";
					installPhase = "cp -r ./init $out";
				};

And produces the same enclave image as before with the following PCRs:

enclave> {
enclave>   "HashAlgorithm": "Sha384 { ... }",
enclave>   "PCR0": "f2fc4cdb0f563103998d6283e9b18e01fa6ac8e5f0625372d9ff36ef18da9bc1da89c266e2db517bbf0ce79670dab1bd",
enclave>   "PCR1": "187d678b9f1988ae987d42a113d67da3bb8ab07a19d46ca2c7dee6e04e47c4280b01d696e575e0400349a78545edde46",
enclave>   "PCR2": "1dc9cd9945ea55fec6095a609e9a3df1ea8a3b988664b1fb8c234bd5b1fdd0daaecf87b9c0d98378977063fc2001c2ba"
enclave> }

This indicates that the problem is not with the init per se.

Weirdly, the PCRs are different from when I use the init from the artifacts:

enclave> {
enclave>   "HashAlgorithm": "Sha384 { ... }",
enclave>   "PCR0": "c7fa718d1c7480a01cd2eb21b110405ce9253168555aaaed656187cd53ffc537547ce736060b0fa1aeb4fac3e026b81a",
enclave>   "PCR1": "f53390d050384663e092ae7c8d12249d7c24869e1f925b0a1c6e7e4e10ef44747441e4cb585bce7b3d5c4a528987e067",
enclave>   "PCR2": "1dc9cd9945ea55fec6095a609e9a3df1ea8a3b988664b1fb8c234bd5b1fdd0daaecf87b9c0d98378977063fc2001c2ba"
enclave> }

which indicates that both inits are not the same, which is surprising for builds that should be reproducible. The gcc command is the exact same as in init’s nix file. Investigating the build pipeline for any issues.

3

Summary of investigations into aws-nitro-util compatibility issues below. All issues are now considered resolved.

init

init just needed to be made executable, it should ideally have been in the aws-nitro-util repo itself but it works if we do it ourselves.

The binaries are not exactly the same as a local build since the init build along with the kernel uses a different nixpkgs version. But the “latest commit” snippet also works just fine.

cmdline

cmdline now works as well, the buildEif derivation was expecting a string as the cmdline while nitro.blobs.${eifArch}.cmdLine was pointing to a file. All that was needed was to read the contents and supply that. In retrospect, the difference was visible in the build logs and some validation could have made it easier to detect and fix.