Introduction

Nix deployments can be very bandwidth-intensive, and in certain deployments such as spacecraft or other very remote systems this can become a major hurdle.

This is the problem DeltaNAR aims to solve.

By computing the delta between the desired deployment state & what already exists in the Nix store on the host we can drastically reduce the bandwidth required to push update closures.

Installation

For closure size reasons DeltaNAR is distributed as 2 separate Nix packages:

The packing program

This has a relatively larger set of dependencies & is not optimised for closure size.

The unpacking program

Optimised for closure size & has as small of a dependency set as possible.

Flakes

{
  description = "DeltaNAR usage";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
    deltanar.url = "github:nixos/adisbladis/deltanar";
    deltanar.inputs.nixpkgs.follows = "nixpkgs";
  };

  outputs =
    {
      self,
      nixpkgs,
    }:
    {
      devShells = forAllSystems (system: let
        pkgs = nixpkgs.legacyPackages.${system};
      in {
        default = pkgs.mkShell {
          packages = [
            deltanar.packages.${system}.pack
            deltanar.packages.${system}.unpack
          ];
        };
      });
    };
}

Classic Nix

You can just as easily use deltanar without using Flakes:

let
  pkgs = import <nixpkgs> { };
  inherit (pkgs) lib;

  deltanar = pkgs.callPackage (builtins.fetchGit {
    url = "https://github.com/adisbladis/deltanar.git";
  }) { };
in
  deltanar.pack

Getting started

This tutorial shows how to:

Set up prerequisites
Create a file containing the delta between what’s on the target host & deployment closure.
Unpack file into a Nix store or a binary cache
Populate the local Nix store

These steps apply to a host called spacecraft.

Creating gcroots

To calculate a diff DeltaNAR needs to know what is already in the store of the system being deployed to.

This is achieved by using a gcroots[1] mechanism mimicking that of Nix, with an additional level of structure imposed: There is one gcroots child directory per host.

Tip

It’s a good idea to symlink the DeltaNAR directory into /nix/var/nix/gcroots/ so the deployment host doesn’t garbage collect closures it requires for delta computation.

Steps

First, create a gcroot directory for host spacecraft:

mkdir -p gcroots/spacecraft

Symlink an already deployed NixOS generation into the gcroots directory:

ln -s /nix/store/5vg80fas99lkn1a5i2bnwgwd3ia3i82m-nixos-system-nixos-26.05pre-git gcroots/spacecraft

Note

DeltaNAR doesn’t contain a mechanism for managing gcroots. This needs to be done either manually or through custom scripting.

Packing

dnar-pack --gcroots ./gcroots --host spacecraft --path /nix/store/7mdg60drrnh0wq1j8hmmbhll47czm107-nixos-system-nixos-26.05pre-git

This will create delta.dnar in the current working directory.

Unpacking

To a Nix store

dnar-unpack nix-store-export | nix-store --import

Will unpack delta.dnar from the current working directory into a local Nix store.

This mode is particularly useful in deployment pipelines.

To a binary cache

dnar-unpack binary-cache --cache my-cache

Will unpack delta.dnar from the current working directory into a local binary cache directory at my-cache with the same layout as nix copy, which can then be imported using nix copy:

nix copy --from file://$(readlink -f my-cache) --all --no-check-sigs

This mode is particularly useful when deploying closures to a remote facility with multiple hosts.

Compression

DeltaNAR files are uncompressed, and compression is left up to the user. To pipe the DeltaNAR output use the special input/output argument -:

dnar-pack ... --out - | xz > delta.dnar.xz
xzcat delta.dnar.xz | dnar-unpack ... --input -

Deduplication

DeltaNAR tries to achieve maximum deduplication by doing multiple levels of analysis of what’s being deployed.

CDC

Individual files in the Nix store are chunked using a content defined chunker.

A file is then transferred as a list of chunks. If a sub-file chunk already exists in the target Nix store (even in another store path), it is taken from the existing chunk, completely avoiding re-sending the data.

A chunk the target is missing is sent as content addressed data, or — when a similar file is already present — as a binary delta against it.

File

To avoid packing a long list of chunk entries for files which are fully identical, a hash per file is also computed. If a file hash matches exactly, its contents will be reused in full.

Delta compression

Deduplication reuses content the target already has byte for byte.

But a file that changed between two closures is often only slightly different from one the target already has.

DeltaNAR exploits this with opportunistic binary delta compression: a chunk the target is missing can be reconstructed from a similar file it already has, so only the difference needs to be sent.

This happens per chunk, as a fallback for the chunks that did not match exactly while a file is packed.

Choosing a reference

For each file, DeltaNAR counts which local files the matched chunks came from and picks the file contributing the most matches as the reference. That file shares the most content with what’s being packed, so it is the most likely to also resemble the chunks that didn’t match.

Encoding delta

Each missing chunk is diffed against the whole reference file. If the resulting delta is smaller than the chunk itself, the chunk is packed as a delta referencing that file plus the delta payload. Otherwise the chunk is sent verbatim as content addressed data.

Reconstruction

The target already has the reference file, so on unpack a delta chunk is reconstructed by applying its delta to that file. Nothing beyond the small delta itself has to be transferred.

DNAR format

The DNAR format is specified using Protobuf.

syntax = "proto3";

package dnar;

option go_package = "github.com/adisbladis/deltanar/dnar";

// Protocol specification for the DeltaNAR format
//
// The DeltaNAR protocol:
// 1. A StreamHeader(len(nar))
// 2. Multiple NAR
// 3. A StreamHeader(len(caChunks))
// 4. Multiple CAChunk
// 5. A PathTrailer

// Sent before a stream of other messages indicating how many messages will follow
message StreamHeader {
    uint64 length = 1;
}

// A file being used as an input to write another file
message FileDescriptor {
    uint32 store_path = 1; // Store path offset in DnarHeader.paths
    string path = 2;
}

// A content addressed chunk
message CAChunk {
    bytes data = 1;
}

// A file within a NAR
message NarFile {
    string path = 1;

    message ChunkDescriptor {
        oneof chunk_type {
            // Read from CA chunk
            CAChunk ca = 1;
            // Read from file descriptor
            FDChunk fd = 2;
            // Reconstruct from a delta against an existing base chunk
            DeltaChunk delta = 4;
        }

        message CAChunk {
            uint64 index = 1; // CA chunk index in chunk stream to read from
        }

        message FDChunk {
            uint64 index = 1; // File descriptor index in DnarHeader.files to read from
            uint64 size = 2;
            uint64 offset = 3;
            bytes digest = 4; // Chunk digest (verify that existing store contents match)
        }

        // A chunk that's not present on the target but can be reconstructed from a binary delta.
        message DeltaChunk {
            uint64 index = 1; // File descriptor index in DnarHeader.files
            uint64 size = 2;
            uint64 offset = 3;
            bytes digest = 4;
            uint64 delta_index = 5; // CA chunk index in the chunk stream holding the delta payload
            uint64 result_size = 6; // Size of the reconstructed chunk
        }
    }

    oneof file_type {
        RegularFile regular = 2;
        DirectoryFile directory = 3;
        SymlinkFile symlink = 4;
    }

    message RegularFile {
        uint64 size = 1;
        repeated ChunkDescriptor chunks = 2;
        bool executable = 3;
    }

    message DirectoryFile {
        // If a directory hash was matched copy the directory from here
        // Note that unlike the chunk index this is an _int64_ where a -1 signifies
        // that the directory does _not_ already exist on the remote.
        int64 from = 1;
    }

    message SymlinkFile {
        string target = 1;
    }
}

// Top level file header
message PathTrailer {
    repeated string paths = 1;
    repeated FileDescriptor files = 2;
}

// NAR
message NAR {
  // Core data fields
  string path = 1;
  repeated NarFile files = 2;

  // Narinfo metadata (extracted from local store using nix path-info --json)
  string narHash = 3;
  uint64 narSize = 4;
  repeated string references = 5;
}

Acknowledgements

DeltaNAR is sponsored by OroraTech🚀

Keyboard shortcuts

DeltaNAR