We are designing the upgrade mechanism for:

  • Efficiency - We use next level technology git to manage the entire root filesystem
  • Help Webconverger Limited the business to move effectively to a subscription model
  • We can easily check the integrity of the filesystem, good for security
  • Git is fast & efficient for upgrades, also good for security
  • We can safely downgrade and roll back to an earlier version
  • Every commit accountable and transparent

The upgrade features requires persistency, i.e. install to a writable medium

  • USB-HDD (.img) version which can upgrade itself or do a install - related issue
  • ISO (.iso) version which can live boot and a install

Upgrade process

Webconverger uses git to maintain the filesystem.

  1. Webconverger downloads a config and looks for fetch-revision=BRANCH, otherwise defaults to master
  2. On boot it fetches upto that branch, unless update-revision=SHA is specified, which is used for mounting exact commits
  3. If there is an update, the kernels upon /live/image/live are updated and a new git-revision based on the fetch branch is applied
  4. On REBOOT the script /usr/share/initramfs-tools/scripts/live in the initrd fuse.git-fs mounts the git-revision SHA1 set upon the /proc/cmdline

17.x UPDATE: Mounting scripts move to /lib/live/boot and /usr/share/initramfs-tools/hooks is how they are included.

Note: After changing anything in /usr/share/initramfs-tools/, re-generate the initrd and commit them like so:

update-initramfs -u -k all

In future we hope to utilise the boot-once option in extlinux, where by if the machine failed to boot from a kernel error, a subsequent boot will revert to the previous "known working" git-revision and kernel.

How do I roll back / downgrade ?

This is largely for debugging, using update-revision=. Supported clients will "fast forward" out of trouble.

No upgrades on Live version since it isn't a writable disk

If the logs say Not a writable boot medium, then you are in Live mode and you cannot upgrade until you've installed to a writable disk.

Two reboots for an update, <48hr delay

Webconverger currently only fetches on boot, so if an update is pushed, an already running system needs effectively two boots, one to fetch, and one to apply the changes to come up on the revision.

In practice a lot of machines are shutdown overnight. So for example an update that is pushed on a Monday afternoon, would be downloaded on the Tuesday boot and applied on the Wednesday boot.

We want to get this time down. For example an option that syncronously checks for new commits upon https://github.com/Webconverger/webc/commits/master, fetches and reboots immediately for security concious deployments.

noupgrade boot API disables upgrades

Disabling upgrades will disable Webconverger from doing a git fetch on boot.

Notes on validation

  • using hashes as update-revision, you'll do a sort of implicit validation
  • signed tags might be useful, but we'll have to have some way to circumvent that so we can push debug revisions without having to tag all of them

Quick notes on testing before a push

  1. Commit to master, make a note of sha1
  2. git push origin master:test before pushing to master, push to a remote branch test
  3. Append fetch-revision=test debug to your test instance of Webconverger
  4. Ensure sha1 was fetched
  5. Reboot and check the update was applied
  6. Test the update one more time
  7. git push the chroot to master to let everyone else enjoy the update
  8. Perhaps tweet the change
  9. Clean up git push origin :test and delete remote branch on https://github.com/Webconverger/webc

Note one should probably work from a branch, e.g. git branch flashtest; git push origin flashtest -u and then merge to master for a push

Checking the size of an upgrade in debug mode

TODOs and issues

  • make fetch happen periodically from /etc/inittab to reduce upgrade delay?

For more background, see these gitupgrades notes.

Does git-fs use a lot of RAM?

Our memory requirements have gone up and by our calculations, it's about 200M of memory usage for git-fs, which isn't a lot.

How to integrate git-fs from scratch

/usr/share/initramfs-tools/hooks/git-fs ensures the git-fs binary is in the initramfs.

/usr/share/initramfs-tools/scripts/live-bottom/00git-fs prepares the /live for mounting.

Notice https://github.com/Webconverger/webc/tree/master/.git-fixups which needs to run before switching.

/lib/live/boot/overlay.sh is where the actual git-fs mount happens.

/etc/webc/upgrade.sh implements the {fetch,update}-revision= API that fetches updates in the background.

OPTIONAL: mount_git function for mounting the .git/ directory for debugging.