This adds support for dm-vertiy on the root filesystem.
Currently only squashfs is supported.
Three new flags are introduced.
* --dm-verity: Enable basic dm-verity support
* --dm-verity-fec NB_ROOTS: Enable forward error correction. Optional
* --dm-verity-sign SCRIPT: Specify signing script for the root hash. Optional
1) lb config rejected multiple checksum types
2) When using the installer, cdrom-checker requires a md5 checksum file,
use 'Check the integrity of the installation media' in the installer
3) The comments in the first lines of the checksum files caused
cdrom-checker to fail the integrity of the image
If grub/splash.png exists, assume the configuration editor intends
to have a grub-specific splash.png, and do not modify theme.txt.
But if syslinux has the only known splash.png, use it for both
syslinux and grub.
(This allows for a hybrid image where the grub side can have e.g. a
16:9 1920x1080 splash.png which gets grub is capable of automatically
scaling, while the syslinux side has a 640x480 splash.png which
effectively must be this fixed size.)
When a package lists contains only packages protected by a test
that doesn't match for the current run, then Expand_package_list
outputs nothing and the following "grep -v" fails because it
has not filtered anything. Avoid this by protecting the "grep -v"
call with "|| true".
Before commit 9f3e5fe8d (Fix the way the .disk/mkisofs file is created)
all these commands (`mkdir`, write to `binary/.disk/mkisofs` and
`xorriso`) were in the same `binary.sh` script. Since that commit, the
write was extracted, to prevent issues with quoting, but the related
mkdir was left in `binary.sh`. This means that the write is now executed
first, and the `mkdir` only afterwards, making the `mkdir` quite pointless.
In practice, this did not break becaue binary_disk also does the same
`mkdir` and runs before `binary_iso`, but if one runs commands manually
and skips `binary_iso`, then this does break.
Even though this is not really a supported usecase, just move the mkdir
outside of `binary.sh`, so it runs *before* the write again as intended.
Moved includes.chroot to includes.chroot_after_packages and added
includes.chroot_before_packages. includes.chroot does still work as before.
We also now use rsync for copying files if it is installed.
This improves runtime and space consumption for large includes.
Gbp-Dch: Short
Closes: #927128
Installation of flatpaks doesn't work with normal chroots.
This patch enables support for using systemd-nspawn in hooks.
Gbp-Dch: Short
Closes: #965953
avoids spitting out warning
> [2020-06-07 22:30:32] lb binary_grub-efi
> P: Begin preparing Grub based EFI support...
> Reading package lists...
> Building dependency tree...
> Reading state information...
> Package grub-efi-amd64-signed is not available, but is referred to by another package.
> This may mean that the package is missing, has been obsoleted, or
> is only available from another source
>
> E: Package 'grub-efi-amd64-signed' has no installation candidate
> W: UEFI Secure Boot disabled due to missing signed Grub/Shim.
this simplifies things to avoid the messy recursion.
it is also a necessary step to implementing handling of common options
like --debug. we need to process all options to decide how to approach
doing things (e.g. with debug messages to be output or not) before actually
performing any work, in order for options like --debug to be reacted to
properly.
also, as things were, options like `--debug` were not being passed along
in the recursive executions, while now that is no longer an issue.
the order of file/folder deletions for `--all`|`--purge`|`--remove`
actions is slightly changed here, but i don't see any issue with that and
it's cleaner to not preserve that.
Gbp-Dch: Short
This reverts commit 665372c19d.
the kali test failed due to their script using a hack of passing along a
custom option after an early terminator through to their auto/config file.
the change implemented here was valuable, but i'll have to look into
revising things to work with what Kali is doing.
Gbp-Dch: Ignore
some scripts temporarily install packages to accomplish some work before
then removing them. the list of packages installed is kept in memory in a
variable.
a weakness of this design is that if a failure occurs or the user cancels,
and then following this the user re-runs `lb build`, letting it try to
pick up and recover from where it left off, that list of packages that had
been installed is lost, resulting in those packages that were installed
then being a permanent part of the chroot.
here we fix this weakness by backing up the list to a file, which is always
read from on removal. thus in a recovery situation, any packages still
installed from a situation like that just described, will be removed upon
the next use of `Remove_package()`.
this is not perfect, since we are having to wait for opportunistic
execution of the remove function. we need to find a suitable place for the
`Cleanup_temp_packages()` function to be used.
- doing so in `Init_config_data()` would not be suitable because we don't
hold the lock when that's run, even if we ignored the hijacking of that
function for taking such action...
- doing it in `Exit()` doesn't seem a good fit either.
- putting it explicitly in every build script just seems a little messy...
perhaps a local exit trap like for removing the lock...?
note that `binary_rootfs` skips running the remove function after installing
tooling, since it just throws the wrapper chroot away, which then leaves the
file around with stale data for the next remove instance to pick up, which
then does not actually remove it because it's not installed. this is not
ideal either... perhaps the optimisation should be removed from that script?
Gbp-Dch: Short
after handling arguments, catch and report any remaining "non-option"
arguments.
for instance users could make the basic mistake of using
`lb config --bootloaders syslinux grub-efi`, i.e. failing to quote the
multiple bootloaders (i actually encountered a user doing this who swore
that "it just worked"). catching and reporting such mistakes could be
valuable to users.
previous behaviour:
```
$ lb config --bootloaders syslinux grub-efi
P: Updating config tree for a debian/buster/amd64 system
P: Symlinking hooks...
```
new behaviour:
```
$ lb config --bootloaders syslinux grub-efi
[2020-05-05 18:56:07] lb config --bootloaders syslinux grub-efi
E: Unexpected argument found: grub-efi
```
Gbp-Dch: Short
debootstrap must obviously exist in the host. we thus must pass 'host'
not 'chroot' such that a warning about needing to install it on your host
system is output, rather than it being added to a list of packages to be
installed, which never occurs in this script.
Gbp-Dch: Ignore
when loopback support was introduced, it initially duplicated the code
for generating a grub2 config, before the duplicated code was removed from
the grub-pc script, effectively thus moving grub config generation to the
loopback feature script.
grub-efi support was added after this.
this results in a misleading filename, since the `binary_loopback_cfg`
script is essential for use of grub-pc|grub-efi, and actually only has a
single line of code on top that's needed for adding actual loopback
support on top. (when grub-pc and grub-efi are not used, the entire script
is still needed for loopback support to work).
so here we rename it to make better sense, and correct/clarify bits of
documentation.
Gbp-Dch: Short
with LB_BOOTLOADER_BIOS and LB_BOOTLOADER_EFI introduced, we can simplify
and correct things here.
previously it was possible for more than one of each type to be added.
Gbp-Dch: Short
the design choice from when EFI support was introduced was to change
`--bootloader` to `--bootloaders`, with users specifying their selection
of BIOS and EFI bootloaders together. at this time there were not even any
decent validation checks being performed, and invalid combinations could
cause some chaos.
since then proper validation was put in place, including checking that
only a single instance of each of BIOS and EFI bootloaders exists in the
selection.
here we tweak things such that we stick with the same option, but we split
the selection up such that we store the BIOS and EFI selections separately
within the saved config file, and offer it up to scripts to help simplify
those scripts.
we must however retain support for splitting from the combined option,
both because we still use it in the combined option, and for backwards
compatibility with older saved configs.
Gbp-Dch: Short
thus far, config bootloader validation only did the basic check that each
bootloader specified was a known and supported bootloader, it did not check
combinations.
it now checks combinations, and strips out the previous "bootloader role"
stuff.
the no-bootloaders warning is duplicated, covering two slightly different
situations (empty string, and whitespace string). this is anticipated to
be just temporary, with this just being the first step in better handling
bootloader selections.
Gbp-Dch: Short
"grub" presumably was what is now called "grub-legacy"
removed both because there is already a proper piece of code adding
bootloader packages.
Gbp-Dch: Ignore
while `lb build` ran the config validation check, spotting invalid configs
and stopping with an error, the major build stages if executed directly did
not, nor did the component stages.
here we move execution of the validation function into the common init
function, with an exported variable used to indicate that validation has
been performed. thus validation is performed no matter what part of the
build system you execute, but only once.
Gbp-Dch: Short
`Init_config_data` is more suited to build scripts than here. note that
it's not used in `config` either. this deliberately does not pass along
arguments to it because `Arguments()` can only handle the basic common
options, not the `clean` set. this is somewhat confusing and causes a
pointless execution of `getopt`. furthermore the function is expanding
slightly further beyond it's original scope, with yet more change coming
that's unsuitable for `clean`, so it makes sense to avoid using the
function here just for the couple of function calls needed.
validation of the config is disabled, as it is not performed currently.
it is not clear if it should be enabled or not for `clean`. it may be
useful to not validate, if we wish to be able to provide users with an
option to be able to delete the config.
Gbp-Dch: Ignore
using `$VERSION` as part of the default `$LB_ISO_PREPARER` means that when
you simply run `lb config` once, this variable is stored as a part of the
string, and replaced on use, but if you run `lb config` twice, it gets
replaced with a fixed version, that is then used in all subsequent builds.
let's replace with a placeholder (`@LB_VERSION@`) that can be used both in
the default, or in user strings, and will be replaced on use only.
this means that subsequent builds will always reflect the actual version
of live-build used.
Gbp-Dch: Ignore
`DATE_UTC_OPTION` is set in `Prepare_config()` for use by scripts, even
though only a few scripts will actually use it, since it allows those
scripts to be cleaner. we may want to possibly extend this as a
`DATE_OPTIONS` variable perhaps as part of enabling proper reproduciblity.
Gbp-Dch: Short
since date is not obtained as UTC, timezone is an important detail of
understanding the given time, which users may want to make use of.
Gbp-Dch: Short
don't construct each part from a fresh "now", which can result in
inaccuracies in the overall date due to "now" drifting over the individual
date calls. instead feed the full date that was obtained back into it when
extracting the component parts.
Gbp-Dch: Ignore
- the definition of $PROGRAM as used in $USAGE strings defined in each
script has been broken for a long time, being simply "lb" when it needs
to be "lb COMMAND".
- `config` changed $PROGRAM to "lb config" thus its output was correct in
this regard unlike everything else, but with the switch to a more
"intelligent" `Man()` function recently, it means that instead of
`man lb config`, what was actually run was `man lb config config`,
which displayed the manpage, then on exiting with `q`, it showed some
sort of index line todo with a "config" search (no exact manpage
match?), for which you had to enter `ctrl+c` to get rid of.
this revises things to fix the issues, minimising change by changing
$PROGRAM to "lb COMMAND", with the frontend overriding this.
Gbp-Dch: Ignore
did not properly consider all usage cases properly in deciding placement.
this captured `--usage` in `$PROGRAM --usage` as the action for instance.
Gbp-Dch: Ignore
these should be bypassing the possibility of running the auto script
surely, just like how this is done for `--all`, otherwise you pass through
the auto file twice.
Gbp-Dch: Ignore
- detect lack of options using actual arg count rather than first arg
being an empty string.
- fix string splitting issues by looping properly on "${@}"
- tidier
Gbp-Dch: Ignore
...alongside printing usage (which is perhaps unnecessary), so that it is
actually clear to users that a problem occurred, and what.
and capture it before option processing of remaining args.
Gbp-Dch: Ignore
- it's only used by the debootstrap script after alternatives were dropped
long ago, so let's move it, avoiding it being loaded for everything
else.
- there's no need to pass printing another message through it.
- there's little point in making the sid distinction if you happen to
decide to build sid, it's a given that it's less stable than stable.
really, is there any need for this at all?
Gbp-Dch: Ignore
in tweaking a previous commit to remove some excessive change before
submission, i mistakenly identified the part of the sed replacement
restored here as being unnecessary to its functionality, but in fact it is.
without it the placeholder is not actually removed.
the lack of removal of the placeholder meant that you ended up with
duplicate copies of the live menu entries.
Gbp-Dch: Ignore
more fitting now that we've moved the advanced installer entries
out to a different submenu, leaving just memtest (and HDT on
syslinux).
the advanced.cfg file is also renamed to utilities.cfg in the syslinux
case, but in a backwards compatible way of moving the user advanced.cfg
file over the new one, if the user provides a file with the old name.
alternatively we could just leave the old name in place, but that would be
a little odd.
Gbp-Dch: Short
This is considered to be more robust.
Two instances remain:
scripts/build/chroot_archives, line 257:
if [ "${LB_APT}" = "aptitude" ] && [ ! $(Chroot chroot "which aptitude") ]
The command is run inside a chroot where the environment might be special,
and would need further testing.
manpages/Makefile, line 42:
@if [ ! -x "$$(which po4a 2>/dev/null)" ]; \
I am insufficiently familiar with makefile syntax to edit this.
instead of cramming the live entries into a string via a layer of functions
for terminating entries with newlines, which we then have to run through
perl to tweak the newlines for correct use with sed... let's write the
entries to a temp file, then use that file in the sed replacement.
the helper functions injecting newlines to the end of entries as they were
built into a long string have obviously become unnecessary and so were
removed. one function was renamed for reasons of consistency and clarity.
the file is initially deleted before use for reasons of wanting to bullet
proof the codebase to work properly under conditions of recovering from
failure/cancellation, `--force` re-running and such.
this removes the last use of perl.
Gbp-Dch: Short
it reported: "possible bashism in scripts/build/binary_loopback_cfg line 284 (should be '.', not 'source')"
which is clearly a misidentification.
Gbp-Dch: Ignore
partly lost in some adjustments that were made to the submitted work,
which was focused on restoring the 'start installer' entry.
there is no need for dynamic setting of these two `source` imports in the
default file, in fact user modifications should also use the fixed import
commands in future.
note that the old placeholders however remain replaced, which inject
precisely this string, for backwards compatibility.
Gbp-Dch: Ignore
during testing i encountered an unexpected error resulting from the
following condition:
- bootstrap was cached
- cache of bootstrap packages was empty (from playing with `lb clean`)
- installer was not none|live
everything works fine up until the main installer script, which comes to
an abrupt halt with an error due to the missing cached bootstrap packages
that it wants to copy.
this situation is easy to unintentionally create, as i managed to do.
here we catch the failure condition, correcting for it.
this is done by checking for the missing cached packages when restoring
the bootstrap from the cache, and skipping this if the packages were
missing, thus forcing the bootstrap to be rebuilt. the packages should
then be found within the cache, allowing the installer stage to complete
successfully.
of course the bootstrap stage will only cache the packages if caching is
enabled, but if caching is disabled and installer enabled, the config
validation will catch that, reporting that problem.
Gbp-Dch: Short
install mode was silent when selinux was not enabled, whilst remove mode
always output a "Begin unmounting..." message. this makes both modes silent
when selinux is not enabled.
this also happens to fix an unintended side effect of
d79c96232b whereby a warning subsequently
was always emitted in remove mode when selinux was not in use, which was
not ideal.
Gbp-Dch: Short
The unconditional SVG to PNG conversion could overwrite a splash.png
provided by the user. Ensure we don't overwrite such a file. But we
still remove the SVG file as syslinux is not able to make use of it.
there are several files of which identical duplicate copies are held in:
- share/bootloaders/extlinux
- share/bootloaders/pxelinux
- share/bootloaders/isolinux
- share/bootloaders/syslinux
it is a pain to maintain this from a development standpoint, having to
copy modified config files into the other directories each time changes
are made and mistakes have been made before due to this.
this creates a new folder share/bootloaders/syslinux_common and moves them
to this new directory.
it also expands the binary_syslinux stage to use it, with it now
constructing the installed set of bootloader files as follows:
1. copy {LB_DIR}/bootloaders/syslinux_common
2. copy {LB_DIR}/bootloaders/{syslinux|isolinux|extlinux|pxelinux} on top
3. copy config/bootloaders/syslinux_common on top
4. copy config/bootloaders/{syslinux|isolinux|extlinux|pxelinux} on top
note, to explain part of the binary_syslinux change, instead of just
copying the correct bootloader folder full of the files, we now make the
target bootloader specific directory, then copy the contents of source
directories into it.
Gbp-Dch: Short
Due to this mistake, the helpers were not called in reverse order
during the removal step. This lead to things like "apt update" failing
because a broken /etc/resolv.conf has been restored before the call
to "chroot_archives remove".
Gbp-Dch: Ignore
introduced by an issue with the implementation of
91d446d93e
the introduced of that commit caused builds to fail doing `apt-get update`
or downloading packages and such.
this tweak fixes the problem.
Gbp-Dch: Ignore
just as most scripts are skipped if their stagefile exists (indicating
that they have already been run to completion), including chroot
preparation scripts in install mode, this implements the same guard for
chroot prep remove mode, such that they exit early if their stagefile
does not exist, indicating that the modification has already been removed.
(also override-able by --force in the same way).
this basically just uses a tweaked copy of Check_stagefile().
Gbp-Dch: Short
no backwards compatibility hack for reading the old var from existing
saved config used because this was previously stored in the alternate
format config/build file.
Gbp-Dch: Short
no backwards compatibility hack for reading the old var from existing
saved config used because this was previously stored in the alternate
format config/build file.
Gbp-Dch: Short
back in v4.0~a6-1 a transition process was started to move the live-build
config to a new format. the new format was INI style, and required
parsing functions to read/write values, compared to the existing format
which was just shell script code setting variables.
this partial transition is the explanation for the existence of the
`New_configuration()` function, and understanding this is important to
understanding the purpose of it - it is not in fact intended for creating
a new configuration, it is just related to the new config format
transition.
the positives of the new format were that it was somewhat cleaner looking,
while the negative was the terrible relative efficiency.
the file `config/build` was created to hold options in this new format.
the transition was only ever completed for a handful of config options:
- architecture
- archive areas and parent archive areas
- live image name
- live image type
a 'configuration version' attribute was also saved, which is not used by
anything.
the bootstrap-mirror and parent-bootstrap-mirror attributes are pointlessly
stored in it seemingly resulting from work done in v4.0~a17-1. (they are
also stored in another config file from which the value is actually used).
it in fact seems to have been a source of confusion for Raphaël in
authoring 44b9b0a650, since the new
`[parent]-distribution-{chroot|binary}` options it introduced were stored
both in `config/bootstrap` and in `config/build`, while only used from the
former. i expect, understandably, that he thought that `config/build` was
just an information file.
Gbp-Dch: Short
the grub-pc image creation code has no business being in binary_iso, it
should be in binary_grub-pc.
it should be noted that the binary_iso script did not even have the
necessary package check for grub-mkimage, while binary_grub-pc did have
it, pointlessly.
Gbp-Dch: Short
live-build might be run from a local folder rather than the system
installation, so the dpkg version number should not override the version
picked up from VERSION
if we care about the possibility of the installed package version
potentially differing from the version of the git checkout, or
whatever, then this should be printed alongside it, as now done.
Gbp-Dch: Ignore
[Raphaël Hertzog: tweak to apply on top of my changes]
thus it can correctly indicate success/fail status instead of always
indicating failure. when a user asks for usage with -u|--usage then we
should exit in success mode rather than failure as when usage in printed
in response to incorrect usage.
Gbp-Dch: Short
this means that the usage goes from:
lb {-h|--help|-u|--usage|-v|--version}
lb COMMAND [OPTIONS]
to:
lb {-h|--help|-u|--usage|-v|--version}
lb [FRONTEND_OPTIONS] COMMAND [COMMAND_OPTIONS]
though it is probably not worth is to update the description in the
manpages...? hmm...
so for instance this matters for color control with --color|--no-color
(you already had full control via environment vars). previously you could
do `lb COMMAND --no-color` to turn off colour, only to find that output
at the frontend level was still coloured (the option is processed at the
command context level, not the frontend), so you might try to instead use
`lb --no-color COMMAND`, only to find that this was not supported. Well
now it is, and used at the frontend level will fully control colour output
(after the command is processed anyway).
the full set of common options are thus available (except --force) at the
frontend level, and thus for instance all Echo_*() helpers used in the
frontend will work correctly after args are processed.
furthermore usage like `lb --color --help` will actually work. (not that
color is used there, but this previously would have failed with the
frontend treating the `--color` argument as the command; that's the point!)
Gbp-Dch: Short
this was previously not done in 8b109ffb96
to keep the renaming simple, but leaving the variable plural is a cause
for confusion.
since this property is stored in the INI style config/build config file
rather than a shell script based one, at the property there is already
singular, there was no need for a backwards compatibility hack.
Gbp-Dch: Short
note that the bit of code removed from source_debian relies upon a
variable LB_TASKS which itself is an old leftover artefact from before
v4.0.
Gbp-Dch: Short
since everywhere where 'apt' is a permitted value, 'apt-get' is also, it
just wasn't listed in the option's documentation and thus was also not
listed in the new validation check.
Gbp-Dch: Short
80aa5ab611 implemented a hack to handle
replacement of LB_LINUX_FLAVOURS with LB_LINUX_FLAVOURS_WITH_ARCH in
config files, but implemented it in the wrong place.
adding a conditional conversion within the config file meant that the old
value would only be read from **new** config files that are created
obviously without it, including re-saved configs if `lb config` were
re-run with additional options (not recommended). any existing value in an
existing config file would actually be ignored.
the right place to read the old value was in the Set_defaults() function
(since renamed).
a second issue also existed with the hack, it failed to excape the `$`
and thus printed the existing value of $LB_LINUX_FLAVOURS into the
conditional check being constructed in the config file, instead of
printing the name of the variable. the check embedded into the config
file thus became this on an amd64 machine:
```
if [ -n "amd64" ]
then
LB_LINUX_FLAVOURS_WITH_ARCH="amd64"
fi
```
which is clearly not what was intended.
Gbp-Dch: Short
while helpful for users to know the defaults, the values printed as the
supposed defaults for most are actually the same values as being
configured, or in some cases a piece of text "autodetected or empty", and
thus the information is completely wrong and actually unhelpful since it
misinforms the user.
fixing this to give the real defaults is very much non-trivial.
as a workaround users wanting to know the default for an option can always:
a. use `lb config` wit no options (or auto) in a clean directory and thus
get a config with all defaults.
b. look at the live-build code.
if they just want to reset an option, they can also just comment it out.
Gbp-Dch: Short
Closes: #904614
(and move the wget options setting down where it should be while at it)
the value of LB_DEBIAN_INSTALLER is now properly checked in the main
validation routine, so we can just directly exit here as a simple safety
check should validation be bypassed.
Gbp-Dch: Short
- add a validation check where an error will be printed
- replace the check done in the grub scripts with one that simple exits
if executed bypassing the validation check
Gbp-Dch: Short
move away from the somewhat config file grouping based organisation to
an alphabetised list, after grouping into script-specific; general;
build-specific and other.
the config file based organisation was a bad choice, making it hard to
find the right place to insert options for instance.
Gbp-Dch: Short
running `lb config --validate` causes the script to stop after running
the validation check on the config compiled at that point, prior to
writing the config to disk.
this gives users the ability to check the validity of a config without
modifying or rewriting the saved config.
note that if users provide new config options alongside --validate, these
are taken into account in the check performed.
the 'check complete' message will not be seen if an error is reported by
the check function, while it will be seen if only warnings are given, but
it would require a redesign of the validation check function to make any
improvement in that area, and it's perhaps not worth it.
Gbp-Dch: Short
it mostly applies defaults where a value does not exist, but does more in
some cases. the new name better reflects its usage and functionality.
Gbp-Dch: Short
whilst some parts of the codebase were set up to work with multiple types
specified, others did not work with it and would not necessarily be easy
to adjust. this thus makes some tweaks to adjust things accordingly.
- option renamed to singular form (maintaining backwards compatibility)
- a validation check has been added
- unnecessary glob style type references fixed
- checks with In_list changed to a direct singular comparison
- typo of type "netboot" written as just "net" fixed (though unreachable
so of no consequence; really the code could be removed but it's trivial)
Gbp-Dch: Short
two parts of the code worked with both comma and space separated lists,
while two others only worked with comma separated.
swapping out commas with spaces when we setup the var in
Set_config_defaults() means that individual scripts no longer need to worry
about it and everything supports both; and that we can avoid the
IFS/OLDIFS mess.
Gbp-Dch: Short
Thore Sommer
Roland Clobus
Ryan Finnie
Luca Boccassi
Raphaël Hertzog
Marcel Partap
Jan Kot
Raphaël Hertzog
Matthijs Kooijman
Victor Gavro
Lyndon Brown
johnraff