Moved includes.chroot to includes.chroot_after_packages and added
includes.chroot_before_packages. includes.chroot does still work as before.
We also now use rsync for copying files if it is installed.
This improves runtime and space consumption for large includes.
Gbp-Dch: Short
Closes: #927128
We want to be resilient to small network hiccups and when you download
a large number of packages, the probability of a network hiccup are
increasing... as are the chances of hitting one of the apt bugs
(cf "Undetermined error"...).
some scripts temporarily install packages to accomplish some work before
then removing them. the list of packages installed is kept in memory in a
variable.
a weakness of this design is that if a failure occurs or the user cancels,
and then following this the user re-runs `lb build`, letting it try to
pick up and recover from where it left off, that list of packages that had
been installed is lost, resulting in those packages that were installed
then being a permanent part of the chroot.
here we fix this weakness by backing up the list to a file, which is always
read from on removal. thus in a recovery situation, any packages still
installed from a situation like that just described, will be removed upon
the next use of `Remove_package()`.
this is not perfect, since we are having to wait for opportunistic
execution of the remove function. we need to find a suitable place for the
`Cleanup_temp_packages()` function to be used.
- doing so in `Init_config_data()` would not be suitable because we don't
hold the lock when that's run, even if we ignored the hijacking of that
function for taking such action...
- doing it in `Exit()` doesn't seem a good fit either.
- putting it explicitly in every build script just seems a little messy...
perhaps a local exit trap like for removing the lock...?
note that `binary_rootfs` skips running the remove function after installing
tooling, since it just throws the wrapper chroot away, which then leaves the
file around with stale data for the next remove instance to pick up, which
then does not actually remove it because it's not installed. this is not
ideal either... perhaps the optimisation should be removed from that script?
Gbp-Dch: Short
- the comparison with LB_ARCHITECTURE seems to be based upon an old idea
of it potentially being intended in the past to hold more than one
arch (when it's name as also plural), but it in fact only holds one.
- similarly LB_BOOTSTRAP_QEMU_ARCHITECTURE is singular, so the comparison
with the set of params is bad.
- storing $@ in a variable before then looping is not a good idea
- script name was not being passed to the warnings
- can simplify by just returning in the valid case
i'm very suspicious about the correctness of the qemu bits, but that
can wait for now.
Gbp-Dch: Short
the design choice from when EFI support was introduced was to change
`--bootloader` to `--bootloaders`, with users specifying their selection
of BIOS and EFI bootloaders together. at this time there were not even any
decent validation checks being performed, and invalid combinations could
cause some chaos.
since then proper validation was put in place, including checking that
only a single instance of each of BIOS and EFI bootloaders exists in the
selection.
here we tweak things such that we stick with the same option, but we split
the selection up such that we store the BIOS and EFI selections separately
within the saved config file, and offer it up to scripts to help simplify
those scripts.
we must however retain support for splitting from the combined option,
both because we still use it in the combined option, and for backwards
compatibility with older saved configs.
Gbp-Dch: Short
thus far, config bootloader validation only did the basic check that each
bootloader specified was a known and supported bootloader, it did not check
combinations.
it now checks combinations, and strips out the previous "bootloader role"
stuff.
the no-bootloaders warning is duplicated, covering two slightly different
situations (empty string, and whitespace string). this is anticipated to
be just temporary, with this just being the first step in better handling
bootloader selections.
Gbp-Dch: Short
The core purpose of `Prepare_config()` is to prepare the config by setting
defaults where not specified. It should not be involved itself in
validation of user options.
This can be an important distinction, for instance in `clean` where we
use this function, but avoid validation to potentially allow for an option
that deletes the config, that should work even in the face of an invalid
config.
Gbp-Dch: Ignore
while `lb build` ran the config validation check, spotting invalid configs
and stopping with an error, the major build stages if executed directly did
not, nor did the component stages.
here we move execution of the validation function into the common init
function, with an exported variable used to indicate that validation has
been performed. thus validation is performed no matter what part of the
build system you execute, but only once.
Gbp-Dch: Short
using `$VERSION` as part of the default `$LB_ISO_PREPARER` means that when
you simply run `lb config` once, this variable is stored as a part of the
string, and replaced on use, but if you run `lb config` twice, it gets
replaced with a fixed version, that is then used in all subsequent builds.
let's replace with a placeholder (`@LB_VERSION@`) that can be used both in
the default, or in user strings, and will be replaced on use only.
this means that subsequent builds will always reflect the actual version
of live-build used.
Gbp-Dch: Ignore
`DATE_UTC_OPTION` is set in `Prepare_config()` for use by scripts, even
though only a few scripts will actually use it, since it allows those
scripts to be cleaner. we may want to possibly extend this as a
`DATE_OPTIONS` variable perhaps as part of enabling proper reproduciblity.
Gbp-Dch: Short
- the definition of $PROGRAM as used in $USAGE strings defined in each
script has been broken for a long time, being simply "lb" when it needs
to be "lb COMMAND".
- `config` changed $PROGRAM to "lb config" thus its output was correct in
this regard unlike everything else, but with the switch to a more
"intelligent" `Man()` function recently, it means that instead of
`man lb config`, what was actually run was `man lb config config`,
which displayed the manpage, then on exiting with `q`, it showed some
sort of index line todo with a "config" search (no exact manpage
match?), for which you had to enter `ctrl+c` to get rid of.
this revises things to fix the issues, minimising change by changing
$PROGRAM to "lb COMMAND", with the frontend overriding this.
Gbp-Dch: Ignore
load the base set of files first, then any user arch/dist specific override
files second.
this is important since whenever config items get moved in the base files,
this can break the override files, if an option specified in one was moved
to a different common file that it loaded after.
Gbp-Dch: Ignore
- it's only used by the debootstrap script after alternatives were dropped
long ago, so let's move it, avoiding it being loaded for everything
else.
- there's no need to pass printing another message through it.
- there's little point in making the sid distinction if you happen to
decide to build sid, it's a given that it's less stable than stable.
really, is there any need for this at all?
Gbp-Dch: Ignore
This is considered to be more robust.
Two instances remain:
scripts/build/chroot_archives, line 257:
if [ "${LB_APT}" = "aptitude" ] && [ ! $(Chroot chroot "which aptitude") ]
The command is run inside a chroot where the environment might be special,
and would need further testing.
manpages/Makefile, line 42:
@if [ ! -x "$$(which po4a 2>/dev/null)" ]; \
I am insufficiently familiar with makefile syntax to edit this.
support for this has not existed since v2.0~a24-1 !!
until 4d22ca948a (v2.0~a24-1), `PROGRAM` was
defined as `PROGRAM="$(basename ${0})"`. then this commit simply got rid
of the definition.
it then flipped back and forth a bit in a few ways, but always sticking
to a simple fixed "live-build" type string after that commit, as it has
been ever since.
the code removed here has thus been junk since then, and the documentation
corrected here out of date since then.
the checking of `LB_CONFIG` is also possibly junk, but i know too little
about it. it was introduced in v1.0~a43-1 in commit
705a4178e7 with a changelog entry of
"Centraly evaluate LH_CONFIG for custom configuration file."
Gbp-Dch: Short
just as most scripts are skipped if their stagefile exists (indicating
that they have already been run to completion), including chroot
preparation scripts in install mode, this implements the same guard for
chroot prep remove mode, such that they exit early if their stagefile
does not exist, indicating that the modification has already been removed.
(also override-able by --force in the same way).
this basically just uses a tweaked copy of Check_stagefile().
Gbp-Dch: Short
no backwards compatibility hack for reading the old var from existing
saved config used because this was previously stored in the alternate
format config/build file.
Gbp-Dch: Short
no backwards compatibility hack for reading the old var from existing
saved config used because this was previously stored in the alternate
format config/build file.
Gbp-Dch: Short
back in v4.0~a6-1 a transition process was started to move the live-build
config to a new format. the new format was INI style, and required
parsing functions to read/write values, compared to the existing format
which was just shell script code setting variables.
this partial transition is the explanation for the existence of the
`New_configuration()` function, and understanding this is important to
understanding the purpose of it - it is not in fact intended for creating
a new configuration, it is just related to the new config format
transition.
the positives of the new format were that it was somewhat cleaner looking,
while the negative was the terrible relative efficiency.
the file `config/build` was created to hold options in this new format.
the transition was only ever completed for a handful of config options:
- architecture
- archive areas and parent archive areas
- live image name
- live image type
a 'configuration version' attribute was also saved, which is not used by
anything.
the bootstrap-mirror and parent-bootstrap-mirror attributes are pointlessly
stored in it seemingly resulting from work done in v4.0~a17-1. (they are
also stored in another config file from which the value is actually used).
it in fact seems to have been a source of confusion for Raphaël in
authoring 44b9b0a650, since the new
`[parent]-distribution-{chroot|binary}` options it introduced were stored
both in `config/bootstrap` and in `config/build`, while only used from the
former. i expect, understandably, that he thought that `config/build` was
just an information file.
Gbp-Dch: Short
Jan Kot
Unit 193
Roland Clobus
Thore Sommer
Raphaël Hertzog
Ryan Finnie
Lyndon Brown
johnraff
Raphaël Hertzog