Instances of:
if [ $(which <command> ]
have been replaced with:
if command -v <command> >/dev/null
which is considered to be more robust in a range of environments.
scripts/build/chroot_archives: line 259:
if [ "${LB_APT}" = "aptitude" ] && [ ! $(Chroot chroot "which aptitude") ]
has been left untouched because the chroot might require a more complex command
which would need more testing.
manpages/Makefile: line 42:
@if [ ! -x "$$(which po4a 2>/dev/null)" ]; \
has been left untouched because I am not sufficiently familiar with makefiles.
commit f811656150 enabled the grub-efi
bootloader by default for amd64|i386 architectures, but failed to
recognise the this bootloader is not supported for hdd|netboot images.
this meants that if a user tried to build such an image without explicitly
specifying the bootloader, excluding grub-efi, their build would fail
with an error in the binary_grub-efi stage.
this fixes the problem by only enabling grub-efi by default on supported
image builds.
677415f6d7 (2007) in v1.0~a2-1 added a hack
relating to the loop-aes-utils package and losetup. this commit bundled
a bunch of changes, it was not specific to the hack, and so info about the
hack is limited to a brief comment included within the related change in
defaults:
```
# Workaround for loop-aes-utils divertion
# (loop-aes-utils' losetup lacks features).
```
though it is very similar to the removed fdisk hack in that it seems that
one package may replace a binary from another, moving the original to a
new location, and this hack gives the user the opportunity to select the
original instead of the one put in its place, for use in LB.
the comment mentions a package called loop-aes-utils as being the package
that performs such a diversion, and that the need for the hack was that
losetup itself lacked features, presumably encryption support, and it is
clear that it is the losetup binary that is the focus of the diversion.
looking into the history of loop-aes-utils a little, this package was
dropped from debian back in 2012 (#680748), favouring encrytion support of
dm-crypt/cryptsetup.
double checking file contents of packages, only the mount package carries
an /sbin/losetup file, so presumably this means that dm-setup/cryptsetup
do not perform such a diversion of losetup (i.e. their use is exclusively
done directly).
since the possible diversion is simply gone, that completely removes any
point in having the hack of giving users choice between losetup and the
diverted one. so let's remove this obsolete hack...
8321653cb3 (from 2007) introduced a hack to
work around bug #445304 in gnu-fdisk for users who may have replaced fdisk
with the classic gnu version. the hack allowed users to select an alternate
fdisk binary to use to work around the buggy binary.
bug #445304 is marked as found in v1.0-1 and fixed in v1.2-1, though may
have been fixe din v1.1. it was marked fixed in 2009.
checking the package archive, gnu-fdisk does not actually exist anymore
in debian, with one exception - it is available for arm64 on sid via
debports, and that version is 1.3 so thus includes the necessary fix
anyway.
it is thus pointless now that we still carry this hack.
Gbp-Dch: Short
Current versions of the project files are built upon versions published
and licensed by Daniel Baumann, but are modified copies of those files and
thus need to be marked as such per licensing requirements (afaik he did
not pass along ownership / licensing rights to anyone when he left the
project). We should also be careful to not be misrepresenting such
modified copies as being attributed to Daniel.
Adding a new copyright line referring to "The Debian Live team" should
suffice for this.
The authorship block in man pages has also similarly been updated.
Notes:
- tweaked a copy of daniel copyright lines stating 2014 instead of 2015.
both of these cases were in files that i had personally introduced in
some of my past merged commits that moved some code around. i don't know
why they stated 2014.
- binary_onie was introduced in 2018, so that has a 2018 date instead of
2016 unlike the rest.
- 'efi-image' is a 3rd-party (Canonical Ltd) work that we bundle, but it
has been modified by 674794a8f4 and
36a3ba7634 so I similarly added a
debian live copyright line.
- 'grub-cpmodules' is similar. it was only changed by the indentation fix
of 36a3ba7634 but modification is
modification, and this does help cover any possible future changes that
might be made.
if you execute the bootstrap stage with no internet connection, you get
the following output:
```
[2020-03-10 19:18:46] lb bootstrap
P: Setting up clean exit handler
[2020-03-10 19:18:46] lb bootstrap_cache restore
[2020-03-10 19:18:46] lb bootstrap_debootstrap
P: Begin bootstrapping system...
P: If the following stage fails, the most likely cause of the problem is with your mirror configuration or a caching proxy.
P: Running debootstrap (download-only)...
I: Retrieving InRelease
I: Retrieving Release
E: Failed getting release file http://deb.debian.org/debian/dists/buster/Release
P: Begin unmounting filesystems...
P: Saving caches...
chroot: failed to run command ‘/usr/bin/env’: No such file or directory
```
the last line looked suspicious. investigating it turns out that there was
a deficiency in the exit handler.
when debootstrap fails to download what it needs due to lack of a
connection, that failure due to `set -e` causes the Exit() handler to kick
in. Part of this includes outputting the "Saving caches..." line, before
then making a call to Save_package_cache(). That in turn runs the following
command:
```
Chroot chroot "apt-get autoclean" || true
```
The Chroot() function includes a line starting with:
```
${_LINUX32} chroot "${CHROOT}" /usr/bin/env
```
which is the source of the last output line.
the reason we see this unexpected output is that with bootstrapping having
failed, there is no /usr/bin/env within the chroot so it is bound to fail.
the fact is, the exit handler has no business trying to pretty much
anything that it does if the bootstrap_debootstrap stage has not
completed.
this implements such a restriction and thus resolves the problem of this
unexpected and confusing output in the described situation.
we will now see:
```
[2020-03-10 19:18:46] lb bootstrap
P: Setting up clean exit handler
[2020-03-10 19:18:46] lb bootstrap_cache restore
[2020-03-10 19:18:46] lb bootstrap_debootstrap
P: Begin bootstrapping system...
P: If the following stage fails, the most likely cause of the problem is with your mirror configuration or a caching proxy.
P: Running debootstrap (download-only)...
I: Retrieving InRelease
I: Retrieving Release
E: Failed getting release file http://deb.debian.org/debian/dists/buster/Release
```
LB_APT_SOURCE_ARCHIVES determines whether or not deb-src entries are
desired to be included in apt's sources.list. here, instead of excuding
them we always include them but commented out where they would previously
have been excluded. this means that if a user later changes their mind and
wants to make use of them all they have to do is uncomment them rather
than add the necessary lines.
Gbp-Dch: Short
Closes: #952929
all scripts use `set -e` which means that if getop fails, the subsequent
error check that would print an error in addition to any printed by getopt
itself would never actually be reached.
the first though here would be to remove the pointless error check, but
getopt does not include the word "error" with an unrecognised option
failure, nor does it use colour to highlight problems, both of which mean
that it is a little lacking in terms of highlighting problems to users.
thus we properly capture and use the exit code here and output an
appropriate message per invalid argument vs getopt internal error.
also, removed the redundant stderr redirection which is already done
by Echo_error().
Gbp-Dch: Short
white is not going to work well on a terminal with a white background,
obviously. We should keep the standard colour and just try applying the
bold.
or do we want to consider a non black/white colour? like blue...?
Gbp-Dch: Short
instead of conditionally writing deb-src lines, it is **much** neater if
we use sed to optionally removed them at the end.
Gbp-Dch: Short
Closes: #952928
- prefer using `which` over hard coded paths
- it is redundant to check that the bin pointed to the return of
`which` exists and is executable, `which` already gives us
assurance of that if it returns true!
- the redirection of output (`2>/dev/null`) seems to be
unnecessary from my testing.
the instances relatnig to fdisk and losetup in functions/defaults.sh have
been left as they are since they get executed by `lb config` which can run
without sudo elevation unlike `lb build` and in that case `which` would
fail to find these binaries resulting in error.
this also fixes a bug showing an error for missing debootstrap - this tool
requires sudo privileges to run and thus is not found via a none elevated
which search.
Gbp-Dch: Short
Closes: #952927
the existing logic for obtaining a list of firmware packages always
downloaded a fresh copy of the archive content file, deleting the file
already in the cache. here we move to actually making use of the cache.
this helps when building multiple times, at least for the same distro. the
package list obtained is rarely going to change after all. it could of
course differ between distros, but the cache is per-distro, as it has
always been.
we of course here switch to caching each of the archive-area files
individually rather than having one file that gets overwritten (or
appended to in the case of when we kept the decompressed file).
Gbp-Dch: Short
Closes: #952911
the existing logic was to decompress the contents file from the downloaded
archive to disk, then process it to obtain a package list. the largest
one by far is for 'main'; 'non-free' and 'contrib' are tiny in comparison.
for sid-amd64 currently, the archive file is 37 MB, while the decompressed
file it contains is 592.3 MB.
we always delete the files and download afresh (currently), and a previous
commit optimised by deleting the files once we're done with them to avoid
wasting disk space leaving them behind.
here we switch to storing the downloaded compressed file to disk instead,
reducing disk space usage (and IO) by hundreds of megabytes; piping the
decompression directly into awk instead of having awk read from the stored
file.
this moves the appending of new items into the list back within the archive
area loop, which is fine since we're replacing the file for each loop now
so the previous issue relating to appending is of no concern.
Gbp-Dch: Short
Closes: #952910
Edit: There were four copies of the same logic to keep in sync;
Originally this patch deduplicated each file, but leaving a copy of
the new function in each, thus reducing the duplication but not
eliminating it. A later patch moved it into a shared function file
following further enhancements to the code in question. This has
since been revised to have the function moved to a shared file here,
which simplifies and gives a cleaner diff.
Gbp-Dch: Short
Closes: #952908
These functions are specific to handling packages stored in the
cache, not other files. They are also always used with the same
`cache/packages.` prefix to the path.
Gbp-Dch: Short
Closes: #952916
only a couple were in use and only by unused echo helpers which have now
themselves been removed, so nothing in this file is needed.
Partial fix for #952880
Gbp-Dch: Short
Closes: #952880
all of these echo helpers are essentially 'logging' functions with output
always intended for stdout/stderr. lack of explicit stdout/stderr direction
means that their output could be captured unintentionally should they be
used within a function designed to construct a string.
Gbp-Dch: Short
Closes: #952879
lack of stderr directed output for the `E:` prefix meant that it would
not appear alongside the message in some use cases
Gbp-Dch: Short
Closes: #952878
available values currently are memtest86+|memtest86|none; "false" is
presumably handled for backwards compatibility
there is no need to handle this in individual scripts. the right place
to handle it is in Set_Defaults as now done
Gbp-Dch: Short
Closes: #952866
`false` and `none` make no sense as choices for this option. Here we
replace `false` with `none`, and remove `true`.
Note that `true` was treated as an alias for netinst (see the changes to
source_disk and and binary_disk).
For backwards compatibility we still allow `true` and `false` by converting
them to `netinst` and `none` respectively, whilst printing a warning to
encourage users to move to `netinst`/`none`.
Gbp-Dch: Short
Closes: #952864
including:
- spaces replaced with tabs for consistency
- alignment of `;;` in some case statements changed for consistency
Gbp-Dch: Short
Closes: #952857
This makes it possible to build an image against a first distribution
(--distribution-chroot) and have the resulting image point to another
distribution (--distribution-binary). We can use this to build against a
snapshot and have the result use the original distribution that was
snapshotted.
Closes: #888507
Luca Boccassi
johnraff
Lyndon Brown
Raphaël Hertzog