Instances of:
if [ $(which <command> ]
have been replaced with:
if command -v <command> >/dev/null
which is considered to be more robust in a range of environments.
scripts/build/chroot_archives: line 259:
if [ "${LB_APT}" = "aptitude" ] && [ ! $(Chroot chroot "which aptitude") ]
has been left untouched because the chroot might require a more complex command
which would need more testing.
manpages/Makefile: line 42:
@if [ ! -x "$$(which po4a 2>/dev/null)" ]; \
has been left untouched because I am not sufficiently familiar with makefiles.
commit f811656150 enabled the grub-efi
bootloader by default for amd64|i386 architectures, but failed to
recognise the this bootloader is not supported for hdd|netboot images.
this meants that if a user tried to build such an image without explicitly
specifying the bootloader, excluding grub-efi, their build would fail
with an error in the binary_grub-efi stage.
this fixes the problem by only enabling grub-efi by default on supported
image builds.
677415f6d7 (2007) in v1.0~a2-1 added a hack
relating to the loop-aes-utils package and losetup. this commit bundled
a bunch of changes, it was not specific to the hack, and so info about the
hack is limited to a brief comment included within the related change in
defaults:
```
# Workaround for loop-aes-utils divertion
# (loop-aes-utils' losetup lacks features).
```
though it is very similar to the removed fdisk hack in that it seems that
one package may replace a binary from another, moving the original to a
new location, and this hack gives the user the opportunity to select the
original instead of the one put in its place, for use in LB.
the comment mentions a package called loop-aes-utils as being the package
that performs such a diversion, and that the need for the hack was that
losetup itself lacked features, presumably encryption support, and it is
clear that it is the losetup binary that is the focus of the diversion.
looking into the history of loop-aes-utils a little, this package was
dropped from debian back in 2012 (#680748), favouring encrytion support of
dm-crypt/cryptsetup.
double checking file contents of packages, only the mount package carries
an /sbin/losetup file, so presumably this means that dm-setup/cryptsetup
do not perform such a diversion of losetup (i.e. their use is exclusively
done directly).
since the possible diversion is simply gone, that completely removes any
point in having the hack of giving users choice between losetup and the
diverted one. so let's remove this obsolete hack...
8321653cb3 (from 2007) introduced a hack to
work around bug #445304 in gnu-fdisk for users who may have replaced fdisk
with the classic gnu version. the hack allowed users to select an alternate
fdisk binary to use to work around the buggy binary.
bug #445304 is marked as found in v1.0-1 and fixed in v1.2-1, though may
have been fixe din v1.1. it was marked fixed in 2009.
checking the package archive, gnu-fdisk does not actually exist anymore
in debian, with one exception - it is available for arm64 on sid via
debports, and that version is 1.3 so thus includes the necessary fix
anyway.
it is thus pointless now that we still carry this hack.
Gbp-Dch: Short
I'd mistook the copyright here to apply to the apt package, but did not
look close enough, it's clearly referring to copyright of the files which
we've just amended to have a "The Debian Live team" notice, which this
should thus surely also have.
Gbp-Dch: Short
Current versions of the project files are built upon versions published
and licensed by Daniel Baumann, but are modified copies of those files and
thus need to be marked as such per licensing requirements (afaik he did
not pass along ownership / licensing rights to anyone when he left the
project). We should also be careful to not be misrepresenting such
modified copies as being attributed to Daniel.
Adding a new copyright line referring to "The Debian Live team" should
suffice for this.
The authorship block in man pages has also similarly been updated.
Notes:
- tweaked a copy of daniel copyright lines stating 2014 instead of 2015.
both of these cases were in files that i had personally introduced in
some of my past merged commits that moved some code around. i don't know
why they stated 2014.
- binary_onie was introduced in 2018, so that has a 2018 date instead of
2016 unlike the rest.
- 'efi-image' is a 3rd-party (Canonical Ltd) work that we bundle, but it
has been modified by 674794a8f4 and
36a3ba7634 so I similarly added a
debian live copyright line.
- 'grub-cpmodules' is similar. it was only changed by the indentation fix
of 36a3ba7634 but modification is
modification, and this does help cover any possible future changes that
might be made.
Prefer downloading the version of the source package that actually
corresponds to the version of the binary. Should apt-update be
run and a package updated, we do not really want to fetch a newer
copy of the source than that of the binary, we want the exact
corresponding version (kinda the whole point of compiling a source
disc that they correspond). If the exact version is no longer
available then it is surely preferable to list it in the missing
list than end up with a newer version.
Gbp-Dch: Short
Closes: #952932
if you execute the bootstrap stage with no internet connection, you get
the following output:
```
[2020-03-10 19:18:46] lb bootstrap
P: Setting up clean exit handler
[2020-03-10 19:18:46] lb bootstrap_cache restore
[2020-03-10 19:18:46] lb bootstrap_debootstrap
P: Begin bootstrapping system...
P: If the following stage fails, the most likely cause of the problem is with your mirror configuration or a caching proxy.
P: Running debootstrap (download-only)...
I: Retrieving InRelease
I: Retrieving Release
E: Failed getting release file http://deb.debian.org/debian/dists/buster/Release
P: Begin unmounting filesystems...
P: Saving caches...
chroot: failed to run command ‘/usr/bin/env’: No such file or directory
```
the last line looked suspicious. investigating it turns out that there was
a deficiency in the exit handler.
when debootstrap fails to download what it needs due to lack of a
connection, that failure due to `set -e` causes the Exit() handler to kick
in. Part of this includes outputting the "Saving caches..." line, before
then making a call to Save_package_cache(). That in turn runs the following
command:
```
Chroot chroot "apt-get autoclean" || true
```
The Chroot() function includes a line starting with:
```
${_LINUX32} chroot "${CHROOT}" /usr/bin/env
```
which is the source of the last output line.
the reason we see this unexpected output is that with bootstrapping having
failed, there is no /usr/bin/env within the chroot so it is bound to fail.
the fact is, the exit handler has no business trying to pretty much
anything that it does if the bootstrap_debootstrap stage has not
completed.
this implements such a restriction and thus resolves the problem of this
unexpected and confusing output in the described situation.
we will now see:
```
[2020-03-10 19:18:46] lb bootstrap
P: Setting up clean exit handler
[2020-03-10 19:18:46] lb bootstrap_cache restore
[2020-03-10 19:18:46] lb bootstrap_debootstrap
P: Begin bootstrapping system...
P: If the following stage fails, the most likely cause of the problem is with your mirror configuration or a caching proxy.
P: Running debootstrap (download-only)...
I: Retrieving InRelease
I: Retrieving Release
E: Failed getting release file http://deb.debian.org/debian/dists/buster/Release
```
the source stage emitted the following output for each packages downloaded.
note the troubling warning at the end.
```
Reading package lists... Done
NOTICE: 'grep' packaging is maintained in the 'Git' version control system at:
https://salsa.debian.org/debian/grep.git
Please use:
git clone https://salsa.debian.org/debian/grep.git
to retrieve the latest (possibly unreleased) updates to the package.
Need to get 1579 kB of source archives.
Get:1 http://deb.debian.org/debian buster/main grep 3.3-1 (dsc) [2038 B]
Get:2 http://deb.debian.org/debian buster/main grep 3.3-1 (tar) [1473 kB]
Get:3 http://deb.debian.org/debian buster/main grep 3.3-1 (diff) [104 kB]
Fetched 1579 kB in 1s (1293 kB/s)
Download complete and in download only mode
W: Download is performed unsandboxed as root as file 'grep_3.3-1.dsc' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)
```
this occurred because the '_apt' user did not have permission to write to
the destination directory and so was falling back to downloading as root
in order to do its work.
prior to 158950b873 all source packages were
downloaded directly to the root of the chroot. that commit changed this to
save them into a new clean directory within it instead. thus to fix the
problem we can simply set the ownership of this new directory to '_apt'.
Gbp-Dch: Short
the check for existence of debootstrap here was completely redundant since
there is a check at the beginning of the file which already outputs an
appropriate error and exists if missing.
the cache restore/save script is not a chroot modification script unlike
the rest of the scripts that it was bunched up with. It is an actual
component part of the chroot build stage.
let's bring clarity to this with improved documentation.
Gbp-Dch: Short
LB_APT_SOURCE_ARCHIVES determines whether or not deb-src entries are
desired to be included in apt's sources.list. here, instead of excuding
them we always include them but commented out where they would previously
have been excluded. this means that if a user later changes their mind and
wants to make use of them all they have to do is uncomment them rather
than add the necessary lines.
Gbp-Dch: Short
Closes: #952929
all scripts use `set -e` which means that if getop fails, the subsequent
error check that would print an error in addition to any printed by getopt
itself would never actually be reached.
the first though here would be to remove the pointless error check, but
getopt does not include the word "error" with an unrecognised option
failure, nor does it use colour to highlight problems, both of which mean
that it is a little lacking in terms of highlighting problems to users.
thus we properly capture and use the exit code here and output an
appropriate message per invalid argument vs getopt internal error.
also, removed the redundant stderr redirection which is already done
by Echo_error().
Gbp-Dch: Short
white is not going to work well on a terminal with a white background,
obviously. We should keep the standard colour and just try applying the
bold.
or do we want to consider a non black/white colour? like blue...?
Gbp-Dch: Short
instead of conditionally writing deb-src lines, it is **much** neater if
we use sed to optionally removed them at the end.
Gbp-Dch: Short
Closes: #952928
- prefer using `which` over hard coded paths
- it is redundant to check that the bin pointed to the return of
`which` exists and is executable, `which` already gives us
assurance of that if it returns true!
- the redirection of output (`2>/dev/null`) seems to be
unnecessary from my testing.
the instances relatnig to fdisk and losetup in functions/defaults.sh have
been left as they are since they get executed by `lb config` which can run
without sudo elevation unlike `lb build` and in that case `which` would
fail to find these binaries resulting in error.
this also fixes a bug showing an error for missing debootstrap - this tool
requires sudo privileges to run and thus is not found via a none elevated
which search.
Gbp-Dch: Short
Closes: #952927
commit d74f2102a0 added a validation check
to chroot_archives for its 'pass' parameter. this was based upon finding
an instance where the wrong value was being submitted to the script and
wanting to ensure such mistakes would be caught.
unfortunately it seems that I made a mistake in misremembering the facts
surrounding the latter issue when constructing the validation check and
failed to double check with testing until it was already merged. a
correction is needed. the set of valid values is not limited to only
'source|binary' but actually 'source|binary|chroot'. I'd misremembered
'chroot' as being a completely invalid value.
the existing logic for obtaining a list of firmware packages always
downloaded a fresh copy of the archive content file, deleting the file
already in the cache. here we move to actually making use of the cache.
this helps when building multiple times, at least for the same distro. the
package list obtained is rarely going to change after all. it could of
course differ between distros, but the cache is per-distro, as it has
always been.
we of course here switch to caching each of the archive-area files
individually rather than having one file that gets overwritten (or
appended to in the case of when we kept the decompressed file).
Gbp-Dch: Short
Closes: #952911
the existing logic was to decompress the contents file from the downloaded
archive to disk, then process it to obtain a package list. the largest
one by far is for 'main'; 'non-free' and 'contrib' are tiny in comparison.
for sid-amd64 currently, the archive file is 37 MB, while the decompressed
file it contains is 592.3 MB.
we always delete the files and download afresh (currently), and a previous
commit optimised by deleting the files once we're done with them to avoid
wasting disk space leaving them behind.
here we switch to storing the downloaded compressed file to disk instead,
reducing disk space usage (and IO) by hundreds of megabytes; piping the
decompression directly into awk instead of having awk read from the stored
file.
this moves the appending of new items into the list back within the archive
area loop, which is fine since we're replacing the file for each loop now
so the previous issue relating to appending is of no concern.
Gbp-Dch: Short
Closes: #952910
addressing an old fixme, should distro and parent-distro be identical (in
a derivative build) we want to avoid wasting effort downloading and
processing the same content files twice.
since parent and non-parent have separate archive-area lists though we
should perhaps not just assume that they are identical or ignore any
differences; thus here in such a situation we ensure that we handle any
archive areas not already done for parent-distro handling in such a case,
while skipping those already done.
i notice that the fixme actually also refers to avoiding actual
overlapping of the cached files, however the cached files are (currently)
always ignored anyway, so this is of no concern. reusing the cached files
is an entirely separate issue.
Gbp-Dch: Short
Closes: #952909
the 'manually add firmware-linux package' bit was stuck inbetween the
parent and non-parent logic, which was especially unhelpful before we
de-duplicated the logic into a common function.
Gbp-Dch: Short
Luca Boccassi
Lyndon Brown
johnraff