Combine the check+create done in each script. (The original functions
are still callable as before, but a new combined `Aquire_lockfile`
function can be called instead, as now used).
Note, a further simplification could be done in removing the passing of
the lock filename in as a parameter since every use of the functions is
with ".lock". The lock functions already have a fallback to ".build/lock"
though. Checking the history, the fallback used to be for a system wide
lock, which was then replaced with this config-tree specific one. As long
as that is not used implicitly by 3rd-party hooks then surely we are free
to change the fallback to ".lock" and further remove passing in a name as
a param...?
history:
db5d2b0dcd0aa8289a37
Gbp-Dch: Short
Closes: #952918
previously this was white+bold. the white aspect was dropped since this
would not be sensible for users with a white background for their terminal.
bold however does not have any effect for me at least so effectively there
is no highlighting at all.
here we reintroduce a colour, one that will work for both black and white
backgrounds of course. purple looks good to me - significantly different
to that used for errors and warnings, and works well with command
highlighting (as enabled separately).
if a script exits due to a failure and `set -e`, we should ensure that an
error message is printed to be clear to the user that something actually
went wrong.
similarly it would be good to print a suitable message should the user
cancel with ctrl+c for instance.
Gbp-Dch: Short
theres no point in creation of stagefiles being kept within a
conditional block of work. if the script completes with success
then it should create its stagefile to thus avoid repeating any
work that it might have done should it get re-run without being
forced.
Gbp-Dch: Short
(part of never completed side-by-side multi archi support)
Several scripts make a call to a function called Check_multiarchitectures,
the purpose of which is to adjust the target directory that certain 'live'
and 'install' files are located in. The idea is that a script sets up
'DESTDIR', 'DESTDIR_LIVE' and 'DESTDIR_INSTALL' as appropriate and then
the script appends a suitable arch dependant postfix to the directory
name, depending upon the arch currently being targetted. This would allow
the script to be run multiple times, each for a different architecture.
This is a part of an implementation of allowing multiple architectures to
sit side by side within the same live image, selectable from the
bootloader menus. (As opposed to multiple architectures mixed within the
same userland).
This is evidently the case both from the fact that:
1) The arch specific postfix chosen in that function depends on a var
called LB_CURRENT_ARCHITECTURE, which is never set. In fact going
back through the git history to the introduction of the function in
0d5ff4ca75, the var (even considering
var name changes) has never been set by anything. So effectively the
call to the function has been entirely redundant all this time.
2) The major build stages do not perform multiple executions of substages
per arch. Thus from this perspective it seems that the support was
never fully implemented.
3) If any doubt remained, there is an old branch called 'tmp-multiarch'
which has a couple of commits making progress with completing support,
such as implementing the above missing pieces.
The above mentioned branch is 10 years old and can be considered abandoned.
It is not clear whether the original author ever intended to complete and
merge this; nor is it at all clear at what stage of completion it was at.
At any rate, imo it is not at all particularly useful to have extra code
and complexity in order to be able to cram multiple environments side by
side in one image, not when CDs/DVDs and even to some extent USB pen drives
are so cheap. And who really needs more than one environment so
desperately on just one such medium.
If this was not enough to justify removal, then there is also the fact that
the support that was implemented has become completely broken over the
years with scripts diverging in terms of the variable names the function
modifies such that they are incompatible with it.
A quick assessment of the state of this latter aspect:
good:
- grub-legacy uses the correct var names so is fine
- memtest similarly good
- installer_debian-installer looks okay
questionable:
- binary_linux-image uses the correct vars but might not select the
right kernel and initrd files to copy (seems to copy all)
bad:
- grub-pc is making a redundant call, after functionality was moved
to the loopback script
- loopback is using the wrong vars (INITFS instead of DESTDIR +
DESTDIR_INSTALL + DESTDIR_LIVE), plus is doing its own amd64+i686
thing anyway, so the function call would achieve nothing anyway.
- syslinux is also using the wrong var names so would not work with
it and is not even making the necessary function call. Also the
install paths are fixed in the hard coded cfg files anyway so this
would need addressing with placeholders and sed replacement, but
then it is not entirely clear how things should work with respect
to install entries and multi-arch anyway, are we having multiple
copies of the installer, one for each target arch and then multiple
copies of the install menus, perhaps under different submenus?
So, this removes the artefacts of this never completed feature.
Gbp-Dch: Short
all echo helpers are used as logging functions with output to go to the
terminal. when used in functions that are designed to return a string
though the message printed would get incorrectly captured.
the previous fix done in e3a987d977 was
stupidly flawed; somehow my testing led me to mistakenly believe that
was adequate, but retesting proves that it was not.
here we create a new FD #3 linked to stdout to output the messages on,
which testing shows works as I had actually intended it.
e.g. here:
```
Foo () { if [ "$1" = "a" ]; then printf "foo\n"; else printf "error\n"; fi; }
```
we get:
```
~$ Foo a
foo
~$ Foo b
error
~$ XX="$(Foo a)"
~$ echo "${XX}"
foo
~$ XX="$(Foo b)"
~$ echo "${XX}"
error
```
and as demonstrated, "error" got incorrectly captured by in the variable
whereas here:
```
exec 3>&1
Foo () { if [ "$1" = "a" ]; then printf "foo\n"; else printf "error\n" >&3; fi; }
```
it is different in the last case:
```
~$ XX="$(Foo b)"
error
~$ echo "${XX}"
```
the error successfully makes it to the terminal, and the variable is an
empty string (with a newline automatically printed).
Gbp-Dch: Short
I asked for such a switch to be added in debootstrap back at the start of
2015 in #775454 as part of a review I undertook of its security. A slightly
modified patch was merged a few months later and made it into version
1.0.69.
A patch was never merged into live-build to make use of it however. Let's
do that now.
The benefit of this, as explained in #775454, is that if we want strong
security (LB_APT_SECURE=true) then should debootstrap not be able to find
the GPG key to verify things with, it will abort with an error instead of
falling back to just https downloads with a warning. Such a warning would
be easy to miss in the log output, and security could potentially be
compromised if this were to happen.
Gbp-Dch: Short
Instances of:
if [ $(which <command> ]
have been replaced with:
if command -v <command> >/dev/null
which is considered to be more robust in a range of environments.
scripts/build/chroot_archives: line 259:
if [ "${LB_APT}" = "aptitude" ] && [ ! $(Chroot chroot "which aptitude") ]
has been left untouched because the chroot might require a more complex command
which would need more testing.
manpages/Makefile: line 42:
@if [ ! -x "$$(which po4a 2>/dev/null)" ]; \
has been left untouched because I am not sufficiently familiar with makefiles.
commit f811656150 enabled the grub-efi
bootloader by default for amd64|i386 architectures, but failed to
recognise the this bootloader is not supported for hdd|netboot images.
this meants that if a user tried to build such an image without explicitly
specifying the bootloader, excluding grub-efi, their build would fail
with an error in the binary_grub-efi stage.
this fixes the problem by only enabling grub-efi by default on supported
image builds.
677415f6d7 (2007) in v1.0~a2-1 added a hack
relating to the loop-aes-utils package and losetup. this commit bundled
a bunch of changes, it was not specific to the hack, and so info about the
hack is limited to a brief comment included within the related change in
defaults:
```
# Workaround for loop-aes-utils divertion
# (loop-aes-utils' losetup lacks features).
```
though it is very similar to the removed fdisk hack in that it seems that
one package may replace a binary from another, moving the original to a
new location, and this hack gives the user the opportunity to select the
original instead of the one put in its place, for use in LB.
the comment mentions a package called loop-aes-utils as being the package
that performs such a diversion, and that the need for the hack was that
losetup itself lacked features, presumably encryption support, and it is
clear that it is the losetup binary that is the focus of the diversion.
looking into the history of loop-aes-utils a little, this package was
dropped from debian back in 2012 (#680748), favouring encrytion support of
dm-crypt/cryptsetup.
double checking file contents of packages, only the mount package carries
an /sbin/losetup file, so presumably this means that dm-setup/cryptsetup
do not perform such a diversion of losetup (i.e. their use is exclusively
done directly).
since the possible diversion is simply gone, that completely removes any
point in having the hack of giving users choice between losetup and the
diverted one. so let's remove this obsolete hack...
8321653cb3 (from 2007) introduced a hack to
work around bug #445304 in gnu-fdisk for users who may have replaced fdisk
with the classic gnu version. the hack allowed users to select an alternate
fdisk binary to use to work around the buggy binary.
bug #445304 is marked as found in v1.0-1 and fixed in v1.2-1, though may
have been fixe din v1.1. it was marked fixed in 2009.
checking the package archive, gnu-fdisk does not actually exist anymore
in debian, with one exception - it is available for arm64 on sid via
debports, and that version is 1.3 so thus includes the necessary fix
anyway.
it is thus pointless now that we still carry this hack.
Gbp-Dch: Short
I'd mistook the copyright here to apply to the apt package, but did not
look close enough, it's clearly referring to copyright of the files which
we've just amended to have a "The Debian Live team" notice, which this
should thus surely also have.
Gbp-Dch: Short
Current versions of the project files are built upon versions published
and licensed by Daniel Baumann, but are modified copies of those files and
thus need to be marked as such per licensing requirements (afaik he did
not pass along ownership / licensing rights to anyone when he left the
project). We should also be careful to not be misrepresenting such
modified copies as being attributed to Daniel.
Adding a new copyright line referring to "The Debian Live team" should
suffice for this.
The authorship block in man pages has also similarly been updated.
Notes:
- tweaked a copy of daniel copyright lines stating 2014 instead of 2015.
both of these cases were in files that i had personally introduced in
some of my past merged commits that moved some code around. i don't know
why they stated 2014.
- binary_onie was introduced in 2018, so that has a 2018 date instead of
2016 unlike the rest.
- 'efi-image' is a 3rd-party (Canonical Ltd) work that we bundle, but it
has been modified by 674794a8f4 and
36a3ba7634 so I similarly added a
debian live copyright line.
- 'grub-cpmodules' is similar. it was only changed by the indentation fix
of 36a3ba7634 but modification is
modification, and this does help cover any possible future changes that
might be made.
Prefer downloading the version of the source package that actually
corresponds to the version of the binary. Should apt-update be
run and a package updated, we do not really want to fetch a newer
copy of the source than that of the binary, we want the exact
corresponding version (kinda the whole point of compiling a source
disc that they correspond). If the exact version is no longer
available then it is surely preferable to list it in the missing
list than end up with a newer version.
Gbp-Dch: Short
Closes: #952932
if you execute the bootstrap stage with no internet connection, you get
the following output:
```
[2020-03-10 19:18:46] lb bootstrap
P: Setting up clean exit handler
[2020-03-10 19:18:46] lb bootstrap_cache restore
[2020-03-10 19:18:46] lb bootstrap_debootstrap
P: Begin bootstrapping system...
P: If the following stage fails, the most likely cause of the problem is with your mirror configuration or a caching proxy.
P: Running debootstrap (download-only)...
I: Retrieving InRelease
I: Retrieving Release
E: Failed getting release file http://deb.debian.org/debian/dists/buster/Release
P: Begin unmounting filesystems...
P: Saving caches...
chroot: failed to run command ‘/usr/bin/env’: No such file or directory
```
the last line looked suspicious. investigating it turns out that there was
a deficiency in the exit handler.
when debootstrap fails to download what it needs due to lack of a
connection, that failure due to `set -e` causes the Exit() handler to kick
in. Part of this includes outputting the "Saving caches..." line, before
then making a call to Save_package_cache(). That in turn runs the following
command:
```
Chroot chroot "apt-get autoclean" || true
```
The Chroot() function includes a line starting with:
```
${_LINUX32} chroot "${CHROOT}" /usr/bin/env
```
which is the source of the last output line.
the reason we see this unexpected output is that with bootstrapping having
failed, there is no /usr/bin/env within the chroot so it is bound to fail.
the fact is, the exit handler has no business trying to pretty much
anything that it does if the bootstrap_debootstrap stage has not
completed.
this implements such a restriction and thus resolves the problem of this
unexpected and confusing output in the described situation.
we will now see:
```
[2020-03-10 19:18:46] lb bootstrap
P: Setting up clean exit handler
[2020-03-10 19:18:46] lb bootstrap_cache restore
[2020-03-10 19:18:46] lb bootstrap_debootstrap
P: Begin bootstrapping system...
P: If the following stage fails, the most likely cause of the problem is with your mirror configuration or a caching proxy.
P: Running debootstrap (download-only)...
I: Retrieving InRelease
I: Retrieving Release
E: Failed getting release file http://deb.debian.org/debian/dists/buster/Release
```
the source stage emitted the following output for each packages downloaded.
note the troubling warning at the end.
```
Reading package lists... Done
NOTICE: 'grep' packaging is maintained in the 'Git' version control system at:
https://salsa.debian.org/debian/grep.git
Please use:
git clone https://salsa.debian.org/debian/grep.git
to retrieve the latest (possibly unreleased) updates to the package.
Need to get 1579 kB of source archives.
Get:1 http://deb.debian.org/debian buster/main grep 3.3-1 (dsc) [2038 B]
Get:2 http://deb.debian.org/debian buster/main grep 3.3-1 (tar) [1473 kB]
Get:3 http://deb.debian.org/debian buster/main grep 3.3-1 (diff) [104 kB]
Fetched 1579 kB in 1s (1293 kB/s)
Download complete and in download only mode
W: Download is performed unsandboxed as root as file 'grep_3.3-1.dsc' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)
```
this occurred because the '_apt' user did not have permission to write to
the destination directory and so was falling back to downloading as root
in order to do its work.
prior to 158950b873 all source packages were
downloaded directly to the root of the chroot. that commit changed this to
save them into a new clean directory within it instead. thus to fix the
problem we can simply set the ownership of this new directory to '_apt'.
Gbp-Dch: Short
the check for existence of debootstrap here was completely redundant since
there is a check at the beginning of the file which already outputs an
appropriate error and exists if missing.
the cache restore/save script is not a chroot modification script unlike
the rest of the scripts that it was bunched up with. It is an actual
component part of the chroot build stage.
let's bring clarity to this with improved documentation.
Gbp-Dch: Short
LB_APT_SOURCE_ARCHIVES determines whether or not deb-src entries are
desired to be included in apt's sources.list. here, instead of excuding
them we always include them but commented out where they would previously
have been excluded. this means that if a user later changes their mind and
wants to make use of them all they have to do is uncomment them rather
than add the necessary lines.
Gbp-Dch: Short
Closes: #952929
all scripts use `set -e` which means that if getop fails, the subsequent
error check that would print an error in addition to any printed by getopt
itself would never actually be reached.
the first though here would be to remove the pointless error check, but
getopt does not include the word "error" with an unrecognised option
failure, nor does it use colour to highlight problems, both of which mean
that it is a little lacking in terms of highlighting problems to users.
thus we properly capture and use the exit code here and output an
appropriate message per invalid argument vs getopt internal error.
also, removed the redundant stderr redirection which is already done
by Echo_error().
Gbp-Dch: Short
Lyndon Brown
Luca Boccassi
johnraff