to protect against simple mistake of using 'all' instead of
'all-except-archives' when manually executing scripts (e.g. during
development) at the bootstrap stage level. (the bootstrap stage does not
and should not use the archives helper).
Gbp-Dch: Ignore
it now covers:
- `lb chroot_apt install-binary`
- `lb chroot_archives {chroot|binary|source} {install|remove}`
by expanding usage from:
`lb chroot_prep {install|remove} HELPERS [ARGS]`
to:
`lb chroot_prep {install|remove} HELPERS [MODE[ MODE..]] [ARGS]`
where `[MODE[ MODE..]]` is an optional set of one or more of:
- archives-chroot, which specifies to use 'chroot' as the first param to
the chroot_archives script
- archives-binary, which specifies to use 'binary'
- archives-source, which specifies to use 'source'
- apt-install-binary, which specified to pass 'install-binary' instead of
'install' to chroot_apt
thus _all_ chroot prep scripts can be run through this helper now!
note, in the case of the binary stage, 'archives' is deliberately not added
to CHROOT_PREP_OTHER, this is not a mistake!
Gbp-Dch: Short
rather than explicitly running one helper after another in the major
build stages, or by hand (e.g. while testing things during development),
they can be run in bulk via this new helper. it essentially just takes a
list of helpers to run and runs them one by one.
it supports running all helpers except chroot_archives because that one
has different parameter requirements to the rest and supporting it would
make things messier.
helper scripts can either be named by their full script name or without
the 'chroot_' prefix for brevity. you can also just specify 'all' to
refer to all helpers (except chroot_archives, per above).
it automatically reverses the order of the list when run in remove mode.
Gbp-Dch: Short
it was not run in install mode so should not be run in remove mode.
(whether it should in fact be run in install mode is another question; as
is whether chroot_tmpfs should be being used)
Gbp-Dch: Short
when used alongside syslinux and when a single kernel flavour is used,
things work correctly. otherwise booting from EFI is broken.
the problem comes from the fact that syslinux, for a single kernel flavour
creates the file /live/vmlinuz, which is used by the minimal EFI grub.cfg
to locate the device and partition containing the live image. when multiple
kernel flavours are used, it instead creates /live/vmlinuz1, /live/vmlinuz2,
etc. which thus is a problem. similarly when syslinux is not used, you are
left only with long filenames for the kernel files, for example
/live/vmlinuz-4.19.0-8-amd64. in these situations grub cannot find the
device containing the image and thus fails to display the boot menu.
the solution here, instead of dynamically changing the filename searched
for depending upon bootloader configuration, switches to doing a search for
the file /.disk/info instead. this file is generated by binary_disk, and
is present for iso, iso-hybrid and hdd images types, though grub-efi cannot
be used for the hdd type. it is not created for the netboot type, but again,
grub-efi is not compatible with that anyway. it is not created for the tar
type, which the grub-efi script does not block as incompatible, but is this
not a mistake?
furthermore, switching to searching for /.disk/info helps avoid issues for
systems that happen to actually include a real /live/vmlinuz path other
than on a removable live disk or CD/DVD, as is the case with a HP system
discussed in #924053.
this patch was written by adrian15sgd@gmail.com, as per the authorship,
who attached it to the #924053 bug discussion. this commit message however
has been re-written by jnqnfe@gmail.com, prior to submission via an MR,
as part of the fix towards the issues reported in #956131.
Gbp-Dch: Short
Closes: #924053
backwards compatibility:
1. the new file will be included alongside any user custom config
2. rather than replace MEMTEST with an actual config entry, we replace it
with a line to import the content of the new file, and thus will work
just as before.
thus no backwards compatible breakage
Gbp-Dch: Short
backwards compatibility:
1. the new install.cfg and install_start.cfg files (chosen
automatically from the install_*gui.cfg and install_*test.cfg
files) will be included alongside any user custom config.
2. the placeholders are now replaced with lines importing these files
thus everything will work just as before, i.e. no backwards
compatibility breakage.
Gbp-Dch: Short
as just done for grub2|loopback
the primary benefit here is that it means that user configs do not
have to carry copies of all files; they just carry the ones they
want to replace (or add).
Gbp-Dch: Short
`$_SOURCE` is always composed of `<foo>/${_BOOTLOADER}`, so we can just use
`${_BOOTLOADER}` as the basename, without calling `basename ${_SOURCE}`
Gbp-Dch: Ignore
...rather than choosing between the default set and a user provided set
1. ensures backwards compatibility after we switch from generation of
certain content to that content being in pre-prepared config files,
and thus no longer writing that config out to existing files.
2. means that user configs do not have to carry copies of all files; they
just carry the ones they want to replace (or add).
Gbp-Dch: Short
this takes a step forward in moving towards the same updated layout as
with syslinux; here we get:
- <live entries>
- Start installer
- Advanced install options...
- <full set of install options>
- Advanced options...
- Memory Diagnostic Tool (memtest86[+])
note that this only affects the default menu. custom configs are not
affected by this change.
further steps to complete the move to the updated layout will follow
later.
"Advanced options..." should perhaps be renamed later.
Gbp-Dch: Short
an official current debian install disc was compared with to achieve
better consistency.
main menu:
- i: for the single "start" entry
advanced submenu:
- g: for the main graphical entry
- i: for the main text-based entry
- x: for the main expert entry
- a: for the main auto entry
- r: for the main rescue entry
- s: for the synth entry
for expert, auto and rescue, the hotkey is given to the graphical entry
where present, otherwise to the text entry.
Gbp-Dch: Short
...for consistency with syslinux config placeholders and improved
clarity of what text is a placeholder.
the old placeholders without the bookends are still replaced for
user configs for backwards compatibility.
the new ones are little used just at the moment but are expected to
become used much more in later commits.
Gbp-Dch: Short
...from the `Set_config_defaults` function, to being done directly
in `build.sh` (the component which is also responsible for loading
functions, loaded at the start of every script, including the front
end).
thus the colouring decision will now correctly...
- apply to the frontend, such as to the 'root privileges needed'
error, the 'no such script' error, and the command name
colouring that I want to add (the most significant issue).
- apply to error messages generated by the `Arguments` and
`Read_conffiles` functions, which are called before
`Set_config_defaults` by scripts.
as things were, due to the comparison with "false", colour would
_always_ be used in these places (unless _COLOR_ERR=false or
_COLOR_OUT=false wrt. the new command highlight, were set in the
environment when executing a script throught the frontend).
this would not be a problem for normal terminal use of course,
besides being inconsistent where color were turned off, but would
be a bit of a problem if redirected to a file.
a re-evaluation of _COLOR is performed in `Set_config_defaults` to
adjust _COLOR_OUT and _COLOR_ERR where necessary, to correctly
respond to _COLOR being set in saved config files (disabled by
default but a user could always enable), after the point of config
files being loaded.
_COLOR can still be controlled from the environment just as before,
overriding both _COLOR_OUT and _COLOR_ERR.
note that this does not address the fact that --color|--no-color
do not work in the frontend and thus will not impact the colouring
of to-be-introduced command highlighting. this needs to be
addressed separately.
Gbp-Dch: Short
d-i removed this in commit 0917b2dde3ff73a204d27dd2f2fffc8a41175ddd
Note: There was inconsistency between grub and syslinux in use of this, with
syslinux not having it on graphical rescue and auto modes while grub entries
did. The patch to fix that has been dropped since we're removing it everywhere
anyway.
(#395040)
Gbp-Dch: Short
progress-linux, as discussed in MR #142 ([1]) is a little known distro,
which appears to be little more than a personal project of the original
author of live-build.
given that, the expense of maintaining all of these old hacks for it
cannot be justified. it is not known whether or not live-build is even
used with respect to it since the author abandoned live-build some
years ago.
also, at least one past change in live-build possibly broke progress-linux
compatibility anyway, which would have required progress-linux users of
live-build to use a custom progress-linux config, or a progress-linux
fork of live-build, and there is no knowing how much of the hacks in this
"upstream" codebase any user of progress-linux currently relies upon.
and again, progress-linux appears to just be a personal project of
Daniel's, with afaik very little userbase. (Daniel seems to be the only
developer working on the project which speaks to how small it is).
[1]: https://salsa.debian.org/live-team/live-build/-/merge_requests/142
Gbp-Dch: Short
when the --fdisk and --losetup options were removed, the entries in the
getopt option list should have remained for backwards compatibility such
that the usage warnings can kick in instead of unknown option errors.
Gbp-Dch: Ignore
this function takes one or more required stage fileS _plural_, and exits
if any are missing (or at least it does now after the refactor).
let's rename it to make things more clear
Gbp-Dch: Short
as suggested by Raphaël
rather than have fixed stagefile filename strings at all in the scripts,
use `$(basename $0)` to use the name of the script (which is the same for
almost all cases anyway, and the stage files are supposed to be almost
exclusively unique per-script). we can thus simplify things by determining
the filename for most use cases within the functions themselves.
this does change the file used by a couple of scripts, affecting backwards
compatibility of executing live-build upon an existing partially or fully
completed build:
- binary_grub-pc used "binary_grub"
- chroot_includes used "includes.chroot"
care had to be taken for the following cases:
- there are some cases like bootstrap_cache, source_debian and
bootstrap_debootstrap which are dealing with more than one file, and/or
otherwise a filename that is not specific to the script itself exactly,
or should not be based upon its name.
- some cases like chroot_cache, bootstrap_cache and
chroot_install-packages need to append something to the end of the name
depending upon which pass/action mode the script is being executed with.
- furthermore in the bootstrap_cache case one of the filenames is used
within the bootstrap_debootstrap and thus needs very careful handling
to be certain that a change in filename of bootstrap_cache does not
break bootstrap_debootstrap.
Gbp-Dch: Short
- avoid all need to pass ".build/" path in stage file names into the
functions
- add a helper to remove a stage file (required to complete the above
properly)
- avoid duplicating filenames within scripts which makes them prone to
mistakes (some instances of which I've actually encountered and had
to fix)
Gbp-Dch: Short
Fixes the following
- Correct version (memtest86/memtest86+) shown instead of fixed 'memtest86+' text
- Ensure correct directory path always used by using replaceable placeholder
Gbp-Dch: Short
all vars affected have been carefully checked to be quite certain
that they are definitely local
where variable is assigned the return value of a function/command, the
local "declaration" is deliberately done on a separate line, since
`local FOO` is actually treated itself as a command rather than a
declaration; will thus always cause $? to be zero, and thus if done on
the same line as such an assignment can not only clobber $? but in doing
so unintentionally blocks failure of a command from triggering the
expected exit from having `set -e`.
also, from testing, i have found that when assigning "${@}" this must be
done on a separate line confusingly as otherwise an error occurs.
Gbp-Dch: Short
The '---' delimiter should appear before the final 'quiet' parameter
(which is used by the debian installer I believe).
This delimiter is added by live-build in syslinux configs, and is present
in both grub2 and syslinux configs in an official debian 7.7 disc image,
suggesting strongly that live-build grub/grub2 menu creation code is in
the wrong here by missing it.
update: this commit previously used -- as was correct at the time, and has
since been updated to use --- per #775128; which was previously tackled in
a separate later commit. the switch to --- was already done for syslinux
(which was not missing the delimiter unlike grub) in
ba6b9adeff
Gbp-Dch: Short
Closes: #775143
I believe that the `quiet` parameter is meant for d-i not the kernel and
thus should be given on the end after a delimiter, as done with syslinux.
Here we switch the order to move it to the end. The addition of the missing
delimiter will be done in a followup commit.
(See #775143)
Gbp-Dch: Short
When building grub2 menu entries the quiet param (meant for d-i)
was excluded from the rescue menu entries instead of expert.
This is the opposite to what is done in the following:
- Menu entries seen in official debian 7.7 disc images (grub2 and syslinux configs)
- Menu entries created for grub (legacy)
- Menu entries created for syslinux
The evidence strongly suggests that the grub2 menu creation was in the wrong!
(See #775143)
Gbp-Dch: Short
`DI_PACKAGES` does not need to include `DI_REQ_PACKAGES` so long as
we pass the latter to apt in the one case where it was not already
being given it.
in fact with it including that sub-list meant that in the other
case where it was being given to apt, it actually just resulted in
duplication.
Gbp-Dch: Short
the chown command needed running within the chroot, since apt-get is being
run within the chroot and _apt might have a different UID there than on
the host.
Gbp-Dch: Short
Closes: #953957
when building within a home directory, as /proc, /sys, etc are mounted
and unmounted into the chroot at various points in the build, corresponding
entries appear and disappear within the side panel of nautilus.
this is obviously undesirable.
use of `-o x-gvfs-hide` resolves this for the most part. i still see items
occasionally pop up and having spent some time experimenting, i'm putting
it down to buggy behaviour on the part of nautilus. (aside from those
appearing when debootstrap is running - debootstrap also needs this fix).
Gbp-Dch: Short
Combine the check+create done in each script. (The original functions
are still callable as before, but a new combined `Aquire_lockfile`
function can be called instead, as now used).
Note, a further simplification could be done in removing the passing of
the lock filename in as a parameter since every use of the functions is
with ".lock". The lock functions already have a fallback to ".build/lock"
though. Checking the history, the fallback used to be for a system wide
lock, which was then replaced with this config-tree specific one. As long
as that is not used implicitly by 3rd-party hooks then surely we are free
to change the fallback to ".lock" and further remove passing in a name as
a param...?
history:
db5d2b0dcd0aa8289a37
Gbp-Dch: Short
Closes: #952918
theres no point in creation of stagefiles being kept within a
conditional block of work. if the script completes with success
then it should create its stagefile to thus avoid repeating any
work that it might have done should it get re-run without being
forced.
Gbp-Dch: Short
(part of never completed side-by-side multi archi support)
Several scripts make a call to a function called Check_multiarchitectures,
the purpose of which is to adjust the target directory that certain 'live'
and 'install' files are located in. The idea is that a script sets up
'DESTDIR', 'DESTDIR_LIVE' and 'DESTDIR_INSTALL' as appropriate and then
the script appends a suitable arch dependant postfix to the directory
name, depending upon the arch currently being targetted. This would allow
the script to be run multiple times, each for a different architecture.
This is a part of an implementation of allowing multiple architectures to
sit side by side within the same live image, selectable from the
bootloader menus. (As opposed to multiple architectures mixed within the
same userland).
This is evidently the case both from the fact that:
1) The arch specific postfix chosen in that function depends on a var
called LB_CURRENT_ARCHITECTURE, which is never set. In fact going
back through the git history to the introduction of the function in
0d5ff4ca75, the var (even considering
var name changes) has never been set by anything. So effectively the
call to the function has been entirely redundant all this time.
2) The major build stages do not perform multiple executions of substages
per arch. Thus from this perspective it seems that the support was
never fully implemented.
3) If any doubt remained, there is an old branch called 'tmp-multiarch'
which has a couple of commits making progress with completing support,
such as implementing the above missing pieces.
The above mentioned branch is 10 years old and can be considered abandoned.
It is not clear whether the original author ever intended to complete and
merge this; nor is it at all clear at what stage of completion it was at.
At any rate, imo it is not at all particularly useful to have extra code
and complexity in order to be able to cram multiple environments side by
side in one image, not when CDs/DVDs and even to some extent USB pen drives
are so cheap. And who really needs more than one environment so
desperately on just one such medium.
If this was not enough to justify removal, then there is also the fact that
the support that was implemented has become completely broken over the
years with scripts diverging in terms of the variable names the function
modifies such that they are incompatible with it.
A quick assessment of the state of this latter aspect:
good:
- grub-legacy uses the correct var names so is fine
- memtest similarly good
- installer_debian-installer looks okay
questionable:
- binary_linux-image uses the correct vars but might not select the
right kernel and initrd files to copy (seems to copy all)
bad:
- grub-pc is making a redundant call, after functionality was moved
to the loopback script
- loopback is using the wrong vars (INITFS instead of DESTDIR +
DESTDIR_INSTALL + DESTDIR_LIVE), plus is doing its own amd64+i686
thing anyway, so the function call would achieve nothing anyway.
- syslinux is also using the wrong var names so would not work with
it and is not even making the necessary function call. Also the
install paths are fixed in the hard coded cfg files anyway so this
would need addressing with placeholders and sed replacement, but
then it is not entirely clear how things should work with respect
to install entries and multi-arch anyway, are we having multiple
copies of the installer, one for each target arch and then multiple
copies of the install menus, perhaps under different submenus?
So, this removes the artefacts of this never completed feature.
Gbp-Dch: Short
I asked for such a switch to be added in debootstrap back at the start of
2015 in #775454 as part of a review I undertook of its security. A slightly
modified patch was merged a few months later and made it into version
1.0.69.
A patch was never merged into live-build to make use of it however. Let's
do that now.
The benefit of this, as explained in #775454, is that if we want strong
security (LB_APT_SECURE=true) then should debootstrap not be able to find
the GPG key to verify things with, it will abort with an error instead of
falling back to just https downloads with a warning. Such a warning would
be easy to miss in the log output, and security could potentially be
compromised if this were to happen.
Gbp-Dch: Short
Instances of:
if [ $(which <command> ]
have been replaced with:
if command -v <command> >/dev/null
which is considered to be more robust in a range of environments.
scripts/build/chroot_archives: line 259:
if [ "${LB_APT}" = "aptitude" ] && [ ! $(Chroot chroot "which aptitude") ]
has been left untouched because the chroot might require a more complex command
which would need more testing.
manpages/Makefile: line 42:
@if [ ! -x "$$(which po4a 2>/dev/null)" ]; \
has been left untouched because I am not sufficiently familiar with makefiles.
677415f6d7 (2007) in v1.0~a2-1 added a hack
relating to the loop-aes-utils package and losetup. this commit bundled
a bunch of changes, it was not specific to the hack, and so info about the
hack is limited to a brief comment included within the related change in
defaults:
```
# Workaround for loop-aes-utils divertion
# (loop-aes-utils' losetup lacks features).
```
though it is very similar to the removed fdisk hack in that it seems that
one package may replace a binary from another, moving the original to a
new location, and this hack gives the user the opportunity to select the
original instead of the one put in its place, for use in LB.
the comment mentions a package called loop-aes-utils as being the package
that performs such a diversion, and that the need for the hack was that
losetup itself lacked features, presumably encryption support, and it is
clear that it is the losetup binary that is the focus of the diversion.
looking into the history of loop-aes-utils a little, this package was
dropped from debian back in 2012 (#680748), favouring encrytion support of
dm-crypt/cryptsetup.
double checking file contents of packages, only the mount package carries
an /sbin/losetup file, so presumably this means that dm-setup/cryptsetup
do not perform such a diversion of losetup (i.e. their use is exclusively
done directly).
since the possible diversion is simply gone, that completely removes any
point in having the hack of giving users choice between losetup and the
diverted one. so let's remove this obsolete hack...
8321653cb3 (from 2007) introduced a hack to
work around bug #445304 in gnu-fdisk for users who may have replaced fdisk
with the classic gnu version. the hack allowed users to select an alternate
fdisk binary to use to work around the buggy binary.
bug #445304 is marked as found in v1.0-1 and fixed in v1.2-1, though may
have been fixe din v1.1. it was marked fixed in 2009.
checking the package archive, gnu-fdisk does not actually exist anymore
in debian, with one exception - it is available for arm64 on sid via
debports, and that version is 1.3 so thus includes the necessary fix
anyway.
it is thus pointless now that we still carry this hack.
Gbp-Dch: Short
Current versions of the project files are built upon versions published
and licensed by Daniel Baumann, but are modified copies of those files and
thus need to be marked as such per licensing requirements (afaik he did
not pass along ownership / licensing rights to anyone when he left the
project). We should also be careful to not be misrepresenting such
modified copies as being attributed to Daniel.
Adding a new copyright line referring to "The Debian Live team" should
suffice for this.
The authorship block in man pages has also similarly been updated.
Notes:
- tweaked a copy of daniel copyright lines stating 2014 instead of 2015.
both of these cases were in files that i had personally introduced in
some of my past merged commits that moved some code around. i don't know
why they stated 2014.
- binary_onie was introduced in 2018, so that has a 2018 date instead of
2016 unlike the rest.
- 'efi-image' is a 3rd-party (Canonical Ltd) work that we bundle, but it
has been modified by 674794a8f4 and
36a3ba7634 so I similarly added a
debian live copyright line.
- 'grub-cpmodules' is similar. it was only changed by the indentation fix
of 36a3ba7634 but modification is
modification, and this does help cover any possible future changes that
might be made.
Prefer downloading the version of the source package that actually
corresponds to the version of the binary. Should apt-update be
run and a package updated, we do not really want to fetch a newer
copy of the source than that of the binary, we want the exact
corresponding version (kinda the whole point of compiling a source
disc that they correspond). If the exact version is no longer
available then it is surely preferable to list it in the missing
list than end up with a newer version.
Gbp-Dch: Short
Closes: #952932
the source stage emitted the following output for each packages downloaded.
note the troubling warning at the end.
```
Reading package lists... Done
NOTICE: 'grep' packaging is maintained in the 'Git' version control system at:
https://salsa.debian.org/debian/grep.git
Please use:
git clone https://salsa.debian.org/debian/grep.git
to retrieve the latest (possibly unreleased) updates to the package.
Need to get 1579 kB of source archives.
Get:1 http://deb.debian.org/debian buster/main grep 3.3-1 (dsc) [2038 B]
Get:2 http://deb.debian.org/debian buster/main grep 3.3-1 (tar) [1473 kB]
Get:3 http://deb.debian.org/debian buster/main grep 3.3-1 (diff) [104 kB]
Fetched 1579 kB in 1s (1293 kB/s)
Download complete and in download only mode
W: Download is performed unsandboxed as root as file 'grep_3.3-1.dsc' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)
```
this occurred because the '_apt' user did not have permission to write to
the destination directory and so was falling back to downloading as root
in order to do its work.
prior to 158950b873 all source packages were
downloaded directly to the root of the chroot. that commit changed this to
save them into a new clean directory within it instead. thus to fix the
problem we can simply set the ownership of this new directory to '_apt'.
Gbp-Dch: Short
the check for existence of debootstrap here was completely redundant since
there is a check at the beginning of the file which already outputs an
appropriate error and exists if missing.
the cache restore/save script is not a chroot modification script unlike
the rest of the scripts that it was bunched up with. It is an actual
component part of the chroot build stage.
let's bring clarity to this with improved documentation.
Gbp-Dch: Short
LB_APT_SOURCE_ARCHIVES determines whether or not deb-src entries are
desired to be included in apt's sources.list. here, instead of excuding
them we always include them but commented out where they would previously
have been excluded. this means that if a user later changes their mind and
wants to make use of them all they have to do is uncomment them rather
than add the necessary lines.
Gbp-Dch: Short
Closes: #952929
all scripts use `set -e` which means that if getop fails, the subsequent
error check that would print an error in addition to any printed by getopt
itself would never actually be reached.
the first though here would be to remove the pointless error check, but
getopt does not include the word "error" with an unrecognised option
failure, nor does it use colour to highlight problems, both of which mean
that it is a little lacking in terms of highlighting problems to users.
thus we properly capture and use the exit code here and output an
appropriate message per invalid argument vs getopt internal error.
also, removed the redundant stderr redirection which is already done
by Echo_error().
Gbp-Dch: Short
- prefer using `which` over hard coded paths
- it is redundant to check that the bin pointed to the return of
`which` exists and is executable, `which` already gives us
assurance of that if it returns true!
- the redirection of output (`2>/dev/null`) seems to be
unnecessary from my testing.
the instances relatnig to fdisk and losetup in functions/defaults.sh have
been left as they are since they get executed by `lb config` which can run
without sudo elevation unlike `lb build` and in that case `which` would
fail to find these binaries resulting in error.
this also fixes a bug showing an error for missing debootstrap - this tool
requires sudo privileges to run and thus is not found via a none elevated
which search.
Gbp-Dch: Short
Closes: #952927
commit d74f2102a0 added a validation check
to chroot_archives for its 'pass' parameter. this was based upon finding
an instance where the wrong value was being submitted to the script and
wanting to ensure such mistakes would be caught.
unfortunately it seems that I made a mistake in misremembering the facts
surrounding the latter issue when constructing the validation check and
failed to double check with testing until it was already merged. a
correction is needed. the set of valid values is not limited to only
'source|binary' but actually 'source|binary|chroot'. I'd misremembered
'chroot' as being a completely invalid value.
the existing logic for obtaining a list of firmware packages always
downloaded a fresh copy of the archive content file, deleting the file
already in the cache. here we move to actually making use of the cache.
this helps when building multiple times, at least for the same distro. the
package list obtained is rarely going to change after all. it could of
course differ between distros, but the cache is per-distro, as it has
always been.
we of course here switch to caching each of the archive-area files
individually rather than having one file that gets overwritten (or
appended to in the case of when we kept the decompressed file).
Gbp-Dch: Short
Closes: #952911
addressing an old fixme, should distro and parent-distro be identical (in
a derivative build) we want to avoid wasting effort downloading and
processing the same content files twice.
since parent and non-parent have separate archive-area lists though we
should perhaps not just assume that they are identical or ignore any
differences; thus here in such a situation we ensure that we handle any
archive areas not already done for parent-distro handling in such a case,
while skipping those already done.
i notice that the fixme actually also refers to avoiding actual
overlapping of the cached files, however the cached files are (currently)
always ignored anyway, so this is of no concern. reusing the cached files
is an entirely separate issue.
Gbp-Dch: Short
Closes: #952909
the 'manually add firmware-linux package' bit was stuck inbetween the
parent and non-parent logic, which was especially unhelpful before we
de-duplicated the logic into a common function.
Gbp-Dch: Short
Edit: There were four copies of the same logic to keep in sync;
Originally this patch deduplicated each file, but leaving a copy of
the new function in each, thus reducing the duplication but not
eliminating it. A later patch moved it into a shared function file
following further enhancements to the code in question. This has
since been revised to have the function moved to a shared file here,
which simplifies and gives a cleaner diff.
Gbp-Dch: Short
Closes: #952908
the archive content file downloaded to obtain a list of firmware packages
is always deleted and downloaded afresh currently. it may not be ideal that
we do not make use of the cache here, however while that remains
unaddressed, we might as well delete the file after we've used it in order
to not pointlessly waste disk space.
note that this file is ~613 MB for sid-amd64 currently.
Gbp-Dch: Short
Closes: #952907
where multiple archive areas are used, the code here on each loop is:
1) fetching the archive area contents file (compressed)
2) **appending** the output to that of the previous loop
3) searching the file for firmware references, adding to the list
since it appends rather than replaces, entries found in each loop get
re-added on each subsequent loop, resulting in duplication in the
resulting list
below I evaluate the possible solutions to explain why I chose the one
I chose, however the reader should not waste too much time worrying about
whether one of the other solution would have actually been better because
things are changed significantly in further commits shortly!
possible solutions:
a) switching to output (>) rather than append (>>), but this might fail
against an existing file
b) removing the file on each loop, but this will complicate any future
caching improvements that might be made here (currently the files are
always deleted and thus downloaded fresh)
c) allow the appending, evaluating the complete file after the loop
solution C warrants consideration of disk space consumption; currently the
compressed 'main' archive (for sid on amd64) expands to 592.3 MB (feb-2020),
'contrib' is 3.1 MB, and 'non-free' is 18.5 MB.
solution C was chosen here; the difference of accumulated file size vs.
max-single was minor enough to not be of particular concern (~613 vs.
~592 MB).
Gbp-Dch: Short
Closes: #952906
These functions are specific to handling packages stored in the
cache, not other files. They are also always used with the same
`cache/packages.` prefix to the path.
Gbp-Dch: Short
Closes: #952916
instead of trying all from derived mirror then falling back to parent upon
failure, which as pointed out by a message printed out can result in a
load of spurious 404 errors; actually get each udeb from the mirror it is
supposed to be retrieved from.
Partial fix for #952914, this is the last commit for it so closes it
Gbp-Dch: Short
Closes: #952914
the existing logic just bundled the entire parent and derived udeb lists
together, ignoring the fact that there might thus be two instances of some
packages, and relying upon getting derived ones first and checking file
existence to avoid handling the overridden parent instances.
here we now actually filter the list of parent udebs to exclude packages
that are to be obtained from the derivative.
this enables avoiding the file existence checking
Partial fix for #952914
Gbp-Dch: Short
move the code that checks whether a version of a given package has already
been obtained (to account for parent and derived both listing the same
package) to guard the copy from cache action also, not just the download
action.
in rare but possible scenarios it would have been possible to end up with
both the parent and derived copies of a package included.
Partial fix for #952914
Gbp-Dch: Short
this script outputs a series of progressive dots as progress is made,
finally terminated with ' done.'. however if an error occurs then the
error would end up being printed directly on the end of this instead of
on a new line since newlines are not printed after each dot. this fixes
this oversight.
Gbp-Dch: Short
Closes: #952883
the '.' progress stuff has been left as is. perhaps that should be removed
since its use is inconsistent compared to other scripts?
Gbp-Dch: Short
Closes: #952882
Avoid all of the duplication for each installer case
Note, what is done for the netboot case (which was previously missing)
still needs addressing
Gbp-Dch: Short
Closes: #952865
LB_DEBIAN_INSTALLER_GUI defines whether or not to provide the graphical
installer. the installer_debian-installer script pays attention to it and
does not download it if not wanted. the actual bootloaders however
(both grub2/loopback and syslinux) ignore it, which leaves broken and
unwanted menu entries. this fixes that.
Gbp-Dch: Short
Closes: #952890
available values currently are memtest86+|memtest86|none; "false" is
presumably handled for backwards compatibility
there is no need to handle this in individual scripts. the right place
to handle it is in Set_Defaults as now done
Gbp-Dch: Short
Closes: #952866
`false` and `none` make no sense as choices for this option. Here we
replace `false` with `none`, and remove `true`.
Note that `true` was treated as an alias for netinst (see the changes to
source_disk and and binary_disk).
For backwards compatibility we still allow `true` and `false` by converting
them to `netinst` and `none` respectively, whilst printing a warning to
encourage users to move to `netinst`/`none`.
Gbp-Dch: Short
Closes: #952864
including:
- spaces replaced with tabs for consistency
- alignment of `;;` in some case statements changed for consistency
Gbp-Dch: Short
Closes: #952857
The specifics of what whould be done for it, beyond the disk info
label, needs addressing. (fixme comment left). (#952854).
Partial fix for #952846
Gbp-Dch: Short
Closes: #952846
update feb-2020: the bug report is marked as fixed in version 5.0-a7-1,
however there is no such patch merged for that tagged release, nor at all
in master, so this properly closes it.
Gbp-Dch: Short
Closes: #776532
Update feb-2020:
Both grub2 and syslinux were affected by this issue. Someone else's
patch fixing syslinux only just recently got merged in
2735f3bd38 but grub2 was still
affected. I had made patches for both in 2015 which never got
merged. This is the grub2 one.
Gbp-Dch: Short
Closes: #952843
all uses of this script pass in a suitable 'pass' param, and a previous
commit added a validation check. Having a fallback to 'binary' should a
param not be supplied is completely unnecessary.
fyi, this script is only ever called from the bootstrap stage and with
'binary' as the param, thus supporting being called with 'source' and
indeed having a 'pass' param at all is surely utterly pointless in itself.
Gbp-Dch: Ignore
18e0a2f325 "rebranded live sources" which
included renaming `source_debian-live` to `source_live` but accidentally
left the old file in place.
(Closes#952834)
apt-ftparchive is not able to differentiate between .deb and .udeb so
we have to install them in different pool directories so that we can
regenerate the Packages files without having the .udeb show up
unexpectedly.
Since binary_package-lists can overwrite the Packages files generated
in installer_debian-installer we have to ensure that it also updates
the Release file created formerly.
Ideally we should find a way to avoid the duplication of this logic.
Gbp-Dch: Full
In the binary stage, chroot is restored from cache/bootstrap, which contains
neither custom repo lists, nor keys. If local packages are present,
chroot_archives will call 'Apt chroot update' after adding custom repo lists
without adding keys. apt-get will then fail instead of warn as of apt version
1.5.
Closes: #941691
This makes it possible to build an image against a first distribution
(--distribution-chroot) and have the resulting image point to another
distribution (--distribution-binary). We can use this to build against a
snapshot and have the result use the original distribution that was
snapshotted.
Closes: #888507
Before Stretch there was an special amd64 kernel in the i386 arch repo.
So if you wanted to install an amd64 kernel alongside an i386 system
you did not need an additional arch repo.
Debian added multiarch support. That way you can install library packages
from multiple architectures on the same machine.
So there is no longer a need for having an amd64 kernel in i386 arch repo.
You can add an amd64 arch repo to an i386 arch system and fetch the amd64
kernel from the am64 arch repo.
live-build can be setup to use several linux kernel flavours in a single
image.
So in the days previous to this patch you could issue:
lb config --linux-flavours "486 amd64"
to use both 486 and amd64 kernel flavours.
Adding additional arch support to linux flavours poses two problems:
* Packages need to have its arch suffix (e.g. amd64:amd64).
If the suffix is not there apt-get insists on search amd64 kernel
package on i386 arch repo and, of course, fails to find it.
* The rest of the code which handles labels (bootloader config files)
or installed filenames (kernel images themselves) do not use the arch suffix.
This patch adds foreign architecture package support to
linux kernel flavours having taken those problems into account.
Practical example usage: i386 system and extra amd64 kernel.
First add amd64 foreign architecture in your i386 system
thanks to:
dpkg --add-architecture amd64
apt-get update
.
Finally enable amd64 kernel from amd64 arch alongside the
i386 system's 686 kernel thanks to:
lb config --architectures i386 --linux-flavours "686 amd64:amd64"
Turns out gcd works fine after adding /boot/grub/grub.cfg in the img,
as that's the path that gets hardcoded, and adding the EFI/debian/
grub.cfg was not necessary, so remove it.
For secured boot in binary_grub-efi, the gcdx64.efi.signed is
the boot loader for removable device, like CD or USB flash drive,
while grubx64.efi.signed is for hard drive. Therefore for live system,
use gcdx64.efi.signed for amd64 and gcdaa64.efi.signed for arm64.
With the merged /usr feature now enabled in debootstrap, we have
/bin and /lib which are symlinks to /usr/bin and /usr/lib. If
the live-build configuration provides files in
config/includes.chroot/lib/ or config/includes.chroot/bin/ then
lb chroot_includes will replace the /lib or /bin symlink with real
directories having only the content provided in chroot.includes and
problems will follow... the build will usually fail later with a
cryptic error message (for example a hook failing to execute
/usr/bin/env).
Work around the issue by creating a tarball of the files to install
and by unpacking that tarball from within the chroot with appropriate
options.
When using Secure Boot, grub2 as built by Debian will now load a config
file from EFI/$VENDOR instead of having EFI/debian hardcoded.
$VENDOR comes from dpkg-vendor or from the user building grub2.
The vendor string is stored in the control metadata as Efi-Vendor, so
retrieve it when building the EFI image.
Only gnupgv is part of the deboostrap set these days, but apt-key needs
the full gpg (with gpg-agent) which is just a recommends.
Instead just drop the key with an .asc suffix in /etc/apt/trusted.gpg.d
which is supported since apt version 1.4
live-build supports preseeding configuration, but the configuration has
to be installed after bootstrapping since it needs the debconf tools to
be applied. But packages that have already been installed and configured
in the bootstrap step will then ignore those preseeded configs.
After applying each preseed file, parse the package list and manually
reconfigure the relevant package(s) so that the configuration will be
applied.
The binary_onie script works on the host, not in the chroot (if used),
so don't check that the required packages for the script are installed
in the chroot (if present) as they won't be useful.
Check instead on the host.
Instead of hard-coding the decompression and compression formats,
detect them at runtime.
Install the required dependencies as well - they were mistakenly left
out.
Gbp-Dch: Ignore
The apt-secure option does not work anymore when building a sid image,
as with apt 1.6 the existing options are no longer enough to get apt
to accept an unsigned repository, which is necessary when using a
local cached repository (offline build).
Pass Acquire::AllowInsecureRepositories "true"; together with the
other options when --apt-secure false is used to fix the issue.
Open Network Install Environment is an open image format used by
networking vendor to ship a standardised image for networking white
box switches.
ONIE hardware takes this image at boot and a script to chain load
into the final environment via kexec. We can support Debian and
derivatives on such systems by packing an ISO which then gets
unpacked, kexec'ed and live-booted.
A base ONIE system can be tested in QEMU by building a VM following
these instrunctions:
https://github.com/opencomputeproject/onie/blob/master/machine/kvm_x86_64/INSTALL
Once built, boot onie-recovery-x86_64-kvm_x86_64-r0.iso in QEMU/libvirt
and on the console there will be the terminal prompt. Check the IP
assigned by libvirt and then scp the live image (ssh access is enabled
as root without password...). Then the .bin can be booted with:
ONIE-RECOVERY:/ # onie-nos-install /tmp/live.hybrid.iso-ONIE.bin
The implementation is inspired by ONIE's own scripts that can be found
at:
https://github.com/opencomputeproject/onie/blob/master/contrib/debian-iso/cook-bits.sh
A new option, --onie (false by default) can be set to true to enable
building this new format in addition to an ISO.
An additional option, --onie-kernel-cmdline can be used to specify
additional options that the ONIE system should use when kexec'ing the
final image.
Note that only iso or hybrid-iso formats are supported.
For more information about the ONIE ecosystem see:
http://onie.org
Signed-off-by: Erik Ziegenbalg <eziegenb@Brocade.com>
Signed-off-by: Luca Boccassi <bluca@debian.org>
The Tianocore reference UEFI implementation, used for example by Qemu,
wants the EFI directory name to be uppercase in the fat32 partition
when Secure Boot is enabled, and will fail to load otherwise.
Support for UEFI Secure Boot is modelled after how it currently works
in Ubuntu and on how it is going to work on Debian.
A minimal bootloader, shim, is used as the first-stage and it then
loads grub. Both have to be signed.
shim-signed is already available in Debian so the filenames are
already established, and the grub2 repository and packaging is common
between the 2 distros so we can already be reasonably sure of what it
is going to be.
So if both are available, copy /usr/lib/shim/shim[x64|aa64].efi.signed
as boot[x64|aa64].efi so that UEFI loads it first, and copy
/usr/lib/grub/[x86_64|arm64]-efi-signed/grub[x64|aa64].efi.signed as
grub[x64|aa64].efi.
This grub2 EFI monolithic image is currently hard-coded in grub2's
repository to look for a config file in efi/debian, so make a copy
of the previously added minimal grub.cfg that loads the real one in
that directory in both the fat32 and ISO 9660 partitions.
The new option --uefi-secure-boot can be set to auto (default,
enable or disable.
In auto, the lack of the signed EFI binaries is intentionally left as a
soft failure - live-build will simply fallback to using the locally
generated non-signed grub2 monolithic EFI binary as the only
bootloader. Given the difficulties surrounding the Secure Boot
signing infrastructure this approach gives the most flexibility and
makes sure things will "just work" once the packages are available,
without the need to change anything in the configuration.
This will also greatly help downstream distributions and users who
want to do self-signing.
The enable or disable options work as expected.
Closes: #821084
On some UEFI implementations, like the AMI found in the Supermicro
X10SDV-TP8F development board, the fat32 partition will be loaded
first and so Grub will set it the root, and then drop to the console
as it cannot find any config on it.
Add a minimal grub.cfg that allows Grub to find the main config on
the ISO 9660 partition and load it.
Closes: #892406
Machines tend to become unresponsive during the mksquashfs step.
Avoid this by lowering the priority of the process.
Thanks: Ronny Standtke for the patch.
Closes: #867539
Commit a15b579652 (#775989) dropped an early exit from the
chroot_archives remove step in case the parent mirror chroot and binary
parameters are the same and introduced a regression, as with the
following live-build now fails when the parent mirror is using a file:/
local apt repository (for example when the build worker is offline and
uses a pre-built cache of packages).
Example config:
lb config --mirror-bootstrap "file:/pkgs" \
--mirror-chroot "file:/pkgs/" \
--mirror-binary "file:/pkgs" \
--parent-mirror-bootstrap "file:/pkgs" \
--parent-mirror-chroot "file:/pkgs/" \
--parent-mirror-binary "file:/pkgs" \
...
with /pkgs being a directory with the packages for the installation and
the apt metadata (Packages/Sources/Release).
The problem is that, with such a setup, the /pkgs directory is bind
mounted inside the chroot as an optimisation in the install step,
and umounted as one of the first actions in the remove step for
chroot_archives.
Before that fix, the script terminated immediately. But now it
progresses and at the end it tries to run apt update inside the chroot
which will fail since the repository directory has been umounted, and
thus the packages and the apt metadata are no longer available, while
still being listed in /etc/apt/sources.list.
The proposed solution is to umount the local directory at the end of
the remove step, rather than at the beginning.
Closes: #891206
Now grub.cfg shows all the kernel options. Before this patch when you
had more than two kernels it only showed the auto option.
Signed-off-by: Raphaël Hertzog <hertzog@debian.org>
To generate an hdd image, binary_hdd first estimates the needed size of
the image using du. By default, when du finds multiple hardlinked copies
of a file, it counts them only once. However, when the target filesystem
is FAT, which does not support hardlinks, these files will take up more
space when finally copying the contents, breaking the build:
P: Copying binary contents into image...
cp: error writing 'chroot/binary.tmp/live/initrd.img-4.9.0-3-amd64': No space left on device
cp: error writing 'chroot/binary.tmp/efi/boot/bootx64.efi': No space left on device
cp: error writing 'chroot/binary.tmp/efi/boot/bootia32.efi': No space left on device
cp: cannot create directory 'chroot/binary.tmp/boot/grub': No space left on device
cp: cannot create directory 'chroot/binary.tmp/isolinux': No space left on device
To fix this, pass --count-links to du when the target is FAT, to make
the space estimation correct.
This problem is exposed by commit 9c974b26b (Instead of renaming kernel
for syslinux, create hardlinks), which might need to be separately fixed
(to not waste space on FAT targets), but binary_hdd should at least
handle hardlinks more gracefully.
Since commit fdc9250bc (Changing package dependency checks within chroot
to work outside as well), Check_package automatically checks for
LB_BUILD_WITH_CHROOT and works inside as well as outside of the chroot,
so no need to check LB_BUILD_WITH_CHROOT before calling them.
Install_package and Remove_package are just a no-op when building
without chroot, so they can also be called unconditionally.
Restore_cache and Save_cache do not check LB_BUILD_WITH_CHROOT but it
it should not hurt to call them when not needed (which already happened
in some cases).
This commit makes all Check_package calls unconditional on
LB_BUILD_WITH_CHROOT.
For binary_syslinux, this fixes the check (which used outdated paths
outside the chroot since 7b6dfd9d1), for binary_grub-efi,
binary_package-lists and chroot_package-lists this simplifies the code
(but also causes the check to become package-based instead of file-based
on apt-based systems), and for binary_loadlin and binary_win32-loader
this adds the check outside the chroot which was previously missing.
* Use only long kernel names.
* Put advanced options in a submenu.
* Use distro-agnostic labels.
* Don't generate entries with kernel version when we have a single
version.
This option lets you use an alternate bootstrap script when running
debootstrap. Thanks to Sjoerd Simons <sjoerd@debian.org> for the initial
patch.
Closes: #790033
Commit e24e4b in debootstrap fixed setup_available to work in the
--foreign case (iotw at the second stage). Unfortunately this breaks
things if components aren't passed to the second stage _and_ your main
component isn't called main.
To fix this, pass --components to both the first and second stage
debootstrap when needed.
Signed-off-by: Sjoerd Simons <sjoerd.simons@collabora.co.uk>
Up to now we created a temporary GPG key that we registered with apt-key
but with the switch to GnuPG 2 by default, this code broke. Now we stop
doing that but we add the “trusted=yes“ attribute in sources.list so
that APT knows that the repository can be trusted even if it's unsigned.
Signed-off-by: Raphaël Hertzog <hertzog@debian.org>
This work is based on debian-cd team work and uses,
as much as possible, the same mkisofs options
than the Debian Installation CD disk does.
It assumes that /boot/grub/grub.cfg (and other design items)
is generated by: binary_loopback_cfg .
It relies on efi-image and grub-cpmodules being setup
as build scripts on live-build package.
In the future event of these two files being moved
to a binary package (they are originally from:
src: live-installer) the binary_grub-efi script would have
to be rewritten to take the new paths into account.
These two scripts simplify the creation of efi images based on grub-efi.
I have decided to simply steal them. If I had to include them thanks to a source package that would have mean that an src repo would have to be defined by default.
TODO: Ask in a bug a RFE so that these two scripts are put into a binary that could be consumed by both live-installer and live-build packages.
The binary parts of grub-pc are left for the original binary_grub-pc.
As a consequence both /boot/grub/grub.cfg and /boot/grub/loopback.cfg files will be present in any Debian Live CD.
This might be useful to be reused from binary_grub-* bootloaders.
* Added: functions/bootloaders.sh . This file adds bootloader functions that are heavily used in efi scenarios where a bootloader can act as a first or an extra bootloader.
Since the introduction of the new switch:
--bootloaders
you can setup it like this:
--bootloaders=syslinux,grub-efi
.
This means that syslinux is the first bootloader and grub-efi is the extra bootloader.
* Added new bootloader functions: Check_Non_First_Bootloader and Check_Non_Extra_Bootloader.
These functions let each one of the bootloaders abort the build because
they cannot perform a role either as a first bootloader or as an extra bootloader.
* Added bootloader functions: Check_First_Bootloader_Role, Check_Extra_Bootloader_Role and Check_Any_Bootloader_Role
These functions let bootloaders to force their default role in a single line.
At the same time many binary bootloaders were rewritten to make use of the new bootloader role functions explained above.
These roles were enforced:
binary_grub-legacy : First bootloader
binary_grub-pc : Either first or extra bootloader
binary_syslinux : Either first or extra bootloader
If a bootloader is tried to be used in a role that it's not meant to be used then the build fails because that might lead to a non-bootable system.
The fix in a294a46fb9 was not enough.
This should finally resolve the problem when a package list ends
up empty (most notably due to #if evaluating to false).
Sponsored-By: Offensive Security
Some BIOSes dont't boot from partitions starting at sector 1024.
Some are even more peculiar and only start from sector 63.
This patch adds an option for the binary_hdd target to manually
configure the partition start.
Development details
--------------------
* This patch has been based originally on: binary_grub2 . It has been
improved thanks to some binary_syslinux bits.
* This patch ensures that binary_loopback_cfg needs is run
before binary_syslinux is run.
The reason is that it reuses some code from binary_syslinux to avoid
problems when binary_syslinux renames the kernel filenames.
* This patch already supports CPU detection
* I haven't tested all the possible scenarios for the script (with the
latest version). When amd64 and 486 Gnu/Linux flavours is used and the
bootloader is syslinux it works ok.
* I have not implemented a disable switch for not generating it.
Usually you always want loopback.cfg to be there.
* Compared to binary_grub2 script I have removed the installation
entries because I did not see any of them in binary_syslinux.
How to test
-----------
These are some steps to easily test if Looback cfg support is working ok.
0) We assume you have generated an iso
1) Make sure you have a partition that Grub understands. Plain ext4 or
vfat should do it.
2) Create directory: /boot/boot-isos/
3) Put the iso file into that directory making sure it has an ISO or iso
extension.
4) Setup your computer to boot from cdrom and use: Super Grub2 Disk
2.00s2 (Hybrid version recommended)
5) Choose Boot manually...
6) Choose Bootable ISOs (in /boot- ... )
7) Choose (the detected) GRUB Loopback Config
(hdN,msodsN)/boot/boot-ios/name-of-the.iso
8) You will be presented your loopback.cfg. Choose anyone of the entries
(unless it does not match your cpu architecture of course).
9) You should boot into your Debian Live without problems (thanks to
findiso boot parametre).
If you ever wanted to test from your grub2 installation instead from
Super Grub2 Disk check: http://www.supergrubdisk.org/wiki/Loopback.cfg
for an example.
Note: OLDIFS use makes IFS to be reset to "" instead to it being unset.
Either we need to detect if old IFS was unset to unset it
or we need a proper way of setting it as a local variable.
Even more IFS it's not currently used in
Check_package (which it's called from: binary_hdd).
we should have a clean way of resetting/unsetting IFS when calling Check_package.
The other approach it's to explicitly define IFS with its default value in the
places inside live-build code where we implicitly suppose that it's going to have
its default value.
Future live-build versions will still allow to use casper,
but its configuration will be done differently by a custom
config tree, rather than embedded and maintenance intensive
code in live-build itself.
Future live-build versions will still allow to use casper,
but its configuration will be done differently by hooks
in the config tree, rather than embedded and maintenance intensive
code in live-build itself.
Lyndon Brown
adrian15
Steven Shiau
Luca Boccassi
johnraff
David Hewitt
Raphaël Hertzog
John Estabrook
Ronny Standtke
Roland Clobus
Adrian Gibanel Lopez
Luca Boccassi
Steven Shiau
Marcel Partap
Raphaël Hertzog
Rohan Garg
Balint Reczey
Matthijs Kooijman
Sjoerd Simons
Sophie Brun
Kristian Klausen
Ben Armstrong
Maximilian Mehnert
Carlos Zuferri
Daniel Baumann
Daniel Reichelt
Raphaël Hertzog