this was previously not done in 8b109ffb96
to keep the renaming simple, but leaving the variable plural is a cause
for confusion.
since this property is stored in the INI style config/build config file
rather than a shell script based one, at the property there is already
singular, there was no need for a backwards compatibility hack.
Gbp-Dch: Short
(and move the wget options setting down where it should be while at it)
the value of LB_DEBIAN_INSTALLER is now properly checked in the main
validation routine, so we can just directly exit here as a simple safety
check should validation be bypassed.
Gbp-Dch: Short
whilst some parts of the codebase were set up to work with multiple types
specified, others did not work with it and would not necessarily be easy
to adjust. this thus makes some tweaks to adjust things accordingly.
- option renamed to singular form (maintaining backwards compatibility)
- a validation check has been added
- unnecessary glob style type references fixed
- checks with In_list changed to a direct singular comparison
- typo of type "netboot" written as just "net" fixed (though unreachable
so of no consequence; really the code could be removed but it's trivial)
Gbp-Dch: Short
progress-linux, as discussed in MR #142 ([1]) is a little known distro,
which appears to be little more than a personal project of the original
author of live-build.
given that, the expense of maintaining all of these old hacks for it
cannot be justified. it is not known whether or not live-build is even
used with respect to it since the author abandoned live-build some
years ago.
also, at least one past change in live-build possibly broke progress-linux
compatibility anyway, which would have required progress-linux users of
live-build to use a custom progress-linux config, or a progress-linux
fork of live-build, and there is no knowing how much of the hacks in this
"upstream" codebase any user of progress-linux currently relies upon.
and again, progress-linux appears to just be a personal project of
Daniel's, with afaik very little userbase. (Daniel seems to be the only
developer working on the project which speaks to how small it is).
[1]: https://salsa.debian.org/live-team/live-build/-/merge_requests/142
Gbp-Dch: Short
this function takes one or more required stage fileS _plural_, and exits
if any are missing (or at least it does now after the refactor).
let's rename it to make things more clear
Gbp-Dch: Short
as suggested by Raphaël
rather than have fixed stagefile filename strings at all in the scripts,
use `$(basename $0)` to use the name of the script (which is the same for
almost all cases anyway, and the stage files are supposed to be almost
exclusively unique per-script). we can thus simplify things by determining
the filename for most use cases within the functions themselves.
this does change the file used by a couple of scripts, affecting backwards
compatibility of executing live-build upon an existing partially or fully
completed build:
- binary_grub-pc used "binary_grub"
- chroot_includes used "includes.chroot"
care had to be taken for the following cases:
- there are some cases like bootstrap_cache, source_debian and
bootstrap_debootstrap which are dealing with more than one file, and/or
otherwise a filename that is not specific to the script itself exactly,
or should not be based upon its name.
- some cases like chroot_cache, bootstrap_cache and
chroot_install-packages need to append something to the end of the name
depending upon which pass/action mode the script is being executed with.
- furthermore in the bootstrap_cache case one of the filenames is used
within the bootstrap_debootstrap and thus needs very careful handling
to be certain that a change in filename of bootstrap_cache does not
break bootstrap_debootstrap.
Gbp-Dch: Short
- avoid all need to pass ".build/" path in stage file names into the
functions
- add a helper to remove a stage file (required to complete the above
properly)
- avoid duplicating filenames within scripts which makes them prone to
mistakes (some instances of which I've actually encountered and had
to fix)
Gbp-Dch: Short
all vars affected have been carefully checked to be quite certain
that they are definitely local
where variable is assigned the return value of a function/command, the
local "declaration" is deliberately done on a separate line, since
`local FOO` is actually treated itself as a command rather than a
declaration; will thus always cause $? to be zero, and thus if done on
the same line as such an assignment can not only clobber $? but in doing
so unintentionally blocks failure of a command from triggering the
expected exit from having `set -e`.
also, from testing, i have found that when assigning "${@}" this must be
done on a separate line confusingly as otherwise an error occurs.
Gbp-Dch: Short
`DI_PACKAGES` does not need to include `DI_REQ_PACKAGES` so long as
we pass the latter to apt in the one case where it was not already
being given it.
in fact with it including that sub-list meant that in the other
case where it was being given to apt, it actually just resulted in
duplication.
Gbp-Dch: Short
Combine the check+create done in each script. (The original functions
are still callable as before, but a new combined `Aquire_lockfile`
function can be called instead, as now used).
Note, a further simplification could be done in removing the passing of
the lock filename in as a parameter since every use of the functions is
with ".lock". The lock functions already have a fallback to ".build/lock"
though. Checking the history, the fallback used to be for a system wide
lock, which was then replaced with this config-tree specific one. As long
as that is not used implicitly by 3rd-party hooks then surely we are free
to change the fallback to ".lock" and further remove passing in a name as
a param...?
history:
db5d2b0dcd0aa8289a37
Gbp-Dch: Short
Closes: #952918
(part of never completed side-by-side multi archi support)
Several scripts make a call to a function called Check_multiarchitectures,
the purpose of which is to adjust the target directory that certain 'live'
and 'install' files are located in. The idea is that a script sets up
'DESTDIR', 'DESTDIR_LIVE' and 'DESTDIR_INSTALL' as appropriate and then
the script appends a suitable arch dependant postfix to the directory
name, depending upon the arch currently being targetted. This would allow
the script to be run multiple times, each for a different architecture.
This is a part of an implementation of allowing multiple architectures to
sit side by side within the same live image, selectable from the
bootloader menus. (As opposed to multiple architectures mixed within the
same userland).
This is evidently the case both from the fact that:
1) The arch specific postfix chosen in that function depends on a var
called LB_CURRENT_ARCHITECTURE, which is never set. In fact going
back through the git history to the introduction of the function in
0d5ff4ca75, the var (even considering
var name changes) has never been set by anything. So effectively the
call to the function has been entirely redundant all this time.
2) The major build stages do not perform multiple executions of substages
per arch. Thus from this perspective it seems that the support was
never fully implemented.
3) If any doubt remained, there is an old branch called 'tmp-multiarch'
which has a couple of commits making progress with completing support,
such as implementing the above missing pieces.
The above mentioned branch is 10 years old and can be considered abandoned.
It is not clear whether the original author ever intended to complete and
merge this; nor is it at all clear at what stage of completion it was at.
At any rate, imo it is not at all particularly useful to have extra code
and complexity in order to be able to cram multiple environments side by
side in one image, not when CDs/DVDs and even to some extent USB pen drives
are so cheap. And who really needs more than one environment so
desperately on just one such medium.
If this was not enough to justify removal, then there is also the fact that
the support that was implemented has become completely broken over the
years with scripts diverging in terms of the variable names the function
modifies such that they are incompatible with it.
A quick assessment of the state of this latter aspect:
good:
- grub-legacy uses the correct var names so is fine
- memtest similarly good
- installer_debian-installer looks okay
questionable:
- binary_linux-image uses the correct vars but might not select the
right kernel and initrd files to copy (seems to copy all)
bad:
- grub-pc is making a redundant call, after functionality was moved
to the loopback script
- loopback is using the wrong vars (INITFS instead of DESTDIR +
DESTDIR_INSTALL + DESTDIR_LIVE), plus is doing its own amd64+i686
thing anyway, so the function call would achieve nothing anyway.
- syslinux is also using the wrong var names so would not work with
it and is not even making the necessary function call. Also the
install paths are fixed in the hard coded cfg files anyway so this
would need addressing with placeholders and sed replacement, but
then it is not entirely clear how things should work with respect
to install entries and multi-arch anyway, are we having multiple
copies of the installer, one for each target arch and then multiple
copies of the install menus, perhaps under different submenus?
So, this removes the artefacts of this never completed feature.
Gbp-Dch: Short
Current versions of the project files are built upon versions published
and licensed by Daniel Baumann, but are modified copies of those files and
thus need to be marked as such per licensing requirements (afaik he did
not pass along ownership / licensing rights to anyone when he left the
project). We should also be careful to not be misrepresenting such
modified copies as being attributed to Daniel.
Adding a new copyright line referring to "The Debian Live team" should
suffice for this.
The authorship block in man pages has also similarly been updated.
Notes:
- tweaked a copy of daniel copyright lines stating 2014 instead of 2015.
both of these cases were in files that i had personally introduced in
some of my past merged commits that moved some code around. i don't know
why they stated 2014.
- binary_onie was introduced in 2018, so that has a 2018 date instead of
2016 unlike the rest.
- 'efi-image' is a 3rd-party (Canonical Ltd) work that we bundle, but it
has been modified by 674794a8f4 and
36a3ba7634 so I similarly added a
debian live copyright line.
- 'grub-cpmodules' is similar. it was only changed by the indentation fix
of 36a3ba7634 but modification is
modification, and this does help cover any possible future changes that
might be made.
addressing an old fixme, should distro and parent-distro be identical (in
a derivative build) we want to avoid wasting effort downloading and
processing the same content files twice.
since parent and non-parent have separate archive-area lists though we
should perhaps not just assume that they are identical or ignore any
differences; thus here in such a situation we ensure that we handle any
archive areas not already done for parent-distro handling in such a case,
while skipping those already done.
i notice that the fixme actually also refers to avoiding actual
overlapping of the cached files, however the cached files are (currently)
always ignored anyway, so this is of no concern. reusing the cached files
is an entirely separate issue.
Gbp-Dch: Short
Closes: #952909
the 'manually add firmware-linux package' bit was stuck inbetween the
parent and non-parent logic, which was especially unhelpful before we
de-duplicated the logic into a common function.
Gbp-Dch: Short
Edit: There were four copies of the same logic to keep in sync;
Originally this patch deduplicated each file, but leaving a copy of
the new function in each, thus reducing the duplication but not
eliminating it. A later patch moved it into a shared function file
following further enhancements to the code in question. This has
since been revised to have the function moved to a shared file here,
which simplifies and gives a cleaner diff.
Gbp-Dch: Short
Closes: #952908
the archive content file downloaded to obtain a list of firmware packages
is always deleted and downloaded afresh currently. it may not be ideal that
we do not make use of the cache here, however while that remains
unaddressed, we might as well delete the file after we've used it in order
to not pointlessly waste disk space.
note that this file is ~613 MB for sid-amd64 currently.
Gbp-Dch: Short
Closes: #952907
where multiple archive areas are used, the code here on each loop is:
1) fetching the archive area contents file (compressed)
2) **appending** the output to that of the previous loop
3) searching the file for firmware references, adding to the list
since it appends rather than replaces, entries found in each loop get
re-added on each subsequent loop, resulting in duplication in the
resulting list
below I evaluate the possible solutions to explain why I chose the one
I chose, however the reader should not waste too much time worrying about
whether one of the other solution would have actually been better because
things are changed significantly in further commits shortly!
possible solutions:
a) switching to output (>) rather than append (>>), but this might fail
against an existing file
b) removing the file on each loop, but this will complicate any future
caching improvements that might be made here (currently the files are
always deleted and thus downloaded fresh)
c) allow the appending, evaluating the complete file after the loop
solution C warrants consideration of disk space consumption; currently the
compressed 'main' archive (for sid on amd64) expands to 592.3 MB (feb-2020),
'contrib' is 3.1 MB, and 'non-free' is 18.5 MB.
solution C was chosen here; the difference of accumulated file size vs.
max-single was minor enough to not be of particular concern (~613 vs.
~592 MB).
Gbp-Dch: Short
Closes: #952906
These functions are specific to handling packages stored in the
cache, not other files. They are also always used with the same
`cache/packages.` prefix to the path.
Gbp-Dch: Short
Closes: #952916
instead of trying all from derived mirror then falling back to parent upon
failure, which as pointed out by a message printed out can result in a
load of spurious 404 errors; actually get each udeb from the mirror it is
supposed to be retrieved from.
Partial fix for #952914, this is the last commit for it so closes it
Gbp-Dch: Short
Closes: #952914
the existing logic just bundled the entire parent and derived udeb lists
together, ignoring the fact that there might thus be two instances of some
packages, and relying upon getting derived ones first and checking file
existence to avoid handling the overridden parent instances.
here we now actually filter the list of parent udebs to exclude packages
that are to be obtained from the derivative.
this enables avoiding the file existence checking
Partial fix for #952914
Gbp-Dch: Short
move the code that checks whether a version of a given package has already
been obtained (to account for parent and derived both listing the same
package) to guard the copy from cache action also, not just the download
action.
in rare but possible scenarios it would have been possible to end up with
both the parent and derived copies of a package included.
Partial fix for #952914
Gbp-Dch: Short
`false` and `none` make no sense as choices for this option. Here we
replace `false` with `none`, and remove `true`.
Note that `true` was treated as an alias for netinst (see the changes to
source_disk and and binary_disk).
For backwards compatibility we still allow `true` and `false` by converting
them to `netinst` and `none` respectively, whilst printing a warning to
encourage users to move to `netinst`/`none`.
Gbp-Dch: Short
Closes: #952864
including:
- spaces replaced with tabs for consistency
- alignment of `;;` in some case statements changed for consistency
Gbp-Dch: Short
Closes: #952857
apt-ftparchive is not able to differentiate between .deb and .udeb so
we have to install them in different pool directories so that we can
regenerate the Packages files without having the .udeb show up
unexpectedly.
Since binary_package-lists can overwrite the Packages files generated
in installer_debian-installer we have to ensure that it also updates
the Release file created formerly.
Ideally we should find a way to avoid the duplication of this logic.
Gbp-Dch: Full
This makes it possible to build an image against a first distribution
(--distribution-chroot) and have the resulting image point to another
distribution (--distribution-binary). We can use this to build against a
snapshot and have the result use the original distribution that was
snapshotted.
Closes: #888507
Lyndon Brown
Raphaël Hertzog
Kristian Klausen
Ben Armstrong