Follow-up for 18ad612b72.
The availability of 'firmware-linux' must be checked inside the chroot
instead of on the host.
This fixes building Bullseye without 'non-free' available (as discovered
by autopkgtest)
progress-linux, as discussed in MR #142 ([1]) is a little known distro,
which appears to be little more than a personal project of the original
author of live-build.
given that, the expense of maintaining all of these old hacks for it
cannot be justified. it is not known whether or not live-build is even
used with respect to it since the author abandoned live-build some
years ago.
also, at least one past change in live-build possibly broke progress-linux
compatibility anyway, which would have required progress-linux users of
live-build to use a custom progress-linux config, or a progress-linux
fork of live-build, and there is no knowing how much of the hacks in this
"upstream" codebase any user of progress-linux currently relies upon.
and again, progress-linux appears to just be a personal project of
Daniel's, with afaik very little userbase. (Daniel seems to be the only
developer working on the project which speaks to how small it is).
[1]: https://salsa.debian.org/live-team/live-build/-/merge_requests/142
Gbp-Dch: Short
this function takes one or more required stage fileS _plural_, and exits
if any are missing (or at least it does now after the refactor).
let's rename it to make things more clear
Gbp-Dch: Short
as suggested by Raphaël
rather than have fixed stagefile filename strings at all in the scripts,
use `$(basename $0)` to use the name of the script (which is the same for
almost all cases anyway, and the stage files are supposed to be almost
exclusively unique per-script). we can thus simplify things by determining
the filename for most use cases within the functions themselves.
this does change the file used by a couple of scripts, affecting backwards
compatibility of executing live-build upon an existing partially or fully
completed build:
- binary_grub-pc used "binary_grub"
- chroot_includes used "includes.chroot"
care had to be taken for the following cases:
- there are some cases like bootstrap_cache, source_debian and
bootstrap_debootstrap which are dealing with more than one file, and/or
otherwise a filename that is not specific to the script itself exactly,
or should not be based upon its name.
- some cases like chroot_cache, bootstrap_cache and
chroot_install-packages need to append something to the end of the name
depending upon which pass/action mode the script is being executed with.
- furthermore in the bootstrap_cache case one of the filenames is used
within the bootstrap_debootstrap and thus needs very careful handling
to be certain that a change in filename of bootstrap_cache does not
break bootstrap_debootstrap.
Gbp-Dch: Short
- avoid all need to pass ".build/" path in stage file names into the
functions
- add a helper to remove a stage file (required to complete the above
properly)
- avoid duplicating filenames within scripts which makes them prone to
mistakes (some instances of which I've actually encountered and had
to fix)
Gbp-Dch: Short
Combine the check+create done in each script. (The original functions
are still callable as before, but a new combined `Aquire_lockfile`
function can be called instead, as now used).
Note, a further simplification could be done in removing the passing of
the lock filename in as a parameter since every use of the functions is
with ".lock". The lock functions already have a fallback to ".build/lock"
though. Checking the history, the fallback used to be for a system wide
lock, which was then replaced with this config-tree specific one. As long
as that is not used implicitly by 3rd-party hooks then surely we are free
to change the fallback to ".lock" and further remove passing in a name as
a param...?
history:
db5d2b0dcd0aa8289a37
Gbp-Dch: Short
Closes: #952918
Current versions of the project files are built upon versions published
and licensed by Daniel Baumann, but are modified copies of those files and
thus need to be marked as such per licensing requirements (afaik he did
not pass along ownership / licensing rights to anyone when he left the
project). We should also be careful to not be misrepresenting such
modified copies as being attributed to Daniel.
Adding a new copyright line referring to "The Debian Live team" should
suffice for this.
The authorship block in man pages has also similarly been updated.
Notes:
- tweaked a copy of daniel copyright lines stating 2014 instead of 2015.
both of these cases were in files that i had personally introduced in
some of my past merged commits that moved some code around. i don't know
why they stated 2014.
- binary_onie was introduced in 2018, so that has a 2018 date instead of
2016 unlike the rest.
- 'efi-image' is a 3rd-party (Canonical Ltd) work that we bundle, but it
has been modified by 674794a8f4 and
36a3ba7634 so I similarly added a
debian live copyright line.
- 'grub-cpmodules' is similar. it was only changed by the indentation fix
of 36a3ba7634 but modification is
modification, and this does help cover any possible future changes that
might be made.
the existing logic for obtaining a list of firmware packages always
downloaded a fresh copy of the archive content file, deleting the file
already in the cache. here we move to actually making use of the cache.
this helps when building multiple times, at least for the same distro. the
package list obtained is rarely going to change after all. it could of
course differ between distros, but the cache is per-distro, as it has
always been.
we of course here switch to caching each of the archive-area files
individually rather than having one file that gets overwritten (or
appended to in the case of when we kept the decompressed file).
Gbp-Dch: Short
Closes: #952911
addressing an old fixme, should distro and parent-distro be identical (in
a derivative build) we want to avoid wasting effort downloading and
processing the same content files twice.
since parent and non-parent have separate archive-area lists though we
should perhaps not just assume that they are identical or ignore any
differences; thus here in such a situation we ensure that we handle any
archive areas not already done for parent-distro handling in such a case,
while skipping those already done.
i notice that the fixme actually also refers to avoiding actual
overlapping of the cached files, however the cached files are (currently)
always ignored anyway, so this is of no concern. reusing the cached files
is an entirely separate issue.
Gbp-Dch: Short
Closes: #952909
the 'manually add firmware-linux package' bit was stuck inbetween the
parent and non-parent logic, which was especially unhelpful before we
de-duplicated the logic into a common function.
Gbp-Dch: Short
Edit: There were four copies of the same logic to keep in sync;
Originally this patch deduplicated each file, but leaving a copy of
the new function in each, thus reducing the duplication but not
eliminating it. A later patch moved it into a shared function file
following further enhancements to the code in question. This has
since been revised to have the function moved to a shared file here,
which simplifies and gives a cleaner diff.
Gbp-Dch: Short
Closes: #952908
the archive content file downloaded to obtain a list of firmware packages
is always deleted and downloaded afresh currently. it may not be ideal that
we do not make use of the cache here, however while that remains
unaddressed, we might as well delete the file after we've used it in order
to not pointlessly waste disk space.
note that this file is ~613 MB for sid-amd64 currently.
Gbp-Dch: Short
Closes: #952907
where multiple archive areas are used, the code here on each loop is:
1) fetching the archive area contents file (compressed)
2) **appending** the output to that of the previous loop
3) searching the file for firmware references, adding to the list
since it appends rather than replaces, entries found in each loop get
re-added on each subsequent loop, resulting in duplication in the
resulting list
below I evaluate the possible solutions to explain why I chose the one
I chose, however the reader should not waste too much time worrying about
whether one of the other solution would have actually been better because
things are changed significantly in further commits shortly!
possible solutions:
a) switching to output (>) rather than append (>>), but this might fail
against an existing file
b) removing the file on each loop, but this will complicate any future
caching improvements that might be made here (currently the files are
always deleted and thus downloaded fresh)
c) allow the appending, evaluating the complete file after the loop
solution C warrants consideration of disk space consumption; currently the
compressed 'main' archive (for sid on amd64) expands to 592.3 MB (feb-2020),
'contrib' is 3.1 MB, and 'non-free' is 18.5 MB.
solution C was chosen here; the difference of accumulated file size vs.
max-single was minor enough to not be of particular concern (~613 vs.
~592 MB).
Gbp-Dch: Short
Closes: #952906
This makes it possible to build an image against a first distribution
(--distribution-chroot) and have the resulting image point to another
distribution (--distribution-binary). We can use this to build against a
snapshot and have the result use the original distribution that was
snapshotted.
Closes: #888507
Roland Clobus
Lyndon Brown
Raphaël Hertzog
Daniel Baumann
Daniel Baumann