all scripts use `set -e` which means that if getop fails, the subsequent
error check that would print an error in addition to any printed by getopt
itself would never actually be reached.
the first though here would be to remove the pointless error check, but
getopt does not include the word "error" with an unrecognised option
failure, nor does it use colour to highlight problems, both of which mean
that it is a little lacking in terms of highlighting problems to users.
thus we properly capture and use the exit code here and output an
appropriate message per invalid argument vs getopt internal error.
also, removed the redundant stderr redirection which is already done
by Echo_error().
Gbp-Dch: Short
white is not going to work well on a terminal with a white background,
obviously. We should keep the standard colour and just try applying the
bold.
or do we want to consider a non black/white colour? like blue...?
Gbp-Dch: Short
instead of conditionally writing deb-src lines, it is **much** neater if
we use sed to optionally removed them at the end.
Gbp-Dch: Short
Closes: #952928
- prefer using `which` over hard coded paths
- it is redundant to check that the bin pointed to the return of
`which` exists and is executable, `which` already gives us
assurance of that if it returns true!
- the redirection of output (`2>/dev/null`) seems to be
unnecessary from my testing.
the instances relatnig to fdisk and losetup in functions/defaults.sh have
been left as they are since they get executed by `lb config` which can run
without sudo elevation unlike `lb build` and in that case `which` would
fail to find these binaries resulting in error.
this also fixes a bug showing an error for missing debootstrap - this tool
requires sudo privileges to run and thus is not found via a none elevated
which search.
Gbp-Dch: Short
Closes: #952927
commit d74f2102a0 added a validation check
to chroot_archives for its 'pass' parameter. this was based upon finding
an instance where the wrong value was being submitted to the script and
wanting to ensure such mistakes would be caught.
unfortunately it seems that I made a mistake in misremembering the facts
surrounding the latter issue when constructing the validation check and
failed to double check with testing until it was already merged. a
correction is needed. the set of valid values is not limited to only
'source|binary' but actually 'source|binary|chroot'. I'd misremembered
'chroot' as being a completely invalid value.
the existing logic for obtaining a list of firmware packages always
downloaded a fresh copy of the archive content file, deleting the file
already in the cache. here we move to actually making use of the cache.
this helps when building multiple times, at least for the same distro. the
package list obtained is rarely going to change after all. it could of
course differ between distros, but the cache is per-distro, as it has
always been.
we of course here switch to caching each of the archive-area files
individually rather than having one file that gets overwritten (or
appended to in the case of when we kept the decompressed file).
Gbp-Dch: Short
Closes: #952911
the existing logic was to decompress the contents file from the downloaded
archive to disk, then process it to obtain a package list. the largest
one by far is for 'main'; 'non-free' and 'contrib' are tiny in comparison.
for sid-amd64 currently, the archive file is 37 MB, while the decompressed
file it contains is 592.3 MB.
we always delete the files and download afresh (currently), and a previous
commit optimised by deleting the files once we're done with them to avoid
wasting disk space leaving them behind.
here we switch to storing the downloaded compressed file to disk instead,
reducing disk space usage (and IO) by hundreds of megabytes; piping the
decompression directly into awk instead of having awk read from the stored
file.
this moves the appending of new items into the list back within the archive
area loop, which is fine since we're replacing the file for each loop now
so the previous issue relating to appending is of no concern.
Gbp-Dch: Short
Closes: #952910
addressing an old fixme, should distro and parent-distro be identical (in
a derivative build) we want to avoid wasting effort downloading and
processing the same content files twice.
since parent and non-parent have separate archive-area lists though we
should perhaps not just assume that they are identical or ignore any
differences; thus here in such a situation we ensure that we handle any
archive areas not already done for parent-distro handling in such a case,
while skipping those already done.
i notice that the fixme actually also refers to avoiding actual
overlapping of the cached files, however the cached files are (currently)
always ignored anyway, so this is of no concern. reusing the cached files
is an entirely separate issue.
Gbp-Dch: Short
Closes: #952909
the 'manually add firmware-linux package' bit was stuck inbetween the
parent and non-parent logic, which was especially unhelpful before we
de-duplicated the logic into a common function.
Gbp-Dch: Short
Edit: There were four copies of the same logic to keep in sync;
Originally this patch deduplicated each file, but leaving a copy of
the new function in each, thus reducing the duplication but not
eliminating it. A later patch moved it into a shared function file
following further enhancements to the code in question. This has
since been revised to have the function moved to a shared file here,
which simplifies and gives a cleaner diff.
Gbp-Dch: Short
Closes: #952908
the archive content file downloaded to obtain a list of firmware packages
is always deleted and downloaded afresh currently. it may not be ideal that
we do not make use of the cache here, however while that remains
unaddressed, we might as well delete the file after we've used it in order
to not pointlessly waste disk space.
note that this file is ~613 MB for sid-amd64 currently.
Gbp-Dch: Short
Closes: #952907
where multiple archive areas are used, the code here on each loop is:
1) fetching the archive area contents file (compressed)
2) **appending** the output to that of the previous loop
3) searching the file for firmware references, adding to the list
since it appends rather than replaces, entries found in each loop get
re-added on each subsequent loop, resulting in duplication in the
resulting list
below I evaluate the possible solutions to explain why I chose the one
I chose, however the reader should not waste too much time worrying about
whether one of the other solution would have actually been better because
things are changed significantly in further commits shortly!
possible solutions:
a) switching to output (>) rather than append (>>), but this might fail
against an existing file
b) removing the file on each loop, but this will complicate any future
caching improvements that might be made here (currently the files are
always deleted and thus downloaded fresh)
c) allow the appending, evaluating the complete file after the loop
solution C warrants consideration of disk space consumption; currently the
compressed 'main' archive (for sid on amd64) expands to 592.3 MB (feb-2020),
'contrib' is 3.1 MB, and 'non-free' is 18.5 MB.
solution C was chosen here; the difference of accumulated file size vs.
max-single was minor enough to not be of particular concern (~613 vs.
~592 MB).
Gbp-Dch: Short
Closes: #952906
These functions are specific to handling packages stored in the
cache, not other files. They are also always used with the same
`cache/packages.` prefix to the path.
Gbp-Dch: Short
Closes: #952916
instead of trying all from derived mirror then falling back to parent upon
failure, which as pointed out by a message printed out can result in a
load of spurious 404 errors; actually get each udeb from the mirror it is
supposed to be retrieved from.
Partial fix for #952914, this is the last commit for it so closes it
Gbp-Dch: Short
Closes: #952914
the existing logic just bundled the entire parent and derived udeb lists
together, ignoring the fact that there might thus be two instances of some
packages, and relying upon getting derived ones first and checking file
existence to avoid handling the overridden parent instances.
here we now actually filter the list of parent udebs to exclude packages
that are to be obtained from the derivative.
this enables avoiding the file existence checking
Partial fix for #952914
Gbp-Dch: Short
move the code that checks whether a version of a given package has already
been obtained (to account for parent and derived both listing the same
package) to guard the copy from cache action also, not just the download
action.
in rare but possible scenarios it would have been possible to end up with
both the parent and derived copies of a package included.
Partial fix for #952914
Gbp-Dch: Short
this script outputs a series of progressive dots as progress is made,
finally terminated with ' done.'. however if an error occurs then the
error would end up being printed directly on the end of this instead of
on a new line since newlines are not printed after each dot. this fixes
this oversight.
Gbp-Dch: Short
Closes: #952883
the '.' progress stuff has been left as is. perhaps that should be removed
since its use is inconsistent compared to other scripts?
Gbp-Dch: Short
Closes: #952882
only a couple were in use and only by unused echo helpers which have now
themselves been removed, so nothing in this file is needed.
Partial fix for #952880
Gbp-Dch: Short
Closes: #952880
all of these echo helpers are essentially 'logging' functions with output
always intended for stdout/stderr. lack of explicit stdout/stderr direction
means that their output could be captured unintentionally should they be
used within a function designed to construct a string.
Gbp-Dch: Short
Closes: #952879
Lyndon Brown
Luca Boccassi