Commit Graph

1156 Commits

Author SHA1 Message Date
Lyndon Brown c2eb40d9b1 grub2|loopback: add hotkey for memtest (m) 2020-04-23 09:12:52 +02:00
Lyndon Brown 89d72410c4 grub2|loopback: add hotkey for single kernel live entry (l) 2020-04-23 09:12:52 +02:00
Lyndon Brown 6e86a6480c grub2|loopback: allow providing hotkey to menu entry creation helpers 2020-04-23 09:12:52 +02:00
Lyndon Brown 40a076bf35 grub2|loopback: use '@' bookended placeholders
...for consistency with syslinux config placeholders and improved
clarity of what text is a placeholder.

the old placeholders without the bookends are still replaced for
user configs for backwards compatibility.

the new ones are little used just at the moment but are expected to
become used much more in later commits.

Gbp-Dch: Short
2020-04-23 09:12:24 +02:00
Lyndon Brown d5d1e35f22 grub2|loopback: remove unnecessary perl based newline escaping for sed
none of the stuff changed here is multi-line so had no reason to use it.

Gbp-Dch: Short
2020-04-22 18:36:47 +02:00
Lyndon Brown bbeed4cb60 build: fix misleading message 2020-04-20 10:16:14 +00:00
Lyndon Brown f0588be19a loadlin: fix missing directory error
fixes an error I experienced in a test build

Gbp-Dch: Short
2020-04-01 19:04:14 +00:00
Lyndon Brown 1716958a8d bootstrap_cache: validate action param 2020-04-01 18:03:20 +00:00
Lyndon Brown 39dae8cdc7 move auto colouring decision
...from the `Set_config_defaults` function, to being done directly
in `build.sh` (the component which is also responsible for loading
functions, loaded at the start of every script, including the front
end).

thus the colouring decision will now correctly...
 - apply to the frontend, such as to the 'root privileges needed'
   error, the 'no such script' error, and the command name
   colouring that I want to add (the most significant issue).
 - apply to error messages generated by the `Arguments` and
   `Read_conffiles` functions, which are called before
   `Set_config_defaults` by scripts.

as things were, due to the comparison with "false", colour would
_always_ be used in these places (unless _COLOR_ERR=false or
_COLOR_OUT=false wrt. the new command highlight, were set in the
environment when executing a script throught the frontend).

this would not be a problem for normal terminal use of course,
besides being inconsistent where color were turned off, but would
be a bit of a problem if redirected to a file.

a re-evaluation of _COLOR is performed in `Set_config_defaults` to
adjust _COLOR_OUT and _COLOR_ERR where necessary, to correctly
respond to _COLOR being set in saved config files (disabled by
default but a user could always enable), after the point of config
files being loaded.

_COLOR can still be controlled from the environment just as before,
overriding both _COLOR_OUT and _COLOR_ERR.

note that this does not address the fact that --color|--no-color
do not work in the frontend and thus will not impact the colouring
of to-be-introduced command highlighting. this needs to be
addressed separately.

Gbp-Dch: Short
2020-03-23 08:06:51 +00:00
Lyndon Brown 85e0681ee8 args: fix a typo
Gbp-Dch: Ignore
2020-03-23 08:06:51 +00:00
jnqnfe a25b77e099 bootloaders: remove old "video=vesa:ywrap,mtrr" kernel param, as done in d-i
d-i removed this in commit 0917b2dde3ff73a204d27dd2f2fffc8a41175ddd

Note: There was inconsistency between grub and syslinux in use of this, with
syslinux not having it on graphical rescue and auto modes while grub entries
did. The patch to fix that has been dropped since we're removing it everywhere
anyway.

(#395040)

Gbp-Dch: Short
2020-03-22 19:43:56 +00:00
Lyndon Brown 757d1e6b48 binary_iso: remove duplicate xorriso param
-J is already in the generic set defined at the start

Gbp-Dch: Short
2020-03-20 16:17:16 +00:00
Lyndon Brown 92425bd99c archives: param is required
Gbp-Dch: Ignore
2020-03-20 11:43:01 +00:00
Lyndon Brown 7c4de2f20d archives: clarify var
Gbp-Dch: Ignore
2020-03-20 11:43:01 +00:00
Lyndon Brown 94a3e184c2 archives: explicitly pass along _PASS to Create_apt_sources_list
Gbp-Dch: Ignore
2020-03-20 11:43:01 +00:00
Lyndon Brown 945a166f75 strip progress-linux distro hacks
progress-linux, as discussed in MR #142 ([1]) is a little known distro,
which appears to be little more than a personal project of the original
author of live-build.

given that, the expense of maintaining all of these old hacks for it
cannot be justified. it is not known whether or not live-build is even
used with respect to it since the author abandoned live-build some
years ago.

also, at least one past change in live-build possibly broke progress-linux
compatibility anyway, which would have required progress-linux users of
live-build to use a custom progress-linux config, or a progress-linux
fork of live-build, and there is no knowing how much of the hacks in this
"upstream" codebase any user of progress-linux currently relies upon.
and again, progress-linux appears to just be a personal project of
Daniel's, with afaik very little userbase. (Daniel seems to be the only
developer working on the project which speaks to how small it is).

[1]: https://salsa.debian.org/live-team/live-build/-/merge_requests/142

Gbp-Dch: Short
2020-03-20 10:19:33 +00:00
jnqnfe 4fa83598a3 grub: remove old and unused splash code
The LB_GRUB_SPLASH variable is populated by the --grub-splash param
but is not actually used for anything.

Gbp-Dch: Short
2020-03-20 10:02:52 +00:00
Lyndon Brown 01a6de2f4c config: fix backwards compatibility break
when the --fdisk and --losetup options were removed, the entries in the
getopt option list should have remained for backwards compatibility such
that the usage warnings can kick in instead of unknown option errors.

Gbp-Dch: Ignore
2020-03-20 09:28:58 +00:00
Lyndon Brown 561f2dcc3a config: fix incomplete rename of --architectures
missed the getopt data update in 8b109ffb96

Gbp-Dch: Ignore
2020-03-20 09:28:58 +00:00
Lyndon Brown fb0790cc43 stagefiles: s/Require_stagefile/Require_stagefiles/
this function takes one or more required stage fileS _plural_, and exits
if any are missing (or at least it does now after the refactor).

let's rename it to make things more clear

Gbp-Dch: Short
2020-03-17 22:59:37 +00:00
Lyndon Brown dadeec9d39 stagefiles: fix doc mistake
missed in final revision of fe9195b59c

Gbp-Dch: Ignore
2020-03-17 22:09:51 +00:00
Lyndon Brown fe9195b59c stagefiles: further robustify with auto filenames
as suggested by Raphaël

rather than have fixed stagefile filename strings at all in the scripts,
use `$(basename $0)` to use the name of the script (which is the same for
almost all cases anyway, and the stage files are supposed to be almost
exclusively unique per-script). we can thus simplify things by determining
the filename for most use cases within the functions themselves.

this does change the file used by a couple of scripts, affecting backwards
compatibility of executing live-build upon an existing partially or fully
completed build:
 - binary_grub-pc used "binary_grub"
 - chroot_includes used "includes.chroot"

care had to be taken for the following cases:
 - there are some cases like bootstrap_cache, source_debian and
   bootstrap_debootstrap which are dealing with more than one file, and/or
   otherwise a filename that is not specific to the script itself exactly,
   or should not be based upon its name.
 - some cases like chroot_cache, bootstrap_cache and
   chroot_install-packages need to append something to the end of the name
   depending upon which pass/action mode the script is being executed with.
 - furthermore in the bootstrap_cache case one of the filenames is used
   within the bootstrap_debootstrap and thus needs very careful handling
   to be certain that a change in filename of bootstrap_cache does not
   break bootstrap_debootstrap.

Gbp-Dch: Short
2020-03-17 18:57:02 +00:00
Lyndon Brown 04d9ee0211 stagefiles: simplify & robustify
- avoid all need to pass ".build/" path in stage file names into the
   functions
 - add a helper to remove a stage file (required to complete the above
   properly)
 - avoid duplicating filenames within scripts which makes them prone to
   mistakes (some instances of which I've actually encountered and had
   to fix)

Gbp-Dch: Short
2020-03-17 18:57:02 +00:00
jnqnfe a773edb813 syslinux: apply kernel version filtering logic to multi-flavour kernel scenarios 2020-03-16 23:08:26 +00:00
jnqnfe 38af959aa5 syslinux: use more dynamic memtest menu config file
Fixes the following
 - Correct version (memtest86/memtest86+) shown instead of fixed 'memtest86+' text
 - Ensure correct directory path always used by using replaceable placeholder

Gbp-Dch: Short
2020-03-16 23:08:26 +00:00
jnqnfe 31fa6abd36 syslinux: add memtest menu entry only if including memtest 2020-03-16 23:08:26 +00:00
jnqnfe 7ffd2288d9 syslinux: add install menu entries only if including installer 2020-03-16 23:08:26 +00:00
Lyndon Brown 7e41b1267c fix another wrong stage file filename 2020-03-16 22:40:23 +00:00
Lyndon Brown c55eb8a0c3 use local scope for private function vars
all vars affected have been carefully checked to be quite certain
that they are definitely local

where variable is assigned the return value of a function/command, the
local "declaration" is deliberately done on a separate line, since
`local FOO` is actually treated itself as a command rather than a
declaration; will thus always cause $? to be zero, and thus if done on
the same line as such an assignment can not only clobber $? but in doing
so unintentionally blocks failure of a command from triggering the
expected exit from having `set -e`.

also, from testing, i have found that when assigning "${@}" this must be
done on a separate line confusingly as otherwise an error occurs.

Gbp-Dch: Short
2020-03-16 22:10:03 +00:00
Lyndon Brown af040d7803 add build & build stage success messages 2020-03-16 21:42:49 +00:00
jnqnfe 24aab15d2a bootloaders: add install with speech synthesis menu entries
Following the official Debian install disks

Gbp-Dch: Short
2020-03-16 20:33:18 +00:00
jnqnfe efbe50233e grub1: refactor install menu creation
This is necessary as the simplest solution to allow a single
speech-synth entry to be added.

Gbp-Dch: Short
2020-03-16 20:33:18 +00:00
jnqnfe 2fc60a1151 syslinux: remove replacement of unused label in menu configs 2020-03-16 16:29:00 +00:00
jnqnfe 8a7bc408e5 grub1: fix consistency
Use the normal appended-params directly in `Grub_live_entry` calls
like the failsafe ones.

Gbp-Dch: Short
2020-03-16 16:29:00 +00:00
jnqnfe aadf8c4cac grub1/grub2/loopback: fix missing delimiter in menu kernel parameter lists
The '---' delimiter should appear before the final 'quiet' parameter
(which is used by the debian installer I believe).

This delimiter is added by live-build in syslinux configs, and is present
in both grub2 and syslinux configs in an official debian 7.7 disc image,
suggesting strongly that live-build grub/grub2 menu creation code is in
the wrong here by missing it.

update: this commit previously used -- as was correct at the time, and has
since been updated to use --- per #775128; which was previously tackled in
a separate later commit. the switch to --- was already done for syslinux
(which was not missing the delimiter unlike grub) in
ba6b9adeff

Gbp-Dch: Short
Closes: #775143
2020-03-16 16:29:00 +00:00
jnqnfe a33938f8d4 grub1/grub2/loopback: fix order of quiet kernel parameters in menus
I believe that the `quiet` parameter is meant for d-i not the kernel and
thus should be given on the end after a delimiter, as done with syslinux.

Here we switch the order to move it to the end. The addition of the missing
delimiter will be done in a followup commit.

(See #775143)

Gbp-Dch: Short
2020-03-16 16:29:00 +00:00
jnqnfe 636e52c36e grub2/loopback: fix misapplication of quiet kernel parameter
When building grub2 menu entries the quiet param (meant for d-i)
was excluded from the rescue menu entries instead of expert.

This is the opposite to what is done in the following:
 - Menu entries seen in official debian 7.7 disc images (grub2 and syslinux configs)
 - Menu entries created for grub (legacy)
 - Menu entries created for syslinux

The evidence strongly suggests that the grub2 menu creation was in the wrong!

(See #775143)

Gbp-Dch: Short
2020-03-16 16:29:00 +00:00
jnqnfe 464c7a6946 grub1: fix wrong variable name 2020-03-16 16:29:00 +00:00
Lyndon Brown 49e68da5ee allow turning off colour
necessary to give control when colour is default enabled

Gbp-Dch: Short
2020-03-16 13:32:16 +00:00
Lyndon Brown 2cf397df66 installer: fix double slash in urls 2020-03-16 01:36:53 +00:00
Lyndon Brown b326c37f92 installer: expand debugging output 2020-03-16 01:36:53 +00:00
Lyndon Brown 2ea6eeefa2 installer: move an exclude removal outside of loop 2020-03-16 01:36:53 +00:00
Lyndon Brown 19e1a5b9c2 installer: tweak temp shell script construction
Gbp-Dch: Ignore
2020-03-16 01:36:53 +00:00
Lyndon Brown 47e136df6f installer: dedup long paths 2020-03-16 01:36:53 +00:00
Lyndon Brown 3fb0bb235d installer: tidy package lists
`DI_PACKAGES` does not need to include `DI_REQ_PACKAGES` so long as
we pass the latter to apt in the one case where it was not already
being given it.

in fact with it including that sub-list meant that in the other
case where it was being given to apt, it actually just resulted in
duplication.

Gbp-Dch: Short
2020-03-16 01:36:53 +00:00
Lyndon Brown 9a878350d3 installer: misc small tweaks
Gbp-Dch: Ignore
2020-03-16 01:36:53 +00:00
Lyndon Brown 2abf7bc65b installer: clear large string once done with
Gbp-Dch: Ignore
2020-03-16 01:36:53 +00:00
Lyndon Brown 1a9a1585ad installer: restore udeb existence check
this was removed as part of b54bdd7fb1
but is actually still needed.

Gbp-Dch: Ignore
2020-03-16 01:36:53 +00:00
Lyndon Brown 7891188557 installer: fix broken loop
Gbp-Dch: Ignore
2020-03-16 01:36:53 +00:00
Lyndon Brown 00a2fabcdf fix wrong description
Gbp-Dch: Ignore
2020-03-15 21:54:37 +00:00
Lyndon Brown a67febdd82 fix missing stage descriptions
Gbp-Dch: Ignore
2020-03-15 21:54:37 +00:00
Lyndon Brown b25db0a4bf source: properly fix _apt permissions warning
the chown command needed running within the chroot, since apt-get is being
run within the chroot and _apt might have a different UID there than on
the host.

Gbp-Dch: Short
Closes: #953957
2020-03-15 21:38:35 +00:00
Steven Shiau 0a353a7b46 Add grub EFI support for armhf arch. 2020-03-15 11:35:19 +00:00
Lyndon Brown 34ceb67718 hide mountpoints from nautilus
when building within a home directory, as /proc, /sys, etc are mounted
and unmounted into the chroot at various points in the build, corresponding
entries appear and disappear within the side panel of nautilus.

this is obviously undesirable.

use of `-o x-gvfs-hide` resolves this for the most part. i still see items
occasionally pop up and having spent some time experimenting, i'm putting
it down to buggy behaviour on the part of nautilus. (aside from those
appearing when debootstrap is running - debootstrap also needs this fix).

Gbp-Dch: Short
2020-03-13 20:28:15 +00:00
Lyndon Brown a46985b08a fix comment typo
Gbp-Dch: Ignore
2020-03-13 16:19:59 +00:00
jnqnfe b27927724a locks: tidy lock acquisition
Combine the check+create done in each script. (The original functions
are still callable as before, but a new combined `Aquire_lockfile`
function can be called instead, as now used).

Note, a further simplification could be done in removing the passing of
the lock filename in as a parameter since every use of the functions is
with ".lock". The lock functions already have a fallback to ".build/lock"
though. Checking the history, the fallback used to be for a system wide
lock, which was then replaced with this config-tree specific one. As long
as that is not used implicitly by 3rd-party hooks then surely we are free
to change the fallback to ".lock" and further remove passing in a name as
a param...?

history:
db5d2b0dcd
0aa8289a37

Gbp-Dch: Short
Closes: #952918
2020-03-13 15:37:38 +00:00
Lyndon Brown 5e423d0851 fix instances of bad stagefile creation circumstances
theres no point in creation of stagefiles being kept within a
conditional block of work. if the script completes with success
then it should create its stagefile to thus avoid repeating any
work that it might have done should it get re-run without being
forced.

Gbp-Dch: Short
2020-03-13 14:42:34 +00:00
Lyndon Brown 3e3e8c2c64 fix instances of broken stagefile handling
mismatches in filenames between check and create

Gbp-Dch: Short
2020-03-13 14:42:34 +00:00
Lyndon Brown 9f42bc1bdc fix splitting bootloader string
the existing solution simply did not work
2020-03-13 14:17:45 +00:00
jnqnfe a6cc72bf6d strip useless multi-arch separate directory stuff
(part of never completed side-by-side multi archi support)

Several scripts make a call to a function called Check_multiarchitectures,
the purpose of which is to adjust the target directory that certain 'live'
and 'install' files are located in. The idea is that a script sets up
'DESTDIR', 'DESTDIR_LIVE' and 'DESTDIR_INSTALL' as appropriate and then
the script appends a suitable arch dependant postfix to the directory
name, depending upon the arch currently being targetted. This would allow
the script to be run multiple times, each for a different architecture.

This is a part of an implementation of allowing multiple architectures to
sit side by side within the same live image, selectable from the
bootloader menus. (As opposed to multiple architectures mixed within the
same userland).

This is evidently the case both from the fact that:
  1) The arch specific postfix chosen in that function depends on a var
     called LB_CURRENT_ARCHITECTURE, which is never set. In fact going
     back through the git history to the introduction of the function in
     0d5ff4ca75, the var (even considering
     var name changes) has never been set by anything. So effectively the
     call to the function has been entirely redundant all this time.
  2) The major build stages do not perform multiple executions of substages
     per arch. Thus from this perspective it seems that the support was
     never fully implemented.
  3) If any doubt remained, there is an old branch called 'tmp-multiarch'
     which has a couple of commits making progress with completing support,
     such as implementing the above missing pieces.

The above mentioned branch is 10 years old and can be considered abandoned.

It is not clear whether the original author ever intended to complete and
merge this; nor is it at all clear at what stage of completion it was at.

At any rate, imo it is not at all particularly useful to have extra code
and complexity in order to be able to cram multiple environments side by
side in one image, not when CDs/DVDs and even to some extent USB pen drives
are so cheap. And who really needs more than one environment so
desperately on just one such medium.

If this was not enough to justify removal, then there is also the fact that
the support that was implemented has become completely broken over the
years with scripts diverging in terms of the variable names the function
modifies such that they are incompatible with it.

A quick assessment of the state of this latter aspect:

  good:
   - grub-legacy uses the correct var names so is fine
   - memtest similarly good
   - installer_debian-installer looks okay

  questionable:
   - binary_linux-image uses the correct vars but might not select the
     right kernel and initrd files to copy (seems to copy all)

  bad:
   - grub-pc is making a redundant call, after functionality was moved
     to the loopback script
   - loopback is using the wrong vars (INITFS instead of DESTDIR +
     DESTDIR_INSTALL + DESTDIR_LIVE), plus is doing its own amd64+i686
     thing anyway, so the function call would achieve nothing anyway.
   - syslinux is also using the wrong var names so would not work with
     it and is not even making the necessary function call. Also the
     install paths are fixed in the hard coded cfg files anyway so this
     would need addressing with placeholders and sed replacement, but
     then it is not entirely clear how things should work with respect
     to install entries and multi-arch anyway, are we having multiple
     copies of the installer, one for each target arch and then multiple
     copies of the install menus, perhaps under different submenus?

So, this removes the artefacts of this never completed feature.

Gbp-Dch: Short
2020-03-13 13:56:21 +00:00
Lyndon Brown 1edc3af346 debootstrap: use --force-check-gpg
I asked for such a switch to be added in debootstrap back at the start of
2015 in #775454 as part of a review I undertook of its security. A slightly
modified patch was merged a few months later and made it into version
1.0.69.

A patch was never merged into live-build to make use of it however. Let's
do that now.

The benefit of this, as explained in #775454, is that if we want strong
security (LB_APT_SECURE=true) then should debootstrap not be able to find
the GPG key to verify things with, it will abort with an error instead of
falling back to just https downloads with a warning. Such a warning would
be easy to miss in the log output, and security could potentially be
compromised if this were to happen.

Gbp-Dch: Short
2020-03-12 14:44:49 +00:00
Luca Boccassi 015e6b65f3 Revert "Test for executables: replace 'which' with more robust 'command -v'"
This reverts commit 2d9ab1f7f8.

Causes test failure due to bashism.
2020-03-12 12:32:26 +00:00
Lyndon Brown c2e73627db grub: fix incorrect error handling 2020-03-12 12:08:33 +00:00
johnraff 2d9ab1f7f8 Test for executables: replace 'which' with more robust 'command -v'
Instances of:
if [ $(which <command> ]
have been replaced with:
if command -v <command> >/dev/null
which is considered to be more robust in a range of environments.

scripts/build/chroot_archives: line 259:
	if [ "${LB_APT}" = "aptitude" ] && [ ! $(Chroot chroot "which aptitude") ]
has been left untouched because the chroot might require a more complex command
which would need more testing.

manpages/Makefile: line 42:
	@if [ ! -x "$$(which po4a 2>/dev/null)" ]; \
has been left untouched because I am not sufficiently familiar with makefiles.
2020-03-12 10:35:57 +00:00
Lyndon Brown 1e0339a4e3 remove obsolete loop-aes-utils related losetup hack
677415f6d7 (2007) in v1.0~a2-1 added a hack
relating to the loop-aes-utils package and losetup. this commit bundled
a bunch of changes, it was not specific to the hack, and so info about the
hack is limited to a brief comment included within the related change in
defaults:
```
# Workaround for loop-aes-utils divertion
# (loop-aes-utils' losetup lacks features).
```
though it is very similar to the removed fdisk hack in that it seems that
one package may replace a binary from another, moving the original to a
new location, and this hack gives the user the opportunity to select the
original instead of the one put in its place, for use in LB.

the comment mentions a package called loop-aes-utils as being the package
that performs such a diversion, and that the need for the hack was that
losetup itself lacked features, presumably encryption support, and it is
clear that it is the losetup binary that is the focus of the diversion.

looking into the history of loop-aes-utils a little, this package was
dropped from debian back in 2012 (#680748), favouring encrytion support of
dm-crypt/cryptsetup.

double checking file contents of packages, only the mount package carries
an /sbin/losetup file, so presumably this means that dm-setup/cryptsetup
do not perform such a diversion of losetup (i.e. their use is exclusively
done directly).

since the possible diversion is simply gone, that completely removes any
point in having the hack of giving users choice between losetup and the
diverted one. so let's remove this obsolete hack...
2020-03-12 10:31:39 +00:00
Lyndon Brown 9930ce1ea5 grub-efi: fix image type check ordering
this should take place before working on efi related stuff

Gbp-Dch: Short
2020-03-12 09:59:44 +00:00
Lyndon Brown a9d8aac959 grub-efi: fix incorrect error handling 2020-03-12 09:59:44 +00:00
Lyndon Brown d9f353c737 remove obsolete fdisk hack
8321653cb3 (from 2007) introduced a hack to
work around bug #445304 in gnu-fdisk for users who may have replaced fdisk
with the classic gnu version. the hack allowed users to select an alternate
fdisk binary to use to work around the buggy binary.

bug #445304 is marked as found in v1.0-1 and fixed in v1.2-1, though may
have been fixe din v1.1. it was marked fixed in 2009.

checking the package archive, gnu-fdisk does not actually exist anymore
in debian, with one exception - it is available for arm64 on sid via
debports, and that version is 1.3 so thus includes the necessary fix
anyway.

it is thus pointless now that we still carry this hack.

Gbp-Dch: Short
2020-03-11 19:06:54 +00:00
Lyndon Brown 7a4a9f94b8 amend copyright & licensing blocks
Current versions of the project files are built upon versions published
and licensed by Daniel Baumann, but are modified copies of those files and
thus need to be marked as such per licensing requirements (afaik he did
not pass along ownership / licensing rights to anyone when he left the
project). We should also be careful to not be misrepresenting such
modified copies as being attributed to Daniel.

Adding a new copyright line referring to "The Debian Live team" should
suffice for this.

The authorship block in man pages has also similarly been updated.

Notes:
 - tweaked a copy of daniel copyright lines stating 2014 instead of 2015.
   both of these cases were in files that i had personally introduced in
   some of my past merged commits that moved some code around. i don't know
   why they stated 2014.
 - binary_onie was introduced in 2018, so that has a 2018 date instead of
   2016 unlike the rest.
 - 'efi-image' is a 3rd-party (Canonical Ltd) work that we bundle, but it
   has been modified by 674794a8f4 and
   36a3ba7634 so I similarly added a
   debian live copyright line.
 - 'grub-cpmodules' is similar. it was only changed by the indentation fix
   of 36a3ba7634 but modification is
   modification, and this does help cover any possible future changes that
   might be made.
2020-03-11 13:51:19 +00:00
jnqnfe 87e9543c98 source: target downloads with corresponding version
Prefer downloading the version of the source package that actually
corresponds to the version of the binary. Should apt-update be
run and a package updated, we do not really want to fetch a newer
copy of the source than that of the binary, we want the exact
corresponding version (kinda the whole point of compiling a source
disc that they correspond). If the exact version is no longer
available then it is surely preferable to list it in the missing
list than end up with a newer version.

Gbp-Dch: Short
Closes: #952932
2020-03-11 11:45:27 +00:00
Lyndon Brown 744141c60f source: fix apt warnings
the source stage emitted the following output for each packages downloaded.
note the troubling warning at the end.

```
Reading package lists... Done
NOTICE: 'grep' packaging is maintained in the 'Git' version control system at:
https://salsa.debian.org/debian/grep.git
Please use:
git clone https://salsa.debian.org/debian/grep.git
to retrieve the latest (possibly unreleased) updates to the package.
Need to get 1579 kB of source archives.
Get:1 http://deb.debian.org/debian buster/main grep 3.3-1 (dsc) [2038 B]
Get:2 http://deb.debian.org/debian buster/main grep 3.3-1 (tar) [1473 kB]
Get:3 http://deb.debian.org/debian buster/main grep 3.3-1 (diff) [104 kB]
Fetched 1579 kB in 1s (1293 kB/s)
Download complete and in download only mode
W: Download is performed unsandboxed as root as file 'grep_3.3-1.dsc' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)
```

this occurred because the '_apt' user did not have permission to write to
the destination directory and so was falling back to downloading as root
in order to do its work.

prior to 158950b873 all source packages were
downloaded directly to the root of the chroot. that commit changed this to
save them into a new clean directory within it instead. thus to fix the
problem we can simply set the ownership of this new directory to '_apt'.

Gbp-Dch: Short
2020-03-10 18:11:21 +00:00
Lyndon Brown e3da1b6f8d debootstrap: remove redundant check
the check for existence of debootstrap here was completely redundant since
there is a check at the beginning of the file which already outputs an
appropriate error and exists if missing.
2020-03-10 17:47:41 +00:00
Lyndon Brown d79fd1e760 archives: fix typo
introduced in ee2bbe4578

Gbp-Dch: Ignore
2020-03-10 17:36:10 +00:00
Lyndon Brown 48df750411 config: improve documentation 2020-03-10 14:12:45 +00:00
Lyndon Brown 24f04a924b chroot: clarify stages
the cache restore/save script is not a chroot modification script unlike
the rest of the scripts that it was bunched up with. It is an actual
component part of the chroot build stage.

let's bring clarity to this with improved documentation.

Gbp-Dch: Short
2020-03-10 12:51:55 +00:00
Lyndon Brown 9f2b7b40e8 archives: always include enabled/disabled deb-src apt entries
LB_APT_SOURCE_ARCHIVES determines whether or not deb-src entries are
desired to be included in apt's sources.list. here, instead of excuding
them we always include them but commented out where they would previously
have been excluded. this means that if a user later changes their mind and
wants to make use of them all they have to do is uncomment them rather
than add the necessary lines.

Gbp-Dch: Short
Closes: #952929
2020-03-10 12:50:39 +00:00
Lyndon Brown ee2bbe4578 archives: simplification
as suggested by Raphaël

Gbp-Dch: Short
2020-03-10 12:49:36 +00:00
Lyndon Brown cf2a9b951c arguments: fix unreachable and poor argument error handling
all scripts use `set -e` which means that if getop fails, the subsequent
error check that would print an error in addition to any printed by getopt
itself would never actually be reached.

the first though here would be to remove the pointless error check, but
getopt does not include the word "error" with an unrecognised option
failure, nor does it use colour to highlight problems, both of which mean
that it is a little lacking in terms of highlighting problems to users.

thus we properly capture and use the exit code here and output an
appropriate message per invalid argument vs getopt internal error.

also, removed the redundant stderr redirection which is already done
by Echo_error().

Gbp-Dch: Short
2020-03-10 12:45:23 +00:00
Lyndon Brown d6096622f9 help/usage: remove pointless vars
build scripts never call Help() and so the empty HELP strings are pointless.
(when called with --help they call Man()).

Closes: #952859
Gbp-Dch: Short
2020-03-10 12:45:23 +00:00
jnqnfe 158950b873 source: downloaded pkgs to a clean directory rather than chroot root
Closes: #952931
2020-03-10 12:44:11 +00:00
jnqnfe feec626274 source: minor tweaks
Gbp-Dch: Ignore
2020-03-10 12:44:11 +00:00
jnqnfe 532144ce66 chroot: improve directory creation/destruction efficiency
Closes: #952924
2020-03-10 12:41:15 +00:00
jnqnfe 0dee07f122 config: rename the config set/check functions for clarity
Gbp-Dch: Short
Closes: #952920
2020-03-10 12:39:37 +00:00
Lyndon Brown b4598b234c tidy script init (4/4) - top level cmd "auto redirect" handling
Partial fix for #952919

Gbp-Dch: Short
Closes: #952919
2020-03-10 12:39:37 +00:00
jnqnfe dff08fa3f7 tidy script init (3/4) - top level commands
Partialfix for #952919

Gbp-Dch: Short
2020-03-10 12:39:37 +00:00
jnqnfe 061d7cb340 tidy script init (2/4) - build stage scripts
Partial fix for #952919

Gbp-Dch: Short
2020-03-10 12:39:37 +00:00
jnqnfe b49abcc1a8 tidy script init (1/4) - arg and config processing
Partial fix for #952919

Gbp-Dch: Short
2020-03-10 12:39:37 +00:00
jnqnfe ed2d70bde1 simplify STAGE checks - use In_List
Closes: #952917
2020-03-10 00:02:43 +00:00
Lyndon Brown 8545c048c6 use actual boolean state for some simple vars
Gbp-Dch: Ignore
2020-03-09 11:21:24 +00:00
Lyndon Brown 093a5ea99c installer: use boolean for clarity
Closes: #952930
2020-03-09 10:58:53 +00:00
Lyndon Brown 7ee59d408e fix consistency in binary execution and existance checking
- prefer using `which` over hard coded paths
 - it is redundant to check that the bin pointed to the return of
   `which` exists and is executable, `which` already gives us
   assurance of that if it returns true!
 - the redirection of output (`2>/dev/null`) seems to be
   unnecessary from my testing.

the instances relatnig to fdisk and losetup in functions/defaults.sh have
been left as they are since they get executed by `lb config` which can run
without sudo elevation unlike `lb build` and in that case `which` would
fail to find these binaries resulting in error.

this also fixes a bug showing an error for missing debootstrap - this tool
requires sudo privileges to run and thus is not found via a none elevated
which search.

Gbp-Dch: Short
Closes: #952927
2020-03-09 10:51:11 +00:00
Lyndon Brown 37f0432e98 fix inadequate chroot_archives validation
commit d74f2102a0 added a validation check
to chroot_archives for its 'pass' parameter. this was based upon finding
an instance where the wrong value was being submitted to the script and
wanting to ensure such mistakes would be caught.

unfortunately it seems that I made a mistake in misremembering the facts
surrounding the latter issue when constructing the validation check and
failed to double check with testing until it was already merged. a
correction is needed. the set of valid values is not limited to only
'source|binary' but actually 'source|binary|chroot'. I'd misremembered
'chroot' as being a completely invalid value.
2020-03-09 10:51:11 +00:00
jnqnfe 8008b57fc2 source: add output of config readme file to source image
To provide an explanation

Gbp-Dch: Short
Closes: #952921
2020-03-05 22:35:52 +00:00
Lyndon Brown 4d0c3a1169 firmware: enable caching for archive content file with firmware lists
the existing logic for obtaining a list of firmware packages always
downloaded a fresh copy of the archive content file, deleting the file
already in the cache. here we move to actually making use of the cache.

this helps when building multiple times, at least for the same distro. the
package list obtained is rarely going to change after all. it could of
course differ between distros, but the cache is per-distro, as it has
always been.

we of course here switch to caching each of the archive-area files
individually rather than having one file that gets overwritten (or
appended to in the case of when we kept the decompressed file).

Gbp-Dch: Short
Closes: #952911
2020-03-05 22:13:57 +00:00
Lyndon Brown a120bc5445 firmware: avoid potentially duplicate work
addressing an old fixme, should distro and parent-distro be identical (in
a derivative build) we want to avoid wasting effort downloading and
processing the same content files twice.

since parent and non-parent have separate archive-area lists though we
should perhaps not just assume that they are identical or ignore any
differences; thus here in such a situation we ensure that we handle any
archive areas not already done for parent-distro handling in such a case,
while skipping those already done.

i notice that the fixme actually also refers to avoiding actual
overlapping of the cached files, however the cached files are (currently)
always ignored anyway, so this is of no concern. reusing the cached files
is an entirely separate issue.

Gbp-Dch: Short
Closes: #952909
2020-03-05 22:13:57 +00:00
Lyndon Brown 94d126c043 firmware: reorder firmware list construction
the 'manually add firmware-linux package' bit was stuck inbetween the
parent and non-parent logic, which was especially unhelpful before we
de-duplicated the logic into a common function.

Gbp-Dch: Short
2020-03-05 22:13:57 +00:00
Lyndon Brown aa745de0bb firmware: de-dup firmware list parsing
Edit: There were four copies of the same logic to keep in sync;
Originally this patch deduplicated each file, but leaving a copy of
the new function in each, thus reducing the duplication but not
eliminating it. A later patch moved it into a shared function file
following further enhancements to the code in question. This has
since been revised to have the function moved to a shared file here,
which simplifies and gives a cleaner diff.

Gbp-Dch: Short
Closes: #952908
2020-03-05 22:13:57 +00:00
Lyndon Brown 4933beffce firmware: delete pointlessly cached (large) file after use
the archive content file downloaded to obtain a list of firmware packages
is always deleted and downloaded afresh currently. it may not be ideal that
we do not make use of the cache here, however while that remains
unaddressed, we might as well delete the file after we've used it in order
to not pointlessly waste disk space.

note that this file is ~613 MB for sid-amd64 currently.

Gbp-Dch: Short
Closes: #952907
2020-03-05 22:13:57 +00:00
Lyndon Brown c53a949325 firmware: fix possible duplication in firmware package lists (inefficiency)
where multiple archive areas are used, the code here on each loop is:
  1) fetching the archive area contents file (compressed)
  2) **appending** the output to that of the previous loop
  3) searching the file for firmware references, adding to the list

since it appends rather than replaces, entries found in each loop get
re-added on each subsequent loop, resulting in duplication in the
resulting list

below I evaluate the possible solutions to explain why I chose the one
I chose, however the reader should not waste too much time worrying about
whether one of the other solution would have actually been better because
things are changed significantly in further commits shortly!

possible solutions:
  a) switching to output (>) rather than append (>>), but this might fail
     against an existing file
  b) removing the file on each loop, but this will complicate any future
     caching improvements that might be made here (currently the files are
     always deleted and thus downloaded fresh)
  c) allow the appending, evaluating the complete file after the loop

solution C warrants consideration of disk space consumption; currently the
compressed 'main' archive (for sid on amd64) expands to 592.3 MB (feb-2020),
'contrib' is 3.1 MB, and 'non-free' is 18.5 MB.

solution C was chosen here; the difference of accumulated file size vs.
max-single was minor enough to not be of particular concern (~613 vs.
~592 MB).

Gbp-Dch: Short
Closes: #952906
2020-03-05 22:13:57 +00:00
Lyndon Brown a951fe7ba6 firmware: construct file location once and reuse 2020-03-05 22:13:57 +00:00
Lyndon Brown c140efe41e firmware: avoid building unnecessary lists 2020-03-05 22:13:57 +00:00
jnqnfe 055907592c cache: clarify and simplify package cache save/restore
These functions are specific to handling packages stored in the
cache, not other files. They are also always used with the same
`cache/packages.` prefix to the path.

Gbp-Dch: Short
Closes: #952916
2020-03-05 22:09:20 +00:00
Lyndon Brown 0869a890ae installer: download udebs directly from correct mirror
instead of trying all from derived mirror then falling back to parent upon
failure, which as pointed out by a message printed out can result in a
load of spurious 404 errors; actually get each udeb from the mirror it is
supposed to be retrieved from.

Partial fix for #952914, this is the last commit for it so closes it

Gbp-Dch: Short
Closes: #952914
2020-03-05 21:34:38 +00:00
Lyndon Brown b54bdd7fb1 installer: filter derived udebs from parent list
the existing logic just bundled the entire parent and derived udeb lists
together, ignoring the fact that there might thus be two instances of some
packages, and relying upon getting derived ones first and checking file
existence to avoid handling the overridden parent instances.

here we now actually filter the list of parent udebs to exclude packages
that are to be obtained from the derivative.

this enables avoiding the file existence checking

Partial fix for #952914

Gbp-Dch: Short
2020-03-05 21:34:38 +00:00
Lyndon Brown 2e3f195c38 installer: robustify udeb inclusion
move the code that checks whether a version of a given package has already
been obtained (to account for parent and derived both listing the same
package) to guard the copy from cache action also, not just the download
action.

in rare but possible scenarios it would have been possible to end up with
both the parent and derived copies of a package included.

Partial fix for #952914

Gbp-Dch: Short
2020-03-05 21:34:38 +00:00
Lyndon Brown 6997f99c9f installer: improve documentation a little
Gbp-Dch: Ignore
2020-03-05 21:34:38 +00:00
jnqnfe ac4ba405df archives: deduplicate apt sourcelist file construction
Geez :O

Gbp-Dch: Short
Closes: #952889
2020-03-05 21:08:53 +00:00
Lyndon Brown 450fb25fbd binary_onie: fix lack of newline on error
this script outputs a series of progressive dots as progress is made,
finally terminated with ' done.'. however if an error occurs then the
error would end up being printed directly on the end of this instead of
on a new line since newlines are not printed after each dot. this fixes
this oversight.

Gbp-Dch: Short
Closes: #952883
2020-03-05 21:04:23 +00:00
Lyndon Brown f3209fead2 binarie_onie: fix missing use of echo helpers
the '.' progress stuff has been left as is. perhaps that should be removed
since its use is inconsistent compared to other scripts?

Gbp-Dch: Short
Closes: #952882
2020-03-05 21:04:23 +00:00
Lyndon Brown c90749056f binary_iso: fix wrong echo helper
Closes: #952881
2020-03-05 21:04:23 +00:00
jnqnfe a45dcf46f6 fix missing use of echo helpers
Closes: #952876
2020-03-05 21:04:22 +00:00
jnqnfe a67e01638f help/usage: fix overly complex script description handling
Closes: #952887
2020-03-05 20:48:49 +00:00
jnqnfe aa858eb2c8 binary_disk: refactor
Avoid all of the duplication for each installer case

Note, what is done for the netboot case (which was previously missing)
still needs addressing

Gbp-Dch: Short
Closes: #952865
2020-03-05 20:42:31 +00:00
Lyndon Brown 8b109ffb96 config: rename --architectures to --architecture
this has only ever supported specifying a single arch, thus was confusing
being plural

Gbp-Dch: Short
Closes: #952892
2020-03-05 20:39:53 +00:00
jnqnfe 49e4608cff chroot: fix redundant usage line
Closes: #952885
2020-03-05 20:38:36 +00:00
jnqnfe 5a00487872 fix capitalisation of some output messages
Closes: #952875
2020-03-05 20:36:54 +00:00
jnqnfe 512dd58240 archives: fix mount local repo comments
Closes: #952873
2020-03-05 20:36:21 +00:00
Lyndon Brown 5c3679c882 add missing shebangs to temp generated shell code files
Closes: #952863
2020-03-05 19:25:13 +00:00
Lyndon Brown 7fb46c464b chroot_dpkg: start-stop-daemon simplification
inspired by what it does for `/usr/sbin/flash-kernel`

Gbp-Dch: Short
Closes: #952891
2020-03-05 11:52:01 +00:00
Lyndon Brown 2f5ae873b0 bootloaders: fix ignoring LB_DEBIAN_INSTALLER_GUI in menu creation
LB_DEBIAN_INSTALLER_GUI defines whether or not to provide the graphical
installer. the installer_debian-installer script pays attention to it and
does not download it if not wanted. the actual bootloaders however
(both grub2/loopback and syslinux) ignore it, which leaves broken and
unwanted menu entries. this fixes that.

Gbp-Dch: Short
Closes: #952890
2020-03-05 11:48:04 +00:00
Lyndon Brown 3840ac2005 memtest: better handle obsolete LB_MEMTEST value
available values currently are memtest86+|memtest86|none; "false" is
presumably handled for backwards compatibility

there is no need to handle this in individual scripts. the right place
to handle it is in Set_Defaults as now done

Gbp-Dch: Short
Closes: #952866
2020-03-05 11:39:57 +00:00
jnqnfe fcafaede19 simplify --debian-installer choices
`false` and `none` make no sense as choices for this option. Here we
replace `false` with `none`, and remove `true`.

Note that `true` was treated as an alias for netinst (see the changes to
source_disk and and binary_disk).

For backwards compatibility we still allow `true` and `false` by converting
them to `netinst` and `none` respectively, whilst printing a warning to
encourage users to move to `netinst`/`none`.

Gbp-Dch: Short
Closes: #952864
2020-03-05 11:36:55 +00:00
Lyndon Brown 6002f89b67 source: fix mistaken append instead of overwrite
Closes: #952862
2020-03-05 11:34:17 +00:00
Lyndon Brown 2511fff259 don't unnecessarily convert exit codes to strings
Closes: #952861
2020-03-05 11:33:12 +00:00
Lyndon Brown f3544ff68e installer: de-dup daily DI url
Closes: #952860
2020-03-05 11:32:14 +00:00
Lyndon Brown 36a3ba7634 fix indentation
including:
 - spaces replaced with tabs for consistency
 - alignment of `;;` in some case statements changed for consistency

Gbp-Dch: Short
Closes: #952857
2020-03-05 11:29:01 +00:00
Lyndon Brown ba08531da3 manpage: fix wrong description for --firmware-binary
--firmware-binary adds firmware to d-i, --firmware-chroot adds to live

Gbp-Dch: Short
2020-03-05 11:28:19 +00:00
jnqnfe c48caf36fd binary: fix missing handling of disk info for netboot case
The specifics of what whould be done for it, beyond the disk info
label, needs addressing. (fixme comment left). (#952854).

Partial fix for #952846

Gbp-Dch: Short
Closes: #952846
2020-03-05 11:22:20 +00:00
jnqnfe 42a9deb871 source: fix missing handling of disk info for netboot case
Partial fix for #952846

Gbp-Dch: Short
2020-03-05 11:22:20 +00:00
Lyndon Brown fde401a9ac installer: fix unquoted string
Gbp-Dch: Ignore
Closes: #952851
2020-03-05 11:18:20 +00:00
jnqnfe 63f374b861 binary_rootfs: fix chmod of squashfs image only being applied when using chroot
Perhaps it is unnecessary in the other case, but not harm applying
in both to be certain.

Gbp-Dch: Short
Closes: #952850
2020-03-05 11:17:04 +00:00
jnqnfe eac4757a89 binary_rootfs: fix deletion of excludes file from wrong location in squashfs after use
Closes: #952849
2020-03-05 11:12:56 +00:00
jnqnfe 769cf6e7b7 bootstrap: only output message where actually applicable
Gbp-Dch: Ignore
Closes: #952848
2020-03-05 11:10:45 +00:00
jnqnfe 849873daca bootstrap: drop obsolete check for `--no-check-gpg` support in debootstrap
This has been present in debootstrap since version 1.0.30.

Gbp-Dch: Short
Closes: #952847
2020-03-05 11:09:41 +00:00
jnqnfe 0d4b464370 binary: fix missing use of chroot_dpkg in binary stage
update feb-2020: the bug report is marked as fixed in version 5.0-a7-1,
however there is no such patch merged for that tagged release, nor at all
in master, so this properly closes it.

Gbp-Dch: Short
Closes: #776532
2020-03-05 11:07:28 +00:00
jnqnfe 2791a0d1e4 grub2: fix using wrong directory in certain cases
Update feb-2020:
Both grub2 and syslinux were affected by this issue. Someone else's
patch fixing syslinux only just recently got merged in
2735f3bd38 but grub2 was still
affected. I had made patches for both in 2015 which never got
merged. This is the grub2 one.

Gbp-Dch: Short
Closes: #952843
2020-03-05 11:06:30 +00:00
Lyndon Brown 9c062e1f8b archives: fix deb-src entry exclusion
the sed commands were missing '-i' to actually modify the file (rather
than output to stdout)

Gbp-Dch: Short
Closes: #952842
2020-03-05 09:46:56 +00:00
Lyndon Brown 666dc9f6f2 bootstrap_archives: remove unnecessary param default
all uses of this script pass in a suitable 'pass' param, and a previous
commit added a validation check. Having a fallback to 'binary' should a
param not be supplied is completely unnecessary.

fyi, this script is only ever called from the bootstrap stage and with
'binary' as the param, thus supporting being called with 'source' and
indeed having a 'pass' param at all is surely utterly pointless in itself.

Gbp-Dch: Ignore
2020-03-05 02:41:48 +00:00
Lyndon Brown d74f2102a0 archives: add 'pass' param validation check
if this had already been in place then it would have caught
bug #952837.

Gbp-Dch: Short
Closes: #952841
2020-03-05 02:41:26 +00:00
Lyndon Brown ae2012953b archives: fix incorrect usage string
Gbp-Dch: Ignore
Closes #952839
2020-03-05 02:40:59 +00:00
jnqnfe f3ebc44e13 archives: fix removal of chroot/root/packages.chroot package list
Gbp-Dch: Short
Closes: #952838
2020-03-05 02:40:42 +00:00
jnqnfe 25b44ac8df source: fix wrong action value for script execution
valid values are binary|source, and source should have been
used here.

Gbp-Dch: Short
Closes: #952837
2020-03-05 02:39:52 +00:00
jnqnfe 6adb5976b6 remove another redundant file
Originally removed in 42fb1e1976
Accidentally restored in bc50443cf0

(Closes #952835)
2020-03-01 08:38:05 +00:00
jnqnfe 1efc6d5adf remove redundant file
18e0a2f325 "rebranded live sources" which
included renaming `source_debian-live` to `source_live` but accidentally
left the old file in place.

(Closes #952834)
2020-03-01 08:37:22 +00:00
David Hewitt 2735f3bd38 binary_syslinux: Acommodate LB_INITRAMFS not live-boot 2020-02-17 10:37:27 +00:00
Raphaël Hertzog 2127283494 Standardize on having the GUI installation first in boot menus
debian-cd changed it that way, we already have it that way in EFI
boot, let's be consistent.
2020-01-23 16:47:04 +01:00
Raphaël Hertzog f64b73a718 Revert some Kali-specific change
That change was erroneously committed as part of
44b9b0a650
2019-12-19 21:05:54 +01:00
Raphaël Hertzog 30b9a391ac Fix "--debian-installer cdrom" and its interaction with binary_package-lists
apt-ftparchive is not able to differentiate between .deb and .udeb so
we have to install them in different pool directories so that we can
regenerate the Packages files without having the .udeb show up
unexpectedly.

Since binary_package-lists can overwrite the Packages files generated
in installer_debian-installer we have to ensure that it also updates
the Release file created formerly.

Ideally we should find a way to avoid the duplication of this logic.

Gbp-Dch: Full
2019-12-19 19:07:59 +01:00
John Estabrook b6ad08c270 Add local archive keys before configuring local package repository
In the binary stage, chroot is restored from cache/bootstrap, which contains
neither custom repo lists, nor keys. If local packages are present,
chroot_archives will call 'Apt chroot update' after adding custom repo lists
without adding keys. apt-get will then fail instead of warn as of apt version
1.5.

Closes: #941691
2019-12-19 18:52:05 +01:00
Ronny Standtke 97e8070b72 Run binary_hooks after binary_grub-efi
That way it becomes possible to customize grub-efi files.

Closes: #940846
2019-12-19 18:24:55 +01:00