Commit Graph

1013 Commits

Author SHA1 Message Date
Lyndon Brown 9bc35759aa config: fix stale live-build version in iso-preparer
using `$VERSION` as part of the default `$LB_ISO_PREPARER` means that when
you simply run `lb config` once, this variable is stored as a part of the
string, and replaced on use, but if you run `lb config` twice, it gets
replaced with a fixed version, that is then used in all subsequent builds.

let's replace with a placeholder (`@LB_VERSION@`) that can be used both in
the default, or in user strings, and will be replaced on use only.

this means that subsequent builds will always reflect the actual version
of live-build used.

Gbp-Dch: Ignore
2020-05-03 23:20:39 +01:00
Lyndon Brown 31cc121c3d config: obey new UTC timestamp control for default ISO volume label 2020-05-03 22:19:49 +01:00
Lyndon Brown 6cf3321d84 config: provide UTC timestamp control
`DATE_UTC_OPTION` is set in `Prepare_config()` for use by scripts, even
though only a few scripts will actually use it, since it allows those
scripts to be cleaner. we may want to possibly extend this as a
`DATE_OPTIONS` variable perhaps as part of enabling proper reproduciblity.

Gbp-Dch: Short
2020-05-03 22:19:43 +01:00
Lyndon Brown 7276d0213f fix usage/help/man bugs
- the definition of $PROGRAM as used in $USAGE strings defined in each
   script has been broken for a long time, being simply "lb" when it needs
   to be "lb COMMAND".
 - `config` changed $PROGRAM to "lb config" thus its output was correct in
   this regard unlike everything else, but with the switch to a more
   "intelligent" `Man()` function recently, it means that instead of
   `man lb config`, what was actually run was `man lb config config`,
   which displayed the manpage, then on exiting with `q`, it showed some
   sort of index line todo with a "config" search (no exact manpage
   match?), for which you had to enter `ctrl+c` to get rid of.

this revises things to fix the issues, minimising change by changing
$PROGRAM to "lb COMMAND", with the frontend overriding this.

Gbp-Dch: Ignore
2020-05-03 01:38:17 +01:00
Lyndon Brown 372a8fd91a args: restore a little tidying
lost in a reversion

Gbp-Dch: Ignore
2020-05-03 01:31:47 +01:00
Lyndon Brown 23ed83e730 config: tweak config file loading order
load the base set of files first, then any user arch/dist specific override
files second.

this is important since whenever config items get moved in the base files,
this can break the override files, if an option specified in one was moved
to a different common file that it loaded after.

Gbp-Dch: Ignore
2020-05-02 20:44:02 +01:00
Lyndon Brown 22d8ac15db options: de-duplicate common option handling 2020-05-02 18:18:49 +01:00
Lyndon Brown 2d7ff4934d improve stagefiles encapsulation
- added and used Stagefiles_dir()
 - added and used Stagefile_exists()

pretty self explanatory.

Gbp-Dch: Short
2020-05-02 14:49:45 +01:00
Lyndon Brown 5f318c5a44 tidy up Echo_breakage()
- it's only used by the debootstrap script after alternatives were dropped
   long ago, so let's move it, avoiding it being loaded for everything
   else.
 - there's no need to pass printing another message through it.
 - there's little point in making the sid distinction if you happen to
   decide to build sid, it's a given that it's less stable than stable.

really, is there any need for this at all?

Gbp-Dch: Ignore
2020-05-02 13:06:17 +01:00
Lyndon Brown 1c74927fe5 config: simplify mode default
all roads lead to debian

i was waiting to get to the work on properly tackling the mode stuff, but
let's just tidy this now...

Gbp-Dch: Ignore
2020-05-02 12:41:35 +01:00
Lyndon Brown f78001144b move and rename Common_config_files()
for better encapsulation, and for consistency with other config file
handling functions in the new location.

Gbp-Dch: Ignore
2020-05-02 02:38:26 +01:00
johnraff 0e5d0483e5 Replace 'which' with 'command -v' to test for the existance of an executable
This is considered to be more robust.

Two instances remain:

scripts/build/chroot_archives, line 257:
if [ "${LB_APT}" = "aptitude" ] && [ ! $(Chroot chroot "which aptitude") ]
The command is run inside a chroot where the environment might be special,
and would need further testing.

manpages/Makefile, line 42:
@if [ ! -x "$$(which po4a 2>/dev/null)" ]; \
I am insufficiently familiar with makefile syntax to edit this.
2020-05-01 09:59:11 +00:00
Lyndon Brown 2c1af527fd config: fix wrong variable name
Gbp-Dch: Ignore
2020-05-01 08:57:44 +00:00
Lyndon Brown a72e7daaf7 config: only try to load FILE.ARCH and FILE.DIST if non-empty strings
to thus avoid trying to load `FILE.`

Gbp-Dch: Ignore
2020-05-01 08:57:44 +00:00
Lyndon Brown 61aa3239c5 config: remove junk config loading code
support for this has not existed since v2.0~a24-1 !!

until 4d22ca948a (v2.0~a24-1), `PROGRAM` was
defined as `PROGRAM="$(basename ${0})"`. then this commit simply got rid
of the definition.

it then flipped back and forth a bit in a few ways, but always sticking
to a simple fixed "live-build" type string after that commit, as it has
been ever since.

the code removed here has thus been junk since then, and the documentation
corrected here out of date since then.

the checking of `LB_CONFIG` is also possibly junk, but i know too little
about it. it was introduced in v1.0~a43-1 in commit
705a4178e7 with a changelog entry of
"Centraly evaluate LH_CONFIG for custom configuration file."

Gbp-Dch: Short
2020-05-01 08:57:44 +00:00
Lyndon Brown 51903da073 init: warn if auto script exists but is not executable 2020-04-30 05:07:02 +00:00
Lyndon Brown 29ab2459c8 a couple tiny doc fixes
Gbp-Dch: Ignore
2020-04-29 10:27:16 +01:00
Lyndon Brown 997a7f73f9 config: remove redundant validation check
this is already covered by the check above it

Gbp-Dch: Ignore
2020-04-26 12:55:18 +00:00
Raphaël Hertzog e7df05cd29 Replace "Help" calls with "Man" and get rid of Help 2020-04-24 16:39:10 +02:00
Raphaël Hertzog fac2d8d903 Make "Man" function smarter to also work for "lb --help" 2020-04-24 16:38:45 +02:00
Raphaël Hertzog 104fa20399 Revert "frontend: properly handle option processing"
This reverts commit 6b7c8ed4bf as it's
breaking multi-value space-separated parameters.
2020-04-24 15:37:13 +02:00
Lyndon Brown d79c96232b stagefiles: guard unnecessary chroot removal
just as most scripts are skipped if their stagefile exists (indicating
that they have already been run to completion), including chroot
preparation scripts in install mode, this implements the same guard for
chroot prep remove mode, such that they exit early if their stagefile
does not exist, indicating that the modification has already been removed.
(also override-able by --force in the same way).

this basically just uses a tweaked copy of Check_stagefile().

Gbp-Dch: Short
2020-04-23 18:26:15 +01:00
Lyndon Brown 500f205073 config: s/LIVE_IMAGE_TYPE/LB_IMAGE_TYPE/
no backwards compatibility hack for reading the old var from existing
saved config used because this was previously stored in the alternate
format config/build file.

Gbp-Dch: Short
2020-04-23 16:54:31 +00:00
Lyndon Brown 83d9145257 config: s/LIVE_IMAGE_NAME/LB_IMAGE_NAME/
no backwards compatibility hack for reading the old var from existing
saved config used because this was previously stored in the alternate
format config/build file.

Gbp-Dch: Short
2020-04-23 16:54:31 +00:00
Lyndon Brown 41738b1932 config: s/LIVE_CONFIGURATION_VERSION/LB_CONFIGURATION_VERSION/ 2020-04-23 16:54:31 +00:00
Lyndon Brown c441c39efe config: revert partial format conversion
back in v4.0~a6-1 a transition process was started to move the live-build
config to a new format. the new format was INI style, and required
parsing functions to read/write values, compared to the existing format
which was just shell script code setting variables.

this partial transition is the explanation for the existence of the
`New_configuration()` function, and understanding this is important to
understanding the purpose of it - it is not in fact intended for creating
a new configuration, it is just related to the new config format
transition.

the positives of the new format were that it was somewhat cleaner looking,
while the negative was the terrible relative efficiency.

the file `config/build` was created to hold options in this new format.

the transition was only ever completed for a handful of config options:
 - architecture
 - archive areas and parent archive areas
 - live image name
 - live image type

a 'configuration version' attribute was also saved, which is not used by
anything.

the bootstrap-mirror and parent-bootstrap-mirror attributes are pointlessly
stored in it seemingly resulting from work done in v4.0~a17-1. (they are
also stored in another config file from which the value is actually used).

it in fact seems to have been a source of confusion for Raphaël in
authoring 44b9b0a650, since the new
`[parent]-distribution-{chroot|binary}` options it introduced were stored
both in `config/bootstrap` and in `config/build`, while only used from the
former. i expect, understandably, that he thought that `config/build` was
just an information file.

Gbp-Dch: Short
2020-04-23 16:54:31 +00:00
Raphaël Hertzog e71db2bd58 Fix version computing when LIVE_BUILD is set
It used to rely on the $LIVE_BUILD/VERSION file which is now gone.
Instead we now rely on the debian/changelog file just like we do in the
Makefile.
2020-04-23 17:00:19 +02:00
Raphaël Hertzog 4c75f80e67 Use descriptive parameter names for Usage() 2020-04-23 16:38:59 +02:00
Lyndon Brown b3bba232ac usage: take exit code as param
thus it can correctly indicate success/fail status instead of always
indicating failure. when a user asks for usage with -u|--usage then we
should exit in success mode rather than failure as when usage in printed
in response to incorrect usage.

Gbp-Dch: Short
2020-04-23 16:32:26 +02:00
Lyndon Brown 6b7c8ed4bf frontend: properly handle option processing
this means that the usage goes from:
 lb {-h|--help|-u|--usage|-v|--version}
 lb COMMAND [OPTIONS]

to:
 lb {-h|--help|-u|--usage|-v|--version}
 lb [FRONTEND_OPTIONS] COMMAND [COMMAND_OPTIONS]

though it is probably not worth is to update the description in the
manpages...? hmm...

so for instance this matters for color control with --color|--no-color
(you already had full control via environment vars). previously you could
do `lb COMMAND --no-color` to turn off colour, only to find that output
at the frontend level was still coloured (the option is processed at the
command context level, not the frontend), so you might try to instead use
`lb --no-color COMMAND`, only to find that this was not supported. Well
now it is, and used at the frontend level will fully control colour output
(after the command is processed anyway).

the full set of common options are thus available (except --force) at the
frontend level, and thus for instance all Echo_*() helpers used in the
frontend will work correctly after args are processed.

furthermore usage like `lb --color --help` will actually work. (not that
color is used there, but this previously would have failed with the
frontend treating the `--color` argument as the command; that's the point!)

Gbp-Dch: Short
2020-04-23 16:29:00 +02:00
Lyndon Brown 8ffe48d8f3 rename LB_BOOTSTRAP_QEMU_ARCHITECTURES to LB_BOOTSTRAP_QEMU_ARCHITECTURE 2020-04-23 16:00:34 +02:00
Raphaël Hertzog 047885bf7e Keep LB_ARCHITECTURES environment variable for backwards compatibility
Custom hook scripts may use the variable. And conditional package lists
can also rely on this variable.
2020-04-23 15:58:59 +02:00
Lyndon Brown 6cc7250954 rename LB_ARCHITECTURES to LB_ARCHITECTURE
this was previously not done in 8b109ffb96
to keep the renaming simple, but leaving the variable plural is a cause
for confusion.

since this property is stored in the INI style config/build config file
rather than a shell script based one, at the property there is already
singular, there was no need for a backwards compatibility hack.

Gbp-Dch: Short
2020-04-23 15:54:19 +02:00
Lyndon Brown f4c8a28a21 config: move control of apt colour
being done in Prepare_config() meant that it would get saved into the
config stored on disk which is not what was intended.

being saved on disk is not a big problem, since the injection of the
dynamically determined option is always done on top of the value read
from the saved config anyway, so you'd always end up with two copies of
the option given to apt, the first being from the saved config, the second
being the more correct dynamically determined one, overriding the first.

so there is no functional change here, just better achieving what was
intended.

Gbp-Dch: Ignore
2020-04-23 15:52:32 +02:00
Lyndon Brown faa51b83c5 config: fix mistake in mirror url tidying
Gbp-Dch: Ignore
2020-04-23 15:51:53 +02:00
Lyndon Brown ba05247438 chroot: make var local
Gbp-Dch: Short
2020-04-23 15:47:00 +02:00
Lyndon Brown 9a91ca9fde config: obsolete old --tasksel option
note that the bit of code removed from source_debian relies upon a
variable LB_TASKS which itself is an old leftover artefact from before
v4.0.

Gbp-Dch: Short
2020-04-23 15:46:06 +02:00
Raphaël Hertzog d414b8fcdb config: obsolete --net-root-path 2020-04-23 15:44:17 +02:00
Lyndon Brown 1ca53bff52 config: obsolete --net-root-* options (except one)
--net-root-path probably needs to go too, but it is being used for
something i don't fully understand currently.

Gbp-Dch: Short
2020-04-23 15:30:41 +02:00
Lyndon Brown 1eee15e852 config: obsolete unused --net-cow-* options 2020-04-23 15:24:50 +02:00
Lyndon Brown c3f0d39675 config: apt-get should probably be an allowed and documented --apt value
since everywhere where 'apt' is a permitted value, 'apt-get' is also, it
just wasn't listed in the option's documentation and thus was also not
listed in the new validation check.

Gbp-Dch: Short
2020-04-23 15:23:30 +02:00
Lyndon Brown c57b8679a4 config: fix broken backwards compatibility hack
80aa5ab611 implemented a hack to handle
replacement of LB_LINUX_FLAVOURS with LB_LINUX_FLAVOURS_WITH_ARCH in
config files, but implemented it in the wrong place.

adding a conditional conversion within the config file meant that the old
value would only be read from **new** config files that are created
obviously without it, including re-saved configs if `lb config` were
re-run with additional options (not recommended). any existing value in an
existing config file would actually be ignored.

the right place to read the old value was in the Set_defaults() function
(since renamed).

a second issue also existed with the hack, it failed to excape the `$`
and thus printed the existing value of $LB_LINUX_FLAVOURS into the
conditional check being constructed in the config file, instead of
printing the name of the variable. the check embedded into the config
file thus became this on an amd64 machine:
```
if [ -n "amd64" ]
then
	LB_LINUX_FLAVOURS_WITH_ARCH="amd64"
fi
```
which is clearly not what was intended.

Gbp-Dch: Short
2020-04-23 15:23:15 +02:00
Lyndon Brown 9ab7a100da config: add missing deprecation warning
Gbp-Dch: Ignore
2020-04-23 15:14:16 +02:00
Lyndon Brown 780ce25288 config: minor reorganisation
Gbp-Dch: Ignore
2020-04-23 15:13:33 +02:00
Lyndon Brown cd68a4f61f config: strip unhelpful comments
Gbp-Dch: Ignore
2020-04-23 15:12:17 +02:00
Lyndon Brown 2d272fff5c config: expand validation coverage
to cover checking available value choices

Gbp-Dch: Short
2020-04-23 15:11:50 +02:00
Lyndon Brown 46bce5a6d1 config: reorganise validation checks
splitting into two parts will make things much tidier

Gbp-Dch: Short
2020-04-23 15:09:56 +02:00
Lyndon Brown 554a8d1108 tidy up grub bootloader compatibility checking
- add a validation check where an error will be printed
 - replace the check done in the grub scripts with one that simple exits
   if executed bypassing the validation check

Gbp-Dch: Short
2020-04-23 15:08:07 +02:00
Lyndon Brown 9b70092f25 config: tidy validation 2020-04-23 15:04:51 +02:00
Lyndon Brown 39572ccf3f config: merge defaults.sh and configuration.sh, part 2
...and second we then change the filename

Gbp-Dch: Short
2020-04-23 15:02:43 +02:00
Lyndon Brown 2d877a2cb9 config: merge defaults.sh and configuration.sh, part 1
the stuff in defaults.sh is only partially to do with setting default
config values, it also does other prep work like centralised comma
separated value to space separated conversion, and validation.

this is split into two parts to keep the diff simple, first we move the
two small functions from configuration.sh into default.sh...

Gbp-Dch: Short
2020-04-23 15:02:38 +02:00
Lyndon Brown 5fb790e43e config: rename Set_config_defaults() to Prepare_config()
it mostly applies defaults where a value does not exist, but does more in
some cases. the new name better reflects its usage and functionality.

Gbp-Dch: Short
2020-04-23 14:52:20 +02:00
Lyndon Brown 7de8a0faa7 config: rename Check_config_defaults() to Validate_config()
this is used after applying user settings on top of the defaults,
so is not specific to checking defaults; it's a validation checker.

Gbp-Dch: Short
2020-04-23 14:51:58 +02:00
Lyndon Brown 430fdc3040 config: move defaults set in wrong function
Gbp-Dch: Ignore
2020-04-23 14:51:28 +02:00
Lyndon Brown 39e4d3e3cb --binary-images can support only a single type
whilst some parts of the codebase were set up to work with multiple types
specified, others did not work with it and would not necessarily be easy
to adjust. this thus makes some tweaks to adjust things accordingly.

 - option renamed to singular form (maintaining backwards compatibility)
 - a validation check has been added
 - unnecessary glob style type references fixed
 - checks with In_list changed to a direct singular comparison
 - typo of type "netboot" written as just "net" fixed (though unreachable
   so of no consequence; really the code could be removed but it's trivial)

Gbp-Dch: Short
2020-04-23 14:51:09 +02:00
Lyndon Brown 6b734df9f3 fix incorrect handling of -c|--conffile
this option actually only has an impact when used for `lb config`; it thus
was incorrectly accepted and captured by Arguments() and the documentation
was wrong.

going back through git history, this has been incorrect since the v0.99 days.

also, reorganised the list alphabetically (no-color was inserted in the
wrong place).

Gbp-Dch: Short
2020-04-23 11:52:13 +01:00
Lyndon Brown db5188aa99 config: support comma and space separated list of source image types 2020-04-23 11:52:13 +01:00
Lyndon Brown d38a097809 config: support comma or space separated archive area lists 2020-04-23 11:52:13 +01:00
Lyndon Brown 673b811627 config: support both comma and space separated to-cache stages 2020-04-23 11:52:13 +01:00
Lyndon Brown f73797158e bootloaders: improve bootloader list handling
two parts of the code worked with both comma and space separated lists,
while two others only worked with comma separated.

swapping out commas with spaces when we setup the var in
Set_config_defaults() means that individual scripts no longer need to worry
about it and everything supports both; and that we can avoid the
IFS/OLDIFS mess.

Gbp-Dch: Short
2020-04-23 11:52:04 +01:00
Lyndon Brown 286caf03be fix outdated grub-legacy bootloader name references 2020-04-23 11:46:36 +01:00
Lyndon Brown 11836c0c18 fix $@ parameter expansion issues
$@ when unquoted is subject to further word splitting. this fixes a bunch
of instances where it was incorrectly being used unquoted.

Gbp-Dch: Short
2020-04-23 09:38:01 +00:00
Lyndon Brown f6a50b6db2 packages: remove auto-installed on removal
i've been noticing that in the binary_syslinux stage a wholebunch of
packages get installed whilst installing isolinux, and then upon removing
the installed packages at the end of the script, apt is listing a large
number of those that were auto installed as no longer in use and can be
removed via `apt autoremove`. this then persists through package
installation and removal steps throughout remaining scripts.

adding `--auto-remove` to `apt-get remove --purge` and similarly
`--purge-unused` to `aptitude purge` instructs apt to get rid of the unused
auto-installed packages at the same time as removing specific package that
brought them in in the first place, and thus resolves the problem.

Gbp-Dch: Short
2020-04-23 09:25:17 +00:00
jnqnfe 314ca3d56a bootloaders: replace use of vga=normal with vga=788 in live menu entries
For consistency with install entries (both in live-build and
official Debian install discs).

Comparing with live-build created installer entries, grub-legacy
and grub2 both favour vga-788 for GUI entries and vga=normal for
test entries, whilst syslinux uses vga-788 for everything.

Gbp-Dch: Short
2020-04-01 15:25:32 +00:00
Lyndon Brown 3c4d07ff18 apt: use its new colour support
apt v2.0.1 introduced support for coloured E:/W:/N: labels. this adds
support to control it based upon our own colour control.

note that with utilities like dpkg we do not do this, but apt only uses
its new colour support automatically when `apt` is used directly, it is
not automatically enabled (per isatty()) for `apt-get`/`aptitude` (the
`apt` developer responsible for adding colour support in response to my
request for it told me that it was deliberately done like this per being
customary to not change behaviour of those tools for compatibility
reasons). colour errors/warnings are useful, so we want to turn it on for
our use of these tools where we can.

Gbp-Dch: Short
2020-03-27 21:05:52 +00:00
Lyndon Brown 39dae8cdc7 move auto colouring decision
...from the `Set_config_defaults` function, to being done directly
in `build.sh` (the component which is also responsible for loading
functions, loaded at the start of every script, including the front
end).

thus the colouring decision will now correctly...
 - apply to the frontend, such as to the 'root privileges needed'
   error, the 'no such script' error, and the command name
   colouring that I want to add (the most significant issue).
 - apply to error messages generated by the `Arguments` and
   `Read_conffiles` functions, which are called before
   `Set_config_defaults` by scripts.

as things were, due to the comparison with "false", colour would
_always_ be used in these places (unless _COLOR_ERR=false or
_COLOR_OUT=false wrt. the new command highlight, were set in the
environment when executing a script throught the frontend).

this would not be a problem for normal terminal use of course,
besides being inconsistent where color were turned off, but would
be a bit of a problem if redirected to a file.

a re-evaluation of _COLOR is performed in `Set_config_defaults` to
adjust _COLOR_OUT and _COLOR_ERR where necessary, to correctly
respond to _COLOR being set in saved config files (disabled by
default but a user could always enable), after the point of config
files being loaded.

_COLOR can still be controlled from the environment just as before,
overriding both _COLOR_OUT and _COLOR_ERR.

note that this does not address the fact that --color|--no-color
do not work in the frontend and thus will not impact the colouring
of to-be-introduced command highlighting. this needs to be
addressed separately.

Gbp-Dch: Short
2020-03-23 08:06:51 +00:00
Lyndon Brown 85e0681ee8 args: fix a typo
Gbp-Dch: Ignore
2020-03-23 08:06:51 +00:00
Lyndon Brown 94a3e184c2 archives: explicitly pass along _PASS to Create_apt_sources_list
Gbp-Dch: Ignore
2020-03-20 11:43:01 +00:00
Lyndon Brown 29d9c23cd2 defaults: enable d-i GUI for all 2020-03-20 10:19:33 +00:00
Lyndon Brown c534ff52a4 defaults: ensure labels have defaults for derivatives 2020-03-20 10:19:33 +00:00
Lyndon Brown 2c14566c69 defaults: tidy mirrors 2020-03-20 10:19:33 +00:00
Lyndon Brown 945a166f75 strip progress-linux distro hacks
progress-linux, as discussed in MR #142 ([1]) is a little known distro,
which appears to be little more than a personal project of the original
author of live-build.

given that, the expense of maintaining all of these old hacks for it
cannot be justified. it is not known whether or not live-build is even
used with respect to it since the author abandoned live-build some
years ago.

also, at least one past change in live-build possibly broke progress-linux
compatibility anyway, which would have required progress-linux users of
live-build to use a custom progress-linux config, or a progress-linux
fork of live-build, and there is no knowing how much of the hacks in this
"upstream" codebase any user of progress-linux currently relies upon.
and again, progress-linux appears to just be a personal project of
Daniel's, with afaik very little userbase. (Daniel seems to be the only
developer working on the project which speaks to how small it is).

[1]: https://salsa.debian.org/live-team/live-build/-/merge_requests/142

Gbp-Dch: Short
2020-03-20 10:19:33 +00:00
Lyndon Brown d6a80d3d4d defaults: purge long unused LB_ROOT
seems to have been obsolete since all the way back at v1.0~a22-1.

history:
 - in 0d0de885e3 it was renamed from
   LIVE_ROOT to LH_ROOT, but also seems to have become completely
   obsolete at this point, and thus mistakenly not actually removed.
   before this it seems to have been used to hold the base directory of
   live-build from which paths were constructed, but then this use was
   removed making it redundant, but it remained in the code.
 - 83bc63f725 renamed to LB_ROOT.
 - a79a5bea10 dropped setting the variable
   only if not already set, in favour of always setting it depending upon
   LB_MODE. but still it remained unused.

Gbp-Dch: Short
2020-03-20 10:19:33 +00:00
jnqnfe 4fa83598a3 grub: remove old and unused splash code
The LB_GRUB_SPLASH variable is populated by the --grub-splash param
but is not actually used for anything.

Gbp-Dch: Short
2020-03-20 10:02:52 +00:00
Lyndon Brown 0e090a65e3 fix -h|--help component script man page redirection
the frontend handles -h|--help directly and correctly redirects to the
man page.

component scripts however fail to load the correct manpage because they
are being directed to `man <script>` instead of `man lb script`.

(affects the top level commands and major build stages which actually have
man pages; the low level components don't and so will always fail anyway).
2020-03-19 22:42:01 +00:00
Lyndon Brown 406accfab9 defaults: remove redundant setting of LIVE_BUILD
this is handled for every script in build.sh. this is not stored in the
saved config or anything, so no need to re-evaluate in
`Set_config_defaults`. this just seems to completely pointless.

Gbp-Dch: Short
2020-03-19 16:33:25 +00:00
Lyndon Brown b7896564c5 defaults: bump checksums to stronger form
md5 & sha1 are not considered secure anymore and thus are of questionable
value here if checksums are wanted.

Gbp-Dch: Short
2020-03-18 14:47:22 +00:00
Raphaël Hertzog bdffaebe12 Minor cleanups in Require_stagefiles 2020-03-18 09:37:22 +01:00
Lyndon Brown fb0790cc43 stagefiles: s/Require_stagefile/Require_stagefiles/
this function takes one or more required stage fileS _plural_, and exits
if any are missing (or at least it does now after the refactor).

let's rename it to make things more clear

Gbp-Dch: Short
2020-03-17 22:59:37 +00:00
Lyndon Brown fe0d2358b9 stagefiles: only list missing stages 2020-03-17 22:59:34 +00:00
Lyndon Brown 3bed138fda stagefiles: avoid stagename in required error
the name of the stage is already printed earlier in the output prior to
the error here being printed. so the error really does not need to include
the script name itself.
2020-03-17 22:57:09 +00:00
Lyndon Brown ea0f6b7810 stagefiles: fix completely wrong require-stages logic
now having investigated my suspicions of the functionality and use of
Require_stagefile(), i conclude that it has been fundamentally broken
all the way back to v1.0~a8-1 (or at least usage of it since v1.0.1-2).

gah. (╯°□°)╯︵ ┻━┻

----

very early on in the history of live-build this function took the name of
a _single_ stage file only and did `exit 1` should the file not be found.
this was simple and clearly accomplished "what was on the tin", so to
speak.

in bd1a5ddc82 (2007, 1.0~a8-1) things got
weird. it was modified to support a list of multiple files. but instead of
being written to cause failure if _any_ of the listed files were missing
as perhaps one might expect, it was instead written to fail only if all
files were missing!

if you jump to the conclusion that i'm talking about a simple flipped
logic from a lack or otherwise of a `!` here, you'd be mistaken; there is
a comment inside the function that could not be more clear about what was
intended by the author - "Find at least one of the required stages"! this
makes me thoroughly confused about what they were thinking.

as we'll get to, this was fundamentally flawed (or at least its later use
was), but furthermore there were other notable issues introduced at this
point (but don't worry too much about these, they've all been addressed):
 - `NAME` was modified in the loop, using the existing value, but nothing
   initially set it...
 - the setting of `NAME` seems related to its use in the subsequent error
   output, yet they are logically separated; it is only set if a file
   exists, while the error is only printed if none exist.
 - it is pointlessly using a messy `CONTINUE="true"` based mechanism,
   when it could just `return 0`.
 - it did not handle correctly the bad use case of no params having been
   supplied.

it doesn't seem to have been entirely thought through, despite its
pervasive use throughout the build system.

note that no change was made in that commit to make actual use of the
new multi-param support. it would not be used until about a year later.

the function has remained largely untouched since then. in
c68c0a2708 a notable change was made to add
an initial setting of `NAME`, which partially addressed one of the above
issues. but it did not really address the issue the change was meant to
solve, since the `NAME` as printed in the error was now the name of the
script when what was really wanted was the name of the stagefile. this was
finally fixed properly in d54990695f.
however the weirdly pointless setting of `NAME` persisted in the loop.

finally i personally just refactored the function in the commit prior to
this one, retaining the same functionality but addressing the remaining
of the above minor implementation issues.

looking at usage of the new functionality introduced in
bd1a5ddc82, it does not seem to have been
until 0cbbde2b96 (2008, almost a year after
it was made possible) that changes were made to finally start making use
of the ability to pass more than one filename at a time to the function,
and it would appear that perhaps the author forgot what it actually was
that the function accomplished when used with multiple params, and failed
to double check.

in this first use of multiple parameters, this commit went from passing
single file names to individual calls to the function to passing the files
in one single call, in a commit the purpose of which was described as
simply tidying things up. it was most certainly not intended to change
stage requirements.

unfortunately, a change in requirements did occur as a result since the
new usage of the function was not accomplishing the same as before. this
change completely broke the stage requirements protection mechanism such
that only a single one of the listed stages needed to have completed for
the check to pass, rather than all as expected.

this flaw made it into release v1.0.1-2 and it has existed every since.

in the very next commit from that one,
6204dc0e6d things got even worse. here we
see the config stage being specified commonly as the first stage listed,
which is still the case today. this means that ever since this commit,
if you've already got a config before building (which you inevitably do,
especially after some later commits introduced automatically creating it
if missing), then all other stage requirements are simply ignored.

so it seems pretty damn clear that this function is accomplishing
completely the wrong objective. it _should_ be checking that _all_ files
given to it exist, not just one or more. ¯\_(ツ)_/¯

this FINALLY addresses this mistake.

(not that i wish to berate the author; i've made silly mistakes of my own
before)
2020-03-17 22:57:09 +00:00
Lyndon Brown 1b09b15277 stagefiles: refactor Require_stagefile()
- count of params is available as $#, we don't need the pipe-to-wc logic.
 - the whole 'CONTINUE' based logic is silly, we can just return once one
   of the files is found.
 - setting of 'NAME' in the loop was completely pointless.
 - the error message for multiple files was not very clear just injecting
   a sequence of words into a sentence.
 - it did not work correctly if no arguments were given (bad usage)

note, you might question whether the functionality of this function is
correct, as did I; this is tackled in a followup commit whilst this
commit retains the existing functionality!

Gbp-Dch: Short
2020-03-17 22:57:04 +00:00
Lyndon Brown fe9195b59c stagefiles: further robustify with auto filenames
as suggested by Raphaël

rather than have fixed stagefile filename strings at all in the scripts,
use `$(basename $0)` to use the name of the script (which is the same for
almost all cases anyway, and the stage files are supposed to be almost
exclusively unique per-script). we can thus simplify things by determining
the filename for most use cases within the functions themselves.

this does change the file used by a couple of scripts, affecting backwards
compatibility of executing live-build upon an existing partially or fully
completed build:
 - binary_grub-pc used "binary_grub"
 - chroot_includes used "includes.chroot"

care had to be taken for the following cases:
 - there are some cases like bootstrap_cache, source_debian and
   bootstrap_debootstrap which are dealing with more than one file, and/or
   otherwise a filename that is not specific to the script itself exactly,
   or should not be based upon its name.
 - some cases like chroot_cache, bootstrap_cache and
   chroot_install-packages need to append something to the end of the name
   depending upon which pass/action mode the script is being executed with.
 - furthermore in the bootstrap_cache case one of the filenames is used
   within the bootstrap_debootstrap and thus needs very careful handling
   to be certain that a change in filename of bootstrap_cache does not
   break bootstrap_debootstrap.

Gbp-Dch: Short
2020-03-17 18:57:02 +00:00
Lyndon Brown 04d9ee0211 stagefiles: simplify & robustify
- avoid all need to pass ".build/" path in stage file names into the
   functions
 - add a helper to remove a stage file (required to complete the above
   properly)
 - avoid duplicating filenames within scripts which makes them prone to
   mistakes (some instances of which I've actually encountered and had
   to fix)

Gbp-Dch: Short
2020-03-17 18:57:02 +00:00
Lyndon Brown bea349c822 exit: fix missing local scope
missed in c55eb8a0c3

Gbp-Dch: Ignore
2020-03-17 17:33:31 +00:00
Lyndon Brown c55eb8a0c3 use local scope for private function vars
all vars affected have been carefully checked to be quite certain
that they are definitely local

where variable is assigned the return value of a function/command, the
local "declaration" is deliberately done on a separate line, since
`local FOO` is actually treated itself as a command rather than a
declaration; will thus always cause $? to be zero, and thus if done on
the same line as such an assignment can not only clobber $? but in doing
so unintentionally blocks failure of a command from triggering the
expected exit from having `set -e`.

also, from testing, i have found that when assigning "${@}" this must be
done on a separate line confusingly as otherwise an error occurs.

Gbp-Dch: Short
2020-03-16 22:10:03 +00:00
Lyndon Brown ca520eb5f0 split stdout & stderr auto colouring
thus for some reason if one is connected to a tty and the other a file,
we still get colour in the tty by default.

in terms of options, --color and --no-color override both, no granular
ones added since it's not worth it imo.

this is backwards compatible with custom configs setting `_COLOR`.

it could be argued that setting $_COLOR to "false" for the auto non-tty
cases is redundant, which it is, but it doesn't hurt to do so; it ensures
that if anything (inc. 3rd-party hooks and such) rely on it that it
remains correct; and ensures that if anything in the future mistakenly
uses $_COLOR instead of $_COLOR_OUT|$_COLOR_ERR that at least that will
only be broken for the use case of only one of stdout|sdterr being a tty.

Gbp-Dch: Ignore
2020-03-16 13:32:16 +00:00
Lyndon Brown 09b279b7bc turn on colour by default
...when stdout+stderr connected to a tty (as opposed for example to being
piped to a log file)

very helpful to have colour such that the red/yellow of errors/warnings
can draw the eye to problems.

Gbp-Dch: Short
2020-03-16 13:32:16 +00:00
Lyndon Brown 49e68da5ee allow turning off colour
necessary to give control when colour is default enabled

Gbp-Dch: Short
2020-03-16 13:32:16 +00:00
Lyndon Brown 24c3582b2c exit: don't convert numeric exit code to string 2020-03-13 16:06:20 +00:00
Lyndon Brown 1cbe8f8aac functions/bootloaders: tidy and simplify 2020-03-13 16:03:48 +00:00
jnqnfe b27927724a locks: tidy lock acquisition
Combine the check+create done in each script. (The original functions
are still callable as before, but a new combined `Aquire_lockfile`
function can be called instead, as now used).

Note, a further simplification could be done in removing the passing of
the lock filename in as a parameter since every use of the functions is
with ".lock". The lock functions already have a fallback to ".build/lock"
though. Checking the history, the fallback used to be for a system wide
lock, which was then replaced with this config-tree specific one. As long
as that is not used implicitly by 3rd-party hooks then surely we are free
to change the fallback to ".lock" and further remove passing in a name as
a param...?

history:
db5d2b0dcd
0aa8289a37

Gbp-Dch: Short
Closes: #952918
2020-03-13 15:37:38 +00:00
Lyndon Brown 346e3e1c36 fix colouring of notice type message prefixes
previously this was white+bold. the white aspect was dropped since this
would not be sensible for users with a white background for their terminal.
bold however does not have any effect for me at least so effectively there
is no highlighting at all.

here we reintroduce a colour, one that will work for both black and white
backgrounds of course. purple looks good to me - significantly different
to that used for errors and warnings, and works well with command
highlighting (as enabled separately).
2020-03-13 15:23:13 +00:00
Lyndon Brown 5a1c875cb8 exit: ensure an appropriate message is printed on unexpected exit
if a script exits due to a failure and `set -e`, we should ensure that an
error message is printed to be clear to the user that something actually
went wrong.

similarly it would be good to print a suitable message should the user
cancel with ctrl+c for instance.

Gbp-Dch: Short
2020-03-13 15:10:29 +00:00
jnqnfe a6cc72bf6d strip useless multi-arch separate directory stuff
(part of never completed side-by-side multi archi support)

Several scripts make a call to a function called Check_multiarchitectures,
the purpose of which is to adjust the target directory that certain 'live'
and 'install' files are located in. The idea is that a script sets up
'DESTDIR', 'DESTDIR_LIVE' and 'DESTDIR_INSTALL' as appropriate and then
the script appends a suitable arch dependant postfix to the directory
name, depending upon the arch currently being targetted. This would allow
the script to be run multiple times, each for a different architecture.

This is a part of an implementation of allowing multiple architectures to
sit side by side within the same live image, selectable from the
bootloader menus. (As opposed to multiple architectures mixed within the
same userland).

This is evidently the case both from the fact that:
  1) The arch specific postfix chosen in that function depends on a var
     called LB_CURRENT_ARCHITECTURE, which is never set. In fact going
     back through the git history to the introduction of the function in
     0d5ff4ca75, the var (even considering
     var name changes) has never been set by anything. So effectively the
     call to the function has been entirely redundant all this time.
  2) The major build stages do not perform multiple executions of substages
     per arch. Thus from this perspective it seems that the support was
     never fully implemented.
  3) If any doubt remained, there is an old branch called 'tmp-multiarch'
     which has a couple of commits making progress with completing support,
     such as implementing the above missing pieces.

The above mentioned branch is 10 years old and can be considered abandoned.

It is not clear whether the original author ever intended to complete and
merge this; nor is it at all clear at what stage of completion it was at.

At any rate, imo it is not at all particularly useful to have extra code
and complexity in order to be able to cram multiple environments side by
side in one image, not when CDs/DVDs and even to some extent USB pen drives
are so cheap. And who really needs more than one environment so
desperately on just one such medium.

If this was not enough to justify removal, then there is also the fact that
the support that was implemented has become completely broken over the
years with scripts diverging in terms of the variable names the function
modifies such that they are incompatible with it.

A quick assessment of the state of this latter aspect:

  good:
   - grub-legacy uses the correct var names so is fine
   - memtest similarly good
   - installer_debian-installer looks okay

  questionable:
   - binary_linux-image uses the correct vars but might not select the
     right kernel and initrd files to copy (seems to copy all)

  bad:
   - grub-pc is making a redundant call, after functionality was moved
     to the loopback script
   - loopback is using the wrong vars (INITFS instead of DESTDIR +
     DESTDIR_INSTALL + DESTDIR_LIVE), plus is doing its own amd64+i686
     thing anyway, so the function call would achieve nothing anyway.
   - syslinux is also using the wrong var names so would not work with
     it and is not even making the necessary function call. Also the
     install paths are fixed in the hard coded cfg files anyway so this
     would need addressing with placeholders and sed replacement, but
     then it is not entirely clear how things should work with respect
     to install entries and multi-arch anyway, are we having multiple
     copies of the installer, one for each target arch and then multiple
     copies of the install menus, perhaps under different submenus?

So, this removes the artefacts of this never completed feature.

Gbp-Dch: Short
2020-03-13 13:56:21 +00:00
Lyndon Brown c54cc2c497 echo: really ensure log messages go to stdout
all echo helpers are used as logging functions with output to go to the
terminal. when used in functions that are designed to return a string
though the message printed would get incorrectly captured.

the previous fix done in e3a987d977 was
stupidly flawed; somehow my testing led me to mistakenly believe that
was adequate, but retesting proves that it was not.

here we create a new FD #3 linked to stdout to output the messages on,
which testing shows works as I had actually intended it.

e.g. here:
```
Foo () { if [ "$1" = "a" ]; then printf "foo\n"; else printf "error\n"; fi; }
```

we get:
```
~$ Foo a
foo
~$ Foo b
error
~$ XX="$(Foo a)"
~$ echo "${XX}"
foo
~$ XX="$(Foo b)"
~$ echo "${XX}"
error
```

and as demonstrated, "error" got incorrectly captured by in the variable

whereas here:
```
exec 3>&1
Foo () { if [ "$1" = "a" ]; then printf "foo\n"; else printf "error\n" >&3; fi; }
```

it is different in the last case:
```
~$ XX="$(Foo b)"
error
~$ echo "${XX}"

```

the error successfully makes it to the terminal, and the variable is an
empty string (with a newline automatically printed).

Gbp-Dch: Short
2020-03-13 13:50:35 +00:00
Luca Boccassi 015e6b65f3 Revert "Test for executables: replace 'which' with more robust 'command -v'"
This reverts commit 2d9ab1f7f8.

Causes test failure due to bashism.
2020-03-12 12:32:26 +00:00
johnraff 2d9ab1f7f8 Test for executables: replace 'which' with more robust 'command -v'
Instances of:
if [ $(which <command> ]
have been replaced with:
if command -v <command> >/dev/null
which is considered to be more robust in a range of environments.

scripts/build/chroot_archives: line 259:
	if [ "${LB_APT}" = "aptitude" ] && [ ! $(Chroot chroot "which aptitude") ]
has been left untouched because the chroot might require a more complex command
which would need more testing.

manpages/Makefile: line 42:
	@if [ ! -x "$$(which po4a 2>/dev/null)" ]; \
has been left untouched because I am not sufficiently familiar with makefiles.
2020-03-12 10:35:57 +00:00
Lyndon Brown 72a87fb70d fix error for default hdd|netboot images
commit f811656150 enabled the grub-efi
bootloader by default for amd64|i386 architectures, but failed to
recognise the this bootloader is not supported for hdd|netboot images.

this meants that if a user tried to build such an image without explicitly
specifying the bootloader, excluding grub-efi, their build would fail
with an error in the binary_grub-efi stage.

this fixes the problem by only enabling grub-efi by default on supported
image builds.
2020-03-12 10:33:13 +00:00