apt-ftparchive is not able to differentiate between .deb and .udeb so
we have to install them in different pool directories so that we can
regenerate the Packages files without having the .udeb show up
unexpectedly.
Since binary_package-lists can overwrite the Packages files generated
in installer_debian-installer we have to ensure that it also updates
the Release file created formerly.
Ideally we should find a way to avoid the duplication of this logic.
Gbp-Dch: Full
In the binary stage, chroot is restored from cache/bootstrap, which contains
neither custom repo lists, nor keys. If local packages are present,
chroot_archives will call 'Apt chroot update' after adding custom repo lists
without adding keys. apt-get will then fail instead of warn as of apt version
1.5.
Closes: #941691
This makes it possible to build an image against a first distribution
(--distribution-chroot) and have the resulting image point to another
distribution (--distribution-binary). We can use this to build against a
snapshot and have the result use the original distribution that was
snapshotted.
Closes: #888507
Before Stretch there was an special amd64 kernel in the i386 arch repo.
So if you wanted to install an amd64 kernel alongside an i386 system
you did not need an additional arch repo.
Debian added multiarch support. That way you can install library packages
from multiple architectures on the same machine.
So there is no longer a need for having an amd64 kernel in i386 arch repo.
You can add an amd64 arch repo to an i386 arch system and fetch the amd64
kernel from the am64 arch repo.
live-build can be setup to use several linux kernel flavours in a single
image.
So in the days previous to this patch you could issue:
lb config --linux-flavours "486 amd64"
to use both 486 and amd64 kernel flavours.
Adding additional arch support to linux flavours poses two problems:
* Packages need to have its arch suffix (e.g. amd64:amd64).
If the suffix is not there apt-get insists on search amd64 kernel
package on i386 arch repo and, of course, fails to find it.
* The rest of the code which handles labels (bootloader config files)
or installed filenames (kernel images themselves) do not use the arch suffix.
This patch adds foreign architecture package support to
linux kernel flavours having taken those problems into account.
Practical example usage: i386 system and extra amd64 kernel.
First add amd64 foreign architecture in your i386 system
thanks to:
dpkg --add-architecture amd64
apt-get update
.
Finally enable amd64 kernel from amd64 arch alongside the
i386 system's 686 kernel thanks to:
lb config --architectures i386 --linux-flavours "686 amd64:amd64"
Turns out gcd works fine after adding /boot/grub/grub.cfg in the img,
as that's the path that gets hardcoded, and adding the EFI/debian/
grub.cfg was not necessary, so remove it.
For secured boot in binary_grub-efi, the gcdx64.efi.signed is
the boot loader for removable device, like CD or USB flash drive,
while grubx64.efi.signed is for hard drive. Therefore for live system,
use gcdx64.efi.signed for amd64 and gcdaa64.efi.signed for arm64.
With the merged /usr feature now enabled in debootstrap, we have
/bin and /lib which are symlinks to /usr/bin and /usr/lib. If
the live-build configuration provides files in
config/includes.chroot/lib/ or config/includes.chroot/bin/ then
lb chroot_includes will replace the /lib or /bin symlink with real
directories having only the content provided in chroot.includes and
problems will follow... the build will usually fail later with a
cryptic error message (for example a hook failing to execute
/usr/bin/env).
Work around the issue by creating a tarball of the files to install
and by unpacking that tarball from within the chroot with appropriate
options.
When using Secure Boot, grub2 as built by Debian will now load a config
file from EFI/$VENDOR instead of having EFI/debian hardcoded.
$VENDOR comes from dpkg-vendor or from the user building grub2.
The vendor string is stored in the control metadata as Efi-Vendor, so
retrieve it when building the EFI image.
Only gnupgv is part of the deboostrap set these days, but apt-key needs
the full gpg (with gpg-agent) which is just a recommends.
Instead just drop the key with an .asc suffix in /etc/apt/trusted.gpg.d
which is supported since apt version 1.4
live-build supports preseeding configuration, but the configuration has
to be installed after bootstrapping since it needs the debconf tools to
be applied. But packages that have already been installed and configured
in the bootstrap step will then ignore those preseeded configs.
After applying each preseed file, parse the package list and manually
reconfigure the relevant package(s) so that the configuration will be
applied.
The binary_onie script works on the host, not in the chroot (if used),
so don't check that the required packages for the script are installed
in the chroot (if present) as they won't be useful.
Check instead on the host.
Instead of hard-coding the decompression and compression formats,
detect them at runtime.
Install the required dependencies as well - they were mistakenly left
out.
Gbp-Dch: Ignore
The apt-secure option does not work anymore when building a sid image,
as with apt 1.6 the existing options are no longer enough to get apt
to accept an unsigned repository, which is necessary when using a
local cached repository (offline build).
Pass Acquire::AllowInsecureRepositories "true"; together with the
other options when --apt-secure false is used to fix the issue.
Open Network Install Environment is an open image format used by
networking vendor to ship a standardised image for networking white
box switches.
ONIE hardware takes this image at boot and a script to chain load
into the final environment via kexec. We can support Debian and
derivatives on such systems by packing an ISO which then gets
unpacked, kexec'ed and live-booted.
A base ONIE system can be tested in QEMU by building a VM following
these instrunctions:
https://github.com/opencomputeproject/onie/blob/master/machine/kvm_x86_64/INSTALL
Once built, boot onie-recovery-x86_64-kvm_x86_64-r0.iso in QEMU/libvirt
and on the console there will be the terminal prompt. Check the IP
assigned by libvirt and then scp the live image (ssh access is enabled
as root without password...). Then the .bin can be booted with:
ONIE-RECOVERY:/ # onie-nos-install /tmp/live.hybrid.iso-ONIE.bin
The implementation is inspired by ONIE's own scripts that can be found
at:
https://github.com/opencomputeproject/onie/blob/master/contrib/debian-iso/cook-bits.sh
A new option, --onie (false by default) can be set to true to enable
building this new format in addition to an ISO.
An additional option, --onie-kernel-cmdline can be used to specify
additional options that the ONIE system should use when kexec'ing the
final image.
Note that only iso or hybrid-iso formats are supported.
For more information about the ONIE ecosystem see:
http://onie.org
Signed-off-by: Erik Ziegenbalg <eziegenb@Brocade.com>
Signed-off-by: Luca Boccassi <bluca@debian.org>
The Tianocore reference UEFI implementation, used for example by Qemu,
wants the EFI directory name to be uppercase in the fat32 partition
when Secure Boot is enabled, and will fail to load otherwise.
Support for UEFI Secure Boot is modelled after how it currently works
in Ubuntu and on how it is going to work on Debian.
A minimal bootloader, shim, is used as the first-stage and it then
loads grub. Both have to be signed.
shim-signed is already available in Debian so the filenames are
already established, and the grub2 repository and packaging is common
between the 2 distros so we can already be reasonably sure of what it
is going to be.
So if both are available, copy /usr/lib/shim/shim[x64|aa64].efi.signed
as boot[x64|aa64].efi so that UEFI loads it first, and copy
/usr/lib/grub/[x86_64|arm64]-efi-signed/grub[x64|aa64].efi.signed as
grub[x64|aa64].efi.
This grub2 EFI monolithic image is currently hard-coded in grub2's
repository to look for a config file in efi/debian, so make a copy
of the previously added minimal grub.cfg that loads the real one in
that directory in both the fat32 and ISO 9660 partitions.
The new option --uefi-secure-boot can be set to auto (default,
enable or disable.
In auto, the lack of the signed EFI binaries is intentionally left as a
soft failure - live-build will simply fallback to using the locally
generated non-signed grub2 monolithic EFI binary as the only
bootloader. Given the difficulties surrounding the Secure Boot
signing infrastructure this approach gives the most flexibility and
makes sure things will "just work" once the packages are available,
without the need to change anything in the configuration.
This will also greatly help downstream distributions and users who
want to do self-signing.
The enable or disable options work as expected.
Closes: #821084
On some UEFI implementations, like the AMI found in the Supermicro
X10SDV-TP8F development board, the fat32 partition will be loaded
first and so Grub will set it the root, and then drop to the console
as it cannot find any config on it.
Add a minimal grub.cfg that allows Grub to find the main config on
the ISO 9660 partition and load it.
Closes: #892406
Machines tend to become unresponsive during the mksquashfs step.
Avoid this by lowering the priority of the process.
Thanks: Ronny Standtke for the patch.
Closes: #867539
Commit a15b579652 (#775989) dropped an early exit from the
chroot_archives remove step in case the parent mirror chroot and binary
parameters are the same and introduced a regression, as with the
following live-build now fails when the parent mirror is using a file:/
local apt repository (for example when the build worker is offline and
uses a pre-built cache of packages).
Example config:
lb config --mirror-bootstrap "file:/pkgs" \
--mirror-chroot "file:/pkgs/" \
--mirror-binary "file:/pkgs" \
--parent-mirror-bootstrap "file:/pkgs" \
--parent-mirror-chroot "file:/pkgs/" \
--parent-mirror-binary "file:/pkgs" \
...
with /pkgs being a directory with the packages for the installation and
the apt metadata (Packages/Sources/Release).
The problem is that, with such a setup, the /pkgs directory is bind
mounted inside the chroot as an optimisation in the install step,
and umounted as one of the first actions in the remove step for
chroot_archives.
Before that fix, the script terminated immediately. But now it
progresses and at the end it tries to run apt update inside the chroot
which will fail since the repository directory has been umounted, and
thus the packages and the apt metadata are no longer available, while
still being listed in /etc/apt/sources.list.
The proposed solution is to umount the local directory at the end of
the remove step, rather than at the beginning.
Closes: #891206
Now grub.cfg shows all the kernel options. Before this patch when you
had more than two kernels it only showed the auto option.
Signed-off-by: Raphaël Hertzog <hertzog@debian.org>
To generate an hdd image, binary_hdd first estimates the needed size of
the image using du. By default, when du finds multiple hardlinked copies
of a file, it counts them only once. However, when the target filesystem
is FAT, which does not support hardlinks, these files will take up more
space when finally copying the contents, breaking the build:
P: Copying binary contents into image...
cp: error writing 'chroot/binary.tmp/live/initrd.img-4.9.0-3-amd64': No space left on device
cp: error writing 'chroot/binary.tmp/efi/boot/bootx64.efi': No space left on device
cp: error writing 'chroot/binary.tmp/efi/boot/bootia32.efi': No space left on device
cp: cannot create directory 'chroot/binary.tmp/boot/grub': No space left on device
cp: cannot create directory 'chroot/binary.tmp/isolinux': No space left on device
To fix this, pass --count-links to du when the target is FAT, to make
the space estimation correct.
This problem is exposed by commit 9c974b26b (Instead of renaming kernel
for syslinux, create hardlinks), which might need to be separately fixed
(to not waste space on FAT targets), but binary_hdd should at least
handle hardlinks more gracefully.
Since commit fdc9250bc (Changing package dependency checks within chroot
to work outside as well), Check_package automatically checks for
LB_BUILD_WITH_CHROOT and works inside as well as outside of the chroot,
so no need to check LB_BUILD_WITH_CHROOT before calling them.
Install_package and Remove_package are just a no-op when building
without chroot, so they can also be called unconditionally.
Restore_cache and Save_cache do not check LB_BUILD_WITH_CHROOT but it
it should not hurt to call them when not needed (which already happened
in some cases).
This commit makes all Check_package calls unconditional on
LB_BUILD_WITH_CHROOT.
For binary_syslinux, this fixes the check (which used outdated paths
outside the chroot since 7b6dfd9d1), for binary_grub-efi,
binary_package-lists and chroot_package-lists this simplifies the code
(but also causes the check to become package-based instead of file-based
on apt-based systems), and for binary_loadlin and binary_win32-loader
this adds the check outside the chroot which was previously missing.
* Use only long kernel names.
* Put advanced options in a submenu.
* Use distro-agnostic labels.
* Don't generate entries with kernel version when we have a single
version.
This option lets you use an alternate bootstrap script when running
debootstrap. Thanks to Sjoerd Simons <sjoerd@debian.org> for the initial
patch.
Closes: #790033
Commit e24e4b in debootstrap fixed setup_available to work in the
--foreign case (iotw at the second stage). Unfortunately this breaks
things if components aren't passed to the second stage _and_ your main
component isn't called main.
To fix this, pass --components to both the first and second stage
debootstrap when needed.
Signed-off-by: Sjoerd Simons <sjoerd.simons@collabora.co.uk>
Up to now we created a temporary GPG key that we registered with apt-key
but with the switch to GnuPG 2 by default, this code broke. Now we stop
doing that but we add the “trusted=yes“ attribute in sources.list so
that APT knows that the repository can be trusted even if it's unsigned.
Signed-off-by: Raphaël Hertzog <hertzog@debian.org>
This work is based on debian-cd team work and uses,
as much as possible, the same mkisofs options
than the Debian Installation CD disk does.
It assumes that /boot/grub/grub.cfg (and other design items)
is generated by: binary_loopback_cfg .
It relies on efi-image and grub-cpmodules being setup
as build scripts on live-build package.
In the future event of these two files being moved
to a binary package (they are originally from:
src: live-installer) the binary_grub-efi script would have
to be rewritten to take the new paths into account.
These two scripts simplify the creation of efi images based on grub-efi.
I have decided to simply steal them. If I had to include them thanks to a source package that would have mean that an src repo would have to be defined by default.
TODO: Ask in a bug a RFE so that these two scripts are put into a binary that could be consumed by both live-installer and live-build packages.
The binary parts of grub-pc are left for the original binary_grub-pc.
As a consequence both /boot/grub/grub.cfg and /boot/grub/loopback.cfg files will be present in any Debian Live CD.
This might be useful to be reused from binary_grub-* bootloaders.
* Added: functions/bootloaders.sh . This file adds bootloader functions that are heavily used in efi scenarios where a bootloader can act as a first or an extra bootloader.
Since the introduction of the new switch:
--bootloaders
you can setup it like this:
--bootloaders=syslinux,grub-efi
.
This means that syslinux is the first bootloader and grub-efi is the extra bootloader.
* Added new bootloader functions: Check_Non_First_Bootloader and Check_Non_Extra_Bootloader.
These functions let each one of the bootloaders abort the build because
they cannot perform a role either as a first bootloader or as an extra bootloader.
* Added bootloader functions: Check_First_Bootloader_Role, Check_Extra_Bootloader_Role and Check_Any_Bootloader_Role
These functions let bootloaders to force their default role in a single line.
At the same time many binary bootloaders were rewritten to make use of the new bootloader role functions explained above.
These roles were enforced:
binary_grub-legacy : First bootloader
binary_grub-pc : Either first or extra bootloader
binary_syslinux : Either first or extra bootloader
If a bootloader is tried to be used in a role that it's not meant to be used then the build fails because that might lead to a non-bootable system.
The fix in a294a46fb9 was not enough.
This should finally resolve the problem when a package list ends
up empty (most notably due to #if evaluating to false).
Sponsored-By: Offensive Security
Some BIOSes dont't boot from partitions starting at sector 1024.
Some are even more peculiar and only start from sector 63.
This patch adds an option for the binary_hdd target to manually
configure the partition start.
Development details
--------------------
* This patch has been based originally on: binary_grub2 . It has been
improved thanks to some binary_syslinux bits.
* This patch ensures that binary_loopback_cfg needs is run
before binary_syslinux is run.
The reason is that it reuses some code from binary_syslinux to avoid
problems when binary_syslinux renames the kernel filenames.
* This patch already supports CPU detection
* I haven't tested all the possible scenarios for the script (with the
latest version). When amd64 and 486 Gnu/Linux flavours is used and the
bootloader is syslinux it works ok.
* I have not implemented a disable switch for not generating it.
Usually you always want loopback.cfg to be there.
* Compared to binary_grub2 script I have removed the installation
entries because I did not see any of them in binary_syslinux.
How to test
-----------
These are some steps to easily test if Looback cfg support is working ok.
0) We assume you have generated an iso
1) Make sure you have a partition that Grub understands. Plain ext4 or
vfat should do it.
2) Create directory: /boot/boot-isos/
3) Put the iso file into that directory making sure it has an ISO or iso
extension.
4) Setup your computer to boot from cdrom and use: Super Grub2 Disk
2.00s2 (Hybrid version recommended)
5) Choose Boot manually...
6) Choose Bootable ISOs (in /boot- ... )
7) Choose (the detected) GRUB Loopback Config
(hdN,msodsN)/boot/boot-ios/name-of-the.iso
8) You will be presented your loopback.cfg. Choose anyone of the entries
(unless it does not match your cpu architecture of course).
9) You should boot into your Debian Live without problems (thanks to
findiso boot parametre).
If you ever wanted to test from your grub2 installation instead from
Super Grub2 Disk check: http://www.supergrubdisk.org/wiki/Loopback.cfg
for an example.
Note: OLDIFS use makes IFS to be reset to "" instead to it being unset.
Either we need to detect if old IFS was unset to unset it
or we need a proper way of setting it as a local variable.
Even more IFS it's not currently used in
Check_package (which it's called from: binary_hdd).
we should have a clean way of resetting/unsetting IFS when calling Check_package.
The other approach it's to explicitly define IFS with its default value in the
places inside live-build code where we implicitly suppose that it's going to have
its default value.
Future live-build versions will still allow to use casper,
but its configuration will be done differently by a custom
config tree, rather than embedded and maintenance intensive
code in live-build itself.
Future live-build versions will still allow to use casper,
but its configuration will be done differently by hooks
in the config tree, rather than embedded and maintenance intensive
code in live-build itself.
debootstrap is the official tool to bootstrap debian,
cdebootstrap has had the one or other bug making it
broken for times during the release cycles.
The extra effort of supporting both debootstrap
and cdebootstrap is hardly worth it since the bootstrap
stage is cached anyway.
If a local repository path is given as a mirror URL lets bind it into the
chroot. The local repository will be unmounted while processing "remove" or
latest by the exit function.
The user might not have created any package list or it might have
created a packages list that end up being empty due to various #if tests.
We should not fail in those cases.
extlinux configuration for the bootable image is installed to
/boot/extlinux, matching the extlinux-install script in Debian.
From the configuration point of view it's still called "syslinux",
the same as the rest of the syslinux family (pxelinux, isolinux etc.)
* Add function to output unique list of foreign architectures from an
expanded package list.
* If foreign architectures are detected, add unique architectures
to dpkg and update apt.
* This requires users to explicitly list at least _one_ package of a
foreign architecture in their package list (e.g. foo:arch) for any other
foreign arch dependencies to be handled appropriately.
In the past, we had lb config for both creating a configuration directory
and updating settings in there. With the rewrite in Python, we're now
changing this finally to the more sane 'init' (create initial 'sample'
configuration directory tree, taking most arguments that lb config did)
and 'config' which will only be a get/set program to work on top of
an already initialized configuration tree.
Otherwise users opting for XFS and JFS filesystems and installing without
network won't have them and the (first) boot can be interrupted due to
this.
Sponsored-by: Offensive Security
It is important for derivatives that the bootstrap cache is setup completely,
it is re-used in several places and thus cannot consist of the parent
distribution only.
Until there's a supportable way to bootstrap from multiple archives at the same time
(and multistrap doesn't work for that as discussed in the past), we're going to run
a lot of duplicated code here to massage the apt sources for the time being.
Raphaël Hertzog
John Estabrook
Ronny Standtke
Roland Clobus
adrian15
Adrian Gibanel Lopez
Luca Boccassi
Steven Shiau
Marcel Partap
Raphaël Hertzog
Rohan Garg
Balint Reczey
Matthijs Kooijman
Sjoerd Simons
Sophie Brun
Kristian Klausen
Ben Armstrong
Maximilian Mehnert
Carlos Zuferri
Daniel Baumann
Daniel Reichelt
Raphaël Hertzog
jnqnfe
Jan Blunck
Simon McVittie
Kiel Christofferson
Jan Blunck