This might have worked by accident depending on the exact layout under the
LOCALDEBS directory (see previous commit for the big picture).
Let's focus on where input files for d-i should be stored:
<LOCALDEBS>/dists/<CODENAME>/local/debian-installer/binary-amd64/*.udeb
Those *.udeb end up being stored under the “local” component[1] (making it
clear they're not from usual components on a Debian mirror), but they are
referenced in Packages for the “main” component[2], possibly replacing a
different version found in the archive.
/dists/<CODENAME>/local/debian-installer/binary-<ARCH>/*.udeb [1]
/dists/<CODENAME>/main/debian-installer/binary-<ARCH>/Packages [2]
This makes it possible for d-i to find those *.udeb automatically, as it
uses a single Packages file.
Mimic what make_disc_trees.pl does, making it possible for hw-detect to
spot not only the /firmware/dep11 directory in netinst images, but also
on installation images with firmware added manually.
Rename pseudo-option to clarify it returns filenames and indices.
Keep generation disabled for bullseye (as an extra safeguard, even if we
have separate, per-release branches anyway): That's helpful information
but bullseye is a stable release, and changing even unofficial material
might not be the best idea. Be cautious by default, it's easy enough to
enable if desired.
The next step for make-firmware-image is to use generate_firmware_patterns
for each firmware package that was detected. That command requires DEP-11
metadata, so needs to know which Packages file each firmware package comes
from.
Instead of building long $pkgfiles and $bp_pkgfiles strings, and passing
them to catz, build lists instead and iterate over them. This means a few
more catz calls, but that also means being able to derive the DEP-11
directory from Packages paths.
There should be no functional changes.
Since that tool might be used without a caller that sources CONF.sh,
be explicit about the components in which firmware packages must be
searched:
- use "main contrib non-free" for bullseye;
- use "main non-free non-free-firmware" for bookworm and above.
non-free can go away once we have all we need in non-free-firmware.
Export some environment variables for generate_firmware_task (also
renaming SUITE to CODENAME).
Treat the output file argument specifically when it's “--list-filenames”:
instead of actually generating a task file with a list of packages (and a
few comments), output the Filename field for each firmware package
matching our criteria.
Move some status messages to STDERR accordingly.
This makes the “find suitable firmware packages” reusable, e.g. by
make-firmware-image.
We have a few fully-free firmware packages in main, and a few downloaders
(e.g. b43-installer and b43legacy-installer) in contrib; the latter might
not be really interesting without an alternative connectivity during the
installation process.
Regarding non-free vs. non-free-firmware, we started moving packages[1] from
the former to the latter so that we could build official installation images
based on main and non-free-firmware only.
1. https://lists.debian.org/debian-boot/2023/01/msg00150.html
At the moment, NON_FREE_COMPONENTS still lists both, but the plan is to move
to non-free-firmware only once all the packages we want are available there.
The relevant variables regarding components include:
- CONTRIB
- NONFREE
- EXTRANONFREE
- FORCE_FIRMWARE
This commit implements:
- using main all the time;
- using contrib if CONTRIB=1;
- using NONFREE_COMPONENTS if NONFREE=1 or EXTRANONFREE=1 or FORCE_FIRMWARE=1.
At the moment it seems the only existing caller is the top-level Makefile,
only when FORCE_FIRMWARE is set. But it seems to make sense to match what
tools/apt-selection does, to avoid any surprises.
There's no dep11 directory for bookworm's non-free-firmware component
for the time being, and we've just decided to be more cautious about
input files, which breaks the bookworm build.
Until the issue is fixed on the appstream.debian.org side, and until dak
can indeed import that directory into the Debian archive, use sid's
non-free-firmware dep11 directory instead of bookworm's.
Move the README.txt creation from generate_firmware_patterns to its
caller, so that it's done just once, and so that added blocks are
counted properly.
Initially it was interesting to pass “--package ALL” to create *.patterns
files for all packages found in the specified Components-* file(s), but
make_disc_trees.pl uses a call per package, so drop support for looping.
Remember whether modalias information was found for the requested package,
and only apply the firmware-sof-signed workaround when that's indeed the
requested package and when modalias information wasn't found (in case that
gets added to DEP-11 metadata at some point in the future). When using the
workaround, include a reminder in the logs.
The output directory's presence is the caller's responsibility; while
it might have been useful during development, actual builds require
make_disc_trees.pl to keep track of blocks getting added, so let it
take care of fulfilling prerequisites.
Without this, apt would try and download way too many indices, resulting
in 128 “Ign:” lines. With this, we're down to 4 “Ign:” lines (Release.gpg
and InRelease, twice).
Funnily enough, that changes apt's behaviour, which suddenly requires
uncompressed Packages to be present.
Some gzip calls have --no-name, which disables both the original name
and timestamp; use -n (disables original name) and -m (disables original
timestamp) for pigz, for all calls.
Some gzip calls have --no-name, which disables both the original name
and timestamp; use -n (disables original name) and -m (disables original
timestamp) for pigz, for all calls.
This makes it possible for hw-detect's install-firmware hook to enable the
right components if it installs firmware packages based on modalias
information. The components found in Contents-firmware indices are only
useful for firmware packages detected by the check-missing-firmware.sh
script.
All three columns could be aligned, but I'm not sure how many characters
should be allocated to the package filename. Alternatively, we could
scratch alignment altogether, and let interested users use column -t on
the file when debugging.
Otherwise failures would go unnoticed, logs would pretend firmware-b43-installer
(the first firmware package getting processed) is indeed installed there while
it's not. See next commit.
Update comments:
- Bump linux-image package to the latest found in bullseye.
- Include snd-soc-sof.*\.ko in the lookup.
- Delete the TODO: the list was built using snd-soc-sof*.ko already,
since a80f013b57.
List the desired non-free sections to use in NONFREE_COMPONENTS in
CONF.sh, default to "non-free non-free-firmware" by default until
things are all migrated.
The dependency tree created by parsing the output of "apt-cache depends" always
generates an array of array. The old representation of a simple dependency as
an hash is gone in favor of an OR with a single entry (so the hash is the only
element within the array).
So all the code in add_missing() that was dealing with the case of the hash
was never triggered.
I tweaked the OR case so that the generated log messages actually
mention "OR" only if it's a true OR dependency (i.e. with multiple
entries in the array).
Everywhere where we call add_missing() on Depends, we must do it also
on Recommends and Suggests based on $add_rec and $add_sug.
Tweak the code that is supposed to stop the recursion to use
the list that we have built but don't be strict about version
constraints on Recommends/Suggests since they can't always
be respected.
Closes: #601203
There's no point in passing them as parameters everywhere.
This changes the behaviour in some places:
1/ When we process the non-explicitly included packages (for COMPLETE=1),
we used to never take Recommends/Suggests into account, now we do. But
the only thing that will be changed is the ordering with some
Recommended packages being included before their depending package.
2/ When we process the EXTRANONFREE image, this one was always including
Recommends and Suggests, and it's now respecting the setting too.
It's required by debian-installer and cryptsetup >= 2.4.0 no longer
recommends it, so it ends up missing in the netinst image.
base-installer then fails to honor the apt-setup request and the
resulting system is unbootable because the initramfs doesn't contain
anything to unlock the encrypted partitions. :-(
Using the new generate_firmware_patterns script, generate a patterns
file in /firmware/dep11/<package>.patterns as we add a firmware
package onto media.
Deal with size changes as these are added/removed, and add them to the
md5sum.txt file on media too.
The source code doesn't use the MODULE_FIRMWARE() macro so the machinery
implemented in src:firmware-nonfree wouldn't help. Extract alias info
manually once, and make note to keep an eye on further updates (e.g. via
a crontab on d-i.debian.org).
The related code seems to have been useful a long time ago (the need to use
an RLE file and to reduce the number of colours to 16 has been dropped as of
syslinux version 3.62). So replace obsolete code with a simple cp command.
Signed-off-by: Wolfgang Schweer <wschweer@arcor.de>
tools/make_disc_trees.pl: Apply patch by Daniel Dickinson to create Packages
files for local/debian-installer. Closes: #523017
Signed-off-by: Holger Levsen <holger@layer-acht.org>
tools/start_new_disc: Apply patch by JH Chatenet to include local Release file
for debian-installer. Closes: #968671
Signed-off-by: Holger Levsen <holger@layer-acht.org>
Shift the extra_image helper function from boot-x86 to common.sh to
help.
Call extra_image with || true for now on boot-arm64 so we don't fail
builds straight away. Remove that || true later once we're happy
things work.
I was just bitten by this problem in Kali where the ISO was not
containing the expected packages because tools/sort_deps was accepting
"xserver-xorg-input-libinput-udeb" as a solution to
"xserver-xorg-input-all | xserver-xorg-input-driver" and the
udeb was among the package already selected for inclusion on the
ISO.
With this change, tools/sort_deps will look for udeb to solve
udeb dependencies and it will look for non-udeb to solve non-udeb
dependencies.
FTR, here's the sort_deps.log extract proving my assertion:
xserver-xorg Dep: (OR xserver-xorg-input-all xorgxrdp xserver-xorg-input-aiptek xserver-xorg-input-elographics xserver-xorg-input-evdev xserver-xorg-input-evdev-udeb xserver-xorg-input-joystick xserver-xorg-input-kbd xserver-xorg-input-libinput xserver-xorg-input-libinput-udeb xserver-xorg-input-mouse xserver-xorg-input-mtrack xserver-xorg-input-multitouch xserver-xorg-input-mutouch xserver-xorg-input-synaptics xserver-xorg-input-void xserver-xorg-input-wacom xserver-xorg-input-xwiimote ) soft_depend 1
xserver-xorg-input-all not included in a useful version, check_backports 0
xorgxrdp not included in a useful version, check_backports 0
xserver-xorg-input-aiptek not included in a useful version, check_backports 0
xserver-xorg-input-elographics not included in a useful version, check_backports 0
xserver-xorg-input-evdev not included in a useful version, check_backports 0
xserver-xorg-input-evdev-udeb not included in a useful version, check_backports 0
xserver-xorg-input-joystick not included in a useful version, check_backports 0
xserver-xorg-input-kbd not included in a useful version, check_backports 0
xserver-xorg-input-libinput not included in a useful version, check_backports 0
xserver-xorg-input-libinput-udeb is included already, acceptable version 0.29.0-1
Add the option to switch to format v2 which uses sha256 everywhere
instead of md5. We can't actually turn this on yet, need to wait for
users to upgrade a bit and get new v2-aware jigdo clients.
But this works as-is for generating v1 until we hit that switch - see
the new JIGOO_CHECKSUM option in CONF.sh
Cyril Brulebois
Steve McIntyre
Raphaël Hertzog
Samuel Thibault
bauen1
Wolfgang Schweer
Holger Levsen