Almost all packages initially identified as interesting for us[1] have
been moved to non-free-firmware. An updated list of packages has been
published[2], which can be summed up this way:
- amd64-microcode and intel-microcode would require specific code
anyway (#1029804);
- zd1211-firmware shows inconsistencies (archive vs. repository).
1. https://lists.debian.org/debian-boot/2023/01/msg00150.html
2. https://lists.debian.org/debian-boot/2023/01/msg00235.html
Therefore, stop looking for non-free firmware packages in non-free,
and concentrate on non-free-firmware only.
Components are ordered “main contrib non-free” almost everywhere, let's
use the standard order here too. There should be no functional changes.
Note: This controls the sources.list file used during the build, but
doesn't control “deb cdrom:” lines in sources.list during or after
installation, which are the result of running apt-cdrom (which sorts
components alphabetically, see its pkgCdrom::ReduceSourcelist function).
WARNING: This is an attempt at making it easier for users to enable
support for local packages. This branch is really believed to make
the whole situation easier and clearer on the long run, but it might
introduce some regressions, or at least behavorial changes depending
on existing configurations.
In particular, users might have a hierarchy under LOCALDEBS that's
different from the one getting documented in eb6d140f48, possibly with
a top-level local/ (see also bfebeded70 and 7ce8d0c28f).
It's advised to move files around to match the documented layout, and
to report any regressions that would be spotted: many tests where run,
trying to torture this patch series as much as possible, imagining
borderline cases, but in the end, that's not a feature that's getting
fully battle-tested since it's obviously not used for official builds.
When local packages are enabled (LOCAL=1), various things happen in the
target /dists directory. One of them is the creation of a /dists/local
symlink, pointing to (/dists/)<CODENAME>. This leads to a very confusing
situation, where “local” is initially about a component, but effectively
ends up looking like a distribution.
It is theoretically possible that cleaning this up might require some
follow-up fixes in other tools, but even that would be better than
keeping bear traps around.
It's possible to set LOCAL=1 to enable support for local packages, which
automatically results in “local” being listed in /.disk/base_components,
which is used to build the --components parameter for debootstrap.
Users might have various sets of *.deb and/or *.udebs, be it over time,
or for various codenames, and it's possible for /dists/<CODENAME>/local
to be entirely missing in the generated image.
In this case, /.disk/base_components must not contain “local”, as that
would break debootstrap; this commit detects and fixes this possible
inconsistency.
This is a regression introduced while trying to catch up with apt's
expectations, in 2183089968c04c4765d2119d176e021431b26353: iterating over
Packages files is done by running the “zcat” utility on Packages.gz and
Packages.xz; replacing Packages.gz with an uncompressed Packages broke
iterating over Packages files referencing udebs.
This might have worked by accident depending on the exact layout under the
LOCALDEBS directory (see previous commit for the big picture).
Let's focus on where input files for d-i should be stored:
<LOCALDEBS>/dists/<CODENAME>/local/debian-installer/binary-amd64/*.udeb
Those *.udeb end up being stored under the “local” component[1] (making it
clear they're not from usual components on a Debian mirror), but they are
referenced in Packages for the “main” component[2], possibly replacing a
different version found in the archive.
/dists/<CODENAME>/local/debian-installer/binary-<ARCH>/*.udeb [1]
/dists/<CODENAME>/main/debian-installer/binary-<ARCH>/Packages [2]
This makes it possible for d-i to find those *.udeb automatically, as it
uses a single Packages file.
Getting support for local packages (*.deb and/or *.udeb) to work
consistently has been a gigantic time sink, never again!
Document the expected layout as precisely as possible, so that
unsuspecting users have a better chance of getting things right!
It looks strange to have NONFREE unset and CONTRIB=1 in the default config,
since packages in contrib are usually there because they depend on non-free
stuff.
Mimic what make_disc_trees.pl does, making it possible for hw-detect to
spot not only the /firmware/dep11 directory in netinst images, but also
on installation images with firmware added manually.
Rename pseudo-option to clarify it returns filenames and indices.
Keep generation disabled for bullseye (as an extra safeguard, even if we
have separate, per-release branches anyway): That's helpful information
but bullseye is a stable release, and changing even unofficial material
might not be the best idea. Be cautious by default, it's easy enough to
enable if desired.
The next step for make-firmware-image is to use generate_firmware_patterns
for each firmware package that was detected. That command requires DEP-11
metadata, so needs to know which Packages file each firmware package comes
from.
Instead of building long $pkgfiles and $bp_pkgfiles strings, and passing
them to catz, build lists instead and iterate over them. This means a few
more catz calls, but that also means being able to derive the DEP-11
directory from Packages paths.
There should be no functional changes.
Since that tool might be used without a caller that sources CONF.sh,
be explicit about the components in which firmware packages must be
searched:
- use "main contrib non-free" for bullseye;
- use "main non-free non-free-firmware" for bookworm and above.
non-free can go away once we have all we need in non-free-firmware.
Export some environment variables for generate_firmware_task (also
renaming SUITE to CODENAME).
Treat the output file argument specifically when it's “--list-filenames”:
instead of actually generating a task file with a list of packages (and a
few comments), output the Filename field for each firmware package
matching our criteria.
Move some status messages to STDERR accordingly.
This makes the “find suitable firmware packages” reusable, e.g. by
make-firmware-image.
We have a few fully-free firmware packages in main, and a few downloaders
(e.g. b43-installer and b43legacy-installer) in contrib; the latter might
not be really interesting without an alternative connectivity during the
installation process.
Regarding non-free vs. non-free-firmware, we started moving packages[1] from
the former to the latter so that we could build official installation images
based on main and non-free-firmware only.
1. https://lists.debian.org/debian-boot/2023/01/msg00150.html
At the moment, NON_FREE_COMPONENTS still lists both, but the plan is to move
to non-free-firmware only once all the packages we want are available there.
The relevant variables regarding components include:
- CONTRIB
- NONFREE
- EXTRANONFREE
- FORCE_FIRMWARE
This commit implements:
- using main all the time;
- using contrib if CONTRIB=1;
- using NONFREE_COMPONENTS if NONFREE=1 or EXTRANONFREE=1 or FORCE_FIRMWARE=1.
At the moment it seems the only existing caller is the top-level Makefile,
only when FORCE_FIRMWARE is set. But it seems to make sense to match what
tools/apt-selection does, to avoid any surprises.
There's no dep11 directory for bookworm's non-free-firmware component
for the time being, and we've just decided to be more cautious about
input files, which breaks the bookworm build.
Until the issue is fixed on the appstream.debian.org side, and until dak
can indeed import that directory into the Debian archive, use sid's
non-free-firmware dep11 directory instead of bookworm's.
Move the README.txt creation from generate_firmware_patterns to its
caller, so that it's done just once, and so that added blocks are
counted properly.
Initially it was interesting to pass “--package ALL” to create *.patterns
files for all packages found in the specified Components-* file(s), but
make_disc_trees.pl uses a call per package, so drop support for looping.
Remember whether modalias information was found for the requested package,
and only apply the firmware-sof-signed workaround when that's indeed the
requested package and when modalias information wasn't found (in case that
gets added to DEP-11 metadata at some point in the future). When using the
workaround, include a reminder in the logs.
The output directory's presence is the caller's responsibility; while
it might have been useful during development, actual builds require
make_disc_trees.pl to keep track of blocks getting added, so let it
take care of fulfilling prerequisites.
Without this, apt would try and download way too many indices, resulting
in 128 “Ign:” lines. With this, we're down to 4 “Ign:” lines (Release.gpg
and InRelease, twice).
Funnily enough, that changes apt's behaviour, which suddenly requires
uncompressed Packages to be present.
Some gzip calls have --no-name, which disables both the original name
and timestamp; use -n (disables original name) and -m (disables original
timestamp) for pigz, for all calls.
Some gzip calls have --no-name, which disables both the original name
and timestamp; use -n (disables original name) and -m (disables original
timestamp) for pigz, for all calls.
This makes it possible for hw-detect's install-firmware hook to enable the
right components if it installs firmware packages based on modalias
information. The components found in Contents-firmware indices are only
useful for firmware packages detected by the check-missing-firmware.sh
script.
All three columns could be aligned, but I'm not sure how many characters
should be allocated to the package filename. Alternatively, we could
scratch alignment altogether, and let interested users use column -t on
the file when debugging.
Otherwise failures would go unnoticed, logs would pretend firmware-b43-installer
(the first firmware package getting processed) is indeed installed there while
it's not. See next commit.
Update comments:
- Bump linux-image package to the latest found in bullseye.
- Include snd-soc-sof.*\.ko in the lookup.
- Delete the TODO: the list was built using snd-soc-sof*.ko already,
since a80f013b57.
Cyril Brulebois
Samuel Thibault