Commit Graph

1375 Commits

Author SHA1 Message Date
Cyril Brulebois 7ce8d0c28f start_new_disc: remove /dists/local commodity symlink
When local packages are enabled (LOCAL=1), various things happen in the
target /dists directory. One of them is the creation of a /dists/local
symlink, pointing to (/dists/)<CODENAME>. This leads to a very confusing
situation, where “local” is initially about a component, but effectively
ends up looking like a distribution.

It is theoretically possible that cleaning this up might require some
follow-up fixes in other tools, but even that would be better than
keeping bear traps around.
2023-01-27 05:24:03 +01:00
Cyril Brulebois ae8490ba70 make_disc_trees.pl: rewrite base_components if needed
It's possible to set LOCAL=1 to enable support for local packages, which
automatically results in “local” being listed in /.disk/base_components,
which is used to build the --components parameter for debootstrap.

Users might have various sets of *.deb and/or *.udebs, be it over time,
or for various codenames, and it's possible for /dists/<CODENAME>/local
to be entirely missing in the generated image.

In this case, /.disk/base_components must not contain “local”, as that
would break debootstrap; this commit detects and fixes this possible
inconsistency.
2023-01-27 05:24:03 +01:00
Cyril Brulebois c24949141b Packages-gen: write compressed Packages indices again
This is a regression introduced while trying to catch up with apt's
expectations, in 2183089968c04c4765d2119d176e021431b26353: iterating over
Packages files is done by running the “zcat” utility on Packages.gz and
Packages.xz; replacing Packages.gz with an uncompressed Packages broke
iterating over Packages files referencing udebs.
2023-01-27 05:24:03 +01:00
Cyril Brulebois bfebeded70 make_disc_trees.pl: merge “local” udebs into “main” explicitly
This might have worked by accident depending on the exact layout under the
LOCALDEBS directory (see previous commit for the big picture).

Let's focus on where input files for d-i should be stored:

    <LOCALDEBS>/dists/<CODENAME>/local/debian-installer/binary-amd64/*.udeb

Those *.udeb end up being stored under the “local” component[1] (making it
clear they're not from usual components on a Debian mirror), but they are
referenced in Packages for the “main” component[2], possibly replacing a
different version found in the archive.

    /dists/<CODENAME>/local/debian-installer/binary-<ARCH>/*.udeb  [1]
    /dists/<CODENAME>/main/debian-installer/binary-<ARCH>/Packages [2]

This makes it possible for d-i to find those *.udeb automatically, as it
uses a single Packages file.
2023-01-27 05:24:03 +01:00
Cyril Brulebois 8bba3df865 make_disc_trees.pl: fix block calculation for --rollback
Per-package metadata files ($p.patterns and $p.component) go together,
and both needed to be accounted for.
2023-01-27 00:37:06 +01:00
Cyril Brulebois 5e39a81adb make_disc_trees.pl: fix block calculation for multi-arch
Per-package metadata files ($p.patterns and $p.component) go together,
and both needed to be accounted for.
2023-01-27 00:37:06 +01:00
Cyril Brulebois 95df8ab524 make-firmware-image: implement firmware/dep11 generation for bookworm and above
Mimic what make_disc_trees.pl does, making it possible for hw-detect to
spot not only the /firmware/dep11 directory in netinst images, but also
on installation images with firmware added manually.

Rename pseudo-option to clarify it returns filenames and indices.

Keep generation disabled for bullseye (as an extra safeguard, even if we
have separate, per-release branches anyway): That's helpful information
but bullseye is a stable release, and changing even unofficial material
might not be the best idea. Be cautious by default, it's easy enough to
enable if desired.
2023-01-26 23:22:40 +01:00
Cyril Brulebois 45db7e1582 generate_firmware_task: build lists of Packages files and iterate on them
The next step for make-firmware-image is to use generate_firmware_patterns
for each firmware package that was detected. That command requires DEP-11
metadata, so needs to know which Packages file each firmware package comes
from.

Instead of building long $pkgfiles and $bp_pkgfiles strings, and passing
them to catz, build lists instead and iterate over them. This means a few
more catz calls, but that also means being able to derive the DEP-11
directory from Packages paths.

There should be no functional changes.
2023-01-26 23:22:40 +01:00
Cyril Brulebois a2c722b735 make-firmware-image: use generate_firmware_task to find firmware
Since that tool might be used without a caller that sources CONF.sh,
be explicit about the components in which firmware packages must be
searched:
 - use "main contrib non-free" for bullseye;
 - use "main non-free non-free-firmware" for bookworm and above.

non-free can go away once we have all we need in non-free-firmware.

Export some environment variables for generate_firmware_task (also
renaming SUITE to CODENAME).
2023-01-26 23:22:40 +01:00
Cyril Brulebois bd13392570 generate_firmware_task: implement support for --list-filenames
Treat the output file argument specifically when it's “--list-filenames”:
instead of actually generating a task file with a list of packages (and a
few comments), output the Filename field for each firmware package
matching our criteria.

Move some status messages to STDERR accordingly.

This makes the “find suitable firmware packages” reusable, e.g. by
make-firmware-image.
2023-01-26 23:22:40 +01:00
Cyril Brulebois 90739b8cde generate_firmware_task: move code around
This is a preparatory step for the following commit, there should be no
functional changes.
2023-01-26 23:22:40 +01:00
Cyril Brulebois 14d2664459 generate_firmware_task: pick and choose components
We have a few fully-free firmware packages in main, and a few downloaders
(e.g. b43-installer and b43legacy-installer) in contrib; the latter might
not be really interesting without an alternative connectivity during the
installation process.

Regarding non-free vs. non-free-firmware, we started moving packages[1] from
the former to the latter so that we could build official installation images
based on main and non-free-firmware only.

 1. https://lists.debian.org/debian-boot/2023/01/msg00150.html

At the moment, NON_FREE_COMPONENTS still lists both, but the plan is to move
to non-free-firmware only once all the packages we want are available there.

The relevant variables regarding components include:
 - CONTRIB
 - NONFREE
 - EXTRANONFREE
 - FORCE_FIRMWARE

This commit implements:
 - using main all the time;
 - using contrib if CONTRIB=1;
 - using NONFREE_COMPONENTS if NONFREE=1 or EXTRANONFREE=1 or FORCE_FIRMWARE=1.

At the moment it seems the only existing caller is the top-level Makefile,
only when FORCE_FIRMWARE is set. But it seems to make sense to match what
tools/apt-selection does, to avoid any surprises.
2023-01-26 23:22:40 +01:00
Cyril Brulebois 4a31c04b81 make_disc_trees.pl: remove workaround for missing bookworm/non-free-firmware/dep11
This directory appeared in the archive earlier today.
2023-01-26 22:24:51 +01:00
Cyril Brulebois ee40a3bf23 make_disc_trees.pl: add workaround for missing bookworm/non-free-firmware/dep11
There's no dep11 directory for bookworm's non-free-firmware component
for the time being, and we've just decided to be more cautious about
input files, which breaks the bookworm build.

Until the issue is fixed on the appstream.debian.org side, and until dak
can indeed import that directory into the Debian archive, use sid's
non-free-firmware dep11 directory instead of bookworm's.
2023-01-25 01:38:34 +01:00
Cyril Brulebois 214e2dd88c make_disc_trees.pl: use write_file() for .component files as well 2023-01-25 01:38:34 +01:00
Cyril Brulebois 957ee269b4 make_disc_trees.pl: create /firmware/dep11/README.txt and count it
Move the README.txt creation from generate_firmware_patterns to its
caller, so that it's done just once, and so that added blocks are
counted properly.
2023-01-25 01:38:34 +01:00
Cyril Brulebois b56af4dbcc generate_firmware_patterns: rewrite to match caller's expectations
Initially it was interesting to pass “--package ALL” to create *.patterns
files for all packages found in the specified Components-* file(s), but
make_disc_trees.pl uses a call per package, so drop support for looping.

Remember whether modalias information was found for the requested package,
and only apply the firmware-sof-signed workaround when that's indeed the
requested package and when modalias information wasn't found (in case that
gets added to DEP-11 metadata at some point in the future). When using the
workaround, include a reminder in the logs.
2023-01-25 01:38:33 +01:00
Cyril Brulebois 9d95f65bd9 generate_firmware_patterns: stop creating the output directory
The output directory's presence is the caller's responsibility; while
it might have been useful during development, actual builds require
make_disc_trees.pl to keep track of blocks getting added, so let it
take care of fulfilling prerequisites.
2023-01-25 01:38:33 +01:00
Cyril Brulebois 2183089968 easy-build.sh: create top-level Release file for local packages
Without this, apt would try and download way too many indices, resulting
in 128 “Ign:” lines. With this, we're down to 4 “Ign:” lines (Release.gpg
and InRelease, twice).

Funnily enough, that changes apt's behaviour, which suddenly requires
uncompressed Packages to be present.
2023-01-23 20:25:21 +01:00
Cyril Brulebois cc53129b15 apt-selection: mark local repositories as trusted
Without this, apt won't like local repositories.
2023-01-23 19:30:42 +01:00
Cyril Brulebois f2a4a832a2 Replace all gzip --best calls with pigz -9nm ones.
Some gzip calls have --no-name, which disables both the original name
and timestamp; use -n (disables original name) and -m (disables original
timestamp) for pigz, for all calls.
2023-01-23 19:30:39 +01:00
Cyril Brulebois 14064e08a1 Replace all gzip -9 calls with pigz -9nm ones.
Some gzip calls have --no-name, which disables both the original name
and timestamp; use -n (disables original name) and -m (disables original
timestamp) for pigz, for all calls.
2023-01-23 19:30:35 +01:00
Cyril Brulebois f7925da6cd make_disc_trees.pl: store .component alongside .patterns for firmware packages
This makes it possible for hw-detect's install-firmware hook to enable the
right components if it installs firmware packages based on modalias
information. The components found in Contents-firmware indices are only
useful for firmware packages detected by the check-missing-firmware.sh
script.
2023-01-22 14:50:09 +01:00
Cyril Brulebois 4308e4666c make_disc_trees.pl: add components to Contents-firmware indices
All three columns could be aligned, but I'm not sure how many characters
should be allocated to the package filename. Alternatively, we could
scratch alignment altogether, and let interested users use column -t on
the file when debugging.
2023-01-22 14:49:06 +01:00
Cyril Brulebois bbda95c283 make_disc_trees.pl: fix misnamed $dist variable
$dist would suggest distribution, an alternative name for suite or
codename (!), let's make it clear we mean component here.
2023-01-21 16:40:56 +01:00
Cyril Brulebois 8c497447f9 generate_firmware_patterns: make sure dep11 metadata files are present
Otherwise we might be failing to generate firmware patterns for some
firmware packages. Being cautious about this lets us submit patches
against dak: https://salsa.debian.org/ftp-team/dak/-/merge_requests/267
2023-01-21 16:35:30 +01:00
Cyril Brulebois a2378ed25d make_disc_trees: create /firmware before adding symlinks there
Otherwise it would only be created after failing to create the first symlink.
See previous commit.
2023-01-20 16:48:55 +01:00
Cyril Brulebois b1cb4c43c2 make_disc_trees: error out if firmware symlink creation fails
Otherwise failures would go unnoticed, logs would pretend firmware-b43-installer
(the first firmware package getting processed) is indeed installed there while
it's not. See next commit.
2023-01-20 16:48:55 +01:00
Cyril Brulebois 5aa89ab269 Add leading / to paths in Contents-firmware indices.
With this change, hw-detect's check-missing-firmware.sh can use grep -f
with a file listing absolute paths to required firmware files.
2023-01-18 22:43:25 +01:00
Cyril Brulebois 81baccabbf Refresh references in comments. 2023-01-18 22:07:26 +01:00
Cyril Brulebois 51bf02a44c Avoid duplicates. 2023-01-18 21:59:31 +01:00
Cyril Brulebois 80d19dcb42 Refresh @sof_aliases for unstable.
Bump to the current linux-image ABI in unstable, and look at
snd-soc-*-sof modules in addition to the snd-soc-sof ones.
2023-01-18 21:46:54 +01:00
Cyril Brulebois fb2eac45eb Refresh @sof_aliases for bullseye.
Update comments:
 - Bump linux-image package to the latest found in bullseye.
 - Include snd-soc-sof.*\.ko in the lookup.
 - Delete the TODO: the list was built using snd-soc-sof*.ko already,
   since a80f013b57.
2023-01-18 21:45:41 +01:00
Cyril Brulebois 139041f332 Reindent without mixing tabs and spaces.
No functional changes.
2023-01-18 21:25:50 +01:00
Cyril Brulebois 723fdc9ee8 Delete TODO.
It was addressed in 897bf4754d.
2023-01-18 21:17:44 +01:00
Steve McIntyre f67691f2a5 Add grub-efi-ia32* to the amd64 package list
More work to support mixed-mode EFI systems
2022-12-29 20:49:56 +00:00
Steve McIntyre beccc326ca Drop verbosity 2022-12-29 04:02:20 +00:00
Steve McIntyre bba0f2116c Stop adding win32-loader stuff, it's dead 2022-12-29 03:57:01 +00:00
Steve McIntyre 8a8e623819 Add 32-bit EFI binaries to all amd64 images too
so we no longer need a multi-arch image for Bay Trail etc.
2022-12-29 03:54:45 +00:00
Steve McIntyre 2ef9e57ec0 Tweak the uuid search setup
Not all arches may have a grub.cfg, it seems. Don't fail there.
2022-11-28 10:24:00 +00:00
Steve McIntyre d99baf29c5 Backport the UUID grub fix to bullseye too 2022-11-26 01:45:49 +00:00
Steve McIntyre 6d1fdf2446 Change how we find the installer when booting via grub-efi
instead of looking for the .disk/info file, use .disk/id/$UUID
instead. Closes: #1024346, #1024720.
2022-11-26 01:45:26 +00:00
Steve McIntyre 32291d88a3 Add a Contents-firmware file
to help with looking up firmware packages in d-i
2022-10-09 01:44:37 +01:00
Steve McIntyre e57aa6fa72 Add non-free-firmware support
List the desired non-free sections to use in NONFREE_COMPONENTS in
CONF.sh, default to "non-free non-free-firmware" by default until
things are all migrated.
2022-10-08 17:48:59 +01:00
Steve McIntyre 8c52820262 Be more verbose when leaving packages out due to size limits 2022-10-07 15:55:12 +01:00
Raphaël Hertzog 805b36d16b Remove tools/sort_deps_edu
Its only purpose was to try to implement the recursive lookup that got
implemented in ca9ac8deac
2022-08-09 19:39:46 +02:00
Raphaël Hertzog e77ade6033 Remove dead code in sort_deps and improve log messages
The dependency tree created by parsing the output of "apt-cache depends" always
generates an array of array. The old representation of a simple dependency as
an hash is gone in favor of an OR with a single entry (so the hash is the only
element within the array).

So all the code in add_missing() that was dealing with the case of the hash
was never triggered.

I tweaked the OR case so that the generated log messages actually
mention "OR" only if it's a true OR dependency (i.e. with multiple
entries in the array).
2022-08-09 19:39:15 +02:00
Raphaël Hertzog ca9ac8deac Implement recursive lookup of Recommends and Suggests
Everywhere where we call add_missing() on Depends, we must do it also
on Recommends and Suggests based on $add_rec and $add_sug.

Tweak the code that is supposed to stop the recursion to use
the list that we have built but don't be strict about version
constraints on Recommends/Suggests since they can't always
be respected.

Closes: #601203
2022-08-09 19:37:48 +02:00
Steve McIntyre 3a13ddb48a Revert "Implement recursive lookup of Recommends and Suggests"
This reverts commit 8513b237af.

This change doesn't seem ready yet - running a weekly build failed
with massive memory usage.
2022-08-08 20:23:13 +01:00
Steve McIntyre 8c86a9b3ed Revert "Remove tools/sort_deps_edu"
This reverts commit cc4e1fa450.
2022-08-08 20:22:44 +01:00