Commit Graph

2562 Commits

Author SHA1 Message Date
Steve McIntyre 20dd2183d2 Remove long-dead docs reference to need-complete-mirror 2023-03-26 23:10:17 +01:00
Steve McIntyre ae8cf8b946 Kill loadlin and the /tools dir 2023-03-26 23:10:03 +01:00
Steve McIntyre def9376769 link.pl: make failures fatal
Don't just log things, we should stop hard...
2023-03-12 18:44:18 +00:00
Steve McIntyre da2e501700 Add a changelog entry for the hardlink change 2023-03-12 18:43:56 +00:00
James Addison d2d51c9179 firmware: use hard links rather than symlinks
When creating on-disc firmware links, use the same link creation logic as archive-area links

Closes #1031696
2023-03-12 18:39:40 +00:00
Cyril Brulebois 5aebb6794a releasing version 3.2.0 2023-02-24 17:04:20 +01:00
Steve McIntyre feb344b3c2 Add iucode-tool to the list of base packages wanted for amd64/i386
so we will have it available for installing CPU microcode.
2023-02-23 23:51:28 +00:00
Steve McIntyre 7131e37390 Add new config option DISABLE_UEFI_32 to control mixed-mode EFI
on x86 builds. Closes: #1031424
2023-02-23 21:46:37 +00:00
Steve McIntyre dcbf4ea088 make-firmware-image: Add a list file
to show what's included in the firmware archives
2023-02-20 12:29:05 +00:00
Cyril Brulebois 7eea34f26f make-firmware-image: fix missing Contents-firmware index (Closes: #1031598) 2023-02-19 03:56:45 +01:00
Steve McIntyre aebf4f0556 make_disc_trees.pl: fix generation of Contents-firmware
The regexp didn't match symlinks in .deb files, let's fix that
2023-02-18 23:33:02 +00:00
Steve McIntyre c5ff679857 Fix noisy error in make_disc_trees.pl
firmware/dep11 only exists on the first image, so don't run
md5_files_for_md5sum on it if doesn't exist
2023-02-18 22:12:42 +00:00
Steve McIntyre ecb6ec4a0a Tweak firmware-image build scripts
Add more debug to generate_firmware_patterns when called with -v
Make make-firmware-image more robust, I hope?
2023-02-09 17:44:28 +00:00
Steve McIntyre 80f7fa39ff Don't break on CD 2 onwards
The $base_components file only matters on disc 1, so check it exists
before trying to do stuff with it.
2023-02-09 01:40:33 +00:00
Cyril Brulebois d7796845f4 Document recent changes. 2023-02-08 12:22:51 +01:00
Cyril Brulebois 6118a758fd make-firmware-image: remove non-free from FIRMWARE_COMPONENTS
See previous commit for rationale.
2023-01-30 02:07:12 +01:00
Cyril Brulebois 3d879d5e46 CONF.sh: remove non-free from NONFREE_COMPONENTS
Almost all packages initially identified as interesting for us[1] have
been moved to non-free-firmware. An updated list of packages has been
published[2], which can be summed up this way:
 - amd64-microcode and intel-microcode would require specific code
   anyway (#1029804);
 - zd1211-firmware shows inconsistencies (archive vs. repository).

 1. https://lists.debian.org/debian-boot/2023/01/msg00150.html
 2. https://lists.debian.org/debian-boot/2023/01/msg00235.html

Therefore, stop looking for non-free firmware packages in non-free,
and concentrate on non-free-firmware only.
2023-01-30 02:01:12 +01:00
Samuel Thibault d94e9eb5b7 Fix missing %install% replacements
E.g. the ontimeout line of spkgtk.cfg includes both the kernel and initrd
paths on the same line, so need several %install% replacements.
2023-01-29 22:05:56 +01:00
Cyril Brulebois 492b085305 make_disc_trees.pl: move local vs. base_components safeguard
We have two safeguards related to the local component now, one if it's
present, one if it's absent: group them via if-then-else.
2023-01-27 07:38:40 +01:00
Cyril Brulebois 2e61e76d86 generate_firmware_task: prettify output
Use a format string to align both columns (Package and Filename fields).
2023-01-27 07:38:40 +01:00
Cyril Brulebois d415c60a12 generate_firmware_patterns: prettify output
Use format strings to align all entries (%4d entries).

Use standard double-space indentation found almost everywhere else.
2023-01-27 07:38:40 +01:00
Cyril Brulebois 59978789f0 apt-selection: order contrib before non-free*
Components are ordered “main contrib non-free” almost everywhere, let's
use the standard order here too. There should be no functional changes.

Note: This controls the sources.list file used during the build, but
doesn't control “deb cdrom:” lines in sources.list during or after
installation, which are the result of running apt-cdrom (which sorts
components alphabetically, see its pkgCdrom::ReduceSourcelist function).
2023-01-27 07:38:40 +01:00
Cyril Brulebois ccaa1a9c69 Merge branch 'clarify-fix-improve-support-for-local-packages'
WARNING: This is an attempt at making it easier for users to enable
support for local packages. This branch is really believed to make
the whole situation easier and clearer on the long run, but it might
introduce some regressions, or at least behavorial changes depending
on existing configurations.

In particular, users might have a hierarchy under LOCALDEBS that's
different from the one getting documented in eb6d140f48, possibly with
a top-level local/ (see also bfebeded70 and 7ce8d0c28f).

It's advised to move files around to match the documented layout, and
to report any regressions that would be spotted: many tests where run,
trying to torture this patch series as much as possible, imagining
borderline cases, but in the end, that's not a feature that's getting
fully battle-tested since it's obviously not used for official builds.
2023-01-27 05:27:02 +01:00
Cyril Brulebois 7ce8d0c28f start_new_disc: remove /dists/local commodity symlink
When local packages are enabled (LOCAL=1), various things happen in the
target /dists directory. One of them is the creation of a /dists/local
symlink, pointing to (/dists/)<CODENAME>. This leads to a very confusing
situation, where “local” is initially about a component, but effectively
ends up looking like a distribution.

It is theoretically possible that cleaning this up might require some
follow-up fixes in other tools, but even that would be better than
keeping bear traps around.
2023-01-27 05:24:03 +01:00
Cyril Brulebois ae8490ba70 make_disc_trees.pl: rewrite base_components if needed
It's possible to set LOCAL=1 to enable support for local packages, which
automatically results in “local” being listed in /.disk/base_components,
which is used to build the --components parameter for debootstrap.

Users might have various sets of *.deb and/or *.udebs, be it over time,
or for various codenames, and it's possible for /dists/<CODENAME>/local
to be entirely missing in the generated image.

In this case, /.disk/base_components must not contain “local”, as that
would break debootstrap; this commit detects and fixes this possible
inconsistency.
2023-01-27 05:24:03 +01:00
Cyril Brulebois c24949141b Packages-gen: write compressed Packages indices again
This is a regression introduced while trying to catch up with apt's
expectations, in 2183089968c04c4765d2119d176e021431b26353: iterating over
Packages files is done by running the “zcat” utility on Packages.gz and
Packages.xz; replacing Packages.gz with an uncompressed Packages broke
iterating over Packages files referencing udebs.
2023-01-27 05:24:03 +01:00
Cyril Brulebois bfebeded70 make_disc_trees.pl: merge “local” udebs into “main” explicitly
This might have worked by accident depending on the exact layout under the
LOCALDEBS directory (see previous commit for the big picture).

Let's focus on where input files for d-i should be stored:

    <LOCALDEBS>/dists/<CODENAME>/local/debian-installer/binary-amd64/*.udeb

Those *.udeb end up being stored under the “local” component[1] (making it
clear they're not from usual components on a Debian mirror), but they are
referenced in Packages for the “main” component[2], possibly replacing a
different version found in the archive.

    /dists/<CODENAME>/local/debian-installer/binary-<ARCH>/*.udeb  [1]
    /dists/<CODENAME>/main/debian-installer/binary-<ARCH>/Packages [2]

This makes it possible for d-i to find those *.udeb automatically, as it
uses a single Packages file.
2023-01-27 05:24:03 +01:00
Cyril Brulebois eb6d140f48 CONF.sh: clarify expected layout below LOCALDEBS
Getting support for local packages (*.deb and/or *.udeb) to work
consistently has been a gigantic time sink, never again!

Document the expected layout as precisely as possible, so that
unsuspecting users have a better chance of getting things right!
2023-01-27 05:24:03 +01:00
Cyril Brulebois 8bba3df865 make_disc_trees.pl: fix block calculation for --rollback
Per-package metadata files ($p.patterns and $p.component) go together,
and both needed to be accounted for.
2023-01-27 00:37:06 +01:00
Cyril Brulebois 5e39a81adb make_disc_trees.pl: fix block calculation for multi-arch
Per-package metadata files ($p.patterns and $p.component) go together,
and both needed to be accounted for.
2023-01-27 00:37:06 +01:00
Cyril Brulebois 9fcb750948 README: clarify link between FORCE_FIRMWARE and NONFREE_COMPONENTS 2023-01-27 00:22:43 +01:00
Cyril Brulebois 88b10ce037 README: clarify both NONFREE and NONFREE_COMPONENTS must be exported 2023-01-27 00:22:43 +01:00
Cyril Brulebois e6b19960aa CONF.sh: a link between FORCE_FIRMWARE and NONFREE_COMPONENTS 2023-01-27 00:22:43 +01:00
Cyril Brulebois ee1efda6de CONF.sh: unset NONFREE_COMPONENTS initially
The idea is to start from a clean slate every time, do the same as for
other variables.
2023-01-27 00:22:43 +01:00
Cyril Brulebois 2674f26060 CONF.sh: unset CONTRIB for consistency
It looks strange to have NONFREE unset and CONTRIB=1 in the default config,
since packages in contrib are usually there because they depend on non-free
stuff.
2023-01-27 00:01:16 +01:00
Cyril Brulebois 95df8ab524 make-firmware-image: implement firmware/dep11 generation for bookworm and above
Mimic what make_disc_trees.pl does, making it possible for hw-detect to
spot not only the /firmware/dep11 directory in netinst images, but also
on installation images with firmware added manually.

Rename pseudo-option to clarify it returns filenames and indices.

Keep generation disabled for bullseye (as an extra safeguard, even if we
have separate, per-release branches anyway): That's helpful information
but bullseye is a stable release, and changing even unofficial material
might not be the best idea. Be cautious by default, it's easy enough to
enable if desired.
2023-01-26 23:22:40 +01:00
Cyril Brulebois 45db7e1582 generate_firmware_task: build lists of Packages files and iterate on them
The next step for make-firmware-image is to use generate_firmware_patterns
for each firmware package that was detected. That command requires DEP-11
metadata, so needs to know which Packages file each firmware package comes
from.

Instead of building long $pkgfiles and $bp_pkgfiles strings, and passing
them to catz, build lists instead and iterate over them. This means a few
more catz calls, but that also means being able to derive the DEP-11
directory from Packages paths.

There should be no functional changes.
2023-01-26 23:22:40 +01:00
Cyril Brulebois a2c722b735 make-firmware-image: use generate_firmware_task to find firmware
Since that tool might be used without a caller that sources CONF.sh,
be explicit about the components in which firmware packages must be
searched:
 - use "main contrib non-free" for bullseye;
 - use "main non-free non-free-firmware" for bookworm and above.

non-free can go away once we have all we need in non-free-firmware.

Export some environment variables for generate_firmware_task (also
renaming SUITE to CODENAME).
2023-01-26 23:22:40 +01:00
Cyril Brulebois bd13392570 generate_firmware_task: implement support for --list-filenames
Treat the output file argument specifically when it's “--list-filenames”:
instead of actually generating a task file with a list of packages (and a
few comments), output the Filename field for each firmware package
matching our criteria.

Move some status messages to STDERR accordingly.

This makes the “find suitable firmware packages” reusable, e.g. by
make-firmware-image.
2023-01-26 23:22:40 +01:00
Cyril Brulebois 90739b8cde generate_firmware_task: move code around
This is a preparatory step for the following commit, there should be no
functional changes.
2023-01-26 23:22:40 +01:00
Cyril Brulebois 14d2664459 generate_firmware_task: pick and choose components
We have a few fully-free firmware packages in main, and a few downloaders
(e.g. b43-installer and b43legacy-installer) in contrib; the latter might
not be really interesting without an alternative connectivity during the
installation process.

Regarding non-free vs. non-free-firmware, we started moving packages[1] from
the former to the latter so that we could build official installation images
based on main and non-free-firmware only.

 1. https://lists.debian.org/debian-boot/2023/01/msg00150.html

At the moment, NON_FREE_COMPONENTS still lists both, but the plan is to move
to non-free-firmware only once all the packages we want are available there.

The relevant variables regarding components include:
 - CONTRIB
 - NONFREE
 - EXTRANONFREE
 - FORCE_FIRMWARE

This commit implements:
 - using main all the time;
 - using contrib if CONTRIB=1;
 - using NONFREE_COMPONENTS if NONFREE=1 or EXTRANONFREE=1 or FORCE_FIRMWARE=1.

At the moment it seems the only existing caller is the top-level Makefile,
only when FORCE_FIRMWARE is set. But it seems to make sense to match what
tools/apt-selection does, to avoid any surprises.
2023-01-26 23:22:40 +01:00
Cyril Brulebois 4a31c04b81 make_disc_trees.pl: remove workaround for missing bookworm/non-free-firmware/dep11
This directory appeared in the archive earlier today.
2023-01-26 22:24:51 +01:00
Cyril Brulebois ee40a3bf23 make_disc_trees.pl: add workaround for missing bookworm/non-free-firmware/dep11
There's no dep11 directory for bookworm's non-free-firmware component
for the time being, and we've just decided to be more cautious about
input files, which breaks the bookworm build.

Until the issue is fixed on the appstream.debian.org side, and until dak
can indeed import that directory into the Debian archive, use sid's
non-free-firmware dep11 directory instead of bookworm's.
2023-01-25 01:38:34 +01:00
Cyril Brulebois 214e2dd88c make_disc_trees.pl: use write_file() for .component files as well 2023-01-25 01:38:34 +01:00
Cyril Brulebois 957ee269b4 make_disc_trees.pl: create /firmware/dep11/README.txt and count it
Move the README.txt creation from generate_firmware_patterns to its
caller, so that it's done just once, and so that added blocks are
counted properly.
2023-01-25 01:38:34 +01:00
Cyril Brulebois b56af4dbcc generate_firmware_patterns: rewrite to match caller's expectations
Initially it was interesting to pass “--package ALL” to create *.patterns
files for all packages found in the specified Components-* file(s), but
make_disc_trees.pl uses a call per package, so drop support for looping.

Remember whether modalias information was found for the requested package,
and only apply the firmware-sof-signed workaround when that's indeed the
requested package and when modalias information wasn't found (in case that
gets added to DEP-11 metadata at some point in the future). When using the
workaround, include a reminder in the logs.
2023-01-25 01:38:33 +01:00
Cyril Brulebois 9d95f65bd9 generate_firmware_patterns: stop creating the output directory
The output directory's presence is the caller's responsibility; while
it might have been useful during development, actual builds require
make_disc_trees.pl to keep track of blocks getting added, so let it
take care of fulfilling prerequisites.
2023-01-25 01:38:33 +01:00
Cyril Brulebois 2183089968 easy-build.sh: create top-level Release file for local packages
Without this, apt would try and download way too many indices, resulting
in 128 “Ign:” lines. With this, we're down to 4 “Ign:” lines (Release.gpg
and InRelease, twice).

Funnily enough, that changes apt's behaviour, which suddenly requires
uncompressed Packages to be present.
2023-01-23 20:25:21 +01:00
Cyril Brulebois cc53129b15 apt-selection: mark local repositories as trusted
Without this, apt won't like local repositories.
2023-01-23 19:30:42 +01:00
Cyril Brulebois f2a4a832a2 Replace all gzip --best calls with pigz -9nm ones.
Some gzip calls have --no-name, which disables both the original name
and timestamp; use -n (disables original name) and -m (disables original
timestamp) for pigz, for all calls.
2023-01-23 19:30:39 +01:00