manuel
/
live-build
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

firmware: save the compressed contents file to disk instead of decompressed 

the existing logic was to decompress the contents file from the downloaded
archive to disk, then process it to obtain a package list. the largest
one by far is for 'main'; 'non-free' and 'contrib' are tiny in comparison.
for sid-amd64 currently, the archive file is 37 MB, while the decompressed
file it contains is 592.3 MB.

we always delete the files and download afresh (currently), and a previous
commit optimised by deleting the files once we're done with them to avoid
wasting disk space leaving them behind.

here we switch to storing the downloaded compressed file to disk instead,
reducing disk space usage (and IO) by hundreds of megabytes; piping the
decompression directly into awk instead of having awk read from the stored
file.

this moves the appending of new items into the list back within the archive
area loop, which is fine since we're replacing the file for each loop now
so the previous issue relating to appending is of no concern.

Gbp-Dch: Short
Closes: #952910
This commit is contained in:
Lyndon Brown 2020-02-12 05:00:08 +00:00 committed by Luca Boccassi
parent a120bc5445
commit 7867641fd0
1 changed files with 12 additions and 11 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

23
functions/firmwarelists.sh
View File

@ -18,20 +18,21 @@ Firmware_List_From_Contents () {
local DISTRO_CHROOT="${2}"
local ARCHIVE_AREAS="${3}"
local CONTENTS_FILE="cache/contents.chroot/contents.${DISTRO_CHROOT}.${LB_ARCHITECTURES}"
# Ensure fresh
rm -f "${CONTENTS_FILE}"
for _ARCHIVE_AREA in ${ARCHIVE_AREAS}
do
local CONTENTS_FILE="cache/contents.chroot/contents.${DISTRO_CHROOT}.${LB_ARCHITECTURES}.gz"
local CONTENTS_URL="${MIRROR_CHROOT}/dists/${DISTRO_CHROOT}/${_ARCHIVE_AREA}/Contents-${LB_ARCHITECTURES}.gz"
wget ${WGET_OPTIONS} "${CONTENTS_URL}" -O - | gunzip -c >> "${CONTENTS_FILE}"
# Ensure fresh
rm -f "${CONTENTS_FILE}"
wget ${WGET_OPTIONS} "${CONTENTS_URL}" -O "${CONTENTS_FILE}"
local PACKAGES
PACKAGES="$(gunzip -c "${CONTENTS_FILE}" | awk '/^lib\/firmware/ { print $2 }' | sort -u )"
FIRMWARE_PACKAGES="${FIRMWARE_PACKAGES} ${PACKAGES}"
# Don't waste disk space preserving since always getting fresh
rm -f "${CONTENTS_FILE}"
done
FIRMWARE_PACKAGES="${FIRMWARE_PACKAGES} $(awk '/^lib\/firmware/ { print $2 }' "${CONTENTS_FILE}" | sort -u)"
# Don't waste disk space preserving since always getting fresh
rm -f "${CONTENTS_FILE}"
}