firmware: fix possible duplication in firmware package lists (inefficiency)

where multiple archive areas are used, the code here on each loop is:
  1) fetching the archive area contents file (compressed)
  2) **appending** the output to that of the previous loop
  3) searching the file for firmware references, adding to the list

since it appends rather than replaces, entries found in each loop get
re-added on each subsequent loop, resulting in duplication in the
resulting list

below I evaluate the possible solutions to explain why I chose the one
I chose, however the reader should not waste too much time worrying about
whether one of the other solution would have actually been better because
things are changed significantly in further commits shortly!

possible solutions:
  a) switching to output (>) rather than append (>>), but this might fail
     against an existing file
  b) removing the file on each loop, but this will complicate any future
     caching improvements that might be made here (currently the files are
     always deleted and thus downloaded fresh)
  c) allow the appending, evaluating the complete file after the loop

solution C warrants consideration of disk space consumption; currently the
compressed 'main' archive (for sid on amd64) expands to 592.3 MB (feb-2020),
'contrib' is 3.1 MB, and 'non-free' is 18.5 MB.

solution C was chosen here; the difference of accumulated file size vs.
max-single was minor enough to not be of particular concern (~613 vs.
~592 MB).

Gbp-Dch: Short
Closes: #952906
This commit is contained in:
Lyndon Brown 2020-02-10 18:49:34 +00:00 committed by Luca Boccassi
parent a951fe7ba6
commit c53a949325
2 changed files with 8 additions and 8 deletions

View File

@ -68,10 +68,10 @@ do
CONTENTS_URL="${LB_PARENT_MIRROR_CHROOT}/dists/${LB_PARENT_DISTRIBUTION_CHROOT}/${_PARENT_ARCHIVE_AREA}/Contents-${LB_ARCHITECTURES}.gz"
wget ${WGET_OPTIONS} "${CONTENTS_URL}" -O - | gunzip -c >> "${CONTENTS_FILE}"
FIRMWARE_PACKAGES="${FIRMWARE_PACKAGES} $(awk '/^lib\/firmware/ { print $2 }' "${CONTENTS_FILE}" | sort -u)"
done
FIRMWARE_PACKAGES="${FIRMWARE_PACKAGES} $(awk '/^lib\/firmware/ { print $2 }' "${CONTENTS_FILE}" | sort -u)"
if echo ${LB_PARENT_ARCHIVE_AREAS} | grep -qs "non-free"
then
# Manually add firmware-linux/non-free meta package
@ -102,9 +102,9 @@ then
CONTENTS_URL="${LB_MIRROR_CHROOT}/dists/${LB_DISTRIBUTION_CHROOT}/${_ARCHIVE_AREA}/Contents-${LB_ARCHITECTURES}.gz"
wget ${WGET_OPTIONS} "${CONTENTS_URL}" -O - | gunzip -c >> "${CONTENTS_FILE}"
FIRMWARE_PACKAGES="${FIRMWARE_PACKAGES} $(awk '/^lib\/firmware/ { print $2 }' "${CONTENTS_FILE}" | sort -u)"
done
FIRMWARE_PACKAGES="${FIRMWARE_PACKAGES} $(awk '/^lib\/firmware/ { print $2 }' "${CONTENTS_FILE}" | sort -u)"
fi
# Drop section and keep package names only

View File

@ -356,10 +356,10 @@ then
CONTENTS_URL="${LB_PARENT_MIRROR_CHROOT}/dists/${LB_PARENT_DISTRIBUTION_CHROOT}/${_PARENT_ARCHIVE_AREA}/Contents-${LB_ARCHITECTURES}.gz"
wget ${WGET_OPTIONS} "${CONTENTS_URL}" -O - | gunzip -c >> "${CONTENTS_FILE}"
FIRMWARE_PACKAGES="${FIRMWARE_PACKAGES} $(awk '/^lib\/firmware/ { print $2 }' "${CONTENTS_FILE}" | sort -u)"
done
FIRMWARE_PACKAGES="${FIRMWARE_PACKAGES} $(awk '/^lib\/firmware/ { print $2 }' "${CONTENTS_FILE}" | sort -u)"
if echo ${LB_PARENT_ARCHIVE_AREAS} | grep -qs "non-free"
then
# Manually add firmware-linux/non-free meta package
@ -390,9 +390,9 @@ then
CONTENTS_URL="${LB_MIRROR_CHROOT}/dists/${LB_DISTRIBUTION_CHROOT}/${_ARCHIVE_AREA}/Contents-${LB_ARCHITECTURES}.gz"
wget ${WGET_OPTIONS} "${CONTENTS_URL}" -O - | gunzip -c >> "${CONTENTS_FILE}"
FIRMWARE_PACKAGES="${FIRMWARE_PACKAGES} $(awk '/^lib\/firmware/ { print $2 }' "${CONTENTS_FILE}" | sort -u)"
done
FIRMWARE_PACKAGES="${FIRMWARE_PACKAGES} $(awk '/^lib\/firmware/ { print $2 }' "${CONTENTS_FILE}" | sort -u)"
fi
# Drop section and keep package names only