[PATCH] ar71xx: Add caldata extraction speedup

Michael Fritscher michael at fritscher.net
Mo Apr 1 08:14:46 CEST 2019


Was für Art von Problemen? Nicht dass es uns beim nächsten Upgrade der
OpenWRT-Version um die Ohren fliegt, weil wir in unseren Scripten
irgendwelche impliziten Timing-Abhängigkeiten drinne haben...

Viele Grüße,
Michael Fritscher

On 27.03.19 18:28, Adrian Schmutzler wrote:
> Dieser Patch hat bei mir überraschend zu Problemen geführt.
> 
> Da es sich um eine Nice-To-Have-Geschichte handelt, werde ich den Patch als "Rejected" markieren und auf den Backport verzichten.
> 
> Grüße
> 
> Adrian
> 
>> -----Original Message-----
>> From: franken-dev [mailto:franken-dev-bounces at freifunk.net] On Behalf Of
>> Adrian Schmutzler
>> Sent: Dienstag, 19. März 2019 17:59
>> To: franken-dev at freifunk.net
>> Subject: [PATCH] ar71xx: Add caldata extraction speedup
>>
>> This is a backport from the OpenWrt master:
>>
>> Reading and writing to and from flash storage is slowed down
>> enormously by some functions which use a block size of 1.
>>
>> This patch reworks the extraction scripts to be much faster and
>> efficient by reading and writing in possibly one big block.
>>
>> Speed comparison @ TP-Link TL-WDR4300 (just manually) results
>> in a time reduction by three orders of magnitude (99.9 %).
>>
>>> time dd if=/dev/mtd3 of=/lib/firmware/test-slow bs=1 count=4096 skip=4096
>> 4096+0 records in
>> 4096+0 records out
>> real    0m 15.85s
>> user    0m 0.06s
>> sys     0m 13.28s
>>
>>> time dd if=/dev/mtd3 of=/lib/firmware/test-fast bs=4096 count=1 skip=4096
>> iflag=skip_bytes
>> 1+0 records in
>> 1+0 records out
>> real    0m 0.02s
>> user    0m 0.00s
>> sys     0m 0.02s
>>
>> Since the boot time increase is relatively substantial, I
>> decided to backport this to the current, openwrt-18.06-based
>> firmware.
>>
>> Signed-off-by: Adrian Schmutzler <freifunk at adrianschmutzler.de>
>>
>> ---
>>
>> Just in case someone is interested in reviewing this for our firmware
>> as long as it stays 18.06 ...
>> ---
>>  ...1-ar71xx-Speed-up-caldata-eeprom-handling.patch | 102
>> +++++++++++++++++++++
>>  ...71xx-Speed-up-mtd-extraction-in-ar71xx.sh.patch |  39 ++++++++
>>  2 files changed, 141 insertions(+)
>>  create mode 100755 build_patches/openwrt/0041-ar71xx-Speed-up-caldata-
>> eeprom-handling.patch
>>  create mode 100755 build_patches/openwrt/0042-ar71xx-Speed-up-mtd-
>> extraction-in-ar71xx.sh.patch
>>
>> diff --git a/build_patches/openwrt/0041-ar71xx-Speed-up-caldata-eeprom-
>> handling.patch b/build_patches/openwrt/0041-ar71xx-Speed-up-caldata-
>> eeprom-handling.patch
>> new file mode 100755
>> index 00000000..363e173a
>> --- /dev/null
>> +++ b/build_patches/openwrt/0041-ar71xx-Speed-up-caldata-eeprom-
>> handling.patch
>> @@ -0,0 +1,102 @@
>> +From df6505e5adfaaf04231e459afac492ffadde69ec Mon Sep 17 00:00:00 2001
>> +From: Adrian Schmutzler <freifunk at adrianschmutzler.de>
>> +Date: Thu, 7 Mar 2019 09:40:08 +0100
>> +Subject: [PATCH 1/2] ar71xx: Speed up caldata/eeprom handling
>> +
>> +Reading and writing to and from flash storage is slowed down
>> +enormously by some functions which use a block size of 1.
>> +
>> +This patch reworks the extraction scripts to be much faster and
>> +efficient by reading and writing in possibly one big block.
>> +
>> +This is based on the initial commit a69e101 for ipq40xx by
>> +Christian Lamparter <chunkeey at gmail.com>.
>> +
>> +Speed comparison @ TP-Link TL-WDR4300 (just manually) results
>> +in a time reduction by three orders of magnitude (99.9 %).
>> +
>> +> time dd if=/dev/mtd3 of=/lib/firmware/test-slow bs=1 count=4096
>> skip=4096
>> +4096+0 records in
>> +4096+0 records out
>> +real    0m 15.85s
>> +user    0m 0.06s
>> +sys     0m 13.28s
>> +
>> +> time dd if=/dev/mtd3 of=/lib/firmware/test-fast bs=4096 count=1 skip=4096
>> iflag=skip_bytes
>> +1+0 records in
>> +1+0 records out
>> +real    0m 0.02s
>> +user    0m 0.00s
>> +sys     0m 0.02s
>> +
>> +Signed-off-by: Adrian Schmutzler <freifunk at adrianschmutzler.de>
>> +---
>> + .../linux/ar71xx/base-files/etc/hotplug.d/firmware/10-ath9k-eeprom  | 6 +++-
>> --
>> + .../ar71xx/base-files/etc/hotplug.d/firmware/11-ath10k-caldata      | 6 +++---
>> + 2 files changed, 6 insertions(+), 6 deletions(-)
>> +
>> +diff --git a/target/linux/ar71xx/base-files/etc/hotplug.d/firmware/10-ath9k-
>> eeprom b/target/linux/ar71xx/base-files/etc/hotplug.d/firmware/10-ath9k-
>> eeprom
>> +index 94bce7d335..208d5f6bff 100644
>> +--- a/target/linux/ar71xx/base-files/etc/hotplug.d/firmware/10-ath9k-eeprom
>> ++++ b/target/linux/ar71xx/base-files/etc/hotplug.d/firmware/10-ath9k-
>> eeprom
>> +@@ -20,7 +20,7 @@ ath9k_eeprom_extract() {
>> + 	[ -n "$mtd" ] || \
>> + 		ath9k_eeprom_die "no mtd device found for partition $part"
>> +
>> +-	dd if=$mtd of=/lib/firmware/$FIRMWARE bs=1 skip=$offset
>> count=$count 2>/dev/null || \
>> ++	dd if=$mtd of=/lib/firmware/$FIRMWARE iflag=skip_bytes bs=$count
>> skip=$offset count=1 2>/dev/null || \
>> + 		ath9k_eeprom_die "failed to extract from $mtd"
>> + }
>> +
>> +@@ -35,7 +35,7 @@ ath9k_ubi_eeprom_extract() {
>> + 	[ -n "$ubi" ] || \
>> + 		ath9k_eeprom_die "no UBI volume found for $part"
>> +
>> +-	dd if=/dev/$ubi of=/lib/firmware/$FIRMWARE bs=1 skip=$offset
>> count=$count 2>/dev/null || \
>> ++	dd if=/dev/$ubi of=/lib/firmware/$FIRMWARE iflag=skip_bytes
>> bs=$count skip=$offset count=1 2>/dev/null || \
>> + 		ath9k_eeprom_die "failed to extract from $ubi"
>> + }
>> +
>> +@@ -62,7 +62,7 @@ ath9k_patch_firmware_mac() {
>> +
>> + 	[ -z "$mac" ] && return
>> +
>> +-	macaddr_2bin $mac | dd of=/lib/firmware/$FIRMWARE conv=notrunc
>> bs=1 seek=2 count=6
>> ++	macaddr_2bin $mac | dd of=/lib/firmware/$FIRMWARE conv=notrunc
>> oflag=seek_bytes bs=6 seek=2 count=1
>> + }
>> +
>> + board=$(board_name)
>> +diff --git a/target/linux/ar71xx/base-files/etc/hotplug.d/firmware/11-ath10k-
>> caldata b/target/linux/ar71xx/base-files/etc/hotplug.d/firmware/11-ath10k-
>> caldata
>> +index d21d4134d4..39551e41a8 100644
>> +--- a/target/linux/ar71xx/base-files/etc/hotplug.d/firmware/11-ath10k-caldata
>> ++++ b/target/linux/ar71xx/base-files/etc/hotplug.d/firmware/11-ath10k-
>> caldata
>> +@@ -10,7 +10,7 @@ ath10kcal_from_file() {
>> + 	local offset=$2
>> + 	local count=$3
>> +
>> +-	dd if=$source of=/lib/firmware/$FIRMWARE bs=1 skip=$offset
>> count=$count 2>/dev/null || \
>> ++	dd if=$source of=/lib/firmware/$FIRMWARE iflag=skip_bytes bs=$count
>> skip=$offset count=1 2>/dev/null || \
>> + 		ath10kcal_die "failed to extract calibration data from $source"
>> + }
>> +
>> +@@ -30,7 +30,7 @@ ath10kcal_extract() {
>> + 	[ "$count" = "$cal_size" ] || \
>> + 		ath10kcal_die "no calibration data found in $part"
>> +
>> +-	dd if=$mtd of=/lib/firmware/$FIRMWARE bs=1 skip=$offset
>> count=$count 2>/dev/null || \
>> ++	dd if=$mtd of=/lib/firmware/$FIRMWARE iflag=skip_bytes bs=$count
>> skip=$offset count=1 2>/dev/null || \
>> + 		ath10kcal_die "failed to extract calibration data from $mtd"
>> + }
>> +
>> +@@ -39,7 +39,7 @@ ath10kcal_patch_mac() {
>> +
>> + 	[ -z "$mac" ] && return
>> +
>> +-	macaddr_2bin $mac | dd of=/lib/firmware/$FIRMWARE conv=notrunc
>> bs=1 seek=6 count=6
>> ++	macaddr_2bin $mac | dd of=/lib/firmware/$FIRMWARE conv=notrunc
>> oflag=seek_bytes bs=6 seek=6 count=1
>> + }
>> +
>> + [ -e /lib/firmware/$FIRMWARE ] && exit 0
>> +--
>> +2.11.0
>> +
>> diff --git a/build_patches/openwrt/0042-ar71xx-Speed-up-mtd-extraction-in-
>> ar71xx.sh.patch b/build_patches/openwrt/0042-ar71xx-Speed-up-mtd-
>> extraction-in-ar71xx.sh.patch
>> new file mode 100755
>> index 00000000..239b79dd
>> --- /dev/null
>> +++ b/build_patches/openwrt/0042-ar71xx-Speed-up-mtd-extraction-in-
>> ar71xx.sh.patch
>> @@ -0,0 +1,39 @@
>> +From 61fcbe7b1065a8aae983063c4f7002b5a562a494 Mon Sep 17 00:00:00 2001
>> +From: Adrian Schmutzler <freifunk at adrianschmutzler.de>
>> +Date: Thu, 7 Mar 2019 09:40:09 +0100
>> +Subject: [PATCH 2/2] ar71xx: Speed up mtd extraction in ar71xx.sh
>> +
>> +Although the amount of data read here is smaller than for the
>> +caldata, there still might be some speed gain compared to reading
>> +bytewise. And there is no harm ...
>> +
>> +Signed-off-by: Adrian Schmutzler <freifunk at adrianschmutzler.de>
>> +---
>> + target/linux/ar71xx/base-files/lib/ar71xx.sh | 4 ++--
>> + 1 file changed, 2 insertions(+), 2 deletions(-)
>> +
>> +diff --git a/target/linux/ar71xx/base-files/lib/ar71xx.sh
>> b/target/linux/ar71xx/base-files/lib/ar71xx.sh
>> +index 2200069c64..2f3a72f732 100755
>> +--- a/target/linux/ar71xx/base-files/lib/ar71xx.sh
>> ++++ b/target/linux/ar71xx/base-files/lib/ar71xx.sh
>> +@@ -16,7 +16,7 @@ ar71xx_get_mtd_offset_size_format() {
>> + 	dev=$(find_mtd_part $mtd)
>> + 	[ -z "$dev" ] && return
>> +
>> +-	dd if=$dev bs=1 skip=$offset count=$size 2>/dev/null | hexdump -v -e
>> "1/1 \"$format\""
>> ++	dd if=$dev iflag=skip_bytes bs=$size skip=$offset count=1 2>/dev/null |
>> hexdump -v -e "1/1 \"$format\""
>> + }
>> +
>> + ar71xx_get_mtd_part_magic() {
>> +@@ -390,7 +390,7 @@ tplink_pharos_v2_get_model_string() {
>> + 	part=$(find_mtd_part 'product-info')
>> + 	[ -z "$part" ] && return 1
>> +
>> +-	dd if=$part bs=1 skip=4360 count=64 2>/dev/null | tr -d '\r\0' | head -n 1
>> ++	dd if=$part iflag=skip_bytes bs=64 skip=4360 count=1 2>/dev/null | tr -d
>> '\r\0' | head -n 1
>> + }
>> +
>> + ar71xx_board_detect() {
>> +--
>> +2.11.0
>> +
>> --
>> 2.11.0


-------------- nächster Teil --------------
Ein Dateianhang mit Binärdaten wurde abgetrennt...
Dateiname   : signature.asc
Dateityp    : application/pgp-signature
Dateigröße  : 833 bytes
Beschreibung: OpenPGP digital signature
URL         : <https://{'listname': 'franken-dev-freifunk.net', 'hostname': 'lists.freifunk.net'}/pipermail/franken-dev-freifunk.net/attachments/20190401/9cdb5b59/attachment.sig>


Mehr Informationen über die Mailingliste franken-dev