[PATCH] ar71xx: Add caldata extraction speedup
Adrian Schmutzler
mail at adrianschmutzler.de
Mo Apr 1 12:39:55 CEST 2019
Hallo Michael,
mit dem Patch ist die MAC-Adressen-Korrektur beim C60 weggeflogen.
Im OpenWrt-Master klappt das aber, das habe ich getestet. Ich vermute, dass das dd in openwrt-18.06 ggf. das oflag=seek_bytes nicht unterstützt o.ä. Genau nachgekuckt habe ich wie gesagt nicht, da die Sache Nice-to-have ist.
Zwecks Timing: Der Vorteil dieses Patches ist, dass die MAC-Adressen eben jetzt so schnell ausgelesen werden können, dass OpenWrt die interfaces nicht erst neu starten muss, um die Adressen zu bekommen. Das ist uns aber alles egal, weil unser Zeug erst danach arbeitet.
Grüße
Adrian
From: Michael Fritscher [mailto:michael at fritscher.net]
Sent: Montag, 1. April 2019 08:15
To: Adrian Schmutzler <mail at adrianschmutzler.de>; franken-dev at freifunk.net
Subject: Re: [PATCH] ar71xx: Add caldata extraction speedup
Was für Art von Problemen? Nicht dass es uns beim nächsten Upgrade der
OpenWRT-Version um die Ohren fliegt, weil wir in unseren Scripten
irgendwelche impliziten Timing-Abhängigkeiten drinne haben...
Viele Grüße,
Michael Fritscher
On 27.03.19 18:28, Adrian Schmutzler wrote:
> Dieser Patch hat bei mir überraschend zu Problemen geführt.
>
> Da es sich um eine Nice-To-Have-Geschichte handelt, werde ich den Patch als "Rejected" markieren und auf den Backport verzichten.
>
> Grüße
>
> Adrian
>
>> -----Original Message-----
>> From: franken-dev [mailto:franken-dev-bounces at freifunk.net] On Behalf Of
>> Adrian Schmutzler
>> Sent: Dienstag, 19. März 2019 17:59
>> To: franken-dev at freifunk.net <mailto:franken-dev at freifunk.net>
>> Subject: [PATCH] ar71xx: Add caldata extraction speedup
>>
>> This is a backport from the OpenWrt master:
>>
>> Reading and writing to and from flash storage is slowed down
>> enormously by some functions which use a block size of 1.
>>
>> This patch reworks the extraction scripts to be much faster and
>> efficient by reading and writing in possibly one big block.
>>
>> Speed comparison @ TP-Link TL-WDR4300 (just manually) results
>> in a time reduction by three orders of magnitude (99.9 %).
>>
>>> time dd if=/dev/mtd3 of=/lib/firmware/test-slow bs=1 count=4096 skip=4096
>> 4096+0 records in
>> 4096+0 records out
>> real 0m 15.85s
>> user 0m 0.06s
>> sys 0m 13.28s
>>
>>> time dd if=/dev/mtd3 of=/lib/firmware/test-fast bs=4096 count=1 skip=4096
>> iflag=skip_bytes
>> 1+0 records in
>> 1+0 records out
>> real 0m 0.02s
>> user 0m 0.00s
>> sys 0m 0.02s
>>
>> Since the boot time increase is relatively substantial, I
>> decided to backport this to the current, openwrt-18.06-based
>> firmware.
>>
>> Signed-off-by: Adrian Schmutzler <freifunk at adrianschmutzler.de <mailto:freifunk at adrianschmutzler.de> >
>>
>> ---
>>
>> Just in case someone is interested in reviewing this for our firmware
>> as long as it stays 18.06 ...
>> ---
>> ...1-ar71xx-Speed-up-caldata-eeprom-handling.patch | 102
>> +++++++++++++++++++++
>> ...71xx-Speed-up-mtd-extraction-in-ar71xx.sh.patch | 39 ++++++++
>> 2 files changed, 141 insertions(+)
>> create mode 100755 build_patches/openwrt/0041-ar71xx-Speed-up-caldata-
>> eeprom-handling.patch
>> create mode 100755 build_patches/openwrt/0042-ar71xx-Speed-up-mtd-
>> extraction-in-ar71xx.sh.patch
>>
>> diff --git a/build_patches/openwrt/0041-ar71xx-Speed-up-caldata-eeprom-
>> handling.patch b/build_patches/openwrt/0041-ar71xx-Speed-up-caldata-
>> eeprom-handling.patch
>> new file mode 100755
>> index 00000000..363e173a
>> --- /dev/null
>> +++ b/build_patches/openwrt/0041-ar71xx-Speed-up-caldata-eeprom-
>> handling.patch
>> @@ -0,0 +1,102 @@
>> +From df6505e5adfaaf04231e459afac492ffadde69ec Mon Sep 17 00:00:00 2001
>> +From: Adrian Schmutzler <freifunk at adrianschmutzler.de <mailto:freifunk at adrianschmutzler.de> >
>> +Date: Thu, 7 Mar 2019 09:40:08 +0100
>> +Subject: [PATCH 1/2] ar71xx: Speed up caldata/eeprom handling
>> +
>> +Reading and writing to and from flash storage is slowed down
>> +enormously by some functions which use a block size of 1.
>> +
>> +This patch reworks the extraction scripts to be much faster and
>> +efficient by reading and writing in possibly one big block.
>> +
>> +This is based on the initial commit a69e101 for ipq40xx by
>> +Christian Lamparter <chunkeey at gmail.com <mailto:chunkeey at gmail.com> >.
>> +
>> +Speed comparison @ TP-Link TL-WDR4300 (just manually) results
>> +in a time reduction by three orders of magnitude (99.9 %).
>> +
>> +> time dd if=/dev/mtd3 of=/lib/firmware/test-slow bs=1 count=4096
>> skip=4096
>> +4096+0 records in
>> +4096+0 records out
>> +real 0m 15.85s
>> +user 0m 0.06s
>> +sys 0m 13.28s
>> +
>> +> time dd if=/dev/mtd3 of=/lib/firmware/test-fast bs=4096 count=1 skip=4096
>> iflag=skip_bytes
>> +1+0 records in
>> +1+0 records out
>> +real 0m 0.02s
>> +user 0m 0.00s
>> +sys 0m 0.02s
>> +
>> +Signed-off-by: Adrian Schmutzler <freifunk at adrianschmutzler.de <mailto:freifunk at adrianschmutzler.de> >
>> +---
>> + .../linux/ar71xx/base-files/etc/hotplug.d/firmware/10-ath9k-eeprom | 6 +++-
>> --
>> + .../ar71xx/base-files/etc/hotplug.d/firmware/11-ath10k-caldata | 6 +++---
>> + 2 files changed, 6 insertions(+), 6 deletions(-)
>> +
>> +diff --git a/target/linux/ar71xx/base-files/etc/hotplug.d/firmware/10-ath9k-
>> eeprom b/target/linux/ar71xx/base-files/etc/hotplug.d/firmware/10-ath9k-
>> eeprom
>> +index 94bce7d335..208d5f6bff 100644
>> +--- a/target/linux/ar71xx/base-files/etc/hotplug.d/firmware/10-ath9k-eeprom
>> ++++ b/target/linux/ar71xx/base-files/etc/hotplug.d/firmware/10-ath9k-
>> eeprom
>> +@@ -20,7 +20,7 @@ ath9k_eeprom_extract() {
>> + [ -n "$mtd" ] || \
>> + ath9k_eeprom_die "no mtd device found for partition $part"
>> +
>> +- dd if=$mtd of=/lib/firmware/$FIRMWARE bs=1 skip=$offset
>> count=$count 2>/dev/null || \
>> ++ dd if=$mtd of=/lib/firmware/$FIRMWARE iflag=skip_bytes bs=$count
>> skip=$offset count=1 2>/dev/null || \
>> + ath9k_eeprom_die "failed to extract from $mtd"
>> + }
>> +
>> +@@ -35,7 +35,7 @@ ath9k_ubi_eeprom_extract() {
>> + [ -n "$ubi" ] || \
>> + ath9k_eeprom_die "no UBI volume found for $part"
>> +
>> +- dd if=/dev/$ubi of=/lib/firmware/$FIRMWARE bs=1 skip=$offset
>> count=$count 2>/dev/null || \
>> ++ dd if=/dev/$ubi of=/lib/firmware/$FIRMWARE iflag=skip_bytes
>> bs=$count skip=$offset count=1 2>/dev/null || \
>> + ath9k_eeprom_die "failed to extract from $ubi"
>> + }
>> +
>> +@@ -62,7 +62,7 @@ ath9k_patch_firmware_mac() {
>> +
>> + [ -z "$mac" ] && return
>> +
>> +- macaddr_2bin $mac | dd of=/lib/firmware/$FIRMWARE conv=notrunc
>> bs=1 seek=2 count=6
>> ++ macaddr_2bin $mac | dd of=/lib/firmware/$FIRMWARE conv=notrunc
>> oflag=seek_bytes bs=6 seek=2 count=1
>> + }
>> +
>> + board=$(board_name)
>> +diff --git a/target/linux/ar71xx/base-files/etc/hotplug.d/firmware/11-ath10k-
>> caldata b/target/linux/ar71xx/base-files/etc/hotplug.d/firmware/11-ath10k-
>> caldata
>> +index d21d4134d4..39551e41a8 100644
>> +--- a/target/linux/ar71xx/base-files/etc/hotplug.d/firmware/11-ath10k-caldata
>> ++++ b/target/linux/ar71xx/base-files/etc/hotplug.d/firmware/11-ath10k-
>> caldata
>> +@@ -10,7 +10,7 @@ ath10kcal_from_file() {
>> + local offset=$2
>> + local count=$3
>> +
>> +- dd if=$source of=/lib/firmware/$FIRMWARE bs=1 skip=$offset
>> count=$count 2>/dev/null || \
>> ++ dd if=$source of=/lib/firmware/$FIRMWARE iflag=skip_bytes bs=$count
>> skip=$offset count=1 2>/dev/null || \
>> + ath10kcal_die "failed to extract calibration data from $source"
>> + }
>> +
>> +@@ -30,7 +30,7 @@ ath10kcal_extract() {
>> + [ "$count" = "$cal_size" ] || \
>> + ath10kcal_die "no calibration data found in $part"
>> +
>> +- dd if=$mtd of=/lib/firmware/$FIRMWARE bs=1 skip=$offset
>> count=$count 2>/dev/null || \
>> ++ dd if=$mtd of=/lib/firmware/$FIRMWARE iflag=skip_bytes bs=$count
>> skip=$offset count=1 2>/dev/null || \
>> + ath10kcal_die "failed to extract calibration data from $mtd"
>> + }
>> +
>> +@@ -39,7 +39,7 @@ ath10kcal_patch_mac() {
>> +
>> + [ -z "$mac" ] && return
>> +
>> +- macaddr_2bin $mac | dd of=/lib/firmware/$FIRMWARE conv=notrunc
>> bs=1 seek=6 count=6
>> ++ macaddr_2bin $mac | dd of=/lib/firmware/$FIRMWARE conv=notrunc
>> oflag=seek_bytes bs=6 seek=6 count=1
>> + }
>> +
>> + [ -e /lib/firmware/$FIRMWARE ] && exit 0
>> +--
>> +2.11.0
>> +
>> diff --git a/build_patches/openwrt/0042-ar71xx-Speed-up-mtd-extraction-in-
>> ar71xx.sh.patch b/build_patches/openwrt/0042-ar71xx-Speed-up-mtd-
>> extraction-in-ar71xx.sh.patch
>> new file mode 100755
>> index 00000000..239b79dd
>> --- /dev/null
>> +++ b/build_patches/openwrt/0042-ar71xx-Speed-up-mtd-extraction-in-
>> ar71xx.sh.patch
>> @@ -0,0 +1,39 @@
>> +From 61fcbe7b1065a8aae983063c4f7002b5a562a494 Mon Sep 17 00:00:00 2001
>> +From: Adrian Schmutzler <freifunk at adrianschmutzler.de <mailto:freifunk at adrianschmutzler.de> >
>> +Date: Thu, 7 Mar 2019 09:40:09 +0100
>> +Subject: [PATCH 2/2] ar71xx: Speed up mtd extraction in ar71xx.sh
>> +
>> +Although the amount of data read here is smaller than for the
>> +caldata, there still might be some speed gain compared to reading
>> +bytewise. And there is no harm ...
>> +
>> +Signed-off-by: Adrian Schmutzler <freifunk at adrianschmutzler.de <mailto:freifunk at adrianschmutzler.de> >
>> +---
>> + target/linux/ar71xx/base-files/lib/ar71xx.sh | 4 ++--
>> + 1 file changed, 2 insertions(+), 2 deletions(-)
>> +
>> +diff --git a/target/linux/ar71xx/base-files/lib/ar71xx.sh
>> b/target/linux/ar71xx/base-files/lib/ar71xx.sh
>> +index 2200069c64..2f3a72f732 100755
>> +--- a/target/linux/ar71xx/base-files/lib/ar71xx.sh
>> ++++ b/target/linux/ar71xx/base-files/lib/ar71xx.sh
>> +@@ -16,7 +16,7 @@ ar71xx_get_mtd_offset_size_format() {
>> + dev=$(find_mtd_part $mtd)
>> + [ -z "$dev" ] && return
>> +
>> +- dd if=$dev bs=1 skip=$offset count=$size 2>/dev/null | hexdump -v -e
>> "1/1 \"$format\""
>> ++ dd if=$dev iflag=skip_bytes bs=$size skip=$offset count=1 2>/dev/null |
>> hexdump -v -e "1/1 \"$format\""
>> + }
>> +
>> + ar71xx_get_mtd_part_magic() {
>> +@@ -390,7 +390,7 @@ tplink_pharos_v2_get_model_string() {
>> + part=$(find_mtd_part 'product-info')
>> + [ -z "$part" ] && return 1
>> +
>> +- dd if=$part bs=1 skip=4360 count=64 2>/dev/null | tr -d '\r\0' | head -n 1
>> ++ dd if=$part iflag=skip_bytes bs=64 skip=4360 count=1 2>/dev/null | tr -d
>> '\r\0' | head -n 1
>> + }
>> +
>> + ar71xx_board_detect() {
>> +--
>> +2.11.0
>> +
>> --
>> 2.11.0
-------------- nächster Teil --------------
Ein Dateianhang mit HTML-Daten wurde abgetrennt...
URL: <https://{'listname': 'franken-dev-freifunk.net', 'hostname': 'lists.freifunk.net'}/pipermail/franken-dev-freifunk.net/attachments/20190401/f9340d00/attachment.html>
-------------- nächster Teil --------------
Ein Dateianhang mit Binärdaten wurde abgetrennt...
Dateiname : nicht verfügbar
Dateityp : application/pgp-signature
Dateigröße : 834 bytes
Beschreibung: nicht verfügbar
URL : <https://{'listname': 'franken-dev-freifunk.net', 'hostname': 'lists.freifunk.net'}/pipermail/franken-dev-freifunk.net/attachments/20190401/f9340d00/attachment.sig>
Mehr Informationen über die Mailingliste franken-dev