<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 2.0cm 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=DE link="#0563C1" vlink="#954F72"><div class=WordSection1><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'>FYI: There is a quite interesting discussion (still ongoing) about the problems with sysupgrade on OpenWrt list.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'>Link here: <a href="http://lists.infradead.org/pipermail/openwrt-devel/2020-May/023336.html">http://lists.infradead.org/pipermail/openwrt-devel/2020-May/023336.html</a><o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'>First mail also below.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><b><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif'>From:</span></b><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif'> openwrt-devel [mailto:openwrt-devel-bounces@lists.openwrt.org] <b>On Behalf Of </b>Michael Jones<br><b>Sent:</b> Mittwoch, 13. Mai 2020 07:18<br><b>To:</b> openwrt-devel <openwrt-devel@lists.openwrt.org><br><b>Subject:</b> [OpenWrt-Devel] Sysupgrade and Failed to kill all processes<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p><div><p class=MsoNormal>I've been investigating a problem with sysupgrade failing with the error message "Failed to kill all processes", and then hanging indefinitely.<o:p></o:p></p><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>This happens maybe once every 10-20 sysupgrades, and it's kind of a pain.<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>So far I've determined this workflow that the sysupgrade command follows. Note, I'm not aiming for 100% accuracy, but just broad strokes.<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><div><p class=MsoNormal><o:p> </o:p></p></div><div><div><p class=MsoNormal>1) /sbin/sysupgrade locates the file to upgrade from on the filesystem, or if the second option to sysupgrade starts with <a href="http://">http://</a>, it downloads the firmware file using wget.<o:p></o:p></p></div><div><p class=MsoNormal>2) /sbn/sysupgrade does some minor validation of various things, and grabs whatever config files it thinks the end user wants to be restored and packs them up into some kind of tarball.<o:p></o:p></p></div><div><p class=MsoNormal>3) sysupgrade sends a message, via ubus, to procd, to initiate the upgrade.<o:p></o:p></p></div><div><p class=MsoNormal>4) Procd does some stuff which I haven't finished completely understanding just yet, but it looks like firmware verification to make sure we don't upgrade to a bad firmware file.<o:p></o:p></p></div><div><p class=MsoNormal>5) It *does not* appear that procd will proactively terminate services until everything (or almost everything) is shut down. Seems like something that should be added to increase reliability.<o:p></o:p></p></div><div><p class=MsoNormal>6) procd replaces itself (execvp systemcall) with the program /sbin/upgraded. This means that procd is *no longer running*, PID 1 is now /sbin/upgraded. So service management is not possible at this point.<o:p></o:p></p></div><div><p class=MsoNormal>7) /sbin/upgraded now acts as PID1. It executes the shell script /lib/upgrade/stage2 with parameters.<o:p></o:p></p></div><div><p class=MsoNormal>8) The shell script loops on all processes, and sends them the TERM signal, and then the KILL signal. See email subjec for problems with this.<o:p></o:p></p></div><div><p class=MsoNormal>9) the shell script creates a new ram filesystem, mounts it, then copies over a very small set of binaries into it.<o:p></o:p></p></div><div><p class=MsoNormal>10) The shell script changes root into the new ram filesystem<o:p></o:p></p></div><div><p class=MsoNormal>11) Inside the ramfilesystem, the shell script writes the upgraded firmware and saved configuration to disk<o:p></o:p></p></div><div><p class=MsoNormal>12) Reboot.<o:p></o:p></p></div></div></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Now that the very rough summary is out of the way, I have 4 questions.<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>1) I notice that the shell script /lib/upgrade/stage2 is doing a tight loop with kill -9 to terminate processes. However, it's only looping a maximum of 10 times, and its going as fast as the shell can loop. <o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>What's to stop this loop from quickly going through every process almost immediately 10 times, before a process that would be about to terminate terminates? The process in question may be handling some kind of IO, so the kernel wouldn't immediately terminate it.<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Shouldn't there be some very brief sleep at the end of each loop iteration to ensure that the processes that are going to practically terminate have done so?<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>2) Why is the behavior on failure to terminate processes to just give up? That leaves devices hanging without any network connectivity. <o:p></o:p></p></div><div><p class=MsoNormal>A reboot with some logging on disk would allow for remote sysupgrades to have some kind of recoverability.<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>3) Is looping over sigkill a reliable way to terminate all processes?<o:p></o:p></p></div><div><p class=MsoNormal>I was under the impression that the only reliable way to ensure all processes terminate is to use cgroups, and put the processes to terminate in the freezer group and then kill them off after they've been frozen. Otherwise you have basically a race condition between the termination of processes and the creation of children. E.g. a fork-bomb could prevent all processes from being terminated.<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>4) Why doesn't procd, prior to execvp the /sbin/upgraded program, shutdown all the services that are running? <o:p></o:p></p></div><div><p class=MsoNormal style='margin-bottom:12.0pt'><br>Maybe I'm just not seeing where it does this, so if that's the case, then I'm happy to be corrected.<o:p></o:p></p></div><div><p class=MsoNormal>But I'm under the impression that when not using cgroups, stopping all services would allow for anything that isn't double forked to be gracefully shutdown and cleaned up after itself.<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div></div></div></body></html>