[WLANware] Freifunk Google Summer of Code proposal

Sun Mar 6 22:31:07 CET 2016

Hi,
I could probably spare my introduction, as I'm not exactly a new face in
Freifunk and OpenWrt development, but as I'm interested in partaking in
this year's Google Summer of Code as a student, I'll give some more
background about myself. Also, I'm writing in English, as I know of a few
people in other, non-German community network projects that are interested
in this proposal.

I'm currently working on my Masters degree in Computer Science at the
University of Lübeck, Germany. I've been involved with the Freifunk
Lübeck project for a few years now and have become an active contributor
to OpenWrt since then. I consider myself a very expecienced C programmer
(but also have learned many other languages over the years). In the past
years I've developed the VPN daemon fastd [1] (which was also the subject
of my Bachelor thesis), and helped start the Gluon project [2], which are
both used by many community network projects like German Freifunk
communities now.

The TL;DR one-sentence version: I'd like to create an alternative to the
UCI config system, providing schema-based, structured configuration with
saner upgrade behaviour.

This proposal is concerned with one of the basic utilities of OpenWrt: the
UCI config system. Over the years, we've come to the conclusion that the
current state of UCI is not really what we need in Freifunk firmwares like
Gluon. The main issue is the upgrade behaviour:

* When setting new default configurations in a new firmware release, we
  don't know if the old setting came from an older firmware's defaults, or
  if a user set the value herself
* Old settings not used anymore need to be removed explicitly or will
  accumulate; these explicit removal commands in upgrade script also
  accumulate if skipping versions on upgrades should be supported
* UCI config files are usually installed by packages, but because of the
  upgrade concept of OpenWrt, there is no sane way to define a pre/post-rm
  script cleaning up and removing the config files when a package is
  removed during a firmware upgrade

I've posted an UCI extension proposal to the openwrt-devel ML last year
[3], but didn't get much positive feedback (but enough to know that other
people have similar issues). Since then, I've moved away from the idea of
creating an API-compatible extension to UCI, and think creating an
alternative to UCI with mostly different concepts would be preferable.
Package maintainers could decide themselves if UCI or this New
Configuration system ("NC", working title) should be used for some
specific application; as proposed in some replies to [3], generating UCI
config from NC would also be a possible approach.

My current ideas are inspired by GNOME's gconf/gsettings systems; I'm also
borrowing some vocabulary from the systemd people. I'll use the following
terms to describe different types of config:

Schema
    The schema defines which config keys exist and which type the
    corresponding values have. It can also define default values and
    presets for these keys. Schema files lie somewhere below /lib and are
    installed by packages which need configuration.

Default values
    Config keys not set by the user may have default values defined by the
    schema.

User configuration
    User configuration is stored in a database file below /etc. Per-user
    configuration in home directories is not part of this proposal (as
    in OpenWrt most config is system-wide), but it would be a possible
    extension.

Presets
    Presets define the default values for the user configuration. This
    differs from the default values: changing the default values on
    upgrades will change the effective value, but changing a preset won't
    (as on initialization/upgrades, non-existing values in the user
    configuration will be initialised from the presets, but existing values
    will be retained.)

Besides upgrades, there are some more issues of UCI I'd like to solve with
NC:

* UCI config files are designed to be both human- and machine-readable/
  writable (also, parsable by shell scripts.) NC would rather use a binary
  format that can be read, written (regarding performance) and stored
  (regarding space) more efficiently, combined with an intuitive
  configuration tool (maybe optionally curses-based)
* UCI configuration has a flat section/key/value structure, NC would allow
  (JSON-like?) structured configuration
* NC would use only a single database for storage of the user configuration
  (JFFS2 doesn't seem to handle lots of small files very efficiently; this
  needs more research)
* UCI only stores strings. This has led to some funny inconsistencies (for
  storing booleans, any of the following values are understood as true:
  1/on/true/yes/enabled.) NC should natively support at least all primitive
  data types of JSON.
* UCI separates the "save" and "commit" stages when changing configuration.
  Changes that have been saved but not committed will already be effective,
  but aren't stored to non-volatile memory till they are committed.
  This can not only be confusing to users, but also leads to bad
  programming practices like making configuration changes in scripts that
  are never supposed to be committed, which is obviously fragile as other
  scripts or the user may commit the config.
  NC would not separate these steps: Configuration should become effective
  only when it is stored permanently.
* Comments in UCI files are possible, but get lost when the file is
  changed using libuci or the uci CLI. NC could support documentation of
  config keys in the schema.

This project would involve designing the configuration formats (schema
files and user configuration), and implementing:
* C library
* CLI configuration tool
* Lua binding
* Some kind of shell support (for accessing NC from initscripts etc.)
* Documentation for all of this

I'm also thinking about adding "active defaults", i.e. default values that
are not statically defined in the schema, but computed by plugin modules.
This would e.g. allow Gluon packages to install complex configuration for
other packages without the need to store this in the user config in /etc.

I'm not totally sure about the time required to implement all of this, but
I suspect I might be finished in less than the 3 months of GSoC coding
time. Therefore, and to give NC some more practical testing, I'm thinking
about implementing a minimal version of [4] (which was also proposed by me)
based on NC. [4] is obviously a very complex project which I believe is too
big to complete during the time of the GSoC, but I think minimal version to
show the soundness of NC would be a nice addition to this project (for
example only supporting static address/route configuration and network
namespaces.)

Of course, if another student wants to work on [4] (and someone wants to
mentor it), I'd rather not take [4] away from them and think of something
else to add to my project in case it gets finished too quickly.

Regarding mentoring, I'd love to get a mentor who is not involved with
Gluon yet, so he or she can make sure what I create is not only useful for
Gluon, but also for the broader OpenWrt community (or even for other Linux
distributions.)

Please let me know what you think of this proposal.

Regards,
Matthias

[1] https://projects.universe-factory.net/projects/fastd/wiki
[2] https://github.com/freifunk-gluon/gluon
[3] https://lists.openwrt.org/pipermail/openwrt-devel/2015-March/031628.html
[4]
https://wiki.freifunk.net/Ideas#Profile-based_network_configuration_for_OpenWrt

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freifunk.net/pipermail/wlanware-freifunk.net/attachments/20160306/63aa4ea1/attachment.sig>