netjsonconfig: convert NetJSON to OpenWRT UCI

2comments

6th October 2015 in Coding Tags: netjson, ninux, open-source, openwisp, openwrt

Update (1st of Dec 2015):
detailed technical documentation is available at netjsonconfig.openwisp.org.

Update (February 2017):
netjsonconfig is now the official configuration engine of the OpenWISP 2 Controller. for more information you can take a look at How to install OpenWISP and Introduction to OpenWISP 2.

netjsonconfig is a python implementation of the NetJSON data interchange format, more specifically the DeviceConfiguration object.

Yesterday I issued a 0.1 beta pre-release and I am now testing it on several different routers.

When I talk about this library some people usually ask me why did you re-implement UCI in JSON?.

A short, over-simplified answer would be: it's a very good approach if you have to mass manage router configurations via web applications.

But to properly explain the benefits of this approach I have to put the discourse in the right context, so I will try to give a longer, more detailed explanation in this post.

My work involves managing thousands OpenWRT access points, ensure their configuration is up to date, monitor them and so on; you cannot do this manually, think about when you need to change settings like VPN termination, or the name of the SSID, you need some sort of automation.
There are a few controllers that are designed to work on these tasks, and there are even a few ones which are released with FLOSS licenses.

The problems with current FLOSS wifi controllers

The need for this library has arisen while working on the OpenWISP Manager project (from now on OWM), a ruby on rails web app that allows to mass manage OpenWRT devices flashed with a specific firmware (OpenWISP Firmware).

OWM has some really cool features, but also some limitations which make it hard to extend and make its usage unsuited in other settings outside the municipal wifi arena.

The worst limitations of OWM in my opinion are the following ones:

  1. the code that generates the OpenWRT UCI configurations is highly coupled with the Ruby on Rails framework, which makes OWM completely useless for all those networking people who don't use ruby, nor rails, which IMHO is the majority of the networking community
  2. the generated configs work only with OpenWISP Firmware; it would have been really interesting if OWM could manage vanilla OpenWRT devices and if it could offer a way to add support for other firmwares
  3. adding new configuration options in the web interface requires to alter the database: writing a rails migration file, adding the field in the UI and then add the generated line in the UCI templates; in 2015 (almost 2016) we can do better than this!
  4. OWM ships too many features in a single codebase, which over the course of time, has become hard to maintain

I have looked at other wifi controllers, but I have found pretty much the same limitations in all of them.

The things that bothers me most is that this kind of software (even OpenWISP) it's all or nothing: you either use the entire stack or choose a different stack, which reminds me of a certain IT company with a fruity logo ;-) ... not nice at all.

I have also studied the NETCONF RFC, which I considered for some time, but abandoned after understanding its complexity are not suited for my own use.
I need something SIMPLE, possibly based on JSON (because it's easier to use in web apps).

The NetJSON approach

I started working on netjsonconfig to overcome the limitations found in OpenWISP Manager and the OpenWISP team greeted the idea and supported it.

Let's see how I intend to overcome the limitations that we have in OWM:

Regarding problem n. 1, netjsonconfig is a standalone, pure python library with very little dependencies (check the requirements.txt). Python is widely used in the networking community, but the library ships an executable command line utility that may be used from shell scripts or other languages as well (by spawning a new process to execute the netjsonconfig utility, eg: the subprocess module in python).

Regarding problem n. 2, netjsonconfig introduces the concept of "backend": in version 0.1 there is an OpenWrt backend, which generates an OpenWRT configuration, the next version will ship an OpenWISP Firmware backend.
It will be possible to add other backends (an AirOS one would be really useful for some community networks) to the library itself, or alternatively, custom backends can be published in separate packages.

Problem n. 3 does not concern netjsonconfig directly because it involves dealing with a database and a web UI, which according to our plan, will be implemented as a separate program which will make use of netjsonconfig behind the scenes.
To be more specific, my idea is to write a reusable django app (see the link to know more about what a reusable django app is) which will be easily integrable in larger projects.
The app will use a single JSON field to store the configurations in NetJSON DeviceConfiguration format.
In a production environment the json field will very likely be stored in a PostgreSQL Binary JSON column, but I want to make sure SQLite works fine too, in order to make installation easy.
Regarding the web UI, it will likely be automatically generated leveraging JSON Schema which both NetJSON (see Device Configuration schema) and netjsonconfig (see OpenWrt custom NetJSON schema) use.
There are in fact a few implementations that automatically generate a good enough web UI from a JSON Schema.
With such a design, adding new configuration options will be just a matter of adding a few lines of code in the netjsonconfig backend and its schema (each backend has a schema);
no more messing with database migrations; no more messing with UI unless extremely necessary.

Update (20st of Dec 2015):
such reusable django app is now released as django-netjsonconfig.

Regarding problem n. 4, netjsonconfig follows the unix philosophy and deals with configurations only, which makes it easy to use, test, document and contribute to.
Remember: solving one problem is easier to get right; trying to solve many complex problems in a single codebase probably means heading for trouble.

Example netjsonconfig usage

Here's a working NetJSON DeviceConfiguration object stored in a file, eg: ./config.json.

{
    "general": {
        "hostname": "netjsonconfig",
        "timezone": "Coordinated Universal Time"
    },
    "interfaces": [
        {
            "name": "lo",
            "type": "loopback",
            "addresses": [
                {
                    "address": "127.0.0.1",
                    "mask": 8,
                    "proto": "static",
                    "family": "ipv4"
                }
            ]
        },
        {
            "name": "eth0",
            "type": "ethernet"
        },
        {
            "name": "br-eth0_1",
            "type": "bridge",
            "bridge_members": [
                "eth0.1",
                "wlan0"
            ]
        },
        {
            "name": "eth0.1",
            "type": "ethernet",
            "addresses": [
                {
                    "address": "172.17.0.1",
                    "gateway": "193.205.218.1",
                    "broadcast": "172.17.0.255",
                    "mask": 24,
                    "proto": "static",
                    "family": "ipv4"
                }
            ]
        },
        {
            "name": "eth0.2",
            "type": "ethernet",
            "addresses": [
                {
                    "address": "193.206.99.160",
                    "gateway": "193.206.99.129",
                    "mask": 25,
                    "proto": "static",
                    "family": "ipv4"
                }
            ]
        },
        {
            "name": "wlan0",
            "type": "wireless",
            "addresses": [
                {
                    "proto": "dhcp",
                    "family": "ipv4"
                }
            ],
            "wireless": {
                "radio": "radio0",
                "mode": "access_point",
                "ssid": "wpa2-personal",
                "encryption": {
                    "enabled": true,
                    "protocol": "wpa2_personal",
                    "ciphers": [
                        "tkip",
                        "ccmp"
                    ],
                    "key": "passphrase012345"
                }
            }
        },
        {
            "name": "wlan1",
            "type": "wireless",
            "addresses": [
                {
                    "address": "172.20.2.1",
                    "mask": 32,
                    "proto": "static",
                    "family": "ipv4"
                },
                {
                    "address": "fd87::1",
                    "mask": 128,
                    "proto": "static",
                    "family": "ipv6"
                }
            ],
            "wireless": {
                "radio": "radio1",
                "mode": "adhoc",
                "ssid": "wbm8-test-5",
                "bssid": "02:b8:c0:00:00:00"
            }
        }
    ],
    "radios": [
        {
            "name": "radio0",
            "phy": "phy0",
            "driver": "mac80211",
            "protocol": "802.11n",
            "channel": 11,
            "channel_width": 20,
            "tx_power": 15,
            "country": "IT",
            "frag": 140,
            "rts": 160
        },
        {
            "name": "radio1",
            "phy": "phy0",
            "driver": "mac80211",
            "protocol": "802.11n",
            "channel": 48,
            "channel_width": 20,
            "tx_power": 4,
            "country": "IT",
            "disabled": true
        }
    ],
    "switch": [
        {
            "name": "switch0",
            "reset": true,
            "enable_vlan": true,
            "vlan": [
                {
                    "device": "switch0",
                    "vlan": 1,
                    "ports": "0t 2 3 4 5"
                },
                {
                    "device": "switch0",
                    "vlan": 2,
                    "ports": "0t 1"
                }
            ]
        }
    ],
    "dns_servers": ["193.204.5.4", "8.8.8.8"],
    "dns_search": ["netjson.org", "openwisp.org"],
    "ntp": {
        "enabled": true,
        "enable_server": false,
        "server": [
            "0.openwrt.pool.ntp.org",
            "1.openwrt.pool.ntp.org",
            "2.openwrt.pool.ntp.org",
            "3.openwrt.pool.ntp.org"
        ]
    },
    "led": [
        {
            "name": "USB1",
            "sysfs": "tp-link:green:usb1",
            "trigger": "usbdev",
            "dev": "1-1.1",
            "interval": 50
        },
        {
            "name": "USB2",
            "sysfs": "tp-link:green:usb2",
            "trigger": "usbdev",
            "dev": "1-1.2",
            "interval": 50
        },
        {
            "name": "WLAN2G",
            "sysfs": "tp-link:blue:wlan2g",
            "trigger": "phy0tpt"
        }
    ],
    "dropbear": [
        {
            "config_name": "dropbear",
            "PasswordAuth": "on",
            "RootPasswordAuth": "on",
            "Port": 22
        }
    ]
}

Install netjsonconfig with

pip install netjsonconfig

With the render method of the OpenWrt backend we will get a uci import compatible output.

The following command:

netjsonconfig --config config.json --backend openwrt --method render

Will return the following output:

package system

config system
	option hostname 'netjsonconfig'
	option timezone 'UTC'

config timeserver 'ntp'
	list server '0.openwrt.pool.ntp.org'
	list server '1.openwrt.pool.ntp.org'
	list server '2.openwrt.pool.ntp.org'
	list server '3.openwrt.pool.ntp.org'
	option enable_server '0'
	option enabled '1'

config led 'led_usb1'
	option dev '1-1.1'
	option interval '50'
	option name 'USB1'
	option sysfs 'tp-link:green:usb1'
	option trigger 'usbdev'

config led 'led_usb2'
	option dev '1-1.2'
	option interval '50'
	option name 'USB2'
	option sysfs 'tp-link:green:usb2'
	option trigger 'usbdev'

config led 'led_wlan2g'
	option name 'WLAN2G'
	option sysfs 'tp-link:blue:wlan2g'
	option trigger 'phy0tpt'

package network

config interface 'lo'
	option dns '193.204.5.4 8.8.8.8'
	option dns_search 'netjson.org openwisp.org'
	option ifname 'lo'
	option ipaddr '127.0.0.1/8'
	option proto 'static'

config interface 'eth0_1'
	option broadcast '172.17.0.255'
	option dns '193.204.5.4 8.8.8.8'
	option dns_search 'netjson.org openwisp.org'
	option gateway '193.205.218.1'
	option ifname 'eth0.1'
	option ipaddr '172.17.0.1/24'
	option proto 'static'
	option type 'bridge'

config interface 'eth0_2'
	option dns '193.204.5.4 8.8.8.8'
	option dns_search 'netjson.org openwisp.org'
	option gateway '193.206.99.129'
	option ifname 'eth0.2'
	option ipaddr '193.206.99.160/25'
	option proto 'static'

config interface 'wlan0'
	option dns '193.204.5.4 8.8.8.8'
	option dns_search 'netjson.org openwisp.org'
	option ifname 'wlan0'
	option proto 'dhcp'

config interface 'wlan1'
	option dns '193.204.5.4 8.8.8.8'
	option dns_search 'netjson.org openwisp.org'
	option ifname 'wlan1'
	option ipaddr '172.20.2.1/32'
	option proto 'static'

config interface 'wlan1_2'
	option dns '193.204.5.4 8.8.8.8'
	option dns_search 'netjson.org openwisp.org'
	option ifname 'wlan1'
	option ip6addr 'fd87::1/128'
	option proto 'static'

config switch
	option enable_vlan '1'
	option name 'switch0'
	option reset '1'

config switch_vlan
	option device 'switch0'
	option ports '0t 2 3 4 5'
	option vlan '1'

config switch_vlan
	option device 'switch0'
	option ports '0t 1'
	option vlan '2'

package wireless

config wifi-device 'radio0'
	option channel '11'
	option country 'IT'
	option frag '140'
	option htmode 'HT20'
	option hwmode '11g'
	option phy 'phy0'
	option rts '160'
	option txpower '15'
	option type 'mac80211'

config wifi-device 'radio1'
	option channel '48'
	option country 'IT'
	option disabled '1'
	option htmode 'HT20'
	option hwmode '11a'
	option phy 'phy0'
	option txpower '4'
	option type 'mac80211'

config wifi-iface
	option device 'radio0'
	option encryption 'psk2+tkip+ccmp'
	option key 'passphrase012345'
	option mode 'ap'
	option network 'wlan0 eth0_1'
	option ssid 'wpa2-personal'

config wifi-iface
	option bssid '02:b8:c0:00:00:00'
	option device 'radio1'
	option mode 'adhoc'
	option network 'wlan1'
	option ssid 'wbm8-test-5'

package dropbear

config dropbear
	option PasswordAuth 'on'
	option Port '22'
	option RootPasswordAuth 'on'

We can also generate a configuration archive that can be restored from the OpenWrt web interface or from the command line with sysupgrade -b archive.tar.gz with the following command:

netjsonconfig --config config.json --backend openwrt --method generate > config.tar.gz

Which will generate an archive named openwrt-config.tar.gz with the following directory structure:

    /etc/
        config/
            dropbear
            network
            wireless
            system

Each file will contain the correct UCI config options relative to its package.

Let's collaborate

That's all I got for now. If you like these ideas and you want to try hack some code with me, get in touch with me, I'll be more than happy to discuss and collaborate!

If you want to keep up to date with the development efforts, follow the OpenWISP twitter account or the watch the netjsonconfig github repository.

Retweet

Comments

  1. 1.

    Ronak said:

    ( on 17th of April 2017 at 14:11 )

    “Hi,

    I have installed openwisp controller using ansible playbook. Now, i am adding the configurations automatically using OPENWRT devices in openwisp file by specifying shared_key so can you suggest me if I want to set limit to add configuration how can i do it?”

  2. 2.

    Federico Capoano said:

    ( on 18th of April 2017 at 16:02 )

    “Hi Ronak,

    for any question regarding OpenWISP, use one of the support channels:
    http://openwisp.org/support.html”

Comments are closed.

Comments have been closed for this post.

« More recent post Older post »

Categories

Let's be social

Popular posts

Latest Comments

  1. “ I got very good results with this, thanks for sharing. ”

    By Yasir Atabani in How to speed up tests with Django and PostgreSQL

  2. “ Hi Amad, for any question regarding OpenWISP, use one of the support channels: http://openwisp.org/support.html ”

    By Federico Capoano in How to install OpenWISP

  3. “ Sir please guid , i have install the ansible-openwisp2 , now how to add the access points . What is the next procedure . Please help. ”

    By Ahmad in How to install OpenWISP

  4. “ Hi Ronak, for any question regarding OpenWISP, use one of the support channels: http://openwisp.org/support.html ”

    By Federico Capoano in netjsonconfig: convert NetJSON to OpenWRT UCI

  5. “ Hi, I have installed openwisp controller using ansible playbook. Now, i am adding the configurations automatically using OPENWRT devices in openwisp file by specifying shared_key so can you suggest me if I want to set limit to add configuration how can i do it? ”

    By Ronak in netjsonconfig: convert NetJSON to OpenWRT UCI

Popular Tags

battlemesh censorship creativity criptography django event fosdem google-summer-of-code ibiza inspiration javascript jquery linux nemesisdesign netjson ninux nodeshot open-source openwisp openwrt performance photo programming python security staticgenerator talk upload wifi wireless-community