Network Topology Visualizer: django-netjsongraph

2comments

3rd March 2016 in Coding Tags: battlemesh, netjson, network, ninux, routing

July 2017 Update: you can now use the automatic openwisp2 installer to deploy the network topology visualizer.

Recently I've been working on a network topology visualizer based on NetJSON, which I named django-netjsongraph.

There are several motivations that led me to write this django app:

  • I liked the idea of being able to visualize and monitor a network by costantly analyzing its topology
  • I needed this kind of functionality for Ninux.org, an wireless community network that aims to build free networks in the Italian peninsula
  • I wanted to show the potential of a standard data interchange format like NetJSON, which makes it easy to visualize and monitor a network topology

Initially I was developing this feature in a larger web app, but then I understood it would have been better to publish it in a separate package in order to make it more reusable and maintainable.

The app is subdivided into several components:

  1. Network Topology Parser
  2. Network Topology Collector
  3. Network Topology Visualizer
  4. HTTP API
  5. Admin Interface

Let's dig into the details about how these components are implemented

1. Network Topology Parser

Most of the logic of this component is in reality implemented as a dedicated python library called netdiff.

Netdiff in charge of parsing different network topology formats and detecting changes over time (new links, changes in metrics, offline links).

Each format is implemented as a parser and the current parsers support the following formats:

  • NetJSON NetworkGraph
  • OLSR (jsoninfo/txtinfo)
  • BATMAN-advanced (jsondoc/txtinfo)
  • BMX6 (q6m)
  • CNML 1.0

Adding support for a new format is just a matter of writing a new parser, add it to the python path and include it in the NETJSONGRAPH_NETDIFF_PARSERS setting.

You might be wondering, where is the OLSRd2 parser?
OLSRd2 ships a netjsoninfo plugin that returns NetJSON output natively, so no special parsers are needed for it, nice isn't it?

2. Network Topology Collector

network topology collector django model

This component is made of traditional django models (as in MVC) and django management commands which take care of using the parser component to detect the current state of the network topology and save it to one of the supported databases (django supports several relational databases).

It is made in a way that should facilitate including django-netjsongraph in larger projects and extend it. At the moment of this writing I haven't tested such a setup yet but it is my intention to do it as soon as possible because is something I really need.

The system allows to define multiple networks and collect topology data for each of them.

FETCH Strategy

The classic way to collect topology data is to fetch topology data from a URL, I called this "FETCH strategy".

This strategy is the simplest to implement but requires more effort from users because they have to publish the topology data on URL which is reachable by the collector. This involves writing scripts that fetch topology data and send it to a server (via SSH) which has a public ip address and a webserver installed.

Many users have requested the possibility to push topology data directly from nodes to the collector in a POST HTTP request.

To accomodate this need I implemented the RECEIVE strategy.

RECEIVE Strategy

This strategy allows nodes to send network topology data in the payload of a POST HTTP request.

This way a simple shell script like the following one is sufficient to start collecting topology data:

    #!/bin/sh
    BASE_URL="https://network-topology-visualizer.mynetwork.org"
    UUID="4e38397a-4254-4521-89a6-045db25b8de6"
    KEY="S9wCfjeFN8irasZNiV7dGcyK68BTES0m"

    COLLECTOR_URL="$BASE_URL/api/receive/$UUID/?key=$KEY"
    DATA=$(echo "/netjsoninfo filter graph ipv4_0" | nc 127.0.0.1 2009)
    curl -s -X POST -d "$DATA" --header "Content-Type: text/plain" $COLLECTOR_URL

The preceding script queries the olsrd2 netjsoninfo plugin, retrieves the NetJSON NetworkGraph and POSTS it to the collector.

As you can see there is a simple authentication mechanism which requires each POST request to include a "key" parameter which must match the topology's key. Each topology object added to the system will have a uuid and a key.

The cool thing about the RECEIVE strategy is that more nodes can send their topology to the collector, this allows ot mitigate the "Single Point of Failure" problem in which the URL that needs to be fetched is not reachable for some reason.

This feature also allows to monitor and visualize the network when it's broken in several smaller parts, like when important links fail (provided that some nodes are connected to the internet with private gateway, if they share the same gateway they won't be able to reach the collector if central links fail).

3. Network Topology Visualizer

network topology collector django model

This component is what allows casual users to see the network and inspect its nodes.

The engine of this component is implemented in a javascript library called netjsongraph.js.

netjsongraph.js is based on the popular d3.js javascript visualization library and takes care of reading the json graph format (NetJSON NetworkGraph) and generate a force-directed graph.

4. HTTP API

network topology collector django model

This component is based on Django REST Framework, its main purpose is to provide HTTP resources for retrieving topology data in NetJSON format.

This API also allow nodes to POST topology data when using the "RECEIVE strategy".

5. Admin

Network Graph, Topology, Link

This is the web interface (based on the django-admin) that allows administrators to add new topologies to the system.

Next steps

The next phase of development of this project will involve adding more topology formats, integrate the app in larger projects and prepare an easy-to-install package with meaningful deafult settings.

Another use case I would like to test is the "generic topology collector" concept proposed at the Battlemesh v8 by Axel Neumann (core developer of BMX routing protocol): if each node of a distance vector routing protocol could return it's known topology in NetJSON, django-netjsongraph could work as a generic topology collector for distance vector routing protocols.

This should already work, but I'm sure that collecting topology data from 2-3 nodes is quite different than doing it for a few hundred nodes and that is exactly what I would really like to test.

August 2017 Update: I had to close comments because of spam. To ask questions, please use one of the OpenWISP Support Channels.

Retweet

Comments

  1. 1.

    Nasrin Akter said:

    ( on 11th of April 2016 at 18:28 )

    I have a question about Network Topology Collector, can you brief me pls?

  2. 2.

    Federico said:

    ( on 12th of April 2016 at 22:21 )

    Send any question to the interop-dev mailing list or open an issue on github.

Comments are closed.

Comments have been closed for this post.

Categories

Let's be social

Popular posts

Latest Comments

  1. I got very good results with this, thanks for sharing.

    By Yasir Atabani in How to speed up tests with Django and PostgreSQL

  2. Hi Amad, for any question regarding OpenWISP, use one of the support channels: http://openwisp.org/support.html

    By Federico Capoano in How to install OpenWISP

  3. Sir please guid , i have install the ansible-openwisp2 , now how to add the access points . What is the next procedure . Please help.

    By Ahmad in How to install OpenWISP

  4. Hi Ronak, for any question regarding OpenWISP, use one of the support channels: http://openwisp.org/support.html

    By Federico Capoano in netjsonconfig: convert NetJSON to OpenWRT UCI

  5. Hi, I have installed openwisp controller using ansible playbook. Now, i am adding the configurations automatically using OPENWRT devices in openwisp file by specifying shared_key so can you suggest me if I want to set limit to add configuration how can i do it?

    By Ronak in netjsonconfig: convert NetJSON to OpenWRT UCI

Popular Tags

battlemesh censorship creativity criptography django event fosdem google-summer-of-code ibiza inspiration javascript jquery linux nemesisdesign netjson ninux nodeshot open-source openwisp openwrt performance photo programming python security staticgenerator talk upload wifi wireless-community