Another short post to share epic news: we recently received an unrestricted grant of 50,000 USD from Google to support the development of Measurement Kit! Specifically, with "support" here we mean paying salary of persons involved (chiefly me, but also others to a smaller extent) and also travels to/from related hackathons.
We intend to use this grant to implement the testing methodology of the Network Diagnostic Tool for Measurement Kit. We are also considering what other network performance methodologies it would be optimal to implement and discussing on how to enable the Neubot software to run those tests.
Today I am pleased to announce that we have recently released two hopefully stable versions of neubot-runtime and neubot-server. These two releases come after many (perhaps too many) months of development aiming at moving forward with Neubot.
I already mentioned before these efforts that basically strive to split Neubot client and Neubot server to allow to develop them independently. In this context, neubot-runtime is the shared piece of software containing the basic classes used by both the client and the server of Neubot. The latest (and first) stable release of neubot-runtime, v0.4.17.0, contains code that was tested and known to run with neubot-server. Not all the code inside neubot-runtime is already compatible with python3, but we're working in this direction.
Instead, neubot-server contains (as the name implies) only Neubot server code. This first release, v0.5.0.0, is based on neubot-runtime 0.4.17.0. One notable feature that we introduced is that now all tests are independent modules loaded at runtime. Work in this direction was already initiated during the release process of Neubot 0.4.16.
We will now start deploying the new neubot-server on mlab test machines and, when we will be satisfied, we will deploy it (or a subsequent version of it) on all mlab machines, for all clients to use it.
This piece of work (moving forward on the server side) allows us to concentrate now on the client side, for which we tentatively scheduled a new release (0.4.17.0) for end of March.
A short post to say that we've recently made available the first and second stable releases of MeasurementKit, the network measurements library for mobile devices to which we've been working for more than two years now!
The first release (0.1.0 released on December 16, 2015) focused on providing an initial stable API for running OONI tests and iOS and Android support.
The second release (0.1.1 released on January 13, 2016) added some bug fixes and improvements.
Such radical changes reflect profound transformations that have been happening in the last 1.5 years.
Basically, at a certain point the focus moved away from the Neubot software tool alone, and started to be measuring the internet from the edges with whatever mean, provided that it was adequate for the task.
This happened, I guess, during the MorFEO project (i.e. in 2014).
It's ironic that in the same post we have a project called Morfeo (i.e. Morpheus) and content sleeping with the fishes, don't you think? (Source: God Father The Game (Wikia)).
Today, preparing my talk for the (and my!) seventh Nexa conference, I reckoned that the time had come to finally attempt to sync up this website with reality.
As said, the new content now reflects reality. In the last two years I have been working mostly on what was initially called libneubot, then libight, and finally MeasurementKit. This product is a portable network measurement library meant to support network-performance and internet-censorship measurements from mobile devices (iOS and Android). It is a new product because Neubot, as a tool written in Python, was not adequate to perform mobile measurements -- a thing that we're interested to do.
And yes this library, MeasurementKit, was significantly shaped by MorFEO. In fact MeasurementKit is jointly developed with the OONI project, which was partner of MorFEO.
Does this mean that Neubot (as a software) will die? No. I am currently working with Davide (another Neubot core developer) and Carmine to, let's say, extract the code that used to be Neubot from the ball of mud that Neubot become during the last two years of my doctorate. We're making a good job at that. We already have a quasi-stable implementation of the Neubot core classes (the "runtime"), we made this portable to Python3, we separated client code and server code, and now server code kinda works using the runtime.
While writing this blog post I was listening to this song. At a certain point it says: "it shouldn't be impossible to use scissors if this is exactly what you want".
The plan for the next one year or so is to focus on MeasurementKit adding cool features and to maintain Neubot and engineer its code to make it simple again to do useful stuff with it. This seems to me at the moment the most rational way to move forward.
And possibly that's what I'll say tomorrow during my lightning talk at Nexa Conference. Now closing this log and focusing on the slides deck. I will likely have more to say in the near future.
Most hacking during these three months period concentrated on MeasurementKit (mainly pre-release cleanups) and NetworkMeter. This is a rather technical post. But, before entering into the details of the merged pull requests, here's some details for humans. I had a great summer vacation time, spanning three weeks all along August, in my home city (Bordighera) and nearby cities (such as Ventimiglia). Of course I never went to the sea, but spent lots of time with friends and dedicated the nights (after 2:00 AM) to hacking. The following photo shows the harbor of Bordighera from a small nearby hill (and reminds me of one famous picture by Monet also authored in Bordighera).
I have also started contributing to network-meter, a shell for running network measurement tools (e.g. Neubot, OONI) and for visualizing their results. This effort is spearheaded by Poly, a OONI volunteer. Also network-meter is on GitHub under the umbrella of the measurement-kit organization.
Il 23 Aprile scorso ho presentato Neubot durante l'evento Digital Championship di Turin. In effetti io non sono un Digital Champions e originariamente infatti avevano invitato a parlare di Neubot Federico Morando che e' il Digital Champion di Novi Ligure. Tuttavia lui non c'era, cosi' alla fine sono andato io.
Durante la mia presentazione ho descritto la storia di Neubot (il focus sulla neutralita' della rete, gli obiettivi di ricerca) e ho parlato degli obiettivi futuri (misure su mobile, empowerment degli utenti via corsi e altri materiale online, scrittura di esperimenti che tengono conto di cio' che e' importante con gli utenti). Qua di seguito il video, che parte proprio nel momento in cui inizio a parlare:
Non ho preso un voto altissimo, ma a fine manifestazione un po' di persone mi hanno comunque fatto i complimenti per il progetto. E' stato anche interessante lo scambio che ho avuto con Marco Boglione (verso la fine del mio intervento) a proposito di open source e modelli di business.
Osservo come al solito con soddisfazione che il tempo che intercorre tra la decisione di postare sul blog di Neubot e il momento in cui posto e' sempre maggiore di due settimane. Alla prossima!
City of Arts and Sciences (Valencia)—Pictures by me (CC BY 3.0)
NNTools2015. In January we held at Nexa a workshop, called NNTools2015, in which we invited people working on network measurements tools and people working on telecommunication policy to discuss ways to increase joint work by tools developers and within different communities. We are currently working to a paper summarizing up the conclusion of the workshop, which we aim to submit to the Internet Policy Review journal.
CAPS proposal. We are also currently working to submit a CAPS proposal, which will focus on making network measurements more accessible to (and understandable by) people and to incorporate into such tools also measurements that people care about. Among the thing we would like to do, in this respect, there is the idea of preparing an online course explaining (in a pop way, not in the traditional academic way!) what network measurements do and why they matter. The idea to do this came out in a discussion with a brilliant Politecnico di Torino student who attended NNTools2015, Mattia. Below, you find a video of me discussing the concept of our proposal during the Paris CAPS Info Day back in February.
/me speaking at CAPS Info Day (Paris)—Video by Lorenzo (CC BY 3.0)
Neubot server and www. Speaking of Neubot proper, I made some progresses towards deciding the future structure of the software project.
Most of the development work in the last three months focused on libight, a library for running network experiments on the mobile platform developed in cooperation with Arturo Filastò (Tor Project / Hermes Center). This library is developed to implement the mobile applications of Neubot and OONI.
Meanwhile, Alessandro has been working to implement a utp engine for libight. Once libight is stable and the utp engine is merged, the plan is to implement a utp test for Neubot using libight. Probably, given the work to run Neubot tests as independent processes (see below), this will be implementing by packaging an executable linking libight with Neubot.
I was also invited at the 3rd Workshop on Internet Censorship Measurements—organized by Meredith Whittaker of M-Lab and held in New York City the 29-30 October 2014—to discuss issues at the intersection between traffic shaping and Internet censorship.
New York City seen from Central Park—Picture by me (CC BY 3.0)
I have also worked on the draft of the journal paper describing Neubot, integrating changes proposed by Juan Carlos and further polishing and refining the text.