private thoughts about life behind data processing

Monthly Archives: elokuu 2017

Why Open Source is a Great Thing for Public Orgranizations

No activity here lately. I have been busy with my other recreational project visitkoli.fi1. This post is based on the speech I had2 in WMO EC-693 Side-Event last May.

Weather and climate business is a somewhat special niche sector of IT-field. The sector is large enough that significant systems can be made. Actually, even the World’s most powerful computers are used to calculate climate scenarios4. But the sector is also small enough that no proper software ecosystem has been arisen for processing and handling the data. The sector also mainly consists of public institutes and large companies which is apt to keep developed software private and bespoke.

And this is why also my employer, Finnish Meteorological Institute (FMI), has a variety of different kind of software to handle weather and climate information.

FMI though, is doing a remarkable effort to build the ecosystem. In 2012, it decided to release its data as open data. In 2013 a great share of all available data was published. And in 2016, FMI decided to release its software as open source.5 In theory the decision concerned all software, old and new. But in practice, every software have to be considered separately. And there’s no resources allocated to the release.

Despite of these minor caveats, some internationally very relevant software has been published. For example SmartMet Server6 (data and product server for MetOcean data) and HIMAN7 (post-processing tool for weather data) are world class software. As a ’product owner’, publishing the SmartMet Server has been my task. It has been hard work. My intention is to tell how hard, in the next post. Before that I will tell you why publishing software as open source is a great thing to do for any public organisation.

First of all, ecosystem is changing. Public organisations or typical companies can’t control the whole value chain anymore. Demands for very specific services have been increased and there are plenty of companies who are happy to fill them. At the same time large companies like IBM and Google are becoming more and more relevant players. So far many public sector’s tasks like weather forecasting or cartography have been so complicated and resource intensive work that only governments have been able to do this. But today, many technology giants have enough resources and know-how to complete the tasks. (Although often they base their work on open source models developed by public sector.)

This, of course, cast some challenges and changes on public organisations who want to pursue in business:

  1. Public organisations need more efficiency in development and operations.
    Even without healthy competition public sector need to achieve more with less resources in the future. 
  2. They need to ensure their authoritative voice.
    In some cases, objective and trustworthy information is vital. Tornado warnings, statistics, news are few examples.
  3. Most importantly, they to ensure an impact of produced information.
    For example, national meteorological and hydrological institutes are successful when weather and climate don’t cause unanticipated unwanted impact to society.

Increasing collaboration is one of the most powerful ways to increase efficiency. For example  are very well connected concerning observation and modelling activities. They have had to. Countries can’t do observations on others ground. And very few country have resources to run global weather forecast model on their own. But there’s a long way from model output to end user information. And closer we come to end user, more protective organisations tend to be.

Why not extend collaboration to upstream as well? Unfortunately international co-operation, what for example weather business requires, is specially challenging due to different cultures, motivation, environment, legislation and so on. Using open source software may lower these barriers a lot. It makes collaboration open and easy. There’s no need for long and burden negotiations (or at least it one topic can be handled easily). No-one fall in vendor lock. If one needs any changes or modifications, they can be done by oneself or ordered from 3rd party.

Open source software enables every organisation to provide their own service without doing things many times.

Ensuring authoritative voice and the impact of produced information is mainly a question how widely and correctly produced information reach relevant audience. Maximal coverage requires several different channels and services. One organisation can’t handle them all. Publishing data and products as open data helps, but in many cases the data is so complicated that information distorts easily. Providing tools to handle and analyse the data empowers 3rd party users to correctly utilize it. Thus, proper tools ensures consistency between information regardless of channel and service providers.

There’s also a one more aspect in open source, which concerns specially public organisations. Software produced by government institute is eventually produced and owned by the whole society. Releasing software as open source is a very simple and neutral way to ensure that everyone can utilise it. Companies, private developers and other countries.

”No one is left behind”8.