X-Road is a software framework that enables direct secure data exchange over internet between state and private organizations (hereafter “members”). It is the Estonian official software platform for data exchange between public and private sector institutions and enterprises. X-Road members can act as data clients or data providers or both. Data service providers make their data accessible for other members or, alternatively, collect new data from others. All state registers are data service providers.
There are no central servers or databases in X-Road: all X-Road members host their own databases and members interact directly with each other. As of October 2017, there are nearly 900 X-Road members who make a total of over 40 million data queries per month. There are over 1500 data services that can be used via the X-Road. The number of X-Road members is estimated to double and the number of data exchange events to increase tenfold in the coming 5 years.
X-Road is managed by Estonian Information System Authority (Riigi Infosüsteemi Amet, RIA). RIA issues access to X-Road and monitors the data traffic between members.
Based on the recent X-Road version 6 software, members’ information systems store logs of all data exchange events. These usage logs are collected by RIA forming a basis for further monitoring and analysis of data traffic patterns in X-Road. Before this project, there were no dedicated tools to manage and analyze the X-Road usage statistics. To fulfill this gap, the X-Road monitoring tools were produced by STACC in order to:
- conveniently store and manage X-Road usage statistics data;
- detect potential misuse and anomalies in the data exchange between X-Road members;
- report X-Road members about their data exchange patterns: data exchange partners, produced and consumed data services, statistics on counts, data size and duration of data exchange events, etc.;
- produce, manage, share, and visualize publicly available open data on the use of X-Road.
To achieve these aims, a multi-component software was developed at STACC including the following main modules.
- Collector and database module retrieves the usage logs from X-Road members’ security servers and writes them into a database.
- Corrector module matches logs from members in client and producer roles, finds and erases duplicates, calculates durations and sizes of queries.
- Analyzer module detects anomalies and potential misuse incidents, the main types of which are: high proportion of failed queries between data exchange parties, significant change in the number, duration, and data size of queries in a given time frame. The Analyzer adaptively learns the incident patterns based on administrator confirmations or rejections of potential incidents.
Reports module produces two types of reports:
- Member reports: reports about the members’ and their subsystems’ X-Road usage patterns including the list and statistics of their data exchange partners, produced and consumed data services, statistics on counts, data size and duration of data exchange events;
- Open reports: monthly summary statistics of X-Road usage over all members including count of members, total count of queries, counts of services and producers, etc.
Open data module produces and delivers publicly available X-Road usage data. The module consists of two submodules:
- Anonymizer removes usage statistics of security authorities and decreases the precision of X-Road usage logs’ timestamps and writes the data into Open data database;
- Graphical web interface enables a user to preview and download X-Road usage statistics. Date and data fields can be filtered and sorted before viewing and downloading. An API is also provided.
Networking module graphically visualizes the communication between X-Road members. The web interface enables to select all members or one specific member, to select the level of details to be shown, and to filter data exchange pairs based on query counts.
Software technologies that were used in this project: Python, MongoDB, PostgreSQL, SQLite, Django, Apache HTTP server, R, RStudio Shiny Server.
General information about the X-Road can be found at: https://www.ria.ee/en/x-road.html
Project contacts:
Kristjan Eljand (kristjan.eljand@www.stacc.ee) – CEO of STACC
Kristjan Herkül (kristjan.herkul@www.stacc.ee) – Project Manager at STACC
Toomas Mölder (toomas.molder@ria.ee) – Project Manager at RIA
