Technical description
This page aims to describe the technical setup of the project bank. Below is a figure that shows the data flow in the system, starting from the SQL database, ending up in the frontend which displays the graphics for the user.
Microsoft SQL Server: This is the data source for the Project Bank. The Project Bank fetches data from an SQL view, currently containing 4.1 million rows where each row contains information about one grant and the project it is related to. Included in this row is the year of the grant, the amount granted, the title of the project, the organisation the grant is related to, and all other data that is seen in the Project Bank.
Feeder: A Java-application that uses Logstash in order to transform and transfer the data to ElasticSearch. According to http://elastic.co/logstash, "Logstash is a free and open server-side data processing pipeline that ingests data from a multitude of sources, transforms it, and then sends it to your favorite 'stash.'". This application is run automatically when the content of the Microsoft SQL database updates.
Elasticsearch: A search engine. According to https://www.elastic.co/elasticsearch/, "Elasticsearch is a distributed, RESTful search and analytics engine capable of addressing a growing number of use cases. As the heart of the Elastic Stack, it centrally stores your data for lightning fast search, fine‑tuned relevancy, and powerful analytics that scale with ease." In the Project Bank, Elasticsearch is used to index all data in order to quickly filter and transform the data into something manageable for the REST API.
Backend/REST API: Written in the Java web service framework Jersey. This module is the interface between the frontend and ElasticSearch. Its responsibility is mainly to receive requests from the frontend, then query ElasticSearch and transform the results to something manageable by the frontend and then returning this to the frontend.
Frontend: The frontend is built in ReactJs and does everything that has to do with visualization and user interface. Here you can filter data and get graphic representations of what the user wants to see in relation to the data available. In order to make the Project Bank searchable in Google, we used the framework Nextjs in order to initially render the pages on the server side. Contentful as a Content Management System is used in order to simplify the process of updating content on the page such as titles or images without having to modify the code.