Integrating Data Science and Relational Systems

In his thesis, Mark Raasveldt, researcher in CWI's Database Architectures group, proposes several novel techniques that make database management systems easier to use and more efficient.

The database research community has made tremendous strides in developing powerful database engines that allow for efficient analytical query processing. However, these powerful systems have gone largely unused by data scientists. This poor adoption is caused primarily by the state of database-client integration: current methods of combining databases with analytical tools are slow and cumbersome.

Instead, data scientists have opted to re-invent database systems by developing a zoo of data management alternatives. These systems perform similar tasks to classical database management systems, but have many of the problems that were solved in the database field decades ago.

Bridging the gap
CWI researcher Mark Raasveldt investigated how to bridge this gap by making database management systems easier to use and more efficient for these workloads. The aim was facilitating an efficient and smooth integration of analytical tools and relational database management systems. Raasveldt’s research focused on the three primary methods for database-client integration: client-server connections, in-database processing and embedding the database inside the client application.

Novel techniques
Raasveldt proposed several novel techniques that improve upon the state-of-the-art. In his thesis, he demonstrated a new client-server protocol that is optimized for bulk-transfer of large data sets. This allows for more efficient large-scale data analysis when using remote servers. He also showcased so-called vectorized user-defined functions, that improve in-database processing efficiency through vectorized execution.

Furthermore, Raasveldt describes MonetDBLite, an embedded version of the MonetDB database system that was also developed at CWI. MonetDBLite was developed for efficient integration with Python and R.

DuckDB
In his thesis, Raasveldt also showcased DuckDB, a new data management system purpose-built for efficient and painless integration with Python and R (and other analytical tools).

Raasveldt defended his PhD thesis Integrating Analytics with Relational Databases at Leiden University. He performed his research at CWI’s Database Architectures group, supervised by Stefan Manegold and Hannes Mühleisen. Raasveldt will continue his career at CWI as a postdoc, working on the further development of DuckDB.

Read more.

Read More

  • Peter Boncz appointed ACM Fellow

    CWI researcher Peter Boncz has been appointed as one of the 2022 fellows of the Association of Computing Machinery (ACM).

  • Winners of the ADS Thesis Awards 2022

    The ADS Thesis Award winners for 2022 are Philipp Sommerhalter and Sarah Kwakkelaar in the BSc category and Krijn Doekemeijer and Julia Sudnik for the MSc category. The winners were announced at the ADS Highlights Event on the 8th of December 2022!

  • Symposium in honor of Cees de Laat and Leon Gommans

    A symposium entitled ‘From Systems and Networks to Complex Cyber Infrastructures’ took place on 9th December 2022 in honor of Cees de Laat (UvA IvI) and Leon Gommans (UvA IvI).