Optimization of Flink distributed system with FPGA-based architecture

Juan Zhang; Jia You; Yang Hao; Xisheng Li; Yin Liu

doi:10.1117/12.3035073

7 August 2024 Optimization of Flink distributed system with FPGA-based architecture

Juan Zhang, Jia You, Yang Hao, Xisheng Li, Yin Liu

Proceedings Volume 13224, 4th International Conference on Internet of Things and Smart City (IoTSC 2024); 132240T (2024) https://doi.org/10.1117/12.3035073
Event: 4th International Conference on Internet of Things and Smart City, 2024, Hangzhou, China

Abstract

With the continuous increase in the volume of Internet of Things (IoT) data, the limited number of servers deployed at the edge poses a pressing challenge for efficiently processing large-scale data streams. This study aims to explore the application of FPGA hardware acceleration technology to enhance the performance of the Apache Flink framework to meet the growing demands of users. The paper advocates the utilization of FPGA as a hardware accelerator for enhancing the performance of the Flink framework. This is achieved by harnessing the power of PCIe technology and integrating seamlessly with the OpenCL standard, which is tailored for heterogeneous systems. The JVM-FPGA communication mechanism is optimized using a data transmission pipeline mechanism. Through experiments and evaluations of two typical computationally intensive tasks, matrix multiplication and vector addition, it is demonstrated that the performance of the new framework is reduced in terms of latency, throughput is increased by 2.77 times, and CPU utilization is significantly enhanced. This study provides an innovative solution for efficiently processing largescale data streams in edge environments, successfully demonstrating that hardware acceleration can significantly improve the performance of the Apache Flink framework under resource constraints in edge environments, better meeting the continuously growing demands of IoT data.

(2024) Published by SPIE. Downloading of the abstract is permitted for personal use only.

Citation Download Citation

Juan Zhang, Jia You, Yang Hao, Xisheng Li, and Yin Liu "Optimization of Flink distributed system with FPGA-based architecture", Proc. SPIE 13224, 4th International Conference on Internet of Things and Smart City (IoTSC 2024), 132240T (7 August 2024); https://doi.org/10.1117/12.3035073

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available

Members: $17.00

Non-members: $21.00 ADD TO CART

PROCEEDINGS
7 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

RIGHTS & PERMISSIONS

Get copyright permission Get copyright permission on Copyright Marketplace

KEYWORDS

Field programmable gate arrays

Data processing

Data transmission

Matrix multiplication

Prototyping

Distributed computing

Internet of things

Show All Keywords

Keywords/Phrases

Search In:

Publication Years