Intel Unveils Deep Learning Library For Apache Spark

Intel announced the open-source BigDL, a Distributed Deep Learning Library for the Apache Spark* open-source cluster-computing framework.

The deep learning library is part of Intel Corporation’s strategy for enabling state-of-the-art Artificial Intelligence in the industry. Announced last November, the strategy detailed Intel’s work to make AI training and tools broadly accessible to developers through the Intel® Nervana™ AI Academy.

The BigDL features an efficient large-scale distributed deep learning library built on Spark architecture that makes deep learning more accessible to big data users and data scientists. BigDL enables the exporting of AI expertise to data scientists now working across thousands of applications in hundreds of fields.

The BigDL can also serve as the unified data analytics platform (Hadoop/Spark) for data storage, processing and mining, feature engineering, machine and deep learning workloads and more. It lets developers write deep learning applications as standard Spark programs that run on top of existing Spark or Hadoop clusters to put deep learning workloads more directly in touch with the data they use. BigDL is already running in the Databricks Spark Platform.

“BigDL is an open-source project, and we encourage all developers to connect with us on the BigDL Github, sample the code and contribute to the project,” said Doug Fisher, senior vice president and general manager of the Software and Services Group at Intel.