[!IMPORTANT] bigdl-llm has now become ipex-llm, and our future development will move to the IPEX-LLM project.

Overview

BigDL seamlessly scales your data analytics & AI applications from laptop to cloud, with the following libraries:

LLM (deprecated - please use IPEX-LLM instead): Optimizaed large language model library for Intel CPU and GPU
Orca: Distributed Big Data & AI (TF & PyTorch) Pipeline on Spark and Ray
Nano: Transparent Acceleration of Tensorflow & PyTorch Programs on Intel CPU/GPU
DLlib: “Equivalent of Spark MLlib” for Deep Learning
Chronos: Scalable Time Series Analysis using AutoML
Friesian: End-to-End Recommendation Systems
PPML: Secure Big Data and AI (with SGX/TDX Hardware Security)

For more information, you may read the docs.

Choosing the right BigDL library

flowchart TD;
    Feature1{{HW Secured Big Data & AI?}};
    Feature1-- No -->Feature2{{Python vs. Scala/Java?}};
    Feature1-- "Yes"  -->ReferPPML([<em><strong>PPML</strong></em>]);
    Feature2-- Python -->Feature3{{What type of application?}};
    Feature2-- Scala/Java -->ReferDLlib([<em><strong>DLlib</strong></em>]);
    Feature3-- "Large Language Model" -->ReferLLM([<em><strong>LLM</strong></em>]);
    Feature3-- "Big Data + AI (TF/PyTorch)" -->ReferOrca([<em><strong>Orca</strong></em>]);
    Feature3-- Accelerate TensorFlow / PyTorch -->ReferNano([<em><strong>Nano</strong></em>]);
    Feature3-- DL for Spark MLlib -->ReferDLlib2([<em><strong>DLlib</strong></em>]);
    Feature3-- High Level App Framework -->Feature4{{Domain?}};
    Feature4-- Time Series -->ReferChronos([<em><strong>Chronos</strong></em>]);
    Feature4-- Recommender System -->ReferFriesian([<em><strong>Friesian</strong></em>]);
    
    click ReferLLM "https://github.com/intel-analytics/ipex-llm"
    click ReferNano "https://github.com/intel-analytics/BigDL-2.x#nano"
    click ReferOrca "https://github.com/intel-analytics/BigDL-2.x#orca"
    click ReferDLlib "https://github.com/intel-analytics/BigDL-2.x#dllib"
    click ReferDLlib2 "https://github.com/intel-analytics/BigDL-2.x#dllib"
    click ReferChronos "https://github.com/intel-analytics/BigDL-2.x#chronos"
    click ReferFriesian "https://github.com/intel-analytics/BigDL-2.x#friesian"
    click ReferPPML "https://github.com/intel-analytics/BigDL-2.x#ppml"
    
    classDef ReferStyle1 fill:#5099ce,stroke:#5099ce;
    classDef Feature fill:#FFF,stroke:#08409c,stroke-width:1px;
    class ReferLLM,ReferNano,ReferOrca,ReferDLlib,ReferDLlib2,ReferChronos,ReferFriesian,ReferPPML ReferStyle1;
    class Feature1,Feature2,Feature3,Feature4,Feature5,Feature6,Feature7 Feature;

Installing

To install BigDL, we recommend using conda environment:
```
conda create -n my_env 
conda activate my_env
pip install bigdl
```
To install latest nightly build, use pip install --pre --upgrade bigdl; see Python and Scala user guide for more details.
To install each individual library, such as Chronos, use pip install bigdl-chronos; see the document website for more details.

Getting Started

Orca

The Orca library seamlessly scales out your single node TensorFlow, PyTorch or OpenVINO programs across large clusters (so as to process distributed Big Data).

<details><summary>Show Orca example</summary> <br/>

You can build end-to-end, distributed data processing & AI programs using Orca in 4 simple steps:

# 1. Initilize Orca Context (to run your program on K8s, YARN or local laptop)
from bigdl.orca import init_orca_context, OrcaContext
sc = init_orca_context(cluster_mode="k8s", cores=4, memory="10g", num_nodes=2) 

# 2. Perform distribtued data processing (supporting Spark DataFrames,
# TensorFlow Dataset, PyTorch DataLoader, Ray Dataset, Pandas, Pillow, etc.)
spark = OrcaContext.get_spark_session()
df = spark.read.parquet(file_path)
df = df.withColumn('label', df.label-1)
...

# 3. Build deep learning models using standard framework APIs
# (supporting TensorFlow, PyTorch, Keras, OpenVino, etc.)
from tensorflow import keras
...
model = keras.models.Model(inputs=[user, item], outputs=predictions)  
model.compile(...)

# 4. Use Orca Estimator for distributed training/inference
from bigdl.orca.learn.tf.estimator import Estimator
est = Estimator.from_keras(keras_model=model)  
est.fit(data=df,
        feature_cols=['user', 'item'],
        label_cols=['label'],
        ...)

</details>

See Orca user guide, as well as TensorFlow and PyTorch quickstarts, for more details.

In addition, you can also run standard Ray programs on Spark cluster using RayOnSpark in Orca.

<details><summary>Show RayOnSpark example</summary> <br/>

You can not only run Ray program on Spark cluster, but also write Ray code inline with Spark code (so as to process the in-memory Spark RDDs or DataFrames) using RayOnSpark in Orca.

# 1. Initilize Orca Context (to run your program on K8s, YARN or local laptop)
from bigdl.orca import init_orca_context, OrcaContext
sc = init_orca_context(cluster_mode="yarn", cores=4, memory="10g", num_nodes=2, init_ray_on_spark=True) 

# 2. Distribtued data processing using Spark
spark = OrcaContext.get_spark_session()
df = spark.read.parquet(file_path).withColumn(...)

# 3. Convert Spark DataFrame to Ray Dataset
from bigdl.orca.data import spark_df_to_ray_dataset
dataset = spark_df_to_ray_dataset(df)

# 4. Use Ray to operate on Ray Datasets
import ray

@ray.remote
def consume(data) -> int:
   num_batches = 0
   for batch in data.iter_batches(batch_size=10):
       num_batches += 1
   return num_batches

print(ray.get(consume.remote(dataset)))

</details>

See RayOnSpark user guide and quickstart for more details.

Nano

You can transparently accelerate your TensorFlow or PyTorch programs on your laptop or server using Nano. With minimum code changes, Nano automatically applies modern CPU optimizations (e.g., SIMD, multiprocessing, low precision, etc.) to standard TensorFlow and PyTorch code, with up-to 10x speedup.

<details><summary>Show Nano inference example</summary> <br/>

You can automatically optimize a trained PyTorch model for inference or deployment using Nano:

model = ResNet18().load_state_dict(...)
train_dataloader = ...
val_dataloader = ...
def accuracy (pred, target):
  ... 

from bigdl.nano.pytorch import InferenceOptimizer
optimizer = InferenceOptimizer()
optimizer.optimize(model,
                   training_data=train_dataloader,
                   validation_data=val_dataloader,
                   metric=accuracy)
new_model, config = optimizer.get_best_model()

optimizer.summary()

The output of optimizer.summary() will be something like:

 -------------------------------- ---------------------- -------------- ----------------------
|             method             |        status        | latency(ms)  |     metric value     |
 -------------------------------- ---------------------- -------------- ----------------------
|            original            |      successful      |    45.145    |        0.975         |
|              bf16              |      successful      |    27.549    |        0.975         |
|          static_int8           |      successful      |    11.339    |        0.975         |
|         jit_fp32_ipex          |      successful      |    40.618    |        0.975*        |
|  jit_fp32_ipex_channels_last   |      successful      |    19.247    |        0.975*        |
|         jit_bf16_ipex          |      successful      |    10.149    |        0.975         |
|  jit_bf16_ipex_channels_last   |      successful      |    9.782     |        0.975         |
|         openvino_fp32          |      successful      |    22.721    |        0.975*        |
|         openvino_int8          |      successful      |    5.846     |        0.962         |
|        onnxruntime_fp32        |      successful      |    20.838    |        0.975*        |
|    onnxruntime_int8_qlinear    |      successful      |    7.123     |        0.981         |
 -------------------------------- ---------------------- -------------- ----------------------
* means we assume the metric value of the traced model does not change, so we don't recompute metric value to save time.
Optimization cost 60.8s in total.

</details> <details><summary>Show Nano Training example</summary> <br/> You may easily accelerate PyTorch training (e.g., IPEX, BF16, Multi-Instance Training, etc.) using Nano:

model = ResNet18()
optimizer = torch.optim.SGD(...)
train_loader = ...
val_loader = ...

from bigdl.nano.pytorch import TorchNano

# Define your training loop inside `TorchNano.train`
class Trainer(TorchNano):
	def train(self):
	# call `setup` to prepare for model, optimizer(s) and dataloader(s) for accelerated training
	model, optimizer, (train_

BigDL

Install / Use

README