Pymilo

PyMilo: Python for ML I/O

Generate Convert Improve

Install / Use

/learn @openscilab/Pymilo

About this skill

Quality Score

0/100

README

Overview

<p align="justify"> PyMilo is an open source Python package that provides a simple, efficient, and safe way for users to export pre-trained machine learning models in a transparent way. By this, the exported model can be used in other environments, transferred across different platforms, and shared with others. PyMilo allows the users to export the models that are trained using popular Python libraries like scikit-learn, and then use them in deployment environments, or share them without exposing the underlying code or dependencies. The transparency of the exported models ensures reliability and safety for the end users, as it eliminates the risks of binary or pickle formats. </p> <table> <tr> <td align="center">PyPI Counter</td> <td align="center"> <a href="https://pepy.tech/projects/pymilo"> <img src="https://static.pepy.tech/badge/pymilo" alt="PyPI Downloads"> </a> </td> </tr> <tr> <td align="center">Github Stars</td> <td align="center"> <a href="https://github.com/openscilab/pymilo"> <img src="https://img.shields.io/github/stars/openscilab/pymilo.svg?style=social&label=Stars"> </a> </td> </tr> </table> <table> <tr> <td align="center">Branch</td> <td align="center">main</td> <td align="center">dev</td> </tr> <tr> <td align="center">CI</td> <td align="center"> <img src="https://github.com/openscilab/pymilo/actions/workflows/test.yml/badge.svg?branch=main"> </td> <td align="center"> <img src="https://github.com/openscilab/pymilo/actions/workflows/test.yml/badge.svg?branch=dev"> </td> </tr> </table> <table> <tr> <td align="center">Code Quality</td> <td align="center"><a href="https://www.codefactor.io/repository/github/openscilab/pymilo"><img src="https://www.codefactor.io/repository/github/openscilab/pymilo/badge" alt="CodeFactor" /></a></td> <td align="center"><a href="https://app.codacy.com/gh/openscilab/pymilo/dashboard?utm_source=gh&utm_medium=referral&utm_content=&utm_campaign=Badge_grade"><img src="https://app.codacy.com/project/badge/Grade/9eeec99ed11f4d9b86af36dc90f5f753"></a></td> </tr> </table>

Installation

PyPI

Check Python Packaging User Guide
Run pip install pymilo==1.6

Source code

Download Version 1.6 or Latest Source
Run pip install .

Conda

Check Conda Managing Package
Update Conda using conda update conda
Run conda install -c openscilab pymilo

Usage

Import/Export

Imagine you want to train a LinearRegression model representing this equation: $y = x_0 + 2x_1 + 3$. You will create data points (X, y) and train your model as follows.

import numpy as np
from sklearn.linear_model import LinearRegression
X = np.array([[1, 1], [1, 2], [2, 2], [2, 3]])
y = np.dot(X, np.array([1, 2])) + 3
# y = 1 * x_0 + 2 * x_1 + 3
model = LinearRegression().fit(X, y)
pred = model.predict(np.array([[3, 5]]))
# pred = [16.] (=1 * 3 + 2 * 5 + 3)

Using PyMilo Export class you can easily serialize and export your trained model into a JSON file.

from pymilo import Export
Export(model).save("model.json")

Export

The Export class facilitates exporting of machine learning models to JSON files.

| Parameter | Description | | ------------- | --------------- | | model | The machine learning model to be exported |

| Property | Description | | ------------ | --------------- | | data | The serialized model data including all learned parameters | | version | The scikit-learn version used to train the model | | type | The type/class name of the exported model |

| Method | Description | | ---------- | --------------- | | save | Save the exported model to a JSON file | | to_json | Return the model as a JSON string representation | | batch_export | Export multiple models to individual JSON files in a directory |

You can check out your model as a JSON file now.

{
    "data": {
        "fit_intercept": true,
        "copy_X": true,
        "n_jobs": null,
        "positive": false,
        "n_features_in_": 2,
        "coef_": {
            "pymiloed-ndarray-list": [
                1.0000000000000002,
                1.9999999999999991
            ],
            "pymiloed-ndarray-dtype": "float64",
            "pymiloed-ndarray-shape": [
                2
            ],
            "pymiloed-data-structure": "numpy.ndarray"
        },
        "rank_": 2,
        "singular_": {
            "pymiloed-ndarray-list": [
                1.618033988749895,
                0.6180339887498948
            ],
            "pymiloed-ndarray-dtype": "float64",
            "pymiloed-ndarray-shape": [
                2
            ],
            "pymiloed-data-structure": "numpy.ndarray"
        },
        "intercept_": {
            "value": 3.0000000000000018,
            "np-type": "numpy.float64"
        }
    },
    "sklearn_version": "1.4.2",
    "pymilo_version": "0.8",
    "model_type": "LinearRegression"
}

You can see all the learned parameters of the model in this file and change them if you want. This JSON representation is a transparent version of your model.

Now let's load it back. You can do it easily by using PyMilo Import class.

from pymilo import Import
model = Import("model.json").to_model()
pred = model.predict(np.array([[3, 5]]))
# pred = [16.] (=1 * 3 + 2 * 5 + 3)

Import

The Import class facilitates importing of serialized models from JSON files, JSON strings, or URLs.

| Parameter | Description | | ------------- | --------------- | | file_adr | Path to the JSON file containing the serialized model | | json_dump | JSON string representation of the serialized model | | url | URL to download the serialized model from |

| Property | Description | | ------------ | --------------- | | data | The deserialized model data | | version | The scikit-learn version of the original model | | type | The type/class name of the imported model |

| Method | Description | | ---------- | --------------- | | to_model | Convert the imported data back to a scikit-learn model | | batch_import | Import multiple models from JSON files in a directory |

This loaded model is exactly the same as the original trained model.

ML streaming

You can easily serve your ML model from a remote server using ML streaming feature of PyMilo.

⚠️ ML streaming feature exists in versions >=1.0

⚠️ In order to use ML streaming feature, make sure you've installed the streaming mode of PyMilo

⚠️ The ML streaming feature is under construction and is not yet considered stable.

You can choose either REST or WebSocket as the communication medium protocol.

Server

Let's assume you are in the remote server and you want to import the exported JSON file and start serving your model through REST protocol!

from pymilo import Import
from pymilo.streaming import PymiloServer, CommunicationProtocol
my_model = Import("model.json").to_model()
communicator = PymiloServer(
    model=my_model,
    port=8000,
    communication_protocol=CommunicationProtocol["REST"],
    ).communicator
communicator.run()

PymiloServer

The PymiloServer class facilitates streaming machine learning models over a network.

| Parameter | Description | | ------------- | --------------- | | port | Port number for the server to listen on (default: 8000) | | host | Host address for the server (default: "127.0.0.1") | | compressor | Compression method from Compression enum | | communication_protocol | Communication protocol from CommunicationProtocol enum |

The compressor parameter accepts values from the Compression enum including NULL (no compression), GZIP, ZLIB, LZMA, or BZ2. The communication_protocol parameter accepts values from the CommunicationProtocol enum including REST or WEBSOCKET.

| Method | Description | | ---------- | --------------- | | init_client | Initialize a new client with the given client ID | | remove_client | Remove an existing client by client ID | | init_ml_model | Initialize a new ML model for a given client | | set_ml_model | Set or update the ML model for a client | | remove_ml_model | Remove an existing ML model for a client | | get_ml_models | Get all ML model IDs for a client | | execute_model | Execute model methods or access attributes | | grant_access | Allow a client to access another client's model | | revoke_access | Revoke access to a client's model | | get_allowed_models | Get models a client is allowed to access |

Now PymiloServer runs on port 8000 and exposes REST API to upload, download and retrieve **at