Deepchecks

<!-- ~ ---------------------------------------------------------------------------- ~ Copyright (C) 2021-2023 Deepchecks (https://www.deepchecks.com) ~ ~ This file is part of Deepchecks. ~ Deepchecks is distributed under the terms of the GNU Affero General ~ Public License (version 3 or later). ~ You should have received a copy of the GNU Affero General Public License ~ along with Deepchecks. If not, see <http://www.gnu.org/licenses/>. ~ ---------------------------------------------------------------------------- ~ -->

<!-- ALL-CONTRIBUTORS-BADGE:START - Do not remove or modify this section -->

<!-- ALL-CONTRIBUTORS-BADGE:END --> <!--- this badge takes quite some space, can re-add it if wanted  ---> <h1 align="center"> Deepchecks - Continuous Validation for AI & ML: Testing, CI & Monitoring </h1>

Deepchecks is a holistic open-source solution for all of your AI & ML validation needs, enabling you to thoroughly test your data and models from research to production.

<a target="_blank" href="https://docs.deepchecks.com/?utm_source=github.com&utm_medium=referral&utm_campaign=readme&utm_content=logo"> <picture> <source media="(prefers-color-scheme: dark)" srcset="docs/source/_static/images/readme/deepchecks_continuous_validation_dark.png"> <source media="(prefers-color-scheme: light)" srcset="docs/source/_static/images/readme/deepchecks_continuous_validation_light.png"> <img alt="Deepchecks continuous validation parts." src="docs/source/_static/images//readme/deepchecks_continuous_validation_light.png"> </picture> </a> <p align="center"> &emsp; <a href="https://www.deepchecks.com/slack">👋 Join Slack</a> &emsp; | &emsp; <a href="https://docs.deepchecks.com/?utm_source=github.com&utm_medium=referral&utm_campaign=readme&utm_content=top_links">📖 Documentation</a> &emsp; | &emsp; <a href="https://deepchecks.com/blog/?utm_source=github.com&utm_medium=referral&utm_campaign=readme&utm_content=top_links">🌐 Blog</a> &emsp; | &emsp; <a href="https://twitter.com/deepchecks">🐦 Twitter</a> &emsp; </p>

🧩 Components

Deepchecks includes:

• Deepchecks Testing (Quickstart, docs):
  • Running built-in & your own custom Checks and Suites for Tabular, NLP & CV validation (open source).
• CI & Testing Management (Quickstart, docs):
  • Collaborating over test results and iterating efficiently until model is production-ready and can be deployed (open source & managed offering).
• Deepchecks Monitoring (Quickstart, docs):
  • Tracking and validating your deployed models behavior when in production (open source & managed offering).

This repo is our main repo as all components use the deepchecks checks in their core. See the Getting Started section for more information about installation and quickstarts for each of the components. If you want to see deepchecks monitoring's code, you can check out the deepchecks/monitoring repo.

⏩ Getting Started

<details close> <summary> <h3> 💻 Installation </h3> </summary>

Deepchecks Testing (and CI) Installation

pip install deepchecks -U --user

For installing the nlp / vision submodules or with conda:

• For NLP: Replace deepchecks with "deepchecks[nlp]", and optionally install alsodeepchecks[nlp-properties]
• For Computer Vision: Replace deepchecks with "deepchecks[vision]".
• For installing with conda, similarly use: conda install -c conda-forge deepchecks.

Check out the full installation instructions for deepchecks testing here.

Deepchecks Monitoring Installation

To use deepchecks for production monitoring, you can either use our SaaS service, or deploy a local instance in one line on Linux/MacOS (Windows is WIP!) with Docker. Create a new directory for the installation files, open a terminal within that directory and run the following:

pip install deepchecks-installer
deepchecks-installer install-monitoring

This will automatically download the necessary dependencies, run the installation process and then start the application locally.

The installation will take a few minutes. Then you can open the deployment url (default is http://localhost), and start the system onboarding. Check out the full monitoring open source installation & quickstart.

Note that the open source product is built such that each deployment supports monitoring of a single model.

</details>

🏃‍♀️ Quickstarts

<details open> <summary> <h4> Deepchecks Testing Quickstart </h4> </summary>

Jump right into the respective quickstart docs:

• Tabular Quickstarts
• NLP Quickstarts
• Vision Quickstarts

to have it up and running on your data.

Inside the quickstarts, you'll see how to create the relevant deepchecks object for holding your data and metadata (Dataset, TextData or VisionData, corresponding to the data type), and run a Suite or Check. The code snippet for running it will look something like the following, depending on the chosen Suite or Check.

from deepchecks.tabular.suites import model_evaluation
suite = model_evaluation()
suite_result = suite.run(train_dataset=train_dataset, test_dataset=test_dataset, model=model)
suite_result.save_as_html() # replace this with suite_result.show() or suite_result.show_in_window() to see results inline or in window
# or suite_result.results[0].value with the relevant check index to process the check result's values in python

The output will be a report that enables you to inspect the status and results of the chosen checks:

<p align="center"> <img src="docs/source/_static/images/readme/model-evaluation-suite.gif" width="600"> </p> </details> <details open> <summary> <h4> Deepchecks Monitoring Quickstart </h4> </summary>

Jump right into the open source monitoring quickstart docs to have it up and running on your data. You'll then be able to see the checks results over time, set alerts, and interact with the dynamic deepchecks UI that looks like this:

<p align="center"> <img src="docs/source/_static/images/general/monitoring-app-ui.gif" width="600"> </p> </details> <details open> <summary> <h4> Deepchecks CI & Testing Management Quickstart </h4> </summary>

Deepchecks managed CI & Testing management is currently in closed preview. Book a demo for more information about the offering.

<p align="center"> <img src="docs/source/_static/images/general/deepchecks-ci-checks.png" width="600"> </p>

For building and maintaining your own CI process while utilizing Deepchecks Testing for it, check out our docs for Using Deepchecks in CI/CD.

</details>

🧮 How does it work?

At its core, deepchecks includes a wide variety of built-in Checks, for testing all types of data and model related issues. These checks are implemented for various models and data types (Tabular, NLP, Vision), and can easily be customized and expanded.

The check results can be used to automatically make informed decisions about your model's production-readiness, and for monitoring it over time in production. The check results can be examined with visual reports (by saving them to an HTML file, or seeing them in Jupyter), processed with code (using their pythonic / json output), and inspected and collaborated on with Deepchecks' dynamic UI (for examining test results and for production monitoring).

<!--- At its core, Deepchecks has a wide variety of built-in Checks and Suites (lists of checks) for all data types (Tabular,