Cengal

<!--  -->

Cengal is a versatile Python library designed for a wide range of applications. To enhance performance, certain modules within Cengal have been implemented using Cython, C/C++, Nim or Go.

Cengal features an extensive codebase with over 200 projects and more than 80,000 lines of code. I am continually focused on developing new features and making improvements. Contributions are very welcome, especially Pull Requests that include unit tests and documentation enhancements.

If you have any questions or would like to participate in discussions, feel free to join the Cengal Discord. Your support and involvement are greatly appreciated as Cengal evolves.

Architecture & API Design Rationale

<details> <summary title="Rationale"><kbd> Rationale </kbd></summary>

The Cengal library adheres to an API design approach used in frameworks such as Qt. For those familiar with the C++ language, I will draw comparisons between the approaches of Qt and the C++ Standard Template Library (STL). The API provided by the STL was designed to significantly reduce the burden on programmers who develop the STL. This decision was logical from the standpoint of marketing the STL among compiler creators. However, this led to the usability of the STL for the user not being great. This is evident in the fact that the STL provides the most minimal possible API, and any conveniences must be programmed anew by each programmer every time - constantly reinventing the wheel. In contrast, Qt uses the opposite approach to API construction: classes have many methods whose purposes are similar, but are aimed at different usage models. This simplifies the use of Qt for users, speeds up the writing of the final code, and avoids many errors that we usually make when we write our own 'bicycles' for the same actions each time (not because the we are not smart, but because we are humans and therefore prone to make mistakes from time to time).

</details>

Cengal compatibility and requirements

<details> <summary title="Compatibility and requirements"><kbd> Compatibility and requirements </kbd></summary>

• Target platforms: Win32, Linux, macOS, Android, iOS, Emscripten
• Target architectures: x64, x86, ARM
• Target interpreters: CPython, PyPy
• Recommended Python versions: 3.8+

</details>

Installation

<details> <summary title="General wheel"><kbd> General wheel </kbd></summary>

To get started with Cengal, you can easily install it along with all the mandatory dependencies by running pip install cengal. This command not only installs Cengal and its required dependencies but also takes care of fetching and installing prebuilt wheels tailored for the Cengal library. These wheels are compatible with both Windows and Linux systems and work seamlessly with both CPython and PyPy interpreters.

</details> <details> <summary title="Wheel without dependencies"><kbd> Wheel without dependencies </kbd></summary>

If you prefer to install Cengal without its dependencies, you can opt for the 'cengal-light' package, which serves as the backend for the 'cengal' package. Simply run pip install cengal-light to get the lightweight version of Cengal.

</details>

Documentation

Cengal Discord

Cengal Documentation

For example Cengal Coroutines Concepts & Usage

<!-- or partially: [Cengal Wiki](https://github.com/FI-Mihej/Cengal/wiki) For example [Wiki: Cengal Coroutines Concepts & Usage](https://github.com/FI-Mihej/Cengal/wiki/Cengal-Coroutines) -->

Stand-Alone Packages for Specific Cengal Modules

<details> <summary title="Rationale"><kbd> Rationale </kbd></summary>

To cater to varying needs and streamline the installation process, I've introduced stand-alone packages for select Cengal modules. These packages are designed to offer users the ability to install specific Cengal functionality without the burden of the library's full set of dependencies.

The core of this approach lies in our 'cengal-light' package, which houses both Python and compiled Cengal modules. The 'cengal' package itself serves as a lightweight shell, devoid of its own modules, but dependent on 'cengal-light[full]' for a complete Cengal library installation with all required dependencies.

For users seeking individual Cengal features or looking to minimize dependencies, our stand-alone packages provide a convenient solution. Each stand-alone package is dedicated to a specific Cengal module and relies on 'cengal-light' as its sole dependency.

</details>

Below, you'll find a list of these stand-alone packages, each corresponding to a distinct Cengal module:

• CengalPolyBuild: A Comprehensive and Hackable Build System for Multilingual Python Packages: Cython (including automatic conversion from Python to Cython), C/C++, Objective-C, Go, and Nim, with ongoing expansions to include additional languages. (Planned to be released soon)
• InterProcessPyObjects (package for cengal.parallel_execution.asyncio.ashared_memory_manager module): High-performance package delivers blazing-fast inter-process communication through shared memory, enabling Python objects to be shared across processes with exceptional efficiency.
• cengal_memory_barriers (package for cengal.hardware.memory.barriers module): Fast crossplatform memory barriers for Python.
• cengal_cpu_info (package for cengal.hardware.cpu.info module): Extended, cached CPU info with consistent output format.
• cengal_app_dir_path_finder (package for cengal.file_system.app_fs_structure.app_dir_path module): Offering a unified API for easy retrieval of OS-specific application directories, enhancing data management across Windows, Linux, and macOS.

Stay tuned for future additions to our collection of stand-alone packages!

Exclusive Features: No Alternatives Online

<details> <summary title="Build system (work in progress)"><kbd> Build system (work in progress) </kbd></summary>

Build system (work in progress)

Automatic hackable build system for your package which supports Python modules made with different languages: Cython (including Python to Cython automatic compilation), C/C++, ObjectiveC, Go, Nim. Other languages support is in progress.

Compiles your code, gather binary artifacts and puts them into your wheel.

Examples

• Compilable Golang module
• Compilable Nim module
• Pure Python module auto-compiled with Cython

</details> <details> <summary title="Fast inter-process communication through shared memory"><kbd> Fast inter-process communication through shared memory </kbd></summary>

Fast inter-process communication through shared memory

blazing-fast inter-process communication through shared memory, enabling Python objects to be shared across processes with exceptional efficiency. By minimizing the need for frequent serialization-deserialization, it enhances overall speed and responsiveness. The package offers a comprehensive suite of functionalities designed to support a diverse array of Python types and facilitate asynchronous IPC, optimizing performance for demanding applications.

from cengal.hardware.memory.shared_memory import *
from cengal.parallel_execution.asyncio.ashared_memory_manager import *

Key Features

• Shared Memory Communication:

  • Enables sharing of Python objects directly between processes using shared memory.
  • Utilizes a linked list of global messages to inform connected processes about new shared objects.

• Lock-Free Synchronization:

  • Uses memory barriers for efficient communication, avoiding slow syscalls.
  • Ensures each process can access and modify shared memory without contention.

• Supported Python Types:

  • Handles various Python data structures including:
    • Basic types: None, bool, 64-bit int, large int (arbitrary precision integers), float, complex, bytes, bytearray,