venvutil - Manage Conda and Pip VENV's with some simple functions and scripts

This is release v1.0.6. This project is continuously evolving, becoming a catch-all for useful tools and shell functions that facilitate working with Python VENV's and LLM's.

venvutil - Manage Conda and Pip VENV's with some simple functions and scripts

Project Overview

Venvutil is a versatile toolset designed to simplify the management of Python virtual environments (VENV) and enhance the workflow for developers working with Python packages and large language models (LLMs). This project provides a collection of scripts and functions that wrap around common tools like Conda and Pip, offering additional features such as logging, environment state freezing, and streamlined package management.

Key Features

Enhanced VENV Management: Provides tools to easily create, manage, and replicate Python virtual environments and a few other tools.
Conda and Pip Logging: Utilizes logging to track changes to VENV's and freeze the state of environments for reproducibility.
Enhanced Logging: Maintains a log on any potentially destructive operations to VENV's when using Pip or Conda to make changes to the VENV.
Robust Configuration Management: All configuration changes are logged and a pip freeze is done before and after the operation to ensure that the virtual environment is frozen.
Git Integration: Automatically detects deleted files from git status to manage deprecated files in installations.
Rollback and Recovery: Offers tools to rollback changes to VENV's, restoring the state of the environment to a previous state using the freeze log.
Cross-Platform Compatibility: Designed to work seamlessly on macOS and Linux systems, with specific optimizations for Apple Silicon.
Additional Tools: Provides additional tools like genmd, filetree, including some simple test scripts to check if your GPU is being utilized - specifically for Apple Silicon.
Build and Installation Scripts: Offers scripts to automate the setup of complex environments, including rebuilding NumPy with Apple Silicon optimizations.
Comprehensive Documentation: Includes detailed instructions and CHANGELOG to help users get started and stay updated with the latest features.

Why Use Venvutil?

Simplify Environment Management: Most common tasks are simplified with this toolset, such as creating, deleting, switching, and cloning environments.
Enhanced Logging: Provides a record of changes to VENV's, making it easy to track and recover from potential issues.
Ensure Reproducibility: By freezing the state of environments, Venvutil helps ensure that your setups are consistent across different machines and setups.
Transparent Tool Wrapping: Provides pass-through wrappers Pip, Conda, and LD to ensure compatibility without compromising security or functionality.

Tested on the following systems

macOS 15.4 (Sequoia)
macOS 13.4 (Monterey)
Red Hat Enterprise Linus 8
Redhat Enterprise Linux 9

Installation Instructions

Prerequisites

Xcode command line utilities.
Bash 4.0 or higher (macOS has Bash 3.2, you will need to build and install or use Brew)
Python 3.11 (Conda will handle this for you when it installs)
Conda latest version
Python packages
- Rich, ticktoken, nltk
Ensure that your system meets the necessary requirements for running Python and Conda.

Running the installer

git clone https://github.com/unixwzrd/venvutil.git venvutil
cd venvutil
bash ./setup.sh install

# For updates from cloned repo (without Python packages)
./setup.sh refresh

By default this installs into $HOME/local/venvutil (configured via setup/setup.cf). You can override this with -d <directory> to install anywhere you wish. The installer will download and update Conda if necessary, along with the python packages listed above. NLTK needs data and that will be downloaded into your home directory into the nltk_data directory.

Thanks for using Venvutil!

Setup Script Enhancements

Manifest hard links (h): The installer manifest supports a hard link type (h) in addition to symbolic links (l). This allows you to specify hard links directly in the manifest file processed by setup/assets.sh.
Shared shell libraries: setup/setuplib is now a real directory whose .sh files are hard links to the canonical bin/shinclude libraries so setup and runtime never drift.

Usage

Tools Overview

extract-chat extracts ChatGPT JSON chatlogs, works with my Safari extension, to extract chat history.
- Extract in either Markdown or HTML format.
- Retains code and references where possible along with some internal metadata.
- May be broken into chunks and fed into a fresh GPT context for continuity.
tokencount: Detailed Documentation TODO
- A tool designed to count tokens in text files, useful for analyzing text data and preparing it for processing with language models.
chunkfile: Detailed Documentation
- A versatile tool for splitting files into chunks with configurable overlap:
  - Split by number of files, lines or size in bytes.
  - Support for overlapping content between chunks (-o)
warehouse and recall: Detailed Documentation
- Tools for managing offline storage:
  - Move files/directories to external storage while maintaining symlinks
  - Support custom storage locations
numpybench: Detailed Documentation
- A benchmarking tool for testing NumPy performance:
  - Tests/validates GPU and Neural Engine acceleration on Apple Silicon with NumPy
genmd: Detailed Documentation
- A script that generates markdown documentation from project files, facilitating easy sharing and collaboration.
filetree: Detailed Documentation
- will produce file hierarchy structure based on file and directories to exclude and include..
core functions provided by venvutil_lib.sh: Detailed Documentation
- Provides a number of useful shell functions for managing aVirtual Environments along with some utility function, such as ptree
- compile wrappers for C++, G++, and LD: Detailed Documentation
  - To help compile many things in the macOS Environment which incorrectly pass the linker the --version flag.

Shell Functions

These are a few of the shell functions provided by venvutil which I find useful. There is more documentation on the functions in the README of the venvutil Tools.

To use the functions and tools, simply source in the venvutil_lib.sh file in your .bashrc. The setup.sh script will handle adding the necessary checks and source statements to your .bashrc file.

venvutil Tools: Detailed Documentation
- A collection of shell functions and scripts for managing Python virtual environments and LLMs.
vhelp: Detailed Documentation
- Integrated help for scripts and functions. vhelp is the main entry point for the help system.
ptree: Detailed Documentation
- A shell function that displays a file tree structure of a directory, highlighting directories that contain certain files.

lenv: Detailed Documentation

Provides a listing of all Pip and Conda managed environments, versions and date last updated.

(base) [unixwzrd@xanax: ~]$ lenv
Date        Python   Environment                   Path
2025-01-27  3.11.11  adv-numpy-daily-pytorch      ~/miniconda3/envs/adv-numpy-daily-pytorch
2025-01-13  3.12.8   base                       * ~/miniconda3
2025-01-25  3.11.11  comp-numpy-daily-pytorch     ~/miniconda3/envs/comp-numpy-daily-pytorch
2025-01-25  3.11.11  comp-numpy-std-pytorch       ~/miniconda3/envs/comp-numpy-std-pytorch
2025-01-02  3.10.16  python-3.10-PA-dev           ~/miniconda3/envs/python-3.10-PA

Venvutil

Install / Use

README