Ratarmount Logo

Random Access To Archived Resources (Ratarmount)

</br> GitHub repo size

</div>

Ratarmount collects all file positions inside a TAR so that it can easily jump to and read from any file without extracting it. It, then, mounts the TAR using mfusepy, a fork of fusepy, for read access just like archivemount. In contrast to libarchive, on which archivemount is based, random access and true seeking is supported. And in contrast to tarindexer, which also collects file positions for random access, ratarmount offers easy access via FUSE and support for compressed TARs.

Capabilities:

Random Access: Care was taken to achieve fast random access inside compressed streams for bzip2, gzip, xz, and zstd and inside TAR files by building indices containing seek points.
Highly Parallelized: By default, all cores are used for parallelized algorithms like for the gzip, bzip2, and xz decoders. This can yield huge speedups on most modern processors but requires more main memory. It can be controlled or completely turned off using the -P <cores> option.
Recursive Mounting: Ratarmount will also mount TARs inside TARs inside TARs, ... recursively into folders of the same name, which is useful for the 1.31TB ImageNet data set.
Mount Compressed Files: You may also mount files with one of the supported compression schemes. Even if these files do not contain a TAR, you can leverage ratarmount's true seeking capabilities when opening the mounted uncompressed view of such a file.
Read-Only Bind Mounting: Folders may be mounted read-only to other folders for usecases like merging a backup TAR with newer versions of those files residing in a normal folder.
Union Mounting: Multiple TARs, compressed files, and bind mounted folders can be mounted under the same mountpoint.
Write Overlay: A folder can be specified as write overlay. All changes below the mountpoint will be redirected to this folder and deletions are tracked so that all changes can be applied back to the archive.
Remote Files and Folders: A remote archive or whole folder structure can be mounted similar to tools like sshfs thanks to the filesystem_spec project. These can be specified with URIs as explained in the section "Remote Files". Supported remote protocols include: FTP, HTTP, HTTPS, SFTP, SSH, Git, Github, S3, Samba v2 and v3, Dropbox, ... Many of these are very experimental and may be slow. Please open a feature request if further backends are desired.

A complete list of supported formats can be found here.

Examples

ratarmount archive.tar.gz to mount a compressed archive at a folder called archive and make its contents browsable.
ratarmount --recursive archive.tar mountpoint to mount the archive and recursively all its contained archives under a folder called mountpoint.
ratarmount folder mountpoint to bind-mount a folder.
ratarmount folder1 folder2 mountpoint to bind-mount a merged view of two (or more) folders under mountpoint.
ratarmount folder archive.zip folder to mount a merged view of a folder on top of archive contents.
ratarmount -o modules=subdir,subdir=squashfs-root archive.squashfs mountpoint to mount an archive subfolder squashfs-root under mountpoint.
ratarmount http://server.org:80/archive.rar folder folder Mount an archive that is accessible via HTTP range requests.
ratarmount ssh://hostname:22/relativefolder/ mountpoint Mount a folder hierarchy via SSH.
ratarmount ssh://hostname:22//tmp/tmp-abcdef/ mountpoint
ratarmount github://mxmlnkn:ratarmount@v0.15.2/tests/ mountpoint Mount a github repo as if it was checked out at the given tag or SHA or branch.
AWS_ACCESS_KEY_ID=01234567890123456789 AWS_SECRET_ACCESS_KEY=0123456789012345678901234567890123456789 ratarmount s3://127.0.0.1/bucket/single-file.tar mounted Mount an archive inside an S3 bucket reachable via a custom endpoint with the given credentials. Bogus credentials may be necessary for unsecured endpoints.

Installation
Supported Formats
1. TAR compressions supported for random access
2. Other supported archive formats
Benchmarks
The Problem
The Solution
Usage

Installation

You can install ratarmount either by simply downloading the AppImage or via pip. The latter might require installing additional dependencies. The latest nightly build AppImage can be found here.

pip install ratarmount

If you want all features, some of which may possibly result in installation errors on some systems, install with:

pip install ratarmount[full]

Installation via AppImage

The AppImage files are attached under "Assets" on the releases page. They require no installation and can be simply executed like a portable executable. If you want to install it, you can simply copy it into any of the folders listed in your PATH.

appImageName=ratarmount-0.15.0-x86_64.AppImage
wget 'https://github.com/mxmlnkn/ratarmount/releases/download/v0.15.0/$appImageName'
chmod u+x -- "$appImageName"
./"$appImageName" --help  # Simple test run
sudo cp -- "$appImageName" /usr/local/bin/ratarmount  # Example installation

Installation via Package Manager

Arch Linux

Arch Linux's AUR offers ratarmount as stable and development package. Use an AUR helper, like yay or paru, to install one of them:

# stable version
paru -Syu ratarmount
# development version
paru -Syu ratarmount-git

Conda

conda install -c conda-forge ratarmount

System Dependencies for PIP Installation (Rarely Necessary)

Python 3.6+, preferably pip 19.0+, FUSE, and sqlite3 are required. These should be preinstalled on most systems.

On Debian-like systems like Ubuntu, you can install/update all dependencies using:

sudo apt install python3 python3-pip fuse sqlite3 unar libarchive13 lzop gcc liblzo2-dev

On macOS, you have to install macFUSE and other optional dependencies with:

brew install macfuse unar libarchive lrzip lzop lz

Ratarmount

Install / Use

README