datalad
Retrieves, versions, and publishes scientific datasets with DataLad and git-annex, and captures computational provenance with datalad run, rerun, and containers-run
Install / Use
npx skills add K-Dense-AI/scientific-agent-skills --skill dataladInstalls into whichever agent you are using.
SKILL.md
Installable skill definition
Quality Score
Category
MarketingSupported Platforms
Our assessment of datalad
datalad scores 100/100 on our quality scale, 7th of 644 Marketing skills we index (top 2%).
Its SKILL.md is 15 KB long, well organised into 22 sections with 10 code examples: a thorough specification that gives an agent plenty to work with.
With 46,441 GitHub stars, it is one of the more widely adopted skills in the catalogue.
Maintenance, license and trust
- The repository was last updated about 2 months ago, so datalad is actively maintained.
- Our last check on 2026-11-13 found the source still online.
- It is released under the MIT license, a permissive license that allows use, modification and commercial use with attribution.
- Its trust signals score 100/100, with no cautions. These come from repository metadata, not a code audit — read the skill file before letting an agent act on it.
Safety scan
No issues foundOur scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful.
AI review by kimi-k2.7-code on 2026-09-25. Automated pattern scan on 2026-09-25. It catches known dangerous patterns, not every risk — read a skill before letting an agent act on it.
datalad compared with similar skills
datalad has the highest quality score among these 4 similar skills, though 4 alternatives have been updated more recently.
| Skill | Score | Stars | Updated | Format |
|---|---|---|---|---|
| datalad (this skill)by K-Dense-AI | 100 | 46.4k | 56d ago | SKILL.md |
| Agent-Reachby Panniantong | 100 | 112.3k | 8d ago | CLAUDE.md |
| x402by browser-use | 100 | 121.0k | 3d ago | SKILL.md |
| algorithmic-artby anthropics | 100 | 177.9k | 55d ago | SKILL.md |
| pptxby anthropics | 100 | 177.9k | 55d ago | SKILL.md |
Frequently asked questions
- How do I install datalad?
- Run
npx skills add K-Dense-AI/scientific-agent-skills --skill datalad. The install tabs above show the steps for each supported agent. - Which AI agents does datalad work with?
- It is written for Universal, as a SKILL.md file. Other agents that read the same format can often use it too.
- Is datalad safe to use?
- Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful. It is MIT-licensed and scores 100/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand.
- Is datalad still maintained?
- The repository was last updated about 2 months ago, so datalad is actively maintained.
Skill content
View source on GitHubname: datalad
description: Retrieve, version, and publish scientific datasets with DataLad and git-annex, and capture computational provenance with datalad run, rerun, and containers-run. Use when cloning or fetching data from OpenNeuro, DANDI, datasets.datalad.org, or any DataLad dataset; when a file in a dataset reads as a broken symlink or a small pointer instead of real data; when an analysis needs a machine-readable record of how each output was produced so it can be re-executed; or when publishing a dataset to siblings such as a GitHub repository plus a storage remote. Also use to decide between DataLad and plain Git for a data-carrying repository.
compatibility: Needs datalad 1.6.x on Python 3.10+, plus git and git-annex 10.x. git-annex is not written in Python but installs as a prebuilt wheel from PyPI (uv pip install git-annex), from a system package manager, or from conda-forge. Container-based provenance also needs datalad-container (1.2.x) and Singularity/Apptainer or Docker. clone, get, and push need network access; credentialed remotes read secrets from the system keyring or from DATALAD_CREDENTIAL_<NAME>_<COMPONENT> environment variables.
license: MIT
allowed-tools: Read Write Edit Bash
metadata:
version: "1.0"
skill-author: Dylan Pulver
DataLad
Overview
DataLad is a data management layer over Git and git-annex. Git tracks the dataset structure, small text files, and the history. git-annex tracks the content of large files, storing each file as a key and keeping the bytes somewhere that is not necessarily the local repository.
That split is the single most important thing to internalise, because it means a freshly
cloned dataset contains the full history and the full file listing while containing almost
none of the data. A 100 TB dataset clones in seconds and occupies a few megabytes. The
bytes arrive only when asked for, per file, with datalad get.
The second thing DataLad adds is provenance. datalad run executes a command and commits
the result together with a machine-readable record of the command, its inputs, and its
outputs. datalad rerun reads that record back and re-executes it. This turns "how was
this figure produced" from an archaeology problem into a command.
When to use DataLad instead of plain Git
Use DataLad when any of the following holds:
- Files are too large for Git to handle comfortably, or the total exceeds what every collaborator wants on disk.
- Data lives in more than one place (a lab server, a cluster scratch, S3, a supercomputer) and you need to know which copies exist.
- The analysis must be re-executable, and a plain commit message is not enough evidence.
- You are consuming published datasets from OpenNeuro, DANDI, or
datasets.datalad.org, which are distributed as DataLad datasets. - The project nests other datasets inside it and you want each one to keep its own independent history.
Use plain Git when the repository is code and text only, everything fits comfortably in Git, and nobody needs partial checkouts. DataLad on top of a small pure-code repository adds indirection without buying anything.
Installation
# git-annex is NOT written in Python but is available from PyPI if you already
# have git itself installed:
uv pip install git-annex
# You can also install it first from the system
# (Debian/Ubuntu: apt install git-annex; macOS: brew install git-annex;
# conda-forge: conda install -c conda-forge git-annex)
uv pip install datalad
uv pip install datalad-container # only for containers-run
datalad wtf --section dependencies # confirm git-annex version is visible
The PyPI git-annex package ships the prebuilt binary as a wheel for Linux, macOS, and
Windows rather than building the Haskell sources, so it installs like any other Python
dependency and can be pinned in the same environment as DataLad. It does not bring git
along with it.
datalad wtf prints the resolved environment and is the first thing to run when behaviour
looks impossible. An old or missing git-annex is behind a large share of confusing errors.
DataLad itself is MIT licensed. git-annex is a separate tool under the AGPL, which matters only if you redistribute a modified git-annex rather than call it.
The failure that bites first: pointers are not data
After datalad clone, annexed files exist as symlinks into .git/annex/objects/ (or as
small pointer files where symlinks are unavailable, such as on Windows or a crippled
filesystem). Nothing has downloaded the content yet.
datalad clone https://github.com/OpenNeuroDatasets/ds000001.git
cd ds000001
ls sub-01/anat/ # the file is listed
python -c "import nibabel; nibabel.load('sub-01/anat/sub-01_T1w.nii.gz')" # fails
datalad get sub-01/anat/sub-01_T1w.nii.gz # now it works
The failure mode to recognise: a tool reports the file as empty, truncated, corrupt, "not
a gzip file", or a broken symlink, and the file size on disk is a few hundred bytes. That
is a pointer, not a corrupted download. Run datalad get before reading data, and treat
"file exists" as insufficient evidence that its content is present.
Before an analysis touches a directory, fetch it explicitly:
datalad get sub-01/ # everything under a path
datalad get -r . # everything, including subdatasets
datalad get -n -r . # subdataset structure only, no file content
datalad status --annex reports how much content is present locally, and
git annex whereis <path> reports which repositories hold a given file. whereis reads
recorded state and does not contact the remotes, so it tells you what git-annex last
learned rather than what is true right now.
See data-access.md for finding datasets, subdataset behaviour, dropping content safely, and repairing a dataset.
Recording provenance with datalad run
datalad run is the reason to reach for DataLad in a methods context. It saves the
command alongside its effect, in the same commit:
datalad run -m "extract brain mask" \
--input "sub-01/anat/sub-01_T1w.nii.gz" \
--output "derivatives/sub-01_brain.nii.gz" \
"bet {inputs} {outputs} -m"
What each part does, and why skipping it hurts:
--inputretrieves the content before running, so the command does not fail on a pointer. It also records the dependency, which is what letsrerunfetch the same inputs on a different machine.--outputunlocks or removes the target first, so git-annex does not refuse to write over content it is protecting. Without it, a second run of the same command commonly fails with a permission error on an annexed file that looks read-only.{inputs}and{outputs}expand to those values.{pwd},{dspath}, and{tmpdir}are also available, and{inputs[0]}indexes individual entries.- The commit message carries a JSON run record between
=== Do not change lines below ===and^^^ Do not change lines above ^^^. Do not hand-edit that block;rerunparses it.
datalad run refuses to start when the dataset has unsaved modifications, because an
unclean starting state makes the record unreliable. Save or discard first, or pass
--explicit to declare that the listed inputs and outputs are the complete story. Check a
command before committing to it with --dry-run basic or --dry-run command.
A run that changes nothing produces no commit, exactly as datalad save does.
Re-executing
datalad rerun # redo the run recorded at HEAD
datalad rerun --report # show what would be done, change nothing
datalad rerun --script recompute.sh # extract the commands instead of running them
datalad rerun --since <commit> -b check <revision> # replay a range onto a new branch
Rerunning onto a branch (-b) is the safe way to test reproducibility: the replay lands
somewhere else, and a diff against the original branch answers whether the outputs came
back identical.
Containers
With the datalad-container extension, register an image once and every subsequent run
records which image produced the outputs:
datalad containers-add fsl --url docker://brainlife/fsl:6.0.4
datalad containers-run -n fsl -m "brain mask in container" \
--input "sub-01/anat/sub-01_T1w.nii.gz" \
--output "derivatives/sub-01_brain.nii.gz" \
"bet {inputs} {outputs} -m"
The image itself is tracked in the dataset, so the software environment travels with the
data and the provenance record rather than living in someone's shell history. When only
one container is configured, -n may be omitted.
See provenance.md for the STAMPED principles and the YODA
project layout, the run record format, --explicit and --assume-ready semantics, and
exporting provenance toward W3C PROV.
Saving and inspecting changes
datalad status # what changed, including subdataset state
datalad save -m "add QC report" path/to/file
datalad save -m "checkpoint" -r # recurse into subdatasets
datalad save -m "small text file" --to-git notes.md
datalad save decides per file whether content goes to Git or to git-annex, following the
dataset's .gitattributes. Force a file into Git with --to-git, which is the right call
for code and small text files that should stay directly readable. The yoda procedure
(datalad create -c yoda) sets this up for code/, README.md, and CHANGELOG.md
automatically.
Creating a dataset
datalad create my_dataset # plain dataset
datalad create -c yoda my_analysis # analysis layout (code/ tracked in Git,
# README.md and CHANGELOG.md preconfigured)
datalad create -d . inputs/raw # register a new subdataset under an existing one
-c yoda applies the analysis project layout described in
provenance.md. -d . is what registers a new dataset as a
subdataset of the parent rather than leaving an unrelated repository inside it.
Publishing
A DataLad dataset is usually published to two places at once: a Git hosting service for the history, and a storage remote for the annexed content.
datalad create-sibling-github myaccount/mydataset
git annex initremote store type=S3 bucket=my-bucket encryption=none autoenable=true
datalad siblings configure -s github --publish-depends store
datalad push --to github
The Git sibling and the storage sibling are created by different tools on purpose. A Git
sibling is a Git remote, and datalad create-sibling-* handles the hosting-service ones.
An S3 bucket (or WebDAV, or an SSH directory) is a git-annex special remote, not a Git
remote, so it is created with git annex initremote. datalad siblings picks the special
remote up afterwards and treats it like any other. Using datalad siblings add --url s3://... here is the mistake this section exists to prevent: --url is a Git remote URL,
S3 is not, and the push --to github below then fails on the --publish-depends hop.
--publish-depends is what stops the common broken publication: a Git repository whose
history references content that was never uploaded, so collaborators clone successfully
and then find every datalad get failing. Declaring the dependency makes the storage
sibling publish first, every time.
datalad push sends both the Git history and, by default (--data auto-if-wanted), the
annexed content the target is configured to want. Pass --data anything to push all
content regardless of the target's preferences.
See publishing.md for RIA stores, special remotes, credential handling, and configuring which sibling holds what.
Freeing disk space
git annex whereis sub-01/ # confirm another copy exists first
datalad drop sub-01/ #
Truncated for display — read the full file on GitHub.
Related Skills
Agent-Reach
112.3kGive your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.
x402
121.0kSet up Browser Use Cloud payments with x402 — pay per request from a crypto wallet (USDC on Base mainnet), no signup or API key.
algorithmic-art
177.9kCreating algorithmic art using p5.js with seeded randomness and interactive parameter exploration. Use this when users request creating art using code, generative art, algorithmic art, flow fields, or particle systems.
pptx
177.9kUse this skill any time a .pptx or .potx file is involved in any way — as input, output, or both. This includes: creating slide decks, pitch decks, or presentations; reading, parsing, or extracting text from any .pptx or .potx file (even if the extracted content will be used elsewhere, like in an em…
Languages
Trust signals
From repository metadata: license, adoption, age and documentation. Not a code audit — see the Safety scan above for what the skill file itself contains.
