herbdl

Explorations in unsupervised learning of herbaria samples using deep learning models for plant species classification.

Project Overview

This repository contains experiments on herbarium specimen classification using SWIN Transformers, CLIP, and BioCLIP models. The project runs on Boston University's Shared Computing Cluster (SCC).

Repository Structure

`datasets/`

Dataset management and preprocessing utilities.

dataset.py - HerbariaClassificationDataset class for loading and preprocessing herbarium images
constants.py - Path definitions for Kaggle 2021 and 2022 herbarium datasets
merge_datasets.py / merge_datasets.ipynb - Tools for combining multiple datasets
Supports flexible label columns (species, family, genus) and integrates with HuggingFace AutoImageProcessor

Key Datasets:

Kaggle Herbarium 2021 dataset
Kaggle Herbarium 2022 dataset

`finetuning/`

Model training and evaluation scripts for multiple architectures.

`SWIN/`

SWIN Transformer model training and evaluation.

SWIN_finetuning.py - Primary training script using HuggingFace Trainer with WandB logging
train.py - Custom training loop with layer freezing support
eval.py - Model evaluation utilities
Base model: microsoft/swin-base-patch4-window12-384

`BioCLIP/`

BioCLIP zero-shot evaluation for biological domain.

zero_shot.py - Zero-shot evaluation on herbarium data
train_evaluation.py - Training set evaluation

`SWIN-CLIP/`

Hybrid model combining SWIN visual features with CLIP text-image alignment.

train.py - Main training script
modular_model.py - Modular architecture implementation
trainer.py - Custom trainer implementation
train_baseline.py - Baseline model training

Configuration:

Model checkpoints saved in output/SWIN/kaggle22/
WandB integration: project herbdl, entity bu-spark-ml
Environment variables control freezing, learning rate schedules, and run identification

`CLIP/`

Zero-shot evaluation experiments using OpenAI CLIP.

CLIP_0shot.ipynb - Primary evaluation notebook
Tests species identification with/without visible text labels
Explores phenology detection (flowers, buds, leaves)
Documents CLIP's OCR behavior on specimen labels

`clustering_viz/`

Interactive visualization of learned representations and outlier detection.

Main Workflows:

kaggle22_clustering.ipynb - Feature extraction, PCA/t-SNE dimensionality reduction, and visualization generation
asteraceae_outliers.ipynb - Euclidean distance-based outlier detection
outlier_detection/asteraceae_outliers.ipynb - Advanced Mahalanobis distance-based outlier detection
generate_thumbnails.py - Pre-generates optimized thumbnails for fast hover previews
index.html - Interactive Plotly-based web interface with filtering, search, and image preview

Features:

Click points to view herbarium specimens (stacks up to 3 images)
Hover preview with optimized thumbnails (10-20x faster loading)
Search/filter controls for species clusters
Axis locking for consistent zoom levels
Outlier visualization with different markers

`descriptions/`

Text description generation and web scraping utilities.

scrape_ncsu.py - Scrape plant descriptions from NCSU database
wikipedia_scrape.py - Extract plant information from Wikipedia
generate_conv.py - Generate conversational descriptions
playground.ipynb - Experimentation notebook

`kaggle_eval/`

Kaggle competition evaluation scripts and results.

evaluation.py - Evaluation metrics computation
CLIP_explain.ipynb - CLIP model interpretability analysis
evaluation_result/ - JSON files with family, genus, and species-level results

`utils/`

General utility scripts.

resize_images.py - Batch image resizing
image_install_parallel.py - Parallel image downloading
notifications.py - Job notification system
compression.sh - Image compression utilities
labeling.ipynb - Data labeling tools

Quick Start

Environment Setup

# Create and activate virtual environment
virtualenv venv
source venv/bin/activate

# Install dependencies for finetuning
pip install -r finetuning/requirements.txt

# Install dependencies for clustering visualization
pip install -r clustering_viz/requirements.txt

Running SWIN Finetuning

cd finetuning/SWIN
python SWIN_finetuning.py \
    --output_dir ../output/SWIN/kaggle22/ \
    --model_name_or_path "microsoft/swin-base-patch4-window12-384" \
    --train_file ../datasets/train_22_scientific.json \
    --do_train --do_eval \
    --per_device_train_batch_size 8 \
    --learning_rate 1e-4 \
    --num_train_epochs 3

Creating Clustering Visualizations

cd clustering_viz
jupyter notebook kaggle22_clustering.ipynb
# Configure checkpoint path and validation dataset
# Generate thumbnails: python generate_thumbnails.py <plot_json_file>
# Update index.html with JSON filepath
# View via SCC OnDemand

Data Paths (SCC)

Base project: /projectnb/herbdl/
Image data: /projectnb/herbdl/data/kaggle-herbaria/
Model checkpoints: /projectnb/herbdl/workspaces/<username>/herbdl/finetuning/output/

Additional Documentation

CLAUDE.md - Detailed project guide for Claude Code
Subdirectory READMEs in datasets/, utils/, CLIP/, and finetuning/output/

License

See LICENSE file for details.

Herbdl

Install / Use

README