ArtSleuth

<div align="center"> <br> <img src="assets/banner.png" alt="ArtSleuth Banner" width="100%" />

<br><br>

         

<br>

<br> </div>

---

<br>

✦ About

ArtSleuth is a computational art-analysis framework that formalises what connoisseurs have done for centuries — examining the physical evidence a painter leaves on a canvas — using machine learning.

Brushstroke directionality, impasto relief, palette temperature, the habitual gestures that reside in the least-scrutinised passages of a painting — drapery folds, background foliage, the rendering of earlobes. These are the signals that distinguish one hand from another, and they map naturally onto what self-supervised vision transformers learn to encode.

<br> <div align="center">

| | Capability | Method | |:---:|:---|:---| | | Stroke orientation, coherence, energy, curvature with patch-level clustering | Structure tensor on image gradients + DINOv2 patch embeddings | | | Period, school, and genre prediction | CLIP embeddings through learned linear heads (period and genre pretrained; school randomly initialised) | | | Embedding-space comparison with temporal plausibility scoring | Cosine similarity with GP-based date estimation | | | Bayesian inference of distinct hands in collaborative paintings | Dirichlet process Gaussian mixture model | | | One-class anomaly scoring with adversarial robustness testing | Mahalanobis distance plus historical forgery simulation | | | Complementary features from two vision transformers | Concatenation at inference; cross-attention available for training | | | Models how an artist's style evolves over decades | Gaussian process regression in embedding space (requires user-supplied dated references) | | | Visual heatmaps highlighting regions the model considers salient | Gradient-based saliency maps |

</div> <br>

---

<br>

✦ What's Novel

ArtSleuth combines several techniques that are typically studied in isolation:

1. Style-Guided Cross-Attention Fusion — CLIP's semantic understanding directs DINOv2's patch-level attention via multi-head cross-attention with learned temperature, producing fused features neither backbone achieves alone.

2. Temporal Style Drift Modelling — Gaussian process regression over time-stamped reference embeddings captures how an artist's hand evolves across decades, reporting temporal plausibility as a separate signal. Requires user-supplied dated references; no bundled data is shipped.

3. Hierarchical Workshop Decomposition — A Dirichlet process Gaussian mixture model automatically infers the number of distinct hands in a painting, replacing flat k-means with art-historically grounded probabilistic clustering.

4. Adversarial Forgery Robustness — Stress-tests detection against simulated historical forgery techniques (artificial aging, style transfer perturbation, material anachronism) at multiple severity levels.

<br>

---

<br>

✦ Quick Start

No installation required?  Try ArtSleuth live on HuggingFace Spaces.   Pretrained weights are on the HuggingFace Hub.

pip install artsleuth

<br>

Python

import artsleuth

result = artsleuth.analyze("judith_slaying_holofernes.jpg")
print(result.summary())

explanation = result.explain()
explanation.save("analysis_overlay.png")

<br>

CLI

artsleuth analyze painting.jpg
artsleuth style painting.jpg --top-k 5
artsleuth compare painting_a.jpg painting_b.jpg
artsleuth workshop painting.jpg
artsleuth robustness painting.jpg -r "Artemisia Gentileschi"
artsleuth benchmark --backbone dinov2 --backbone fusion
artsleuth demo

<br>

---

<br>

✦ Architecture

%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#1A2E48', 'primaryTextColor': '#F0F0F0', 'primaryBorderColor': '#9DC0D8', 'lineColor': '#9DC0D8', 'secondaryColor': '#1A2E48', 'tertiaryColor': '#1A2E48', 'edgeLabelBackground': '#0D1117', 'clusterBkg': '#0D1117', 'clusterBorder': '#9DC0D8', 'titleColor': '#9DC0D8'}}}%%

graph TD
    %% Temporal drift is optional (off by default; needs user TemporalRegistry data)
    Input["Artwork Image"] --> Resize["Resize · Crop · Normalise"]

    Resize --> Patches["Patch Extraction"]
    Resize --> FullImage["Full-Image Encoding"]

    Patches --> DINO["DINOv2"]
    FullImage --> CLIPEnc["CLIP"]

    DINO --> Brushstroke["Brushstroke Analysis"]
    DINO --> WorkshopNode["Workshop Decomposition"]
    CLIPEnc --> Style["Style Classification"]

    DINO --> Concat["Feature Concatenation"]
    CLIPEnc --> Concat

    Concat --> Attribution["Attribution Scoring"]
    Attribution -.->|optional| Temporal["Temporal Drift Model"]

    Concat --> Forgery["Forgery Detection"]
    Concat -.->|optional| Adversarial["Adversarial Robustness"]

    DINO --> Explain["Saliency Maps"]

    Forgery --> Report["Analysis Report"]
    Attribution --> Report
    Temporal -.-> Report
    Style --> Report
    Brushstroke --> Report
    WorkshopNode --> Report
    Explain --> Report

    Report --> WebUI["Web UI"]
    Report --> CLI["CLI"]
    Report --> MCP["MCP Server"]

_{The inference pipeline concatenates CLIP + DINOv2 embeddings. Cross-attention fusion is available as a training-time architecture (used in the benchmark fine-tuning) but is not part of the default inference path.}

<br> <div align="center">

| Backbone | Strength | Used For | |:---|:---|:---| | DINOv2 | Fine-grained texture and structure | Brushstroke analysis · patch embeddings | | CLIP | Semantic-stylistic understanding | Style classification · style embeddings | | Concat | Complementary feature combination | Attribution · forgery detection |

</div>

_{Default backbone sizes: DINOv2 ViT-B/14 + CLIP ViT-L/14. First run downloads ~1–2 GB of pretrained weights from HuggingFace Hub.}

<br>

---

<br>

✦ Benchmark

Linear probe and end-to-end evaluation on the full WikiArt dataset (81 444 images, 80/20 split, seed 42):

<div align="center">

| Backbone | Style Acc | Style F1 | Artist Acc | Artist Top-5 | Genre Acc | |:---|:---:|:---:|:---:|:---:|:---:| | DINOv2 · ViT-B/14 | 57.5 % | 0.553 | 64.7 % | 90.9 % | 71.0 % | | CLIP · ViT-L/14 | 67.1 % | 0.656 | 74.6 % | 95.9 % | 75.0 % | | Fusion · frozen | 65.0 % | 0.633 | 71.0 % | 94.2 % | 74.2 % | | Fusion · fine-tuned † | 71.6 % | 0.703 | 77.8 % | 96.2 % | 75.1 % | | Fusion · e2e † | 72.7 % | — | 79.0 % | 96.9 % | 76.6 % |

</div>

_{Top three rows: logistic-regression linear probes on frozen backbones (macro-averaged across 27 styles, 129 artists, 11 genres). Reproducible via benchmarks/wikiart.py. † Bottom two rows: separate training runs with partial backbone unfreezing (last 3 transformer blocks), multi-task CE + supervised contrastive loss, AdamW, mixed-precision (5 epochs, effective batch 64). Training code not included in this repository; these numbers are reported for context.}

<br>

Reproduce the frozen linear-probe benchmarks:

pip install artsleuth[benchmarks]
artsleuth benchmark --device cuda

<br> <details> <summary>&nbsp;Forgery detection validation (one-class authentication)</summary> <br>

For each of 125 named artists (≥ 80 works, excluding the catch-all "Unknown Artist" category), we fit a Mahalanobis-distance reference model from 80 % of their authenticated WikiArt works, then test whether held-out genuine paintings score lower (closer to the reference distribution) than impostor paintings by other artists. ROC-AUC = 1.0 means perfect separation; 0.5 means chance.

<div align="center">

| | DINOv2 ViT-B/14 | CLIP ViT-L/14 | Fused (concat) | |:---|:---:|:---:|:---:| | Mean AUC | 0.873 | 0.958 | 0.897 | | Median AUC | 0.895 | 0.970 | 0.918 | | AUC ≥ 0.95 | 28 / 125 artists | 81 / 125 artists | 36 / 125 artists | | AUC ≥ 0.90 | 62 / 125 artists | 113 / 125 artists | 75 / 125 artists |

</div> <br>

Top 15 and bottom 5 by fused AUC:

<div align="center">

| Artist | Works | DINOv2 | CLIP | Fused | |:---|:---:|:---:|:---:|:---:| | Sam Francis | 317 | 1.000 | 1.000 | 1.000 | | Antoine Blanchard | 170 | 1.000 | 1.000 | 1.000 | | Gene Davis | 155 | 1.000 | 1.000 | 1.000 | | Fra Angelico | 167 | 0.995 | 0.990 | 0.998 | | Juan Gris | 196 | 0.993 | 0.996 | 0.998 | | Frans Hals | 176 | 0.992 | 0.999 | 0.997 | | Édouard Cortès | 214 | 0.992 | 1.000 | 0.995 | | El Greco | 159 | 0.981 | 1.000 | 0.993 | | Fernand Léger | 223 | 0.987 | 0.976 | 0.993 | | Anthony van Dyck | 163 | 0.985 | 1.000 | 0.992 | | Maxime Maufra | 119 | 0.991 | 0.998 | 0.991 | | Joshua Reynolds | 200 | 0.976 | 0.999 | 0.989 | | Henri Fantin-Latour | 105 | 0.977 | 1.000 | 0.989 | | Ivan Aivazovsky | 577 | 0.980 | 0.998 | 0.986 | | Gustave Moreau | 83 | 0.965 | 1.000 | 0.983 | | … | | | | | | Salvador Dalí | 479 | 0.675 | 0.876 | 0.725 | | Vasily Polenov | 225 | 0.684 | 0.926 | 0.723 | | Jacek Malczewski | 91 | 0.659 | 0.961 | 0.709 | | Mikhail Vrubel | 95 | 0.618 | 0.823 | 0.654 | | M. C. Escher | 126 | 0.610 | 0.899 | 0.649 |

</div>

_{Mahalanobis-distance one-class classification on WikiArt (125 named artists, 80/20 split, equal genuine/impostor test sets, seed 42). Artists with distinctive visual signatures (El Greco, Fra Angelico, ukiyo-e prints) approach perfect separation; stylistically versatile artists (Dalí, Escher) are harder to model as a single distribution. Full per-artist results in artsleuth/benchmarks/forgery_validation_results.json.}

</details> <br>

---

<br>

✦ Related Work & Honest Limitations

Automated art classification has a rich history, and ArtSleuth builds on the shoulders of work we want to acknowledge properly.

<br>