gemma4-heretical

One-command setup for an uncensored (abliterated) Gemma 4 31B model in Ollama, with optional MLX support for native Apple Silicon inference.

Uses the best available community abliteration: trohrbaugh/gemma-4-31b-it-heretic-ara, produced by Heretic using Arbitrary-Rank Ablation (ARA).

| Metric | Stock | Abliterated | |--------|-------|-------------| | Refusals (out of 100) | 98 | 5 | | KL divergence | — | 0.012 (near-zero quality loss) |

Quick start (Ollama)

Requires Ollama 0.20.2+ and ~33 GB disk/RAM (Q8_0).

git clone https://github.com/pmarreck/gemma4-heretical
cd gemma4-heretical
./get-gemma4-heretical
ollama run gemma4-heretical

Smaller quantizations are available if RAM is tight:

./get-gemma4-heretical Q4_K_M    # ~19 GB
./get-gemma4-heretical Q6_K      # ~25 GB
./get-gemma4-heretical IQ4_NL    # ~17 GB (imatrix)

What does get-gemma4-heretical actually do?

Community GGUF uploads of Gemma 4 models ship with an incorrect chat template (wrong turn delimiters), causing the model to output --- on repeat instead of actual responses. Ollama has built-in Gemma 4 support via RENDERER gemma4 / PARSER gemma4, but the HuggingFace GGUFs don't set it.

This script:

1. Pulls the GGUF from HuggingFace via ollama pull
2. Re-registers it with the correct Gemma 4 renderer/parser metadata
3. That's it — no re-download, no conversion, just a metadata fix

MLX (Apple Silicon native)

For native Metal inference via mlx-vlm, convert the model to MLX format:

nix develop   # provides Python 3.12 + auto-installs dependencies
mlx_vlm.convert \
  --hf-path trohrbaugh/gemma-4-31b-it-heretic-ara \
  --mlx-path ./gemma4-heretical-mlx-8bit \
  -q --q-bits 8

Then serve it as an OpenAI-compatible API:

mlx_vlm.server --model ./gemma4-heretical-mlx-8bit --port 8080

See OPENCLAW_SETUP.md for connecting this to OpenClaw or any OpenAI-compatible client.

Note: Some users report that MLX-converted models can feel slightly less capable than their GGUF equivalents, especially at lower quantizations. This is likely due to differences in quantization methods — GGUF benefits from imatrix-calibrated quantization that better preserves critical weights, whereas MLX quantization historically lacked this. At 8-bit the gap should be minimal, but YMMV. See also omlx, a tool that may help streamline MLX LLM inference on Macs.

DIY abliteration

Want to run Heretic yourself instead of using the pre-made model? The abliterate script handles it (requires ~62 GB RAM for bf16, several hours on MPS):

nix develop
./abliterate
# ... wait for 200 optimization trials ...
./convert-to-ollama

Note: the abliterate script includes a monkey-patch for PEFT #3129 — PEFT doesn't yet recognize Gemma 4's Gemma4ClippableLinear module.

Credits

• Gemma 4 by Google DeepMind
• Heretic by Philipp Emanuel Weidmann — the abliteration tool
• trohrbaugh/gemma-4-31b-it-heretic-ara — the abliterated weights
• jfiekdjdk/gemma-4-31b-it-heretic-ara-gguf — imatrix GGUF quantization

License

Scripts in this repo are MIT. Model weights are subject to Google's Gemma license.