decancer

A library that removes common unicode confusables/homoglyphs from strings.

• Its core is written in Rust and utilizes a form of Binary Search to ensure speed!
• By default, it's capable of filtering 222,557 (19.98%) different unicode codepoints like:
  • All whitespace characters
  • All diacritics, this also eliminates all forms of Zalgo text
  • Most leetspeak characters
  • Most homoglyphs
  • Several emojis
• Unlike other packages, this package is unicode bidi-aware where it also interprets right-to-left characters in the same way as it were to be rendered by an application!
• Its behavior is also highly customizable to your liking!

<!---[ begin DECANCER_GLOBAL ]--->

• And it's available in the following languages:
  • Rust
  • JavaScript (Node.js/Browser)
  • Java
  • C/C++
  • Go
  • Python (unofficial)

<!---[ end ]--->

Installation

<!---[ begin DECANCER_GLOBAL ]---> <details> <summary><b>Rust (v1.65 or later)</b></summary> <!---[ end, begin DECANCER_RUST ]--->

In your Cargo.toml:

decancer = "3.3.3"

<!---[ end, begin DECANCER_GLOBAL ]---> </details> <details> <summary><b>JavaScript (Node.js)</b></summary> <!---[ end, begin DECANCER_JS ]--->

In your shell:

npm install decancer

In your code (CommonJS):

const decancer = require('decancer')

In your code (ESM):

import decancer from 'decancer'

<!---[ end, begin DECANCER_GLOBAL ]---> </details> <details> <summary><b>JavaScript (Browser)</b></summary>

In your code:

<script type="module">
  import init from 'https://cdn.jsdelivr.net/gh/null8626/decancer@v3.3.3/bindings/wasm/bin/decancer.min.js'

  const decancer = await init()
</script>

</details> <details> <summary><b>Java</b></summary>

As a JAR file

You can download the latest JAR file here.

As a dependency

In your build.gradle:

repositories {
  mavenCentral()
  maven { url 'https://jitpack.io' }
}

dependencies {
  implementation 'io.github.null8626:decancer:3.3.3'
}

In your pom.xml:

<repositories>
  <repository>
    <id>central</id>
    <url>https://repo.maven.apache.org/maven2</url>
  </repository>
  <repository>
    <id>jitpack.io</id>
    <url>https://jitpack.io</url>
  </repository>
</repositories>

<dependencies>
  <dependency>
    <groupId>io.github.null8626</groupId>
    <artifactId>decancer</artifactId>
    <version>3.3.3</version>
  </dependency>
</dependencies>

Building from source

Windows:

> git clone https://github.com/null8626/decancer.git --branch v3.3.3 --depth 1
> cd decancer/bindings/java
> powershell -NoLogo -NoProfile -NonInteractive -Command "Expand-Archive -Path .\bin\bindings.zip -DestinationPath .\bin -Force"
> gradle build -x test

macOS/Linux:

git clone https://github.com/null8626/decancer.git --branch v3.3.3 --depth 1
cd decancer/bindings/java
unzip ./bin/bindings.zip -d ./bin
chmod +x ./gradlew
./gradlew build -x test

Tip: You can shrink the size of the resulting JAR file by removing binaries in the bin directory for the platforms you don't want to support.

</details> <details> <summary><b>C/C++</b></summary> <!---[ end, begin DECANCER_NATIVE ]--->

Download

• Header file
• Download for ARM64 macOS (11.0+, Big Sur+)
• Download for ARM64 iOS
• Download for Apple iOS Simulator on ARM6
• Download for ARM64 Android
• Download for ARM64 Windows MSVC
• Download for ARM64 Linux (kernel 4.1, glibc 2.17+)
• Download for ARM64 Linux with MUSL
• Download for ARMv6 Linux (kernel 3.2, glibc 2.17)
• Download for ARMv5TE Linux (kernel 4.4, glibc 2.23)
• Download for ARMv7-A Android
• Download for ARMv7-A Linux (kernel 4.15, glibc 2.27)
• Download for ARMv7-A Linux, hardfloat (kernel 3.2, glibc 2.17)
• Download for 32-bit Linux w/o SSE (kernel 3.2, glibc 2.17)
• Download for 32-bit MSVC (Windows 7+)
• Download for 32-bit FreeBSD
• Download for 32-bit Linux (kernel 3.2+, glibc 2.17+)
• Download for PPC64LE Linux (kernel 3.10, glibc 2.17)
• Download for RISC-V Linux (kernel 4.20, glibc 2.29)
• Download for S390x Linux (kernel 3.2, glibc 2.17)
• Download for SPARC Solaris 11, illumos
• Download for Thumb2-mode ARMv7-A Linux with NEON (kernel 4.4, glibc 2.23)
• Download for 64-bit macOS (10.12+, Sierra+)
• Download for 64-bit iOS
• Download for 64-bit MSVC (Windows 7+)
• Download for 64-bit FreeBSD
• Download for 64-bit illumos
• Download for 64-bit Linux (kernel 3.2+, glibc 2.17+)
• Download for 64-bit Linux with MUSL

Building from source

Building from source requires Rust v1.65 or later.

git clone https://github.com/null8626/decancer.git --branch v3.3.3 --depth 1
cd decancer/bindings/native
cargo build --release

And the binary files should be generated in the target/release directory.

<!---[ end, begin DECANCER_GLOBAL ]---> </details> <details> <summary><b>Go (v1.17 or later)</b></summary> <!---[ end, begin DECANCER_GO ]--->

Building requires Rust v1.65 or later. Windows systems also require a MinGW compiler to be readily available.

In your shell:

git clone https://github.com/null8626/decancer.git --branch v3.3.3 --depth 1
cd decancer/bindings/go
sudo -E "PATH=$PATH" go generate
go install

For most platforms, go generate will require elevated administrator permissions as decancer's native binding