Kagome v2

Kagome is an open source Japanese morphological analyzer written in pure Go. It can tokenize Japanese text into words and analyze parts of speech, with dictionaries embedded in the binary for easy deployment.

[!NOTE] Key features (Improvements from v1):

• Self-contained binaries with embedded dictionaries (MeCab-IPADIC, UniDic)
• Multiple segmentation modes for different use cases
• RESTful API server mode for production use
• WebAssembly support for browser environments
• C library API for FFI integration (Python, PHP, and other languages)

Index

• Basic Usage
  • Command line
  • As a Go library
  • As a C library
  • More examples
• Install
• Commands
  • Tokenize command
  • Server command
    • RESTful API
    • Web App
  • Lattice command
  • Sentence command
• Dictionaries
• Segmentation modes
• Docker
• WebAssembly
• Use from other languages (FFI)
• Reference
• License

Basic Usage

Command line

% kagome -h
Japanese Morphological Analyzer -- github.com/ikawaha/kagome/v2
usage: kagome <command>
The commands are:
   [tokenize] - command line tokenize (*default)
   server - run tokenize server
   lattice - lattice viewer
   sentence - tiny sentence splitter
   version - show version

tokenize [-file input_file] [-dict dic_file] [-userdict user_dic_file] [-sysdict (ipa|uni)] [-simple false] [-mode (normal|search|extended)] [-split] [-json]
  -dict string
    	dict
  -file string
    	input file
  -json
    	outputs in JSON format
  -mode string
    	tokenize mode (normal|search|extended) (default "normal")
  -simple
    	display abbreviated dictionary contents
  -split
    	use tiny sentence splitter
  -sysdict string
    	system dict type (ipa|uni) (default "ipa")
  -udict string
    	user dict

% # piped standard input
% echo "すもももももももものうち" | kagome
すもも	名詞,一般,*,*,*,*,すもも,スモモ,スモモ
も	助詞,係助詞,*,*,*,*,も,モ,モ
もも	名詞,一般,*,*,*,*,もも,モモ,モモ
も	助詞,係助詞,*,*,*,*,も,モ,モ
もも	名詞,一般,*,*,*,*,もも,モモ,モモ
の	助詞,連体化,*,*,*,*,の,ノ,ノ
うち	名詞,非自立,副詞可能,*,*,*,うち,ウチ,ウチ
EOS

• For more details, see the Commands section.

As a Go library

You can integrate Kagome into your Go applications as follows:

# Install Kagome module
go get github.com/ikawaha/kagome/v2

package main

import (
  "fmt"
  "strings"

  "github.com/ikawaha/kagome-dict/ipa"
  "github.com/ikawaha/kagome/v2/tokenizer"
)

func main() {
  t, err := tokenizer.New(ipa.Dict(), tokenizer.OmitBosEos())
  if err != nil {
    panic(err)
  }
  // wakati (simple word splitting/segmentation)
  fmt.Println("---wakati---")
  seg := t.Wakati("すもももももももものうち")
  fmt.Println(seg)

  // tokenize w/ morphological analysis
  fmt.Println("---tokenize---")
  tokens := t.Tokenize("すもももももももものうち")
  for _, token := range tokens {
    features := strings.Join(token.Features(), ",")
    fmt.Printf("%s\t%v\n", token.Surface, features)
  }
}

output:

---wakati---
[すもも も もも も もも の うち]
---tokenize---
すもも	名詞,一般,*,*,*,*,すもも,スモモ,スモモ
も	助詞,係助詞,*,*,*,*,も,モ,モ
もも	名詞,一般,*,*,*,*,もも,モモ,モモ
も	助詞,係助詞,*,*,*,*,も,モ,モ
もも	名詞,一般,*,*,*,*,もも,モモ,モモ
の	助詞,連体化,*,*,*,*,の,ノ,ノ
うち	名詞,非自立,副詞可能,*,*,*,うち,ウチ,ウチ

As a C library

Kagome is written in pure Go but can be compiled as a C shared library and used from other languages via FFI (Foreign Function Interface).

See the "Use from other languages (FFI)" section below for details and examples.

More examples

We provide various examples demonstrating how to use Kagome in different scenarios:

• Examples directory
• Examples in GoDoc

Install

To get the kagome command line tool, choose your preferred installation method below:

• Go (recommended)

  go install github.com/ikawaha/kagome/v2@latest
  

• Homebrew

  # macOS and Linux (for both AMD64 and Arm64)
  brew install ikawaha/kagome/kagome
  

• Manual Install

  • For manual installation, download and extract the appropriate archived file for your OS and architecture from the releases page.
  • Note that the extracted binary must be placed in an accessible directory with execution permission.

• Docker/Docker Compose

  • See the Docker section below

Commands

Major sub-commands of kagome command line tool.

Tokenize command

% # interactive/REPL mode
% kagome
すもももももももものうち
すもも	名詞,一般,*,*,*,*,すもも,スモモ,スモモ
も	助詞,係助詞,*,*,*,*,も,モ,モ
もも	名詞,一般,*,*,*,*,もも,モモ,モモ
も	助詞,係助詞,*,*,*,*,も,モ,モ
もも	名詞,一般,*,*,*,*,もも,モモ,モモ
の	助詞,連体化,*,*,*,*,の,ノ,ノ
うち	名詞,非自立,副詞可能,*,*,*,うち,ウチ,ウチ
EOS

% # piped standard input
% echo "すもももももももものうち" | kagome
すもも  名詞,一般,*,*,*,*,すもも,スモモ,スモモ
も      助詞,係助詞,*,*,*,*,も,モ,モ
もも    名詞,一般,*,*,*,*,もも,モモ,モモ
も      助詞,係助詞,*,*,*,*,も,モ,モ
もも    名詞,一般,*,*,*,*,もも,モモ,モモ
の      助詞,連体化,*,*,*,*,の,ノ,ノ
うち    名詞,非自立,副詞可能,*,*,*,うち,ウチ,ウチ
EOS

% # JSON output
% # (For jq command see https://jqlang.org/)
% echo "猫" | kagome -json | jq .
[
  {
    "id": 286994,
    "start": 0,
    "end": 1,
    "surface": "猫",
    "class": "KNOWN",
    "pos": [
      "名詞",
      "一般",
      "*",
      "*"
    ],
    "base_form": "猫",
    "reading": "ネコ",
    "pronunciation": "ネコ",
    "features": [
      "名詞",
      "一般",
      "*",
      "*",
      "*",
      "*",
      "猫",
      "ネコ",
      "ネコ"
    ]
  }
]

% # word splitting/segmentation only (equivalent to "wakati" functionality)
% echo "すもももももももものうち" | kagome -json | jq -r '[.[].surface] | join("/")'
すもも/も/もも/も/もも/の/うち

% # Extract only pronunciations using jq (for Text-to-Speech purposes, etc.)
% echo "私ははにわよわわわんわん" | kagome -json | jq -r '.[].pronunciation'
ワタシ
ワ
ハニワ
ヨ
ワ
ワ
ワンワン

Server command

For continuous usage, kagome provides a server mode to decouple the startup time of the tokenizer.

RESTful API

Start a server and try to access the "/tokenize" endpoint.

% kagome server &
% curl -XPUT localhost:6060/tokenize -d'{"sentence":"すもももももももものうち", "mode":"normal"}' | jq .

Web App

Start a server and access http://localhost:6060 in your browser.

% kagome server &

<video src="https://github.com/user-attachments/assets/172d8b0c-be9a-4ee5-bdff-315a0eeee85b" controls width="100%"></video>

[!IMPORTANT] The demo web application uses graphviz to draw a lattice. You need graphviz to be installed on your system.

[!TIP] Kagome can be compiled to WebAssembly (wasm) and run locally in a web browser as well. For details, see the WebAssembly section.

• Wasm Demo: https://ikawaha.github.io/kagome/

Lattice command

A debug tool of tokenize process outputs a lattice in graphviz dot format.

% kagome lattice 私は鰻 | dot -Tpng -o lattice.png

Sentence command

Split long text into sentences:

% echo "吾輩は猫である。名前はまだ無い。" | kagome sentence
吾輩は猫である。
名前はまだ無い。

This command is useful if a single line of data is too lengthy, and you want to avoid errors such as bufio.Scanner: token too long.

% echo "吾輩は猫である。名前はまだ無い。" | kagome -json | jq -r '[.[].surface] | join("/")'
吾輩/は/猫/で/ある/。/名前/は/まだ/無い/。

% echo "吾輩は猫である。名前はまだ無い。" | kagome sentence | kagome -json | jq -r '[.[].surface] | join("/")'
吾輩/は/猫/で/ある/。
名前/は/まだ/無い/。

This command is equivalent to the -split option of the tokenize command.

% echo "吾輩は猫である。名前はまだ無い。" | kagome -split -json | jq -r '[.[].surface] | join("/")'
吾輩/は/猫/で/ある/。
名前/は/まだ/無い/。

Dictionaries

• Currently supported dictionaries by default.

  |dict| source | package | |:---|:---|:---| |MeCab IPADIC| mecab-ipadic-2.7.0-20070801 | github.com/ikawaha/kagome-dict/ipa| |UniDIC| unidic-mecab-2.1.2_src | github.com/ikawaha/kagome-dict/uni |

• Experimental Features

  |dict|source|package| |:---|:---|:---| |mecab-ipadic-NEologd|mecab-ipadic-neologd| github.com/ikawaha/kagome-ipa-neologd| |Korean MeCab|mecab-ko-dic-2.1.1-20180720 | github.com/ikawaha/kagome-dict-ko|

[!NOTE] For more details and differences between the dictionaries, see the wiki.

Segmentation modes

Similar to Kuromoji, Kagome also supports various *segmentation modes