Make it compatible with both python 2.x and 3.x.

This is an extremely simple Python wrapper for SRILM: http://www.speech.sri.com/projects/srilm/

Install SRILM

• vim Makefile
  • SRILM = $(PWD) # or a specific direcotry
  • MACHINE_TYPE := i686-m64 # output from uname -i
• vim common/Makefile.machine.MACHINE_TYPE
  • NO_TCL = 1 -> NO_TCL = X
  • GAWK = /usr/bin/awk -> GAWK = /usr/bin/gawk # output from which gawk
  • compile with -fPIC
    • ADDITIONAL_CFLAGS = -fopenmp -> ADDITIONAL_CFLAGS = -fopenmp -fPIC
    • ADDITIONAL_CXXFLAGS = -fopenmp -> ADDITIONAL_CXXFLAGS = -fopenmp -fPIC
• make World & make test
• export SRILM/bin:SRILM/bin/i686-m64:$PATH
• which ngram-count # test

Basically it lets you load a SRILM-format ngram model into memory, and then query it directly from Python.

Right now this is extremely bare-bones, just enough to do what I needed, no fancy infrastructure at all. Feel free to send patches though if you extend it!

Requirements:

• SRILM
• Cython

Installation:

• Edit setup.py so that it can find your SRILM build files.
• To install in your Python environment, use: python setup.py install To just build the interface module: python setup.py build_ext --inplace which will produce srilm.so, which can be placed on your PYTHONPATH and accessed as 'import srilm'.

Usage:

from srilm import LM

Use lower=True if you passed -lower to ngram-count. lower=False is

default.

lm = LM("path/to/model/from/ngram-count", lower=True)

Compute log10(P(brown | the quick))

Note that the context tokens are in reverse order, as per SRILM's

internal convention. I can't decide if this is a bug or not. If you

have a model of order N, and you pass more than (N-1) words, then

the first (N-1) entries in the list will be used. (I.e., the most

recent (N-1) context words.)

lm.logprob_strings("brown", ["quick", "the"])

We can also compute the probability of a sentence (this is just

a convenience wrapper):

log10 P(The | <s>)

+ log10 P(quick | <s> The)

+ log10 P(brown | <s> The quick)

lm.total_logprob_strings(["The", "quick", "brown"])

Internally, SRILM interns tokens to integers. You can convert back

and forth using the .vocab attribute on an LM object:

idx = lm.vocab.intern("brown") print idx assert lm.vocab.extern(idx) == "brown"

.extern() returns None if an idx is unused for some reason.

There's a variant of .logprob_strings that takes these directly,

which is probably not really any faster, but sometimes is more

convenient if you're working with interned tokens anyway:

lm.logprob(lm.vocab.intern("brown"), [lm.vocab.intern("quick"), lm.vocab.intern("the"), ])

There are detect "magic" tokens that don't actually represent anything

in the input stream, like <s> and <unk>. You can detect them like

assert lm.vocab.is_non_word(lm.intern("<s>")) assert not lm.vocab.is_non_word(lm.intern("brown"))

Sometimes it's handy to have two models use the same indices for the

same words, i.e., share a vocab table. This can be done like:

lm2 = LM("other/model", vocab=lm.vocab)

This gives the index of the highest vocabulary word, useful for

iterating over the whole vocabulary. Unlike the Python convention

for describing ranges, this is the inclusive maximum:

lm.vocab.max_interned()

And finally, let's put it together with an example of how to find

the max-probability continuation:

argmax_w P(w | the quick)

by querying each word in the vocabulary in turn:

context = [lm.vocab.intern(w) for w in ["quick", "the"]] best_idx = None best_logprob = -1e100

Don't forget the +1, because Python and SRILM disagree about how

ranges should work...

for i in xrange(lm.vocab.max_interned() + 1): logprob = lm.logprob(i, context) if logprob > best_logprob: best_idx = i best_logprob = logprob best_word = lm.vocab.extern(best_idx) print "Max prob continuation: %s (%s)" % (best_word, best_logprob)