POMDPs

| Linux | Mac OS X | Windows | |-----------------|---------------------|-------------------------| | | | |

This package provides a core interface for working with Markov decision processes (MDPs) and partially observable Markov decision processes (POMDPs). The POMDPTools package acts as a "standard library" for the POMDPs.jl interface, providing implementations of commonly-used components such as policies, belief updaters, distributions, and simulators.

Our goal is to provide a common programming vocabulary for:

1. Expressing problems as MDPs and POMDPs.
2. Writing solver software.
3. Running simulations efficiently.

POMDPs.jl integrates with other ecosystems:

• Python can be used to define and solve MDPs and POMDPs via the quickpomdps package or through tables directly via pyjulia.
• POMDPTools provides two-way integration with CommonRLInterface and therefore with the JuliaReinforcementLearning packages.
• The SymbolicMDPs package provides an interface to work with PDDL models.

For a detailed introduction, check out our Julia Academy course! For help, please post in GitHub Discussions tab. We welcome contributions from anyone! See CONTRIBUTING.md for information about contributing.

Installation

POMDPs.jl and associated solver packages can be installed using Julia's package manager. For example, to install POMDPs.jl and the QMDP solver package, type the following in the Julia REPL:

using Pkg; Pkg.add("POMDPs"); Pkg.add("QMDP")

Quick Start

To run a simple simulation of the classic Tiger POMDP using a policy created by the QMDP solver, you can use the following code (note that POMDPs.jl is not limited to discrete problems with explicitly-defined distributions like this):

using POMDPs, QuickPOMDPs, POMDPTools, QMDP

m = QuickPOMDP(
    states = ["left", "right"],
    actions = ["left", "right", "listen"],
    observations = ["left", "right"],
    initialstate = Uniform(["left", "right"]),
    discount = 0.95,

    transition = function (s, a)
        if a == "listen"
            return Deterministic(s) # tiger stays behind the same door
        else # a door is opened
            return Uniform(["left", "right"]) # reset
        end
    end,

    observation = function (s, a, sp)
        if a == "listen"
            if sp == "left"
                return SparseCat(["left", "right"], [0.85, 0.15]) # sparse categorical distribution
            else
                return SparseCat(["right", "left"], [0.85, 0.15])
            end
        else
            return Uniform(["left", "right"])
        end
    end,

    reward = function (s, a)
        if a == "listen"
            return -1.0
        elseif s == a # the tiger was found
            return -100.0
        else # the tiger was escaped
            return 10.0
        end
    end
)

solver = QMDPSolver()
policy = solve(solver, m)

rsum = 0.0
for (s,b,a,o,r) in stepthrough(m, policy, "s,b,a,o,r", max_steps=10)
    println("s: $s, b: $([s=>pdf(b,s) for s in states(m)]), a: $a, o: $o")
    global rsum += r
end
println("Undiscounted reward was $rsum.")

For more examples and examples with visualizations, reference the Examples and Gallery of POMDPs.jl Problems sections of the documentaiton.

Documentation and Tutorials

In addition to the above-mentioned Julia Academy course, detailed documentation and examples can be found here.

Supported Packages

Many packages use the POMDPs.jl interface, including MDP and POMDP solvers, support tools, and extensions to the POMDPs.jl interface. POMDPs.jl and all packages in the JuliaPOMDP project are fully supported on Linux. OSX and Windows are supported for all native solvers*, and most non-native solvers should work, but may require additional configuration.

Tools:

POMDPs.jl itself contains only the core interface for communicating about problem definitions; these packages contain implementations of commonly-used components:

| Package | Build | Coverage | |-------------------|----------------------|------------------| | POMDPTools (hosted in this repository) | | | | ParticleFilters | | |

Implemented Models:

Many models have been implemented using the POMDPs.jl interface for various projects. This list contains a few commonly used models:

| Package | Build | Coverage | |-------------------|----------------------|------------------| | POMDPModels || | | LaserTag | | | | RockSample | | | | TagPOMDPProblem | | | | DroneSurveillance | | | | ContinuumWorld | | | | VDPTag2 | | [![codecov](https://codecov.io/github/JuliaPOMDP/VDPTag2.jl/gra