ProgressMeter.jl

Progress meter for long-running operations in Julia

Installation

Within julia, execute

using Pkg; Pkg.add("ProgressMeter")

Usage

Progress meters for tasks with a pre-determined number of steps

This works for functions that process things in loops or with map/pmap/reduce:

using Distributed
using ProgressMeter

@showprogress dt=1 desc="Computing..." for i in 1:50
    sleep(0.1)
end

@showprogress pmap(1:10) do x
    sleep(0.1)
    x^2
end

@showprogress reduce(1:10) do x, y
    sleep(0.1)
    x + y
end

The first incantation will use a minimum update interval of 1 second, and show the ETA and final duration. If your computation runs so quickly that it never needs to show progress, no extraneous output will be displayed.

The @showprogress macro wraps a for loop, comprehension, @distributed for loop, or map/pmap/reduce as long as the object being iterated over implements the length method and will handle continue correctly.

using Distributed
using ProgressMeter

@showprogress @distributed for i in 1:10
    sleep(0.1)
end

result = @showprogress desc="Computing..." @distributed (+) for i in 1:10
    sleep(0.1)
    i^2
end

In the case of a @distributed for loop without a reducer, an @sync is implied.

You can also control progress updates and reports manually:

function my_long_running_function(filenames::Array)
    n = length(filenames)
    p = Progress(n; dt=1.0)   # minimum update interval: 1 second
    for f in filenames
        # Here's where you do all the hard, slow work
        next!(p)
    end
end

For tasks such as reading file data where the progress increment varies between iterations, you can use update!:

using ProgressMeter

function readFileLines(fileName::String)
    file = open(fileName,"r")

    seekend(file)
    fileSize = position(file)

    seekstart(file)
    p = Progress(fileSize; dt=1.0)   # minimum update interval: 1 second
    while !eof(file)
        line = readline(file)
        # Here's where you do all the hard, slow work

        update!(p, position(file))
    end
end

The core methods Progress(), ProgressThresh(), ProgressUnknown(), and their updaters are also thread-safe, so can be used with Threads.@threads, Threads.@spawn etc.:

using ProgressMeter
p = Progress(10)
Threads.@threads for i in 1:10
    sleep(2*rand())
    next!(p)
end
finish!(p)

and the @showprogress macro also works

using ProgressMeter
@showprogress Threads.@threads for i in 1:10
    sleep(2*rand())
end

using ProgressMeter
n = 10
p = Progress(n)
tasks = Vector{Task}(undef, n)
for i in 1:n
    tasks[i] = Threads.@spawn begin
        sleep(2*rand())
        next!(p)
    end
end
wait.(tasks)
finish!(p)

Progress bar style

Optionally, a description string can be specified which will be prepended to the output, and a progress meter M characters long can be shown. E.g.

p = Progress(n; desc="Computing initial pass...")

will yield

Computing initial pass...53%|███████████████████████████                       |  ETA: 0:09:02

in a manner similar to python-progressbar.

Also, other properties can be modified through keywords. The glyphs used in the bar may be specified by passing a BarGlyphs object as the keyword argument barglyphs. The BarGlyphs constructor can either take 5 characters as arguments or a single 5 character string. E.g.

p = Progress(n; dt=0.5, barglyphs=BarGlyphs("[=> ]"), barlen=50, color=:yellow)

will yield

Progress: 53%[==========================>                       ]  ETA: 0:09:02

It is possible to give a vector of characters that acts like a transition between the empty character and the fully filled character. For example, definining the progress bar as:

p = Progress(n; dt=0.5,
             barglyphs=BarGlyphs('|','█', ['▁' ,'▂' ,'▃' ,'▄' ,'▅' ,'▆', '▇'],' ','|',),
             barlen=10)

might show the progress bar as:

Progress:  34%|███▃      |  ETA: 0:00:02

where the last bar is not yet fully filled.

Progress meters for tasks with a target threshold

Some tasks only terminate when some criterion is satisfied, for example to achieve convergence within a specified tolerance. In such circumstances, you can use the ProgressThresh type:

prog = ProgressThresh(1e-5; desc="Minimizing:")
for val in exp10.(range(2, stop=-6, length=20))
    update!(prog, val)
    sleep(0.1)
end

Progress meters for tasks with an unknown number of steps

Some tasks only terminate when some non-deterministic criterion is satisfied. In such circumstances, you can use the ProgressUnknown type:

prog = ProgressUnknown(desc="Titles read:")
for val in ["a" , "b", "c", "d"]
    next!(prog)
    if val == "c"
        finish!(prog)
        break
    end
    sleep(0.1)
end

This will display the number of calls to next! until finish! is called.

If your counter does not monotonically increases, you can also set the counter by hand.

prog = ProgressUnknown(desc="Total length of characters read:")
total_length_characters = 0
for val in ["aaa" , "bb", "c", "d"]
    global total_length_characters += length(val)
    update!(prog, total_length_characters)
    if val == "c"
        finish!(prog)
        break
    end
    sleep(0.5)
end

Alternatively, you can display a "spinning ball" symbol by passing spinner=true to the ProgressUnknown constructor.

prog = ProgressUnknown(desc="Working hard:", spinner=true)
while true
    next!(prog)
    rand(1:2*10^8) == 1 && break
end
finish!(prog)

By default, finish! changes the spinner to a ✓, but you can use a different character by passing a spinner keyword to finish!, e.g. passing spinner='✗' on a failure condition:

let found=false
    prog = ProgressUnknown(desc="Searching for the Answer:", spinner=true)
    for tries in 1:10^8
        next!(prog)
        if rand(1:2*10^8) == 42
            found=true
            break
        end
    end
    finish!(prog, spinner = found ? '✓' : '✗')
end

In fact, you can completely customize the spinner character by passing a string (or array of characters) to animate as a spinner argument to next!:

prog = ProgressUnknown(desc="Burning the midnight oil:", spinner=true)
while true
    next!(prog, spinner="🌑🌒🌓🌔🌕🌖🌗🌘")
    rand(1:10^8) == 0xB00 && break
end
finish!(prog)

(Other interesting-looking spinners include "⌜⌝⌟⌞", "⠋⠙⠹⠸⠼⠴⠦⠧⠇⠏", "🕐🕑🕒🕓🕔🕕🕖🕗🕘🕙🕚🕛", "▖▘▝▗'", and "▁▂▃▄▅▆▇█".)

Printing additional information

You can also print and update information related to the computation by using the showvalues keyword. The following example displays the iteration counter and the value of a dummy variable x below the progress meter:

x,n = 1,10
p = Progress(n)
for iter in 1:10
    x *= 2
    sleep(0.5)
    next!(p; showvalues = [("iteration count",iter), ("x",x)])
end

In the above example, the data passed to showvalues is evaluated even if the progress bar is not updated. To avoid this unnecessary computation and reduce the overhead, you can alternatively pass a zero-argument function as a callback to the showvalues keyword.

x,n = 1,10
p = Progress(n)
generate_showvalues(iter, x) = () -> [("iteration count",iter), ("x",x)]
for iter in 1:10
    x *= 2
    sleep(0.5)
# unlike `showvalues=generate_showvalues(iter, x)()`, this version only evaluate the function when necessary
next!(p; showvalues = generate_showvalues(iter, x))
end

Showing average time per iteration

You can include an average per-iteration duration in your progress meter by setting the optional keyword argument showspeed=true when constructing a Progress, ProgressUnknown, or ProgressThresh.

x,n = 1,10
p = Progress(n; showspeed=true)
for iter in 1:10
    x *= 2
    sleep(0.5)
    next!(p; showvalues = [(:iter,iter), (:x,x)])
end

will yield something like:

Progress:  XX%|███████████████████████████           |  ETA: XX:YY:ZZ (12.34  s/it)

instead of

Progress:  XX%|███████████████████████████                         |  ETA: XX:YY:ZZ

Conditionally disabling a progress meter

In addition to the showspeed optional keyword argument, all the progress meters also support the optional enabled keyword argument. You can use this to conditionally disable a progress bar in cases where you want less verbose output or are using another progress bar to track progress in looping over a function that itself uses a progress bar.

function my_awesome_slow_loop(n::Integer; show_progress=true)
    p = Progress(n; enabled=show_progress)
    for i in 1:n
        sleep(0.1)
        next!(p)
    end
end

const SHOW_PROGRESS_BARS = parse(Bool, get(ENV, "PROGRESS_BARS", "true"))

m = 100
# let environment variable disable outer loop progress bar
p = Progress(m; enabled=SHOW_PROGRESS_BARS)
for i in 1:m
    # disable inner loop progress bar since we are tracking progress in the outer loop
    my_awesome_slow_loop(i; show_progress=false)
    next!(p)
end

ProgressMeter with additional information in Jupyter

Jupyter notebooks/lab does not allow one to overwrite only parts of the output of cell. In releases up through 1.2, progress bars are printed repeatedly to the output. Starting with release xx, by default Jupyter clears the output of a cell, but this will remove all output from the cell. You can restore previous behavior by calling ProgressMeter.ijulia_behavior(:append). You can enable it again by calling ProgressMeter.ijulia_behavior(:clear), which will also disable the warning message.

Tips for parallel programming

For re