PyHum

---

May 2020

This is now legacy code, written in python 2.7 and not actively worked on or supported. If you improve the code, please submit a pull request and I will incorporate

I will no longer respond to issues or emails concerning this project, but if you find it useful, I am happy about that

---

PyHum

A Python framework for reading and processing data from a Humminbird low-cost sidescan sonar

Project website here for more details

PyHum is an open-source project dedicated to provide a generic Python framework for reading and exporting data from Humminbird(R) instruments, carrying out rudimentary radiometric corrections to the data, classify bed texture, and produce some maps on aerial photos and kml files for google-earth

1. read Humminbird DAT and associated SON files
2. export data
3. carry out rudimentary radiometric corrections to data, and
4. classify bed texture using the algorithm detailed in Buscombe, Grams, Smith, (2015) "Automated riverbed sediment classification using low-cost sidescan sonar", Journal of Hydraulic Engineering, 10.1061/(ASCE)HY.1943-7900.0001079, 06015019.
5. produce some maps on GeoTIFF and kml files

The software is designed to read Humminbird data (.SON, .IDX, and .DAT files) and works on both sidescan and downward-looking echosounder data, where available.

Please cite!

If you use PyHum in your published work, please cite the following papers:

1. Buscombe, D., Grams, P.E., and Smith, S. (2015) "Automated riverbed sediment classification using low-cost sidescan sonar", Journal of Hydraulic Engineering, 10.1061/(ASCE)HY.1943-7900.0001079, 06015019.

2. Buscombe, D., 2017, Shallow water benthic imaging and substrate characterization using recreational-grade sidescan-sonar. ENVIRONMENTAL MODELLING & SOFTWARE 89, 1-18.

Contributing & Credits

Primary Developer | Daniel Buscombe ------ | --------------- | Northern Arizona University | Flagstaff, AZ 86011 | daniel.buscombe@nau.edu

Co-Developer | Daniel Hamill ------ | --------------- | Department of Watershed Sciences | Utah State University | Logan, UT 84322 | hamill.daniel@gmail.com

Version: 1.4.5 | Revision: Jan, 2018

Thanks to the following individuals for debugging, supplying data and thoughtful suggestions:

Adam Kaeser
Cam Bodine
Amberle Jones
Phil Whittle
Jacob Berninger
Doug Newcomb
Jereme Gaeta
Ian Nesbit
Allen Aven
Liam Zarri
Daphne Tuzlak
Rick Debbout
Vikram Umnithan
Matt Marineau
Kathryn Ford

Please Read

1. This software has been tested with Python 2.7 on Linux Fedora 16 & 20, Ubuntu 12 -- 17, Windows 7. Python 3 is not yet supported

2. This software has (so far) been used only with Humminbird 700, 800, 900, 1100, HELIX, MEGA and ONIX series instruments.

3. PyHum is not yet tested with SOLIX, ICE, ION, PMAX systems. Please make example data available and we'll see what we can do!

Contents

The programs in this package are as follows:

1. read script to read Humminbird DAT and associated SON files, export data, and produce some rudimentary plots

2. correct script to read Humminbird data (output from 'read') and perform some radiometric corrections and produce some rudimentary plots

3. rmshadows read output 'correct', and remove dark shadows in scans caused by shallows, shorelines, and attenuation of acoustics with distance

4. texture script to read radiometrically corrected Humminbird data in MAT format (output from pyhum_correct.py) and perform a textural analysis using the spectral method of Buscombe et al (forthcoming) and produce some rudimentary plots

5. map script to generate a point cloud (X,Y,sidescan intensity), save it to ascii format file, grid it and make a raster overlay on an aerial image (pulled automatically from the ESRI GIS image server), and a kml file for showing the same thing in google-earth

6. map_texture script to generate a point cloud (X,Y,texture lengthscale - calculated using pyhum_texture), save it to ascii format file, grid it and make a raster overlay on an aerial image (pulled automatically from the ESRI GIS image server), and a kml file for showing the same thing in google-earth

7. e1e2 script to analyse the first (e1, 'roughness') and second (e2, 'hardness') echo returns from the high-frequency downward looking echosounder, and generate generalised acoustic parameters for the purposes of point classification of submerged substrates/vegetation. The processing accounts for the absorption of sound in water, and does a basic k-means cluster of e1 and e2 coefficients into specified number of 'acoustic classes'. This code is based on code by Barb Fagetter (blueseas@oceanecology.ca). Georeferenced parameters are saved in csv form, and optionally plots and kml files are generated

8. gui A graphical user interface which essentially serves as a 'wrapper' to the above functions, allowing graphical input of processing options and sequential analysis of the data using PyHum modules

These are all command-line programs which take a number of input (some required, some optional). Please see the individual files for a comprehensive list of input options

<!----> <!--*Automated bed sediment measurement, Colorado River in Glen Canyon*-->

Setup

PyHum only works in python 2.X. Python 3 is not yet supported.

Installing in a conda virtual env (recommended)

In a conda (miniconda/anaconda) python 2 environment:

Linux:

conda create --name pyhum python=2
source activate pyhum
conda install gdal
conda install -c conda-forge basemap-data-hires -y
conda install scipy numpy scikit-image
pip install simplekml sklearn pandas dask
pip install joblib toolz cython
conda install -c conda-forge pyresample -y ### or if that fails: pip install pyresample
pip install git+https://github.com/dbuscombe-usgs/PyHum.git --no-deps
python -c"import PyHum;PyHum.dotest()" 
source deactivate pyhum 

Windows:

conda create --name pyhum python=2
activate pyhum
conda install gdal
conda install -c conda-forge basemap-data-hires -y
conda install scipy numpy scikit-image
pip install simplekml sklearn pandas dask
pip install joblib toolz cython
conda install -c conda-forge pyresample -y ### or if that fails: pip install pyresample==1.1.4
conda install numpy==1.12.0
pip install git+https://github.com/dbuscombe-usgs/PyHum.git --no-deps
pip install matplotlib --upgrade

Then run the test, and finally deactivate the venv ::

python -c"import PyHum;PyHum.dotest()" 
deactivate pyhum

If you get gdal/osgeo/ogr/os errors, install GDAL (Windows only)::

1. Go to: https://www.lfd.uci.edu/~gohlke/pythonlibs/#gdal
2. Download GDAL‑2.2.3‑cp27‑cp27m‑win_amd64.whl
3. install using pip:

pip install GDAL‑2.2.3‑cp27‑cp27m‑win_amd64.whl

Installing as a library accessible outside of virtual env

Prerequisite

pip install Cython

1. From PyPI::

pip install PyHum

2. the latest 'bleeding edge' (pre-release) version directly from github::

pip install git+https://github.com/dbuscombe-usgs/PyHum.git

(Windows users) install git from here: https://git-scm.com/download/win

3. from github repo clone::

git clone --depth 1 git@github.com:dbuscombe-usgs/PyHum.git
cd PyHum
python setup.py install

or a local installation:

python setup.py install --user

4. linux users, using a virtual environment:

virtualenv venv
source venv/bin/activate
pip install numpy
pip install Cython
pip install basemap --allow-external basemap --allow-unverified basemap
pip install PyHum --no-deps
python -c "import PyHum; PyHum.test.dotest()"
deactivate (or source venv/bin/deactivate)

The results will live in "venv/lib/python2.7/site-packages/PyHum". Note for Fedora linux users: you need the geos-devel package for basemap, and the blas and libpack libraries for scipy

Running the test

A test can be carried out by running the supplied script. From the command line (terminal)::

python -c"import PyHum;PyHum.dotest()" 

or (if python3 is your default python)::

python2 -c"import PyHum;PyHum.dotest()" 

Using the GUI

From the command line (terminal)::

python -c "import PyHum; PyHum.gui()"

Using PyHum

Inputs to the program are a .DAT file (e.g. R0089.DAT) and a folder of .SON and .IDX files (e.g. /my/folder/R0089). The program will read the .SON files with or without the accompanying .IDX files, but will be faster if the .IDX files are present.

PyHum is modular so can be called from within a python or ipython console, from an IDE (such as IDLE or Spyder), or by running a script.

The following example script could be saved as, for example "proc_mysidescandata.py" and run from the command line using

python proc_mysidescandata.py -i C:\MyData\R0087.DAT -s C:\MyData\R0087

import sys, getopt

from Tkinter import Tk
from tkFileDialog import askopenfilename, askdirectory

import PyHum
import os

if __name__ == '__main__': 

    argv = sys.argv[1:]
    humfile = ''; sonpath = ''
    
    # parse inputs to variables
    try:
       opts, args = getopt.getopt(argv,"hi:s:")
    except getopt.GetoptError:
         print 'error'
         sys.exit(2)
    for opt, arg in opts:
       if opt == '-h':
         print 'help'
         sys.exit()
       elif opt in ("-i"):
          humfile = arg
       elif opt in ("-s"):
          sonpath = arg

    # prompt user to supply file if no input file given
    if not humfile:
       print 'An input file is required!!!!!!'
       Tk().withdraw() # we don't want a full GUI, so keep the root win