restapi

This is a Python API for working with ArcGIS REST API, ArcGIS Online, and Portal/ArcGIS Enterprise. This package has been designed to work with arcpy when available, or the included open source module pyshp. It will try to use arcpy if available for some data conversions, otherwise will use open source options. Also included is a subpackage for administering ArcGIS Server Sites. This is updated often, so continue checking here for new functionality!

Why would you use this package?

Esri currently provides the ArcGIS API for Python which provides complete bindings to the ArcGIS REST API. This package has less coverage of the REST API, but has many convience functions not available in the ArcGIS API for Python and has a strong focus on downloading and querying data. This package will also support older versions of Python (i.e. 2.7.x) whereas Esri's package only supports 3.x.

Release History

Release History

Installation

restapi is supported on Python 2.7 and 3.x. It can be found on Github and PyPi. To install using pip:

pip install bmi-arcgis-restapi

After installation, it should be available to use in Python:

import restapi

A note about arcpy

By default, restapi will import Esri's arcpy module if available. However, this module is not required to use this package. arcpy is only used when available to write data to disk in esri specific formats (file geodatabase, etc) and working with arcpy Geometries. When arcpy is not availalbe, the pyshp module is used to write data (shapefile format only) and work with shapefile.Shape objects (geometry). Also worth noting is that open source version is much faster than using arcpy.

That being said, there may be times when you want to force restapi to use the open source version, even when you have access to arcpy. Some example scenarios being when you don't need to write any data in an Esri specific format, you want the script to execute very fast, or you are working in an environment where arcpy may not play very nicely (Flask, Django, etc.). To force restapi to use the open source version, you can simply create an environment variable called RESTAPI_USE_ARCPY and set it to FALSE or 0. This variable will be checked before attempting to import arcpy.

Here is an example on how to force open source at runtime:

import os
os.environ['RESTAPI_USE_ARCPY'] = 'FALSE'

# now import restapi
import restapi

requests.exceptions.SSLError

If you are seeing requests.exceptions.SSLError exceptions in restapi >= 2.0, this is probaly due to a change in handling servers without valid SSL certificates. Because many ArcGIS Server instances are accessed using SSL with a self-signed certificate, or through a MITM proxy like Fiddler, restapi < 2.0 defaulted to ignoring SSL errors by setting the request client's verify option to False. The new default behavior is to enable certificate verification. If you are receiving this error, you are probably accessing your server with a self-signed certificate, or through a MITM proxy. If that is not the case, you should investigate why you are seeing SSL errors, as there would likely be an issue with the server configuration, or some other security issues.

To mimic the previous behavior in the newer versions of restapi, there are 2 options - disable certificate verification (less secure), or build a custom CA bundle which includes any self-signed certificates needed to access your server (more secure). Both of these can be done using the new restapi.RequestClient() feature.

import restapi
import requests
session = requests.Session()
client = restapi.RequestClient(session)
restapi.set_request_client(client)

# Disable verification
client.session.verify = False

# -or-

# Use custom CA bundle
client.session.verify = '/path/to/certfile'

Since verify = False is a commonly used setting when dealing with ArcGIS Server instances, it's also possible to use an environment variable. The variable must be set before restapi is imported.

os.environ['RESTAPI_VERIFY_CERT'] = 'FALSE'
import restapi

Connecting to an ArcGIS Server

samples for this section can be found in the connecting_to_arcgis_server.py file.

One of the first things you might do is to connect to a services directory (or catalog):

import restapi

# connect to esri's sample server 6
rest_url = 'https://sampleserver6.arcgisonline.com/arcgis/rest/services'

# connect to restapi.ArcServer instance 
ags = restapi.ArcServer(rest_url)

>>> # get folder and service properties
>>> print('Number of folders: {}'.format(len(ags.folders)))
>>> print('Number of services: {}\n'.format(len(ags.services)))
>>> 
>>> # walk thru directories
>>> for root, services in ags.walk():
>>>     print('Folder: {}'.format(root))
>>>     print('Services: {}\n'.format(services))

Number of folders: 13
Number of services: 60

Folder: None
Services: ['911CallsHotspot/GPServer', '911CallsHotspot/MapServer', 'Census/MapServer', 'CharlotteLAS/ImageServer', 'CommercialDamageAssessment/FeatureServer', 'CommercialDamageAssessment/MapServer', 'CommunityAddressing/FeatureServer', 'CommunityAddressing/MapServer', '<...>', 'Water_Network/MapServer', 'Wildfire/FeatureServer', 'Wildfire/MapServer', 'WindTurbines/MapServer', 'World_Street_Map/MapServer', 'WorldTimeZones/MapServer']

Folder: AGP
Services: ['AGP/Census/MapServer', 'AGP/Hurricanes/MapServer', 'AGP/USA/MapServer', 'AGP/WindTurbines/MapServer']

Folder: Elevation
Services: ['Elevation/earthquakedemoelevation/ImageServer', 'Elevation/ESRI_Elevation_World/GPServer', 'Elevation/GlacierBay/MapServer', 'Elevation/MtBaldy_Elevation/ImageServer', 'Elevation/WorldElevations/MapServer']

# ..etc

Connecting to a child service

from an ArcServer instance, you can connect to any service within that instance by providing a child path or wildcard search match.

# connect to a specific service
# using just the service name (at the root)
usa = ags.getService('USA') #/USA/MapServer -> restapi.common_types.MapService

# using the relative path to a service in a folder
census = ags.getService('AGP/Census') #/AGP/Census/MapServer -> restapi.common_types.MapService

# can also just use the service name, but be weary of possible duplicates
infastructure = ags.getService('Infrastructure') #/Energy/Infrastructure/FeatureServer -> restapi.common_types.FeatureService

# using a wildcard search
covid_cases = ags.getService('*Covid19Cases*') #/NYTimes_Covid19Cases_USCounties/MapServer -> restapi.common_types.MapService

check outputs:

>>> for service in [usa, census, infastructure, covid_cases]:
>>>     print('name: "{}"'.format(service.name))
>>>     print('repr: "{}"'.format(repr(service)))
>>>     print('url: {}\n'.format(service.url))

name: "USA"
repr: "<MapService: USA/MapServer>"
url: https://sampleserver6.arcgisonline.com/arcgis/rest/services/USA/MapServer

name: "Census"
repr: "<MapService: AGP/Census/MapServer>"
url: https://sampleserver6.arcgisonline.com/arcgis/rest/services/AGP/Census/MapServer

name: "Infrastructure"
repr: "<FeatureService: Energy/Infrastructure/FeatureServer>"
url: https://sampleserver6.arcgisonline.com/arcgis/rest/services/Energy/Infrastructure/FeatureServer

name: "NYTimes_Covid19Cases_USCounties"
repr: "<MapService: NYTimes_Covid19Cases_USCounties/MapServer>"
url: https://sampleserver6.arcgisonline.com/arcgis/rest/services/NYTimes_Covid19Cases_USCounties/MapServer

Working with layers

samples for this section can be found in the working_with_layers.py file.

You can connect to MapServiceLayers or FeatureLayers directly by passing the url, or by accessing from the parent FeatureService or MapService.

also query the layer and get back arcpy.da Cursor like access

# get access to the "Cities" layer from USA Map Service
cities = usa.layer('Cities') # or can use layer id: usa.layer(0)

# query the map layer for all cities in California with population > 100000
where = "st = 'CA' and pop2000 > 100000"

# the query operation returns a restapi.FeatureSet or restapi.FeatureCollection depending on the return format
featureSet = cities.query(where=where)

# get result count, can also use len(featureSet)
print('Found {} cities in California with Population > 100K'.format(featureSet.count))

# print first feature (restapi.Feature).  The __str__ method is pretty printed JSON
# can use an index to fetch via its __getitem__ method
print(featureSet[0])

exceeding the maxRecordCount

In many cases, you may run into issues due to the maxRecordCount set for a service, which by default is 1000 features. This limit can be exceeded by using the exceed_limit parameter. If exceed_limit is set to True, the query_in_chunks method will be called internally to keep fetching until all queried features are returned.

# fetch first 1000
first1000 = cities.query()
print('count without exceed_limit: {}'.format(first1000.count)) # can also use len()

# fetch all by exceeding limit
allCities = cities.query(exceed_limit=True)
print('count without exceed_limit: {}'.format(len(allCities)))