Package for working with ArcGIS REST API
Project description
restapi
This is a Python API for working with ArcGIS REST API, ArcGIS Online, and Portal/ArcGIS Enterprise. This package has been designed to work with arcpy when available, or the included open source module pyshp. It will try to use arcpy if available for some data conversions, otherwise will use open source options. Also included is a subpackage for administering ArcGIS Server Sites. This is updated often, so continue checking here for new functionality!
Why would you use this package?
Esri currently provides the ArcGIS API for Python which provides complete bindings to the ArcGIS REST API. This package has less coverage of the REST API, but has many convience functions not available in the ArcGIS API for Python. This package will also support older versions of Python (i.e. 2.7.x) whereas Esri's package only supports 3.x.
Release History
Installation
restapi
is supported on Python 2.7 and 3.x. It can be found on Github and PyPi. To install using pip:
pip install bmi-arcgis-restapi
After installation, it should be available to use in Python:
import restapi
A note about arcpy
By default, restapi
will import Esri's arcpy
module if available. However, this module is not required to use this package. arcpy
is only used when available to write data to disk in esri specific formats (file geodatabase, etc) and working with arcpy
Geometries. When arcpy
is not availalbe, the pyshp module is used to write data (shapefile format only) and work with shapefile.Shape
objects (geometry). Also worth noting is that open source version is much faster than using arcpy
.
That being said, there may be times when you want to force restapi
to use the open source version, even when you have access to arcpy
. Some example scenarios being when you don't need to write any data in an Esri specific format, you want the script to execute very fast, or you are working in an environment where arcpy
may not play very nicely (Flask, Django, etc.). To force restapi
to use the open source version, you can simply create an environment variable called RESTAPI_USE_ARCPY
and set it to FALSE
or 0
. This variable will be checked before attempting to import arcpy
.
Here is an example on how to force open source at runtime:
import os os.environ['RESTAPI_USE_ARCPY'] = 'FALSE' # now import restapi import restapi
requests.exceptions.SSLError
If you are seeing requests.exceptions.SSLError
exceptions in restapi
>= 2.0, this is probaly due to a change in handling servers without valid SSL certificates. Because many ArcGIS Server instances are accessed using SSL with a self-signed certificate, or through a MITM proxy like Fiddler, restapi
< 2.0 defaulted to ignoring SSL errors by setting the request
client's verify
option to False
. The new default behavior is to enable certificate verification. If you are receiving this error, you are probably accessing your server with a self-signed certificate, or through a MITM proxy. If that is not the case, you should investigate why you are seeing SSL errors, as there would likely be an issue with the server configuration, or some other security issues.
To mimic the previous behavior in the newer versions of restapi
, there are 2 options - disable certificate verification (less secure), or build a custom CA bundle which includes any self-signed certificates needed to access your server (more secure). Both of these can be done using the new restapi.RequestClient() feature.
import restapi import requests session = requests.Session() client = restapi.RequestClient(session) restapi.set_request_client(client) # Disable verification client.session.verify = False # -or- # Use custom CA bundle client.session.verify = '/path/to/certfile'
Since verify = False
is a commonly used setting when dealing with ArcGIS Server instances, it's also possible to use an environment variable. The variable must be set before restapi
is imported.
os.environ['RESTAPI_VERIFY_CERT'] = 'FALSE' import restapi
Connecting to an ArcGIS Server
One of the first things you might do is to connect to a services directory (or catalog):
Connect to external services
# connect NOAA ArcGIS Server Instance rest_url = 'https://gis.ngdc.noaa.gov/arcgis/rest/services' # no authentication is required, so no username and password are supplied ags = restapi.ArcServer(rest_url) # get folder and service properties print('Number of folders: {}'.format(len(ags.folders))) print('Number of services: {}'.format(len(ags.services))) # walk thru directories for root, folders, services in ags.walk(): print(root) print(folders) print(services) print('\n')
Connecting to a map service from within the ArcServer object
# access "ahps_gauges" service (stream gauges) gauges = ags.getService('ahps_gauges') print(gauges.url) #print(MapService url # print(layer names print(gauges.list_layers()) # access "observed river stages" layer lyr = gauges.layer('observed_river_stages') #not case sensitive, also supports wildcard search (*) # list fields from col layer print(lyr.list_fields())
You can also query the layer and get back arcpy.da Cursor like access
# run search cursor for gauges in California # (maximimum limit may be 1000 records, can use get_all=True to exceed transfer limit) # can filter fields by putting a field list, can use actual shape field name to get # geometry or use the ArcGIS-like token "SHAPE@" # all fields are gathered by the default ("*") and fields can be filtered by providing a list query = "state = 'CA'" for row in lyr.cursor(where=query, fields=['SHAPE@', u'gaugelid', u'status', u'location']): print(row) # Note: can also do this from the MapService level like this: # cursor = gauges.cursor('observed_river_stages', where=query)
The layer can also be exported to a shapefile or KMZ
# export Nebraska "College/University" layer to feature class # make scratch folder first folder = os.path.join(os.environ['USERPROFILE'], r'Desktop\restapi_test_data') if not os.path.exists(folder): os.makedirs(folder) # export layer to shapefile (can also call from Map Service) output = os.path.join(folder, 'California_Stream_Gauges.shp') lyr.layer_to_fc(output, where=query, sr=102100) #override spatial reference with web mercator # export to KMZ kmz = output.replace('.shp', '.kmz') lyr.layer_to_kmz(kmz, where=query)
Clipping a layer is also easy
# clip lyr by polygon (Sacramento area) esri_json = {"rings":[[[-121.5,38.6],[-121.4,38.6], [-121.3,38.6],[-121.2,38.6], [-121.2,38.3],[-121.5,38.3], [-121.5,38.6]]], "spatialReference": {"wkid":4326,"latestWkid":4326}} # clip by polygon and filter fields (can use polygon shapefile or feature class as well) sac = os.path.join(folder, 'Sacramento_gauges.shp') lyr.clip(esri_json, sac, fields=['gaugelid', 'location'])
You can also connect to a MapService directly
url = 'http://gis.srh.noaa.gov/arcgis/rest/services/ahps_gauges/MapServer' gauges = restapi.MapService(url)
Working with Feature Layers
query examples
# create FeatureLayer url = 'https://services.arcgis.com/V6ZHFr6zdgNZuVG0/arcgis/rest/services/Hazards_Uptown_Charlotte/FeatureServer/0' hazards = restapi.FeatureLayer(url) # QUERY EXAMPLES # query all features, to fetch all regardless of `maxRecordCount` # use `exceed_limit=true` keyword arg fs = hazards.query() print('All Hazards Count: {}'.format(fs.count)) # query features that are "High" Priority high_priority = hazards.query(where="Priority = 'High'") print('High Priority Hazards count: {}'.format(high_priority.count))
download features
# download features - choosing a geodatbase output will bring over domain # info (when you have access to arcpy), whereas a shapefile output will # just bring over the domain values shp = os.path.join(test_data_folder, 'hazards.shp') # export layer to shapefile in WGS 1984 projection hazards.export_layer(shp, outSR=4326)
feature editing
add features using FeatureLayer.addFeatures()
# add new records via FeatureLayer.addFeatures() desc = "restapi edit test" new_ft = { "attributes": { "HazardType": "Flooding", "Description": desc, "SpecialInstructions": None, "Status": "Active", "GlobalID": "416f04e5-0ae9-4444-8d0c-d4e9b44e7f87", "Priority": "Moderate" }, "geometry": create_random_coordinates() } # add new feature results = hazards.addFeatures([new_ft]) print(results)
using restapi
cursors
restapi
also supports cursors similar to what you get when using arcpy
. However, these work directly with the REST API and JSON features while also supporting arcpy
and shapefile
geometry types. See the below example on how to use an insertCursor
to add new records:
# add 3 new features using an insert cursor # using this in a "with" statement will call applyEdits on __exit__ fields = ["SHAPE@", 'HazardType', "Description", "Priority"] with hazards.insertCursor(fields) as irows: for i in range(3): irows.insertRow([create_random_coordinates(), "Wire Down", desc, "High"])
records can be updated with an updateCursor
and a where clause. Note that the OBJECTID
field must be included in the query to indicate which records will be updated. The OID@
field token can be used to retreive the objectIdFieldName
:
# now update records with updateCursor whereClause = "Description = '{}'".format(desc) with hazards.updateCursor(["Priority", "OID@"], where=whereClause) as rows: for row in rows: row[0] = "Low" rows.updateRow(row)
Deleting features can be done with a simple where
clause:
# now delete the records we added hazards.deleteFeatures(where=whereClause)
We can also add attachments
# add attachment, get new OID from add results oid = result.addResults[0] # must get an OID to add attachment to # download python image online and add it to the featuer we just added above url = 'http://www.cis.upenn.edu/~lhuang3/cse399-python/images/pslytherin.png' im = urllib.urlopen(url).read() tmp = os.path.join(os.path.dirname(sys.argv[0]), 'python.png') with open(tmp, 'wb') as f: f.write(im) # add attachment incidents.addAttachment(oid, tmp) os.remove(tmp) # get attachment info from service and download it attachments = incidents.attachments(oid) for attachment in attachments: print(attachment) print attachment.contentType, attachment.size) attachment.download(folder) # folder is a user specified output directory
Update feature and delete features
# update the feature we just added adds[0]['attributes']['address'] = 'Address Not Available' adds[0]['attributes']['objectid'] = oid incidents.updateFeatures(adds) # now delete feature incidents.deleteFeatures(oid)
Offline capabilities (Sync)
# if sync were enabled, we could create a replica like this: # can pass in layer ID (0) or name ('incidents', not case sensative) replica = fs.createReplica(0, 'test_replica', geometry=adds[0]['geometry'], geometryType='esriGeometryPoint', inSR=4326) # now export the replica object to file geodatabase (if arcpy access) or shapefile with hyperlinks (if open source) restapi.exportReplica(replica, folder)
Working with Image Services
url = 'http://pca-gis02.pca.state.mn.us/arcgis/rest/services/Elevation/DEM_1m/ImageServer' im = restapi.ImageService(url) # clip DEM geometry = {"rings":[[ [240006.00808044084, 4954874.19629429], [240157.31010183255, 4954868.8053006204], [240154.85966611796, 4954800.0316874133], [240003.55764305394, 4954805.4226145679], [240006.00808044084, 4954874.19629429]]], "spatialReference":{"wkid":26915,"latestWkid":26915}} tif = os.path.join(folder, 'dem.tif') im.clip(geometry, tif) # test point identify x, y = 400994.780878, 157878.398217 elevation = im.pointIdentify(x=x, y=y, sr=103793) print(elevation)
Geocoding
# hennepin county, MN geocoder henn = 'http://gis.hennepin.us/arcgis/rest/services/Locators/HC_COMPOSITE/GeocodeServer' geocoder = restapi.Geocoder(henn) # find target field, use the SingleLine address field by default geoResult = geocoder.findAddressCandidates('353 N 5th St, Minneapolis, MN 55403') # export results to shapefile print('found {} candidates'.format(len(geoResult)) geocoder.exportResults(geoResult, os.path.join(folder, 'target_field.shp')) # Esri geocoder esri_url = 'http://sampleserver1.arcgisonline.com/ArcGIS/rest/services/Locators/ESRI_Geocode_USA/GeocodeServer' esri_geocoder = restapi.Geocoder(esri_url) # find candidates using key word arguments (**kwargs) to fill in locator fields, no single line option candidates = esri_geocoder.findAddressCandidates(Address='380 New York Street', City='Redlands', State='CA', Zip='92373') print('Number of address candidates: {}'.format(len(candidates))) for candidate in candidates: print(candidate.location) # export results to shapefile out_shp = os.path.join(folder, 'Esri_headquarters.shp') geocoder.exportResults(candidates, out_shp)
Geoprocessing Services
# test esri's drive time analysis GP Task gp_url = 'http://sampleserver1.arcgisonline.com/ArcGIS/rest/services/Network/ESRI_DriveTime_US/GPServer/CreateDriveTimePolygons' gp = restapi.GPTask(gp_url) # get a list of gp parameters (so we know what to pass in as kwargs) print('\nGP Task "{}" parameters:\n'.format(gp.name) for p in gp.parameters: print('\t', p.name, p.dataType) point = {"geometryType":"esriGeometryPoint", "features":[ {"geometry":{"x":-10603050.16225853,"y":4715351.1473399615, "spatialReference":{"wkid":102100,"latestWkid":3857}}}], "sr":{"wkid":102100,"latestWkid":3857}} # run task, passing in gp parameters as keyword arguments (**kwargs) gp_res = gp.run(Input_Location=str(point), Drive_Times = '1 2 3', inSR = 102100) # returns a GPResult() object, can get at the first result by indexing (usually only one result) # can test if there are results by __nonzero__() if gp_res: result = gp_res.results[0] # this returned a GPFeatureRecordSetLayer as an outputParameter, so we can export this to polygons print('\nOutput Result: "{}", data type: {}\n'.format(result.paramName, result.dataType)) # now export the result value to fc (use the value property of the GPResult object from run()) drive_times = os.path.join(folder, 'drive_times.shp') restapi.exportFeatureSet(drive_times, gp_res.value)
A note about input Geometries
restapi will try to use arcpy first if you have it, otherwise will defer to open source. Both support the reading of shapefiles to return the first feature back as a restapi.Geometry object
It also supports arcpy Geometries and shapefile.Shape() objects
>>> shp = r'C:\TEMP\Polygons.shp' # a shapefile on disk somewhere >>> geom = restapi.Geometry(shp) >>> print(geom.envelope()) -121.5,38.3000000007,-121.199999999,38.6000000015
Token Based Security
restapi also supports secured services. This is also session based, so if you sign in once to an ArcGIS Server Resource (on the same ArcGIS Site), the token will automatically persist via the IdentityManager().
There are 3 ways to authticate:
# **kwargs for all accessing all ArcGIS resources are # usr -- username # pw -- password # token -- token (as string or restapi.Token object) # proxy -- url to proxy # secured url secured_url = 'http://some-domain.com/arcgis/rest/services' # 1. username and password ags = restapi.ArcServer(url, 'username', 'password') # token is generated and persists # 2. a token that has already been requested ags = restapi.ArcServer(url, token=token) # uses a token that is already active # 3. via a proxy (assuming using the standard esri proxy) # this will forward all subsequent requests through the proxy ags = restapi.ArcServer(url, proxy='http://some-domain.com/proxy.ashx')
You can even just generate a token and let the IdentityManager handle the rest. It is even smart enough to handle multiple tokens for different sites:
# login to instance 1 usr = 'username' pw = 'password' # urls to two different ArcGIS Server sites url_1 = 'http://some-domain.com/arcserver1/rest/services' url_2 = 'http://domain2.com/arcgis/rest/services' # generate tokens tok1 = restapi.generate_token(url_1, usr, pw) tok2 = restapi.generate_token(url_2, usr, pw) # now we should be able to access both ArcGIS Server sites via the IdentityManager arcserver1 = restapi.ArcServer(url_1) # tok1 is automatically passed in and handled arcserver2 = restapi.ArcServer(url_2) # tok2 is used here
The admin Subpackage
restapi also contains an administrative subpackage (warning: most functionality has not been tested!). You can import this module like this:
from restapi import admin
Connecting to a Portal
url = 'https://domain.gis.com/portal/home' portal = admin.Portal(url, 'username', 'password') # get servers servers = portal.servers # stop sample cities service server = servers[0] service = server.service('SampleWorldCities.MapServer') service.stop()
To connect to an ArcGIS Server instance that you would like to administer you can do the following:
# test with your own servers url = 'localhost:6080/arcgis/admin/services' #server url usr = 'username' pw = 'password' # connect to ArcGIS Server instance arcserver = admin.ArcServerAdmin(url, usr, pw)
To list services within a folder, you can do this:
folder = arcserver.folder('SomeFolder') # supply name of folder as argument for service in folder.iter_services(): print(service.serviceName, service.configuredState # can stop a service like this # service.stop() # or start like this # service.start() print('\n' * 3) # show all services and configured state (use iter_services to return restapi.admin.Service() object!) for service in arcserver.iter_services(): print(service.serviceName, service.configuredState)
Security
You can set security at the folder or service level. By default, the addPermssion() method used by Folder and Service objects will make the service unavailable to the general public and only those in the administrator role can view the services. This is done by setting the 'esriEveryone' principal "isAllowed" value to false. You can also assign permissions based on roles.
arcserver.addPermission('SomeFolder') # by default it will make private True # now make it publically avaiable (unsecure) arcserver.addPermission('SomeFolder', private=False) # secure based on role, in this case will not allow assessor group to see utility data # assessor is name of assessor group role, Watermain is folder to secure arcserver.addPermission('Watermain', 'assessor', False) # note, this can also be done at the folder level: folder = arcserver.folder('Watermain') folder.addPermission('assessor', False)
Stopping and Starting Services
Services can easily be started and stopped with this module. This can be done from the ArcServerAdmin() or Folder() object:
# stop all services in a folder arcserver.stopServices(folderName='SomeFolder') # this can take a few minutes # look thru the folder to check the configured states, should be stopped for service in arcserver.folder('SomeFolder').iter_services(): print(service.serviceName, service.configuredState) # now restart arcserver.startServices(folderName='SomeFolder') # this can take a few minutes # look thru folder, services should be started for service in arcserver.folder('SomeFolder').iter_services(): print(service.serviceName, service.configuredState) # to do this from a folder, simply get a folder object back folder = arcserver.folder('SomeFolder') folder.stopServices() for service in folder.iter_services(): print(service.serviceName, service.configuredState)
Updating Service Properties
The admin package can be used to update the service definitions via JSON. By default, the Service.edit() method will pass in the original service definition as JSON so no changes are made if no arguments are supplied. The first argument is the service config as JSON, but this method also supports keyword arguments to update single properties (**kwargs). These represent keys of a the dictionary in Python.
# connect to an individual service (by wildcard) - do not need to include full name, just # enough of the name to make it a unique name query service = arcserver.service('SampleWorldCities') #provide name of service here # get original service description description = service.description # now edit the description only by using description kwarg (must match key exactly to update) service.edit(description='This is an updated service description') # edit description again to set it back to the original description service.edit(description=description)
There are also some helper methods that aren't available out of the box from the ArcGIS REST API such as enabling or disabling extensions:
# disable Feature Access and kml downloads service.disableExtensions(['FeatureServer', 'KmlServer']) # you can also list enabled/disabled services print(service.enabledExtensions) # [u'KmlServer', u'WFSServer', u'FeatureServer'] service.disabledExtensions # [u'NAServer', u'MobileServer', u'SchematicsServer', u'WCSServer', u'WMSServer'] # Edit service extension properites # get an extension and view its properties fs_extension = service.getExtension('FeatureServer') print(fs_extension) # will print as pretty json
For Service objects, all properties are represented as pretty json. Below is what the FeatureService Extension looks like:
{ "allowedUploadFileTypes": "", "capabilities": "Query,Create,Update,Delete,Uploads,Editing", "enabled": "true", "maxUploadFileSize": 0, "properties": { "allowGeometryUpdates": "true", "allowOthersToDelete": "false", "allowOthersToQuery": "true", "allowOthersToUpdate": "false", "allowTrueCurvesUpdates": "false", "creatorPresent": "false", "dataInGdb": "true", "datasetInspected": "true", "editorTrackingRespectsDayLightSavingTime": "false", "editorTrackingTimeInUTC": "true", "editorTrackingTimeZoneID": "UTC", "enableOwnershipBasedAccessControl": "false", "enableZDefaults": "false", "maxRecordCount": "1000", "realm": "", "syncEnabled": "false", "syncVersionCreationRule": "versionPerDownloadedMap", "versionedData": "false", "xssPreventionEnabled": "true", "zDefaultValue": "0" }, "typeName": "FeatureServer" }
Setting properties for extensions is also easy:
# set properties for an extension using helper method, use **kwargs for setting capabilities service.setExtensionProperties('FeatureServer', capabilities='Query,Update,Delete,Editing') # verify changes were made print(fs_extension.capabilities # 'Query,Update,Delete,Editing' # alternatively, you can edit the service json directly and call the edit method # change it back to original settings fs_extension.capabilities = 'Query,Create,Update,Delete,Uploads,Editing' service.edit() # verify one more time... print(fs_extension.capabilities) # 'Query,Create,Update,Delete,Uploads,Editing'
Access the Data Store
You can iterate through the data store items easily to read/update/add items:
# connect to the server's data store ds = arcserver.dataStore # iterate through all items of data store for item in ds: print(item.type, item.path # if it is an enterprise database connection, you can get the connection string like this if item.type == 'egdb': print(item.info.connectionString) # else if a folder, print(server path elif item.type == 'folder': print(item.info.path) print('\n')
User and Role Stores
When viewing usernames/roles you can limit the number of names returned using the "maxCount" keyword argument. To view and make changes to Role Store:
# connect to role store rs = arcserver.roleStore # print roles for role in rs: print(role) # find users within roles for role in rs: print(role, 'Users: ', rs.getUsersWithinRole(role)) # add a user to role rs.addUsersToRole('Administrators', 'your-domain\\someuser') # remove user from role rs.removeUsersFromRole('Administrators', 'your-domain\\someuser') # remove an entire role rs.removeRole('transportation')
To view and make changes to the User Store:
# connect to user store us = arcserver.userStore # get number of users print(len(us) # iterate through first 10 users for user in us.searchUsers(maxCount=10): print(user) # add new user us.addUser('your-domain\\someuser', 'password') # assign roles by using comma separated list of role names us.assignRoles('your-domain\\someuser', 'Administrators,Publishers') # get privileges from user us.getPrivilegeForUser('your-domain\\someuser') # remove roles from user us.removeRoles('your-domain\\someuser', 'Administrators,Publishers')
Log Files
You can easily query server log files like this:
import restapi import datetime # query log files (within last 3 days), need to convert to milliseconds threeDaysAgo = restapi.date_to_mil(datetime.datetime.now() - relativedelta(days=3)) for log in arcserver.queryLogs(endTime=threeDaysAgo, pageSize=25): print(log.time for message in log: print(message) print('\n')
A note about verbosity
When using the admin subpackage you will likely be making changes to services/permissions etc. On operations that change a configuration, the @passthrough decorator will report back if the operation is successful and return results like this:
{u'status': u'SUCCESS'}
The printing of these messages can be shut off by changing the global "VERBOSE" variable so these messages are not reported. This can be disabled like this:
admin.VERBOSE = False
Advanced Usage
RequestClient
From version 2.0, it is possible to use a custom requests.Session()
instance. This instance can be defined globally for all requests made by restapi
, or it can be passed on each function call as a restapi.RequestClient()
object. This can be useful if different parameters are needed to access different servers.
Use this functionality to access servers behind HTTP or SOCKS proxies, to disable certificate validation or use custom CA certificates, or if additional authentication is needed. Refer to the requests.Session() documentation for details
# Create a restapi.RequestClient() object. custom_session = requests.Session() custom_client = restapi.RequestClient(custom_session) # Customize the client proxies = { “http”: “http://10.10.10.10:8000”, “https”: “http://10.10.10.10:8000”, } custom_client.session.proxies = proxies custom_client.session.headers['Source-Client'] = 'Custom' # Use the client for an individual call rest_url = 'https://gis.ngdc.noaa.gov/arcgis/rest/services' arcserver = restapi.ArcServer(rest_url, client=custom_client) # Set a different client as client as restapi's default global_session = requests.Session() global_client = restapi.RequestClient(global_session) global_client.session.headers['Source-Client'] = 'Global' restapi.set_request_client(global_client) # Now any call made by restapi will use the custom client arcserver = restapi.ArcServer(rest_url) # The global client can also be accessed directly restapi.requestClient.headers['Another-Header'] = 'Header is here'
Any session objects which extend requests.Session()
should be supported, for example, pypac.PACSession().
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Filename, size | File type | Python version | Upload date | Hashes |
---|---|---|---|---|
Filename, size bmi-arcgis-restapi-2.0.2.tar.gz (486.4 kB) | File type Source | Python version None | Upload date | Hashes View |