Some Coveted Programming Skills

To follow-up on an earlier post I wrote this year about GIS Skills, I recently read a story in Readwrite Enterprise outlining 15 programming skills most coveted by employers (http://readwrite.com/2013/04/09/15-programming-skills-most-coveted-by-employers). While their story was aimed at the development community, two of the 15 skills outlined in their story are definitely needed by GIS Analysts and are skills I regularly see in newer job advertisements:

  • SQL * (the #1 most coveted skill)
  • Python (Skill #10)

If your aim is to become a GIS developer, than the additional other programming skills listed in their article apply, namely:

  • ASP.net (Skill #14) **
  • PHP (Skill #11)
  • C# (Skill #7) ***
  • XML (Skill #6)
  • JavaScript (Skill #4)
  • HTML (Skill #3)

* Also the number one preferred skill I listed in my earlier post. So you’ve heard it from at least two different sources now.

** ASP.net is really a framework that you’d develop in using Visual Basic.net or C#, so I’m not sure it’s really in the same category as the other skills mentioned in their article.

*** While you can certainly develop GIS applications using Visual Basic.net, most GIS developers use C#. So if you’re just learning to program, learn C#. If you’re already well entrenched in Visual Basic from years of experience with prior versions (like me), then keep developing in VB.net unless your lack of C# experience starts to interfere with employment prospects.

You’ll also note that their list does not contain references to Flash/Flex and/or Silverlight. Both Flash and Silverlight were all the rage in web-based GIS application development the last few years, however the rather sudden appearance of mobile devices (with their lack of support for Flash and Silverlight) really killed advancement of those technologies as people moved towards GIS websites that would work in the mobile space. Thus these days HTML5 and JavaScript are the preferred development languages for web-based GIS, as these technologies are supported on the major mobile operating systems.

Advertisements

A Maxent Script Tool for ArcGIS

As part of my PhD research at the University of Arizona where I study biogeography, biodiversity and macroecology, I have been part of a group looking at large-scale biodiversity questions for New World plants. In this role, I have been responsible for generating many species distribution models using Stephen Phillip’s Maxent software (and R with the Dismo package). I also use ArcGIS quite often to prepare environmental data for Maxent modeling, and wanted a way to perform and integrate Maxent modeling into my ArcGIS workflows.

Maxent Tool User InterfaceWhen working with data and analysis, my rule is that if I have to do something complicated more than twice, it’s generally worth it to develop an automated workflow. So for my species distribution modeling needs, I wanted a Maxent script tool that I could use as a standalone geoprocessing tool or embed within a ModelBuilder model for use more complicated workflows. For example, one of my most common workflow is iterating through a list of species and running a species distribution model for each one.

To accomplish this, I wrote an ArcGIS script tool in Python that collects a variety of common Maxent parameters in a typical ArcGIS geoprocessing user interface and populates a command string to execute Stephen Phillip’s maxent.jar Java application via a Python call to execute a system command. The resulting script tool uses the arcpy model, so it works like any other ArcGIS geoprocessing script tool. It can operate as a standalone tool to generate a species distribution model for a single species, or can be placed into a ModelBuilder workflow and linked to input parameters as shown in the example below.

Maxent Tool in a ModelBuilder Model

In this example, the script tool is parameterized with a species occurrence dataset obtained from the iterator object in which each dataset is named genus_species.csv. The tool then generates a species distribution model for each species and saves the model to a separate directory within a designated output directory, naming each directory by the genus and species name specified in the name of the input occurrence dataset.

A partial example of the Python script that obtains the parameters from the input datasets is shown below.

Maxent Tool Python Script

These parameters are then appended to a string object that is a system command call:

myCommand = "java -mx512m -jar \"" + maxent + "\" -e \"" + climatedataFolder + "\""
myCommand += " -s \"" + csvFile + "\" -o \"" + newOutputFolder + "\""
myCommand += " outputformat=" + optOutputFormat.lower() + " outputfiletype=" + optOutputFileType.lower()

Finally the command is executed via a call to the os.system Python interface that executes the maxent.jar Javascript application with the appropriate options:

result = os.system(myCommand)

When the maxent.jar Java application launches, you’ll see the typical Maxent GUI that you would see if you launched maxent.jar directly and ran a model manually. Except in this case the maxent.jar application launches and starts the model automatically, as shown below.:
Maxent Running

Of course there are some additional script lines that are not shown in the examples above add other maxent parameters to the command string and evaluate the result of the os.system call to determine whether the command executed correctly or not.

However, I have placed a copy of a simple ArcGIS Toolbox and the associated Python script in a zipfile that you can download for your own modeling uses at the link below. There are different versions for each version of ArcGIS you may be using. While newer versions of ArcGIS can read older toolboxes, older versions of ArcGIS cannot read newer toolboxes. So be sure to download the file below that is most appropriate for your ArcGIS version.

Maxent Tool in ArcCatalogIf you like the tool, please let me know if you have any questions or comments/ideas. This is a fairly simple first implementation that I built to get through a lot of modeling quickly. However, I have a few ideas for additions to the tool that would enable it to handle species occurrence point data directly (instead of using saved CSVs) and perhaps leverage Maxent cache (.mxe) files instead of using the environmental data ASC files for each model iteration. At the moment, I’m too busy to add these, but it may be something I add in the future to round out the tool better.

Speedy Attribute Editing with a Custom ArcMap Add-In

One of my colleagues, TerraSystems Southwest, is assisting The Nature Conservancy in Tucson with the Sonoran Desert Disturbed Lands Analysis project. As part of their scope of work, they were asked to assess the level and type of land disturbance for over 100,000 1-square mile hexagons covering the Sonoran Desert.

As you might expect, editing the attributes of 100,000+ polygons is not a trivial task, and TerraSystems approached me to help find a way to make their editing task faster. The solution we came up with was a custom ArcMap add-in that would enable them to quickly edit the attributes for a selected polygon and move to the next feature for assessment.

The ArcGIS 10.0 and 10.1 add-in was developed in Visual Studio 2010 using esri ArcObjects, and contains a custom toolbar, tools, buttons, and dockable window forms. The Add-in leverages the ArcGIS add-in framework for simple deployment and updating.

TNC Custom Add-in Toolbar

The Add-in quickly edits the attributes of customizable fields to record the following information about each hexagon:

  • The disturbance amount (i.e. DistAmnt) of type Double
  • The disturbance type (i.e. DistType) of type String at least 2 characters wide. (The editor records an abbreviation of the disturbance type)
  • The order (priority) in which you want to edit the polygons (i.e. HexID), of type Integer
  • A flag (i.e. DistFlag) indicating that a polygon needs further review, of type Integer (stores a null, 0 or 1)

The Add-in allows users to identify the layer and fields in which to record these attributes, so it is not hard-coded to a particular layer or field name. In addition, the Add-in records the layer and field assignments to an XML file for reloading in future editing sessions, so users only need to configure the Add-in once.

TNC Custom Add-in Screen

Moreover, it was important to also ensure that the disturbance level and type attributes were not hard-coded, as it may be advantageous to add additional disturbance types in the future. In response, the Add-in was written to load both the disturbance levels and types from XML files that can be easily accessed from within the Add-in to make changes to the attributes the Add-in records for each feature.

XML Screenshot

The main strength of this Add-in is time savings during editing, users can select a feature and quickly edit multiple attributes using two different editing methods:

  • The Add-in dynamically* maps keys on their keyboard to enable power users to press a number key to set the feature’s level of disturbance, followed by a letter key to set the feature’s type of disturbance. Users can also press the space-bar to flag a feature as needing follow-up.
  • Alternatively, users can open the dockable Quick Edit window that contains dynamically* added buttons that represent disturbance levels and disturbance types. Users simply click one disturbance level button and another disturbance type button to set the feature’s attributes. Alternatively, users can click a button to flag the feature for follow-up.

TNC Custom Add-in Screen 2

* The buttons and keys are dynamically mapped when the Add-in loads. So if users edit the XML files to register new disturbance levels and types, the Add-in will dynamically map a new key and add new buttons to the QuickEdit Window.

Once both the feature’s disturbance level and type have been set (either by pressing custom mapped keys or using the Quick Edit Window), the Add-in will automatically* select the next feature in the edit series (using the field users selected as their Priority field when they configured the Add-in) and automatically* zoom to the next feature for inspection.

* Some times there are editing situations in which users prefer to not have the Add-in automatically select the next feature and/or zoom to the next feature. So both of these functions can be turned on or off in the Add-in’s configuration window.

Finally the Add-in contains help documentation showing users how to install, use and customize the Add-in.

TNC Custom Add-in Screen 3

By a conservative estimate, this Add-in shaves approximately 7 seconds off of each editing operation. Multiplied by the 100,000+ hexagons, the Add-in has saved nearly 200 hours on this project and countless hours on future projects that can leverage its customizable framework for future conservation assessment programs.

For more information about the Add-in see our website:

http://www.johndonoghue.net/gis/selected-projects/tnc-land-disturbance-editor-arcmap-custom-add-in/

A Script Tool for Unzipping Files in ArcGIS

I was recently working with the SWReGAP data DVDs and wanted to develop an ArcGIS geoprocessing model to copy the DVD data, clip the rasters to the state of Arizona boundary and reproject them.

Because the DVDs contained a raster dataset for each species that was stored as a zip file, I wanted a way to decompress each raster within my model rather than simply decompressing all the zip files to a folder on my hard drive and using the uncompressed files in my model. (This is because each DVD was filled to capacity with 2+ GB zip files, so decompressing the DVD contents to my hard disk first filled up the remaining space on my hard drive. So I decided that I should decompress each file individually and delete it when I was finished.)

So I set about developing a script tool to decompress each zip file so that I could incorporate the script tool into my ArcGIS geoprocessing model. I tried using the zipfile libraries built into Python, but ran into problems decompressing these files because each zip file was too large for the readfile buffer used by the Python routines.

This led me to 7-Zip, an open source to WinZip. While 7-Zip contains a user interface, I was particularly interested in the 7-Zip Command Line Version as this made it possible for me to call the command line from my ArcGIS script too.

With the 7-Zip Command Line Version installed and a few lines of Python, I had a script tool that was capable of extracting a zip file and could be included in an ArcGIS geoprocessing model. I then developed a second script to decompress a whole folder of zip files, and a third script for zipping a feature class.

import os
import arcpy
infile = arcpy.GetParameterAsText(0)
outpath = arcpy.GetParameterAsText(1)

# create 7zip command
zipcommand = “7za e -tzip \”%s\” -o\”%s\”” % (infile, outpath)

try:
      # execute command
os.system(zipcommand)
arcpy.AddMessage(“Finished”)
except Exception as e:
      print e.message
arcpy.AddError(e.message)

I then added the script to a toolbox in ArcGIS as a script tool and set two parameters: 1) Zipfile Name as a file, and 2) Output Location as a folder. When clicked, the script tool appears like the one below:

I was pleased enough with the resulting tool that I developed a simple model that incorporated the tool into an iterator to loop through a folder of zip files and extract each one. That model is illustrated below:

Finally, I also created a second script tool that decompresses an entire folder of zipfiles to a specified location.

Since these have been helpful in automating my workflow I thought I would offer them to the GIS community as a zip file that contains a folder with an ArcGIS Toolbox, two script tools and a sample mode. You can download them from my website at: http://www.johndonoghue.net/ecology/resources.html

To use the ArcGIS Script tools:

There are a couple of things you need to do before the tools will work. Follow the simple steps below to obtain a copy of 7-Zip Command Line Version and modify your system paths.

  • Download and install 7-Zip Command Line Version. When you download the application it will consist of a zip file. decompress the zip file to a folder on your hard disk. I highly recommend extracting the files to a folder in your root directory named something like C:\7Zip or the default C:\7za920. If you extract them into a folder inside your program files folder on 64-bit systems, you may not be able to run the scripts as Python cannot find the executable even with the path statement.
  • Add the path to your 7-Zip installation to your Windows PATH environment variables – add it to both the System PATH variable and the User PATH variable . For example, my path was: C:\7za920; (see http://www.computerhope.com/issues/ch000549.htm if you need help doing this).
  • Finally,download the  Tools from my website and extract them to a folder on your hard drive. You should then be able to use them as you would any ArcGIS script .

I hope you find these tools helpful in your work.

Are Some Parts of ArcGIS Getting More Difficult to Use?

I got a call from a colleague who was having dificulty getting ArcGIS to calculate the value of a field in an attribute table. He was building a model in ModelBuilder and wanted to perform a conditional (If Then) calculatuion to evaluate whether a field’s value was Zero (0) before he tried to divide another value with it.

In response to his question, I quickly provided him the following VBA code for his field calculator tool:

Pre-Logic Script Code:

if ([SUM_weeks] <> 0) then
    newval = [SUM_deliv] / [SUM_weeks]
else
   newval = .001
end if

Assignment Code:

[NewField] = newval
Field Calculation Example with VBA

Field Calculation Example with VBA

While this worked fine, he mentioned that he would rather have Python code, since it is rumored that esri is removing VBA in future releases of ArcGIS. In response, I initially tried to simply convert the Pre-Logic code block to the following Python code:

if (!SUM_weeks! <> 0):
    newval = !SUM_deliv! / !SUM_weeks!
else:
    newval = .001

Much to my surprise this code generated a parsing error, and try as I did, I could not change it in a way that would work, despit the fact that it appears to be perfectly valid Python code.

After reviewing some portions of the ArcGIS Desktop help file, I learned that in order for me to execute the same statement in Python, I needed to write a Python function and then pass the field values into that function. So I composed the following Python code to send my colleague:

Pre-Logic Script Code:

def DoMath(field1, field2):
   if (field2 == 0):
       newval = .001
   else:
      newval = field1 / field2
   return newval

Assignment Code:

DoMath(!SUM_delive!, !SUM_weeks_!)
Field Calculation Example with Python

Field Calculation Example with Python

This worked fine but I was really surprised that I could not just use the field names within the code block. We’ve always been able to do this in esri projects. We coudl do it in VBA and even Avenue (for those who remember the ArcView 3.x Query Builder), AML allowed this too.  But not anymore, why?

As a long time esri user, this kind of stuff concerns me. I’m not worried that they going to lose their customer base or market share. But it seems that as ArcGIS has grown more powerful some portions of it have grown more difficult for casual users. Replacing dialogs and functions that worked well for casual users with Python code blocks works well for programmers and power users, but it leaves casual users to struggle with stuff they used to know how to do.

Users shouldn’t have to write a Python function and then pass an attribute field value into that function to edit a field. This kind of editing used to be easier and I hope someone at esri figures this out and simplifies it again.

P.S.  Don’t get me wrong, it’s not that I hate change or dislike Python; I use it often. I should point out that there are some wonderful simplications to field editing in ArcGIS 9.x and 10 that esri programmers should be commended for. I greatly enjoy and make frequent use of the Calculate Geometry functions and appreciate not haveing to write the VBA code blocks that I used to write to add X and Y coordinates to attribute tables or calculate polygon areas. Actually I only wrote the code once and then saved the query expressions for subsequent uses, but still, it’s much easier to use the newer Calculate Geometry functions.

How I’d like ArcGIS to Work with SQL Server 2008

When Micosoft announced that SQL Server 2008 would include spatial data types I was excited. I’ve been a SQL Server user for a long time and have always preferred its management interface over those of the other DBMS I’ve tried. So the idea of spatial data types was exciting. Though there weren’t many tools out there to manage spatial data in SQL Server at first, I knew that ESRI would include support in the next update to ArcGIS, which they did at 9.3.

It seemed that GIS vendors such as Manifold beat them to it and included support for Katmai as soon as a pre-release build was available that included the spatial components. But I’m an ESRI user; and I’ve been one for a long time. So I knew that 9.3 would include support for SQL Server 2008.

And it did, except that it requires registering that SQL Server 2008 database in ArcSDE. Come on guys, why do we need ArcSDE to work with simple geometry? As a consultant I can’t always convince a client to go with ArcSDE for a variety of reasons:  they can’t afford it, they don’t have the IT resources to support it, etc. There are lots of times when adding the ArcSDE business tables to a SQL Server database schema is overkill to store some points lines and polygons. There are times when I’m lucky enough just to convince a client to add a couple of fields to a database. ArcSDE has a lot of benefits and there are plenty of times when I do recommend it and implement it in client sites, but ArcSDE shouldn’t always be necessary to work with simple spatial data in a relational database.

I’m a big ESRI fan and I strongly believe that they offer superior GIS tools. However, sometimes the simple approach is best. I’m not a Manifold GIS user, but I do like the way they’ve handled using SQL Server 2008 in their desktop GIS application. The video below shows the kind of integration I was hoping to see at 9.3 and at 9.3.1.

http://www.manifold.net/video/mfdsql08.wmv

Besides the obvious direct connection to SQL Server without RDBMS middle-ware. The connection supports read, write and even table modification from the client. I’d love to see this kind of support for SQL Server 2008 spatial data types in ArcGIS. Maybe 9.4?