Some Coveted Programming Skills

To follow-up on an earlier post I wrote this year about GIS Skills, I recently read a story in Readwrite Enterprise outlining 15 programming skills most coveted by employers (http://readwrite.com/2013/04/09/15-programming-skills-most-coveted-by-employers). While their story was aimed at the development community, two of the 15 skills outlined in their story are definitely needed by GIS Analysts and are skills I regularly see in newer job advertisements:

  • SQL * (the #1 most coveted skill)
  • Python (Skill #10)

If your aim is to become a GIS developer, than the additional other programming skills listed in their article apply, namely:

  • ASP.net (Skill #14) **
  • PHP (Skill #11)
  • C# (Skill #7) ***
  • XML (Skill #6)
  • JavaScript (Skill #4)
  • HTML (Skill #3)

* Also the number one preferred skill I listed in my earlier post. So you’ve heard it from at least two different sources now.

** ASP.net is really a framework that you’d develop in using Visual Basic.net or C#, so I’m not sure it’s really in the same category as the other skills mentioned in their article.

*** While you can certainly develop GIS applications using Visual Basic.net, most GIS developers use C#. So if you’re just learning to program, learn C#. If you’re already well entrenched in Visual Basic from years of experience with prior versions (like me), then keep developing in VB.net unless your lack of C# experience starts to interfere with employment prospects.

You’ll also note that their list does not contain references to Flash/Flex and/or Silverlight. Both Flash and Silverlight were all the rage in web-based GIS application development the last few years, however the rather sudden appearance of mobile devices (with their lack of support for Flash and Silverlight) really killed advancement of those technologies as people moved towards GIS websites that would work in the mobile space. Thus these days HTML5 and JavaScript are the preferred development languages for web-based GIS, as these technologies are supported on the major mobile operating systems.

A Maxent Script Tool for ArcGIS

As part of my PhD research at the University of Arizona where I study biogeography, biodiversity and macroecology, I have been part of a group looking at large-scale biodiversity questions for New World plants. In this role, I have been responsible for generating many species distribution models using Stephen Phillip’s Maxent software (and R with the Dismo package). I also use ArcGIS quite often to prepare environmental data for Maxent modeling, and wanted a way to perform and integrate Maxent modeling into my ArcGIS workflows.

Maxent Tool User InterfaceWhen working with data and analysis, my rule is that if I have to do something complicated more than twice, it’s generally worth it to develop an automated workflow. So for my species distribution modeling needs, I wanted a Maxent script tool that I could use as a standalone geoprocessing tool or embed within a ModelBuilder model for use more complicated workflows. For example, one of my most common workflow is iterating through a list of species and running a species distribution model for each one.

To accomplish this, I wrote an ArcGIS script tool in Python that collects a variety of common Maxent parameters in a typical ArcGIS geoprocessing user interface and populates a command string to execute Stephen Phillip’s maxent.jar Java application via a Python call to execute a system command. The resulting script tool uses the arcpy model, so it works like any other ArcGIS geoprocessing script tool. It can operate as a standalone tool to generate a species distribution model for a single species, or can be placed into a ModelBuilder workflow and linked to input parameters as shown in the example below.

Maxent Tool in a ModelBuilder Model

In this example, the script tool is parameterized with a species occurrence dataset obtained from the iterator object in which each dataset is named genus_species.csv. The tool then generates a species distribution model for each species and saves the model to a separate directory within a designated output directory, naming each directory by the genus and species name specified in the name of the input occurrence dataset.

A partial example of the Python script that obtains the parameters from the input datasets is shown below.

Maxent Tool Python Script

These parameters are then appended to a string object that is a system command call:

myCommand = "java -mx512m -jar \"" + maxent + "\" -e \"" + climatedataFolder + "\""
myCommand += " -s \"" + csvFile + "\" -o \"" + newOutputFolder + "\""
myCommand += " outputformat=" + optOutputFormat.lower() + " outputfiletype=" + optOutputFileType.lower()

Finally the command is executed via a call to the os.system Python interface that executes the maxent.jar Javascript application with the appropriate options:

result = os.system(myCommand)

When the maxent.jar Java application launches, you’ll see the typical Maxent GUI that you would see if you launched maxent.jar directly and ran a model manually. Except in this case the maxent.jar application launches and starts the model automatically, as shown below.:
Maxent Running

Of course there are some additional script lines that are not shown in the examples above add other maxent parameters to the command string and evaluate the result of the os.system call to determine whether the command executed correctly or not.

However, I have placed a copy of a simple ArcGIS Toolbox and the associated Python script in a zipfile that you can download for your own modeling uses at the link below. There are different versions for each version of ArcGIS you may be using. While newer versions of ArcGIS can read older toolboxes, older versions of ArcGIS cannot read newer toolboxes. So be sure to download the file below that is most appropriate for your ArcGIS version.

Maxent Tool in ArcCatalogIf you like the tool, please let me know if you have any questions or comments/ideas. This is a fairly simple first implementation that I built to get through a lot of modeling quickly. However, I have a few ideas for additions to the tool that would enable it to handle species occurrence point data directly (instead of using saved CSVs) and perhaps leverage Maxent cache (.mxe) files instead of using the environmental data ASC files for each model iteration. At the moment, I’m too busy to add these, but it may be something I add in the future to round out the tool better.

Speedy Attribute Editing with a Custom ArcMap Add-In

One of my colleagues, TerraSystems Southwest, is assisting The Nature Conservancy in Tucson with the Sonoran Desert Disturbed Lands Analysis project. As part of their scope of work, they were asked to assess the level and type of land disturbance for over 100,000 1-square mile hexagons covering the Sonoran Desert.

As you might expect, editing the attributes of 100,000+ polygons is not a trivial task, and TerraSystems approached me to help find a way to make their editing task faster. The solution we came up with was a custom ArcMap add-in that would enable them to quickly edit the attributes for a selected polygon and move to the next feature for assessment.

The ArcGIS 10.0 and 10.1 add-in was developed in Visual Studio 2010 using esri ArcObjects, and contains a custom toolbar, tools, buttons, and dockable window forms. The Add-in leverages the ArcGIS add-in framework for simple deployment and updating.

TNC Custom Add-in Toolbar

The Add-in quickly edits the attributes of customizable fields to record the following information about each hexagon:

  • The disturbance amount (i.e. DistAmnt) of type Double
  • The disturbance type (i.e. DistType) of type String at least 2 characters wide. (The editor records an abbreviation of the disturbance type)
  • The order (priority) in which you want to edit the polygons (i.e. HexID), of type Integer
  • A flag (i.e. DistFlag) indicating that a polygon needs further review, of type Integer (stores a null, 0 or 1)

The Add-in allows users to identify the layer and fields in which to record these attributes, so it is not hard-coded to a particular layer or field name. In addition, the Add-in records the layer and field assignments to an XML file for reloading in future editing sessions, so users only need to configure the Add-in once.

TNC Custom Add-in Screen

Moreover, it was important to also ensure that the disturbance level and type attributes were not hard-coded, as it may be advantageous to add additional disturbance types in the future. In response, the Add-in was written to load both the disturbance levels and types from XML files that can be easily accessed from within the Add-in to make changes to the attributes the Add-in records for each feature.

XML Screenshot

The main strength of this Add-in is time savings during editing, users can select a feature and quickly edit multiple attributes using two different editing methods:

  • The Add-in dynamically* maps keys on their keyboard to enable power users to press a number key to set the feature’s level of disturbance, followed by a letter key to set the feature’s type of disturbance. Users can also press the space-bar to flag a feature as needing follow-up.
  • Alternatively, users can open the dockable Quick Edit window that contains dynamically* added buttons that represent disturbance levels and disturbance types. Users simply click one disturbance level button and another disturbance type button to set the feature’s attributes. Alternatively, users can click a button to flag the feature for follow-up.

TNC Custom Add-in Screen 2

* The buttons and keys are dynamically mapped when the Add-in loads. So if users edit the XML files to register new disturbance levels and types, the Add-in will dynamically map a new key and add new buttons to the QuickEdit Window.

Once both the feature’s disturbance level and type have been set (either by pressing custom mapped keys or using the Quick Edit Window), the Add-in will automatically* select the next feature in the edit series (using the field users selected as their Priority field when they configured the Add-in) and automatically* zoom to the next feature for inspection.

* Some times there are editing situations in which users prefer to not have the Add-in automatically select the next feature and/or zoom to the next feature. So both of these functions can be turned on or off in the Add-in’s configuration window.

Finally the Add-in contains help documentation showing users how to install, use and customize the Add-in.

TNC Custom Add-in Screen 3

By a conservative estimate, this Add-in shaves approximately 7 seconds off of each editing operation. Multiplied by the 100,000+ hexagons, the Add-in has saved nearly 200 hours on this project and countless hours on future projects that can leverage its customizable framework for future conservation assessment programs.

For more information about the Add-in see our website:

http://www.johndonoghue.net/gis/selected-projects/tnc-land-disturbance-editor-arcmap-custom-add-in/

Bing Maps?

Please note, the following text is an editorial comment. If you’re looking for information about Bing Maps, see http://www.discoverbing.com

 

Today Microsoft announces another name for Virtual Earth – Bing Maps. When they previously changed the name of Virtual Earth to Live Search Maps, I understood their intention of aligning their consumer mapping application (Virtual Earth) with the other consumer applications in the Microsoft Live Search family. However I personally disliked the name Live Search Maps; it was too long and unfriendly. So I’ve always continued to refer to the service as Virtual Earth (as did many other people).

This time Microsoft is attempting a much larger rebranding strategy around a new search platform Bing.com. Bing is an attempt to gain some ground on Google in the search arena. While Bing is a hip name, the rebranding does make me wonder what Microsoft is thinking. I’m sure a large part of the change is hope that Bing will become a household verb like Google. Somewhere a Microsoft ad agency is dreaming about people asking one another, “did you Bing it? What is it? Where is it?”

Microsoft is throwing g a lot of money ($80 to $100 million) at this rebranding campaign to get consumers to question whether Internet searching is giving them what they want. Unless they’ve got some fantastic features to show consumers an alternate reality that is markedly better, Bing will probably not work. Google achieved search dominance because they had a superior product. To supplant them in the search arena, Bing must be better, not just fancier. A fancy updated user interface with web site previews inside pop-up windows and a multi-million dollar marketing campaign did not help Ask.com. So what kind of bling does Bing have that is going to make me switch?

As an aside, while the name Bing is better than Kumo (another name in contention by Microsoft), when I read the single syllable word “Bing” after the word Microsoft, I instantly thought of another single-syllable word “Bob” – another famous Microsoft failure. I’m not trying to be funny; the association was instantaneous and completely uncontrived.

From the perspective of a geospatial professional I think Microsoft is trying to deal with an identity crisis. If we look at the suite of mapping related products/services offered by Microsoft there are:

  • Live Search Maps (Formerly Virtual Earth)
  • Microsoft Virtual Earth APIs – which were used to interact with Live Search Maps
  • MapPoint Web Service

In the near future Bing rebranding will change:

  • Live Search Maps = Bing Maps
  • Microsoft Virtual Earth API = Bing Maps for Enterprise

Since the MapPoint Web Service name is not being changed, I’m guessing that this does not bode well for its future.

I am curious about whether there are plans to align all of the Windows Live products (Messenger, Mail, Calendar, Photos, Spaces, SkyDrive, etc.) around the Bing brand. Otherwise, it seems there will still be an identify crisis when it comes to consumer products. I am supposed to use Bing to search, but use Live for everything else.) Not to tout the obvious, but gMail, gTalk, Google Docs, etc. all unite around the name Google. How will Bing and Live become associated with Microsoft without millions of dollars of marketing?

I do think it’s a good idea for Microsoft to fix its geospatial product identify crisis and align their search related offerings around one brand. However, I think the product/service naming could be tighter. For example, Google’s mapping environment is named Google Maps. Their API is named the Google Maps API. Their 3D version is named Google Earth. It’s all very intuitive. It’s why I liked the name “Virtual Earth”, it worked.

If Microsoft wants to align their geospatial offerings around a fun (and potentially verb-able) product name like Bing, then I suggest they name their primary mapping products “Bing Earth” and “Bing Earth for Enterprise”. The API would then be called “Bing Earth API”. While they are at it, they could rename the desktop application MapPoint to “Bing Earth Studio” or “Bing Earth Desktop”. Sure Google would be mad, but is the name Bing Maps any different than Google Maps? Besides, Microsoft was already using the word Earth in Virtual Earth, so there’s already a precedent for some overlap.