Re-­engineering the Business Process of Desert Tortoise Data Collection

John Donoghue, Technology Consultant for Ironwood Consulting

Ironwood Consulting Inc. (Ironwood) is a biological resources consulting firm in Southern California, and specializes in biological resource management, including desert tortoise permitting and mitigation. Ironwood is known for its successful work on large-­scale solar development projects in the Mojave Desert. Most recently, Ironwood was responsible for the pre-­project surveys, permitting, relocation, and monitoring of tortoises on four large-­scale solar energy projects: Desert Sunlight (3,000 acres), Silver State South (2,400 acres with 104 tortoises), Stateline (1,500 acres with 41 tortoises), and Dry Lake SEZ (serving three solar energy developer clients over a combined 2,000 acres with 45 tortoises). These projects combined required the successful translocation and monitoring of over 200 desert tortoises.

The problem
The large-­scale solar development projects Ironwood was managing required collecting a great deal of field data. During active desert tortoise field seasons, over 70 biologists would be collecting location, health, and other data on desert tortoises moving throughout project sites and offsite mitigation areas. This data was used to determine each tortoise’s territory, ascertain its health and identify other tortoises it may be interacting with. The information was necessary to help plan and ensure the successful translocation of tortoises from the project development site to nearby mitigation preserves, where tortoises would continue to be monitored to confirm they were adjusting to their new homes after the move.

Field biologists worked in remote locations that were often out of cellular voice and data access most of the time. Historically, field biologists collected data on paper forms that were later transcribed into Excel for analysis and reporting. Later, Ironwood adopted a system that used Pendragon Forms for offline field-­based digital data entry. This data was later synced through a server into a Microsoft Access database for storage and use. While the Pendragon Forms system accelerated the process of collecting and digitizing field data, biologists were aggravated by its antiquated interface and wanted forms that worked like the other iOS and Android applications they regularly used, with features that included scrolling pages, advanced data entry constraints on fields, photographs, GPS support and more.

Problems

  • Previous interface lacked modern features and user experience
  • Some data was entered into more than one form to record a single activity
  • Data entry errors were common
  • Data required time-­consuming and expensive manual QA/QC
  • Data available to system users was not relational and could not easily address questions asked by project managers
  • Projects hosted in separate databases with different schemas
  • Lack of standard database schema hampered efforts to gain efficiencies and prevented meta-­analyses across projects

The Pendragon/Microsoft Access system also required field biologists to enter some of the same data into different Pendragon forms, which resulted in many data entry errors and required a great deal of oversight and time-­consuming manual QA/QC. With Ironwood simultaneously managing multiple large-­scale solar projects, the time consuming manual processes that the Pendragon/Microsoft Access system required became very costly. Furthermore, the system wasn’t designed to support multiple end users, and the data products produced by the system were not designed to quickly answer the questions project managers were regularly asked by the solar development clients and agencies. Since each project was hosted in its own separate Access database with varying schemas to accommodate the peculiarities of each project, the absence of a standardized multi-­project database made analyses across projects impractical.

Developing a New Solution
To address these limitations, Ironwood decided to design an entirely new system that could support hosting multiple projects with multiple simultaneous users, while providing a greater variety of data products to address the different needs of field-­biologists, project managements, clients, and agencies. Ironwood began by meeting with their biologists in charge of desert tortoise management to review the shortcomings of the current system and discuss the envision the opportunities a new system could generate. The information from this and subsequent meetings resulted in a new database design that comprised a core set of tables and table fields that were designed to manage the data for multiple projects, and could accommodate all previously collected tortoise data, as well as incoming data for all upcoming desert tortoise projects. To ensure the database could support multiple simultaneous internal and external users working on different projects, the new database was implemented in Microsoft SQL Server. This also provided a foundation to support integrations with GIS other business systems.

iFormBuilder
In looking to replace Pendragon Forms, Ironwood reviewed a number of alternative iOS and Android applications for field-­based data entry. Ultimately, iFormBuilder was selected because it provided several benefits: there were iFormBuilder applications for both iOS and Android; the form building process in iFormBuilder was very intuitive, and the resulting forms had the modern user interface and user experience the field biologists desired. iFormBuilder forms also offered many features Ironwood needed, such as data entry constraints, data input masks, context sensitive field visibility, field validation, GPS support and in-form photos. Finally, the field biologist workflows were often best implemented as sub-­forms, and iFormBuilder sub-­forms were more intuitive to field biologists than the previous Pendragon Forms.

Integrating iFormBuilder and SQL Server
A fundamental goal for Ironwood’s new system was automating previously manual processes, and automating the transfer of data from iFormBuilder’ forms to Ironwood’s SQL Server database was a critical component. Fortunately, iFormBuilder offered an XML Post Data feature that pushes a copy of each record’s data from iFormBuilder to a user defined web page every time a new record is successfully uploaded to the iFormBuilder server.

Ironwood developed an ASP.net web application that contained a set of URL endpoints to receive XML data posted from iFormBuilder. The ASP.net application parsed the incoming XML data and inserted data as new records into Ironwood’s SQL Server database. During this process, the ASP.net application also performed QA/QC on incoming data, standardized data formats, and transformed the location information contained in the posted data into Open Geospatial Consortium (OCG) compliant spatial data for use in Ironwood’s ArcGIS system and other applications. Finally, the ASP.net application sent each user synching data to iFormBuilder an email that informed them of their successful data sync and identified any potential data errors found by the QA/QC processes.

Another goal of the new system was to re-­engineer the data collection business processes to remove duplicate data entry processes and reduce the number of digital forms needed. The previous Pendragon forms were designed to populate single database tables. Field biologists often had to enter some of the same data into different Pendragon forms as they performed a single activity, such as a health assessment – in which biologists completed one Pendragon form to record a tortoise location, and another Pendragon form to record tortoise health data.

In contrast, Ironwood designed its iFormBuilder forms around the activities biologists performed during their tortoise survey and monitoring work. As the data was received by Ironwood’s ASP.net application during syncing, the ASP.net application would automatically populate the appropriate tables. Therefore, a field biologist could use an activity specific iForm to record data for multiple associated activities, and the ASP.net application would sort the received data and populate the data into each relevant table.

Leveraging Better Data
To help provide more expedient and broader access to the data gathered in iFormBuilder, Ironwood also developed an ASP.net website that served as a central hub for field biologists, project managers, clients, and agencies. The resulting web portal supported multiple users and projects, and provided access to data as soon as it was synced from iFormBuilder and pushed into Ironwood’s SQL Server database. The web portal provided dashboards where users could view tortoise statistics and interactive maps of tortoise locations that were dynamically generated from the database using data from the most recent iFormBuilder application sync.

The web portal provided data in different formats for the various types of users the system supported. Field biologists and agencies accessed the portal to download dynamically generated Excel and GPX files that contained the most recently recorded tortoise locations and other information. Project Managers and Clients downloaded dynamically generated KML data for mapping in Google Earth, and viewed dynamic charts and statistics to review project’s status and look for potential issues. Unpublished URL endpoints allowed the data to be brought into ArcGIS Online, Google Earth and other systems.

In addition, by storing the location data as OGC compliant spatial points, the database could be directly accessed by GIS systems, and was persistently linked to Ironwood’s esri ArcGIS system where Python scripts automated daily cartographic quality maps used by field biologists, project managers, and clients.

Results
The better data input constraints provided in iFormBuilder significantly reduced Ironwood’s QA/QC needs. Despite having developed an automated backend QA/QC process, Ironwood later found that 95% of the data pushed into its database had no potential QA/QC issues identified by that process as the errors were constrained during data-­entry in iForms.

Field biologists were especially pleased with the new system. The iFormBuilder forms provided a modern user interface and user experience they expected, and iFormBuilder features such as designating the default device keyboard for fields, data input masks, and context sensitive field visibility, significantly increased the speed with which biologists could enter data into iForms, while being confident that data was accurate.

Results

  • Considerably fewer data errors
  • Increased adoption of digital forms
  • Greater trust in data
  • Much faster data turn-­around
  • Significantly lower data management costs
  • Improved client confidence in data management process
  • Increased client satisfaction
  • Improved agency cooperation
  • Stronger client relationships

The iFormBuilder XML Push workflow has greatly benefited field biologists by ensuring that the data the biologists recorded and synched was instantly added to Ironwood’s SQL Server database and biologists were immediately informed of its availability.

By developing a data portal website, Ironwood was able to leverage iFormBuilder and SQL Server to provide biologists, project managers, clients and agencies with access to all data anytime they needed it. Clients have praised the system and are using it regularly to keep informed on the progress of the mitigation programs for their projects. By making data collection processes transparent and providing clients with readily accessible data in the formats they needed, Ironwood was able to better communicate the data collection effort with its clients which reinforced the client’s trust in Ironwood’s data collection processes.

Though considerable effort was involved in reaching agreement with on a new database design and data input forms that bridged the gap between the needs of the field biologists and Ironwood’s end-­users, by implementing an integrated iFormBuilder, SQL Server and ASP.net solution, Ironwood was able to reengineer the business process of desert tortoise data collection. While the entire system took time to develop, the major benefits were realized when the system was fully implemented on multiple projects, where it essentially ran itself with very little intervention, resulting in significantly reduced data management and reporting costs, ultimately transforming a once cost-­intensive activity into a financial and practical benefit to Ironwood’s clients.

Some Coveted Programming Skills

To follow-up on an earlier post I wrote this year about GIS Skills, I recently read a story in Readwrite Enterprise outlining 15 programming skills most coveted by employers (http://readwrite.com/2013/04/09/15-programming-skills-most-coveted-by-employers). While their story was aimed at the development community, two of the 15 skills outlined in their story are definitely needed by GIS Analysts and are skills I regularly see in newer job advertisements:

  • SQL * (the #1 most coveted skill)
  • Python (Skill #10)

If your aim is to become a GIS developer, than the additional other programming skills listed in their article apply, namely:

  • ASP.net (Skill #14) **
  • PHP (Skill #11)
  • C# (Skill #7) ***
  • XML (Skill #6)
  • JavaScript (Skill #4)
  • HTML (Skill #3)

* Also the number one preferred skill I listed in my earlier post. So you’ve heard it from at least two different sources now.

** ASP.net is really a framework that you’d develop in using Visual Basic.net or C#, so I’m not sure it’s really in the same category as the other skills mentioned in their article.

*** While you can certainly develop GIS applications using Visual Basic.net, most GIS developers use C#. So if you’re just learning to program, learn C#. If you’re already well entrenched in Visual Basic from years of experience with prior versions (like me), then keep developing in VB.net unless your lack of C# experience starts to interfere with employment prospects.

You’ll also note that their list does not contain references to Flash/Flex and/or Silverlight. Both Flash and Silverlight were all the rage in web-based GIS application development the last few years, however the rather sudden appearance of mobile devices (with their lack of support for Flash and Silverlight) really killed advancement of those technologies as people moved towards GIS websites that would work in the mobile space. Thus these days HTML5 and JavaScript are the preferred development languages for web-based GIS, as these technologies are supported on the major mobile operating systems.

Social Networks and Ecological Food Webs

I read an interesting recent article that discussed a digital “autopsy” of the social network Friendster. Friendster was the grandfather of the social network, preceding both MySpace and Facebook. At its peak it had over 100 million users and a $30 million buyout offer from Google (which they turned down). However, following some technical problems and a site redesign in 2009, the network suffered a catastrophic loss of traffic as users moved to other networks like Facebook.

In this “autopsy” the authors looked into the reasons behind Friendsters “death” and discussed two general reasons for the failure:

  1. The first factor is a cost-benefit ratio: when the costs, in terms of time and effort exerted by being a member of the social network are greater than the benefits obtained, the conditions are set for users to leave.
  2. This is where the network’s topology is important. If a huge number of people in the network have only two friends, then when one of them leaves, the other is left alone on the network and will likely exit as well. This results in a cascade of users leaving the network. However, if people have many friends on the network (many connections), the loss of a single friend is unlikely to result in their exit from the network and unlikely to trigger a cascade of exiting users.

The authors concluded that the “cause of death” was due to declining cost-benefit ratios experienced by users as a result of the technical problems and site redesign. However, it was exacerbated by the network structure.

So, the key point here is that the resilience of the network is determined by the number of connections that each person in the network has. So for any given network, the proportion of the network containing a certain number of friends fraction of the network with a certain number of friends with few connections can be a crucial indicator of the network’s overall resilience from cascades.

In ecology, food webs are “networks” that describe a biological community in terms of the interactions between consumers and resources. The interactions are modeled as connections between consumers and resources that relate to the represent the transfer of energy and matter between the two. The interactions are diagrammed as lines connecting the consumers and their resources – which could be predators and prey, herbivores and plants, plants and soil, etc.

What’s interesting about the Friendster article is its very strong parallel to ecological systems like food webs. Both social networks and food webs need many connections for stability and resilience. Like Friendster, food webs with few participants are highly vulnerable to collapse or sudden radical changes in their structure (called a “trophic cascade“), in which the loss or addition of one or more key participants effectively changes the entire food web by altering key links between network participants. (For example, it’s impossible to play the Kevin Bacon game if he were not part of the network).

For social networks, the key to stability is each person having many Friends so that the loss of any single friend from the network is buffered by the many connections that each person has with other friends. For food webs, stability comes from a similar structure: they need many participants in the food web (the network) to create many different pathways between resources. Having many participants requires preserving biodiversity. If the biodiversity of a system is altered too much by the loss of a top predator or the introduction of an invasive species, the “network” can change. When this happens, vital connections between species can be broken and other connections can arise when newly introduced species competes for the same resources that were formerly linked only to native species. If the network does not contain many participants and has few connections, even small changes in the network will have drastic effects – in the worst case, it can result in a trophic cascade, such as the one that “killed” Friendster.

 

Featured image from: http://coralreef.noaa.gov/education/educators/resourcecd/posters/images/iyor_foodweb.jpg

A Maxent Script Tool for ArcGIS

As part of my PhD research at the University of Arizona where I study biogeography, biodiversity and macroecology, I have been part of a group looking at large-scale biodiversity questions for New World plants. In this role, I have been responsible for generating many species distribution models using Stephen Phillip’s Maxent software (and R with the Dismo package). I also use ArcGIS quite often to prepare environmental data for Maxent modeling, and wanted a way to perform and integrate Maxent modeling into my ArcGIS workflows.

Maxent Tool User InterfaceWhen working with data and analysis, my rule is that if I have to do something complicated more than twice, it’s generally worth it to develop an automated workflow. So for my species distribution modeling needs, I wanted a Maxent script tool that I could use as a standalone geoprocessing tool or embed within a ModelBuilder model for use more complicated workflows. For example, one of my most common workflow is iterating through a list of species and running a species distribution model for each one.

To accomplish this, I wrote an ArcGIS script tool in Python that collects a variety of common Maxent parameters in a typical ArcGIS geoprocessing user interface and populates a command string to execute Stephen Phillip’s maxent.jar Java application via a Python call to execute a system command. The resulting script tool uses the arcpy model, so it works like any other ArcGIS geoprocessing script tool. It can operate as a standalone tool to generate a species distribution model for a single species, or can be placed into a ModelBuilder workflow and linked to input parameters as shown in the example below.

Maxent Tool in a ModelBuilder Model

In this example, the script tool is parameterized with a species occurrence dataset obtained from the iterator object in which each dataset is named genus_species.csv. The tool then generates a species distribution model for each species and saves the model to a separate directory within a designated output directory, naming each directory by the genus and species name specified in the name of the input occurrence dataset.

A partial example of the Python script that obtains the parameters from the input datasets is shown below.

Maxent Tool Python Script

These parameters are then appended to a string object that is a system command call:

myCommand = "java -mx512m -jar \"" + maxent + "\" -e \"" + climatedataFolder + "\""
myCommand += " -s \"" + csvFile + "\" -o \"" + newOutputFolder + "\""
myCommand += " outputformat=" + optOutputFormat.lower() + " outputfiletype=" + optOutputFileType.lower()

Finally the command is executed via a call to the os.system Python interface that executes the maxent.jar Javascript application with the appropriate options:

result = os.system(myCommand)

When the maxent.jar Java application launches, you’ll see the typical Maxent GUI that you would see if you launched maxent.jar directly and ran a model manually. Except in this case the maxent.jar application launches and starts the model automatically, as shown below.:
Maxent Running

Of course there are some additional script lines that are not shown in the examples above add other maxent parameters to the command string and evaluate the result of the os.system call to determine whether the command executed correctly or not.

However, I have placed a copy of a simple ArcGIS Toolbox and the associated Python script in a zipfile that you can download for your own modeling uses at the link below. There are different versions for each version of ArcGIS you may be using. While newer versions of ArcGIS can read older toolboxes, older versions of ArcGIS cannot read newer toolboxes. So be sure to download the file below that is most appropriate for your ArcGIS version.

Maxent Tool in ArcCatalogIf you like the tool, please let me know if you have any questions or comments/ideas. This is a fairly simple first implementation that I built to get through a lot of modeling quickly. However, I have a few ideas for additions to the tool that would enable it to handle species occurrence point data directly (instead of using saved CSVs) and perhaps leverage Maxent cache (.mxe) files instead of using the environmental data ASC files for each model iteration. At the moment, I’m too busy to add these, but it may be something I add in the future to round out the tool better.

Speedy Attribute Editing with a Custom ArcMap Add-In

One of my colleagues, TerraSystems Southwest, is assisting The Nature Conservancy in Tucson with the Sonoran Desert Disturbed Lands Analysis project. As part of their scope of work, they were asked to assess the level and type of land disturbance for over 100,000 1-square mile hexagons covering the Sonoran Desert.

As you might expect, editing the attributes of 100,000+ polygons is not a trivial task, and TerraSystems approached me to help find a way to make their editing task faster. The solution we came up with was a custom ArcMap add-in that would enable them to quickly edit the attributes for a selected polygon and move to the next feature for assessment.

The ArcGIS 10.0 and 10.1 add-in was developed in Visual Studio 2010 using esri ArcObjects, and contains a custom toolbar, tools, buttons, and dockable window forms. The Add-in leverages the ArcGIS add-in framework for simple deployment and updating.

TNC Custom Add-in Toolbar

The Add-in quickly edits the attributes of customizable fields to record the following information about each hexagon:

  • The disturbance amount (i.e. DistAmnt) of type Double
  • The disturbance type (i.e. DistType) of type String at least 2 characters wide. (The editor records an abbreviation of the disturbance type)
  • The order (priority) in which you want to edit the polygons (i.e. HexID), of type Integer
  • A flag (i.e. DistFlag) indicating that a polygon needs further review, of type Integer (stores a null, 0 or 1)

The Add-in allows users to identify the layer and fields in which to record these attributes, so it is not hard-coded to a particular layer or field name. In addition, the Add-in records the layer and field assignments to an XML file for reloading in future editing sessions, so users only need to configure the Add-in once.

TNC Custom Add-in Screen

Moreover, it was important to also ensure that the disturbance level and type attributes were not hard-coded, as it may be advantageous to add additional disturbance types in the future. In response, the Add-in was written to load both the disturbance levels and types from XML files that can be easily accessed from within the Add-in to make changes to the attributes the Add-in records for each feature.

XML Screenshot

The main strength of this Add-in is time savings during editing, users can select a feature and quickly edit multiple attributes using two different editing methods:

  • The Add-in dynamically* maps keys on their keyboard to enable power users to press a number key to set the feature’s level of disturbance, followed by a letter key to set the feature’s type of disturbance. Users can also press the space-bar to flag a feature as needing follow-up.
  • Alternatively, users can open the dockable Quick Edit window that contains dynamically* added buttons that represent disturbance levels and disturbance types. Users simply click one disturbance level button and another disturbance type button to set the feature’s attributes. Alternatively, users can click a button to flag the feature for follow-up.

TNC Custom Add-in Screen 2

* The buttons and keys are dynamically mapped when the Add-in loads. So if users edit the XML files to register new disturbance levels and types, the Add-in will dynamically map a new key and add new buttons to the QuickEdit Window.

Once both the feature’s disturbance level and type have been set (either by pressing custom mapped keys or using the Quick Edit Window), the Add-in will automatically* select the next feature in the edit series (using the field users selected as their Priority field when they configured the Add-in) and automatically* zoom to the next feature for inspection.

* Some times there are editing situations in which users prefer to not have the Add-in automatically select the next feature and/or zoom to the next feature. So both of these functions can be turned on or off in the Add-in’s configuration window.

Finally the Add-in contains help documentation showing users how to install, use and customize the Add-in.

TNC Custom Add-in Screen 3

By a conservative estimate, this Add-in shaves approximately 7 seconds off of each editing operation. Multiplied by the 100,000+ hexagons, the Add-in has saved nearly 200 hours on this project and countless hours on future projects that can leverage its customizable framework for future conservation assessment programs.

For more information about the Add-in see our website:

http://www.johndonoghue.net/gis/selected-projects/tnc-land-disturbance-editor-arcmap-custom-add-in/

Landing a GIS Job and GIS Skills Development in 2013

Note: This post is a 2013 update to my earlier post on Learning GIS and Getting a GIS Job – Some Tips and Tricks which was originally posted in 2009.

In 2009 I wrote a post on how to obtain a GIS job that focused on pathways to learn GIS, critical skills every GIS analyst should know and ways to augment traditional learning pathways to set you apart from other job candidates. Even though this post is well over 3 years old, it’s still read by people interested in the topic and I still receive comments.

I recently re-read the post and decided that an update was in order. Software has changed and old skills need to be replaced with new skills to keep up with the evolving GIS software industry. For brevity, I’ve focused this new post on changes and revisions to the recommendations I made in my earlier post.  I’ve also added links to resources and elaborated on some topics I introduced last time. So, please be sure to read that post before this one so you have the complete picture for what’s changed.

For simplicity, I’ve focused on skills needed for a GIS Analyst/Specialist position. GIS developers and GIS database manager positions have a different set of skills that may overlap with some of these, but also include specific skills that are not part of the everyday toolkit of GIS Analysts and are not included here.

Learning Pathways

There are a variety of reasons why you want to learn GIS. You may be employed in a non-GIS field and are learning GIS as part of your job. Maybe you are switching careers or retraining yourself to find a new job in the growing geospatial industry. Perhaps you are a student who wants to learn GIS to support your field of study, or you are a student majoring in GIS or geography with aspirations of landing a job doing GIS.

Depending upon your goal, there are different pathways to get started learning GIS:

Formal GIS/Geography Degree

If you want to obtain a bachelor’s or master’s degree majoring in GIS or geography, you should start at a community college or university where you can take accredited GIS courses that will apply to your major’s requirements.

Professional, Continuing Education

If you’re seeking GIS skills for your current job or want to re-tool yourself for immediate employment, you may be better served by finding a professional certificate program in GIS at a local community college or university extension. These programs are usually non-accredited, which means that they will not likely count towards satisfying the requirements for a formal degree in GIS, but they are a great way to learn GIS quickly and gain readily employable skills to help you enter the marketplace. In addition, many professional certificate programs are taught by professionals working in the field, so you will get first-hand information about how GIS is used by the people who use it daily. These programs are also fantastic places to network with potential future employers.

If the professional certificate track is what you need, check with local community colleges and universities in your area to see if they have a professional, extended, or continuing education certificate program in GIS or geospatial sciences. If there is no school in your area offering certificates or degrees in GIS, there are a number of excellent online programs available today, including:

This list is by no means complete, so if you know of other programs not listed, please add them as a comment to this post.

Both URISA and ESRI maintain a list of universities as well, so you may also want to check their websites as well:

There are also a number of specific free short-courses and technical workshops available on the ESRI Virtual Campus website.

Learn What Employers Want

Do not necessarily trust that your GIS certificate or degree program will teach you all the skills you need to know to land a GIS job. I’ve interviewed too many candidates who graduated from a GIS certificate program but only knew how to operate ArcGIS. While proficiency with ArcGIS is a critical skillset to have as a GIS analyst, there are a number of other hard and soft skills/techniques you should know to be a worthy job candidate.

Minimum Skills

At a minimum you should be proficient with the use of ESRI ArcGIS software (and extensions such as Spatial and 3D Analyst) and you should be able to produce cartographically pleasing outputs. However, GIS Analysts and Technicians are also expected to know the following: 

  • Cartographic production
  • Data creation
  • Geodatabase design and creation
  • Data editing
  • Topology
  • Projections
  • GIS analysis
  • Geoprocessing
  • GPS data capture
  • Aerial photography interpretation
  • Georeferencing
  • Data conversion
  • Metadata implementation

Employers also expect GIS Analysts to have experience with:

  • At least one non-spatial, database management system (Microsoft Access, Oracle, MySQL, etc.)
  • Microsoft Office (Excel, Word, Outlook, PowerPoint and Access).

Finally, most job advertisements also require GIS Analysts to have a Bachelor’s degree in Geography, Geographic Information Systems (GIS), Cartography, Computer Science or a related discipline. (Related disciplines can be landscape architecture, urban planning, geology, etc.). In many cases employers will accept a degree in an unrelated field with evidence of completion of a GIS certification program or demonstrable experience with GIS. While GIS technician positions often require very minimal years of experience, it’s typical to see employers require at least 3 years of GIS experience for GIS Analysts.  However, don’t let these numbers dissuade you from applying for any position. In most cases these are “desired” qualifications for a “perfect” candidate. “Perfect” candidates are rare, so employers frequently accept lesser qualified candidates, and it’s here that you can shine with your experience and skills.

Preferred Skills

In addition to the minimum skill sets, the following skills are commonly listed in advertisements for GIS Analyst positions:

  • SQL
  • Programming/scripting with Python
  • ArcGIS Server / SDE
  • Model builder
  • Trimble Pathfinder
  • ArcGIS Mobile
  • AutoCAD and integration of GIS and CAD
  • Adobe Photoshop
  • Adobe Acrobat Pro
  • Adobe Illustrator
  • Experience with web-application development (Net, Java, Python, PHP, HTML, JavaScript or Flex)

Soft Skills

Finally, it’s not just about technical abilities; employers also look for GIS people who:

  • Can to learn new technologies, skills and software platforms/extensions quickly with minimal oversight
  • Are good at problem solving and troubleshooting
  • Are self-motivated and pro-active
  • Can multitask and balance priorities with work load
  • Can be flexible in working on unplanned projects/events
  • Can work both independently and function well in a team environment
  • Are able to communicate complex GIS concepts to entry level users.
  • Show attention to detail and process
  • Show initiative in working efficiently and seeking out work
  • Have a customer focused/value added approach
  • Have strong verbal and written communication skills

Domain expertise, don’t just be a GIS jockey, know how to apply it to some government or industry such as environmental analysis, utilities, planning and urban development, marketing, etc. Whatever your interest, learn how to use GIS in your area. It will make you a more valuable and interesting person than just someone who knows how to use ArcGIS.

Read GIS Magazines and Websites

GIS website portals and magazines are a great way to get a sense of what direction the industry is moving, which topics are hot and how people and organizations are implementing GIS. They’re also useful for keeping up with the latest GIS-related news. So, explore some of the magazines and websites that are available. Some of the more well-known sources include:

Magazines

Portals

Attend User Group Meetings

I said this in my earlier post, but I’ll say it again because it’s important: Virtually everyone I know working in GIS got their first or subsequent job as a result of networking at a local or regional user group. Find one and attend them often. Ask questions, talk to people – these are (or will be) your colleagues and peers.

To get started, check the following websites to see if there is a user group in your area:

If there isn’t a local or regional User Group in your area, start one. There are probably lots of GIS people working in your area that would love to attend a user group. This is a great way to make contacts in the GIS industry very quickly. Many agencies can get the use of their meeting rooms for free. Most local vendors will jump at the chance to market their product to a room full of potential customers, so vendors can be helpful in filling presentation slots and getting the word out. With a little bit of tact you can usually get a vendor to sponsor snacks if they are presenting.

 

Get Real-World Experience and Build a Portfolio

In my earlier post I mentioned the need to get real-world experience to set you apart from other job candidates who only have GIS classwork experience. Today, more schools are integrating GIS into their curricula, and more schools offer GIS courses and certificates than they did a few years ago. This means there are potentially more people competing for a GIS job with you, so what will set you apart from the others is your experience.

So get real-world experience, read what I wrote in my earlier post as it’s important to develop real-world skills that will set you apart from your competition.

In addition, build a portfolio of your work that you can take to interviews. My first portfolio consisted of printed 8.5 x 11 copies of maps that I had made, scripts I had written, and reports or documents I had developed. I placed these items in clear plastic sleeves made for a 3-hole binder and brought the binder with me to interviews. Later I made PDFs of my maps, reports and scripts and burned them to CDs that I would give to potential employers when interviewing – but I still brought my binder so I could show these items during the interview.

So create a portfolio and bring it with you to interviews. It provides a great way to show off your discuss your experience and show off your cartographic skills. These will set you apart from most of the competition that will just show up to an interview with a copy of their resume.

Auxiliary skills

These days everyone applying for a GIS job can use ArcGIS Desktop. Sorry but that skill alone doesn’t make you special anymore (it doesn’t make me special anymore either). So potential employers are going to be asking what else you can do.

Databases

Any GIS person worth their salt knows how to set up and manage a database. Whether it’s Microsoft Access, SQL Server, Oracle, MySQL, or PostGreSQL, if you don’t know how to use a database, learn to.

If you are not familiar with databases, then learn Microsoft Access. It’s a good database system that’s easy to learn and use. Plus many organizations commonly use it and many people already have it on their computers – as it comes with many versions of Microsoft Office.

If you don’t have a copy of Microsoft Access and can’t justify buying one, Open Office Base offers many of the same features as Microsoft Access and can serve as a suitable replacement for learning about databases. You can download Open Office at: http://www.openoffice.org/

From there you can graduate to the workgroup and enterprise systems such as Microsoft SQL Server, Oracle, MySQL, PostGreSQL, etc. All of these database management systems offer free versions for development or personal use that you can download to get experience using them. See the links below for each software application:

Model Builder

There are many actions we routinely perform in GIS which can be manually repetitive. Consider the example where you are given a disk with 100 shapefiles that are in the WGS84 projection and you need to import them into your GIS data library (a geodatabase). To do this you’ll need to re-project them to the coordinate system you use (NAD 83, UTM Zone 11). You could do this manually in ArcCatalog or ArcMap but it would be better to construct a workflow that could automate the process and do it 100 times.

Model Builder is perfect for this kind of task. ModelBuilder is an application you use to create, edit, and manage workflows that string together sequences of geoprocessing tools, in which the tools feed the output of one tool into another tool as input. ModelBuilder is a visual programming language for building these workflows.

Why learn it? Model Builder is a key GIS Analysis skill because:

  • Performing actions like manually importing 100 shapefiles is tedious and not a good use of your time.
  • Complicated analyses often require a lot of geoprocessing actions and you’ll often find that you may need to repeat an analysis. If you need to run a complex analysis again and you didn’t build a model, you’ll have to re-do all your geoprocessing tasks manually and hope you took really good notes of your processing steps.
  • If you’re working with large datasets, geoprocessing tasks can take a long time. In this case, it’s best to construct a model to perform these tasks after hours instead of doing each task step by step (by hand) and waiting long periods for the output.
  • Models can be incorporated into ArcGIS Toolboxes and easily distributed to other users so they can quickly repeat an analysis.
  • Models can be uploaded to ArGIS Servers to enable processing of files on a server or in the cloud.
  • Finally, savvy employers know about Model Builder and it shows up as either a required or desired skill in job advertisements.

To get started learning Model Builder, visit the following links:

ArcGIS Server

If you are going after a GIS job in a large organization with an existing GIS program, there’s a chance that you’ll be asked about your experience with ESRI’s ArcGIS Server product. ArcGIS Server is a web server application that allows you to share GIS resources (maps, globes, tools, geoprocessing workflows, etc.) over the web. These resources are hosted on the ArcGIS Serve to facilitate sharing data that is centrally managed, supports multiple users, and contains the most up-to-date information.

ArcGIS Server skills are in demand these days and it’s a key skill to learn if you can. However, learning ArcGIS Server isn’t easy. Installing, managing and operating ArcGIS Server requires knowing a little about how to manage a file server and web server and active directory permissions.  Realistically the best way to learn about ArcGIS Server is to take a class. Many in-person and online programs offer a course in ArcGIS Server; ESRI’s Virtual Campus has courses in ArcGIS Server as well.

If you’re not familiar with ArcGIS Server, visit the links below to get a general introduction to ArcGIS Server and its capabilities.

If you can’t get experience working with ArcGIS Server, one thing you can do (in addition to reviewing the links above) is to register for a free personal account on www.arcgis.com. ArcGIS.com is a subscription based online platform for sharing geographic information and maps. ESRI offers a free personal account that provides limited access to features and 2 GB of storage space. With this account and a copy of ArcGIS you can upload data and maps to ArcGIS.com and experience a little bit of what it’s like to work with an ArcGIS Server from a user perspective.

In addition, reviewing the samples and information on the ArcGIS Server Javascript API provides a perspective on how to program web applications for ArcGIS Server.

Excel

Yeah, it’s not and ESRI product, but Microsoft Excel can be a critical component of your data scrubbing workflow. I use it often to view, manipulate and clean data before I import it into ArcGIS because it’s powerful, fast and can be used to make many data changes quickly and easily.

When learning GIS, we work with prepared datasets that have been cleaned and optimized to work in our exercises with minimal issues. In the real-world, data is messy; it contains a mixture of values in a variety of formats. Knowing how to quickly clean up a dataset is an essential GIS skill. So I use excel to change data from UPPERCASE to lowercase to replace portions of data with other values, to trim data, and to combine and split data. However, I rarely make these changes manually. Instead I use powerful functions in Excel that automate these corrections across rows and columns.

While Excel has many functions, I typically use only several for my data cleansing. So learn how to use functions in Excel; how to write them, copy and paste them. In particular, start with the following functions, as they are most commonly used to manipulate data:

  • Concatenate
  • Left
  • Right
  • Mid
  • Trim
  • Substitute
  • Replace
  • Upper
  • Lower

To learn more about Excel functions, simply search Google or Bing for “learning excel functions”. A good tutorial for getting started is also here: http://blogs.mccombs.utexas.edu/the-most/2009/05/07/learn-excel-functions/

Scripting

In my earlier post I recommended learning a scripting language because knowing how to write a script comes in handy when you need to scrub huge databases or perform actions like correcting addresses for better geocoding accuracy, reformatting data, exporting and importing large datasets, and other manually repetitive activities GIS people are routinely tasked with.

Back in 2009, I suggested that Visual Basic for Applications (VBA) was a good way to get started. Since that post ESRI has deprecated support for VBA in ArcGIS 10.x and VBA is not the preferred way to script in their latest product lines.

Today, scripting in ESRI is all about Python. Python is a free, cross-platform, open-source programming language that is widely used and supported in the open source and esri GIS environments.  It is become the scripting language of choice for geoprocessing users and ESRI has fully embraced Python for ArcGIS. So I would advise anyone in GIS who doesn’t know Python to learn Python. Here are some links to get started learning about Python in ArcGIS:

Finally, while ESRI may not support VBA anymore, VBA is still widely used in Microsoft’s Office product line though it’s being slowly replaced by Microsoft’s Visual Studio Tools for Applications: http://msdn.microsoft.com/en-us/library/cc175562(v=vs.90).aspx

 

Reports for ArcGIS

In my previous post, I wrote about the virtues of learning Crystal Reports. Crystal Reports is a powerful reporting tool that can read databases and spreadsheets, perform queries, sort and summarize data and generate professional looking PDF or live reports. Crystal Reports used to be included with ArcGIS Desktop, however at ArcGIS 9.3, ESRI stopped including the full Crystal Reports product; and the Crystal Reports Wizard is no longer included in ArcGIS 10.x.

As a replacement, ESRI now offers a reporting application built into ArcGIS Desktop that provides much of the same functionality that Crystal Reports offered. However the reporting application cannot be accessed or used outside of ArcGIS Desktop and must be used from the ArcGIS Desktop user interface.

You may be wondering whether it is still important to learn Reports for ArcGIS. Yes, because it’s not always about the map; in fact, it often isn’t about a map at all. I know us GIS people got into GIS because we like to make great looking maps, but the mistake we make is thinking that everyone else likes maps too. When I started working in government GIS, I was amazed at how often I was asked to create a report showing the results of my analyses (for example, a report containing a table summarizing the number of acres of different vegetation types that will be impacted by a proposed development). No map; no polygons on a nice aerial photo – just a table of numbers representing the results of a geographic analysis. There was still a lot of cool GIS analysis that went into generating the report, but the end product wasn’t a map.

While the software may have changed, the need hasn’t  so I’ll continue to recommend learning how to design and create reports of your GIS analyses. Today, we would accomplish this with Reports for ArcGIS. So to learn how to use the integrated reporting tools, see the following links:

2013 Conclusion

While I’ve written this post within the context of obtaining skills you need to land a GIS job, these are equally appropriate skills for those already employed in GIS. The industry continues to change rapidly, and you can get rusty pretty fast if you’re not keeping up. Each new software version brings new features that offer new opportunities for learning a new skill or adding to our existing knowledge base.  For example, I recently learned how to write and deploy ArcObjects applications in ESRI’s new ad-in framework and plan to learn how to use the new Python ad-in framework in ArcGIS 10.1.

Learning these employable GIS skills is also a very good investment. GIS jobs are still ranked very high in job growth outlook – the Bureau of Labor Statistics lists an expected 35% growth in jobs for “Geographers” (http://www.bls.gov/ooh/life-physical-and-social-science/geographers.htm). Whether you’re looking for your first GIS job or looking to move to a new organization, these skills will make you much more marketable than your typical competition.

Finally, I’ve tried to identify those skills that I view as important for obtaining and maintaining a GIS position.  I’ve also focused on skills a GIS Analyst or Specialist should know. If you’ve read this and can think of other important skills I didn’t mention, please leave a comment to benefit the larger community.

Thanks for reading.

Les Miserables Look Down, Look Down the PhD Version

In the opening scene of the new Les Miserables movie, Jean Valjean (played by Hugh Jackman) is one of many shackled prisoners trying pulling a large ship into a dry dock during rough seas. The work song, “Look down, look down” is quite dramatic and the whole scene is cinematically spectacular and a powerful opening to a great film.

After reading the latest edition of PhD Comics’ parody of Les Miserables, I thought back to the opening scene in the movie and the prisoner work song. The song’s parallels with PhD student life were striking.

Thus, in the sprit of the the PhD Comics parody, I present a PhD student version of “Look down, look down”. Apologies to Victor Hugo, and all musical productions of Les Miserables.

Look down, look down
Don’t look em in the eye
Ad-vi-sor comes
Don’t let em see you cry

Look down, look down
Your funding’s gone away
No grant renewed
You’ll have to stay T.A.

Look down, look down
Rejected you have been
Reviewed with scorn
Submission number ten

Look down, look down
Your data seems all wrong
Your test has failed
Your research takes too long

Look down, look down
No correlation here
P value high
This is your greatest fear

Look down, look down
Your findings don’t add up
Reject H-1
Perhaps you should give up

Look down, look down
Your database has crashed
There’s no backup
Your research has been trashed

Look down, look down
Your thesis is a mess
You can’t finish
You might have to confess

Look down, look down
Your research goals not met
One more review
You can’t defend just yet

Look down, look down
Not yet a PhD
May never come
You’re not as good as he

Look down, look down
Five years are all but gone
Too late to quit
You’ll have to carry on

Look down, look down
Don’t look ‘em in the eye
Look down, look down,
You’re here until you die