Skip to content

ArcGIS REST API / OpenLayers / Unit Testing = Fun In The Sun

Until today, I had never truly appreciated the value of unit testing. I recently had the need to bring these ArcGIS REST controls, designed for version 2.6 of OpenLayers, into the current development version. Having no real idea how to get started on this process, I looked to the unit tests as a guide to what needed to be changed. One might be asking why this was necessary, when the team over at Avencia just put together a great ArcGIS REST Plugin that has made its way into the trunk for the upcoming 2.8 release. The answer is that both plugins do different things well. The older AGSControls can perform ‘Identify’ and ‘Geoprocessing’ operations rather well, while the Avencia plugin does a great job at displaying and querying a subset of a layer resource.

In any event, the Test.AnotherWay suite, used by OpenLayers, provides an easy-to-navigate interface for debugging javascript code.

In two steps I was able to begin the debugging process.

First, adding the unit test for the AGSControl to the ‘list-tests.html’ file located in the ‘tests’ folder of a development version of OpenLayers. This unit test, written by the developer, needs to manually downloaded and incorporated into the standard series of tests that come with OpenLayers. As we can see from the image below, this particular test was written as an html file and placed into the ‘Control’ sub-directory of the ‘tests’ directory.

Adding a link to the unit test for use by Test.AnotherWay

Above: Adding a link to the unit test for use by Test.AnotherWay

After adding the test, we open up ‘run-tests.html’ in the browser and select ‘AgsControl’ from the list. After the test has executed, the results are provided to us. With the red light of failure burning bright, we might think to abandon all hope. We are, however, given the cause and location of the failure, an invaluable clue as to where to start debugging. Time to soldier on.

Above: Unit Test Failure

Above: Unit Test Failure

Using these test results as a road map, even I can eventually debug a plugin. The green light of success offers a reassuring reminder that all is well in the GIS world.

Above: Successful Unit Test

Above: Successful Unit Test

I’ve taken away a few things from this experience. Firstly, I’m again deeply impressed by the time and effort that developers in the Open Source community are putting into these projects. The only reason that I could even dream of modifying any of this source code is due to the fact that the developer of the AGSControls provided such detailed unit tests. These allowed me to wrap my brain around what was going on with the code, and how it could be updated. Taking the time to not only write code, but to also provide tools so that others can understand it and modify it with ease is something that I think I’ll always be impressed with. And of course, I’ll be continuing to rely on unit tests as debugging tools as i continue my exploration of javascript programming.

In the words of Dave Bouwman, who has a whopping fourteen posts in his tag cloud on the subject:

Unit testing is quite possibly the single best practice for ensuring that your code is bug free (or very nearly bug free!).

His ‘fundamentals’ article provides a great introduction on the subject: which can be read here.

Google Motion Charts: From Data To Information

I’ve been playing with the motion chart gadget in Google’s AJAX API Playground recently, and have found it to be an extremely interesting tool to use. In order to produce data rich charts, all that is necessary is to import a tabular datasheet into google docs and call on it within a javascript function.

The spreadsheet itself requires:

  • The first column be entity data (place names, events, etc.)
  • The second column to be formatted as ‘Date’
  • The third and following columns to contain your chart-worthy data.

A simple example can be found below depicting historical weather station information for three cities in the Western United States. This information was downloaded from the Western Regional Climate Center and imported manually into a google docs spreadsheet. As we can see, we have a variety of different ways to explore the data, and hopefully, synthesize them into useful information for decision makers in a best case scenario.


Above: Markers indicate the cities in which weather station data was collected.


Above: Google Motion Chart Gadget. NOTE: The data are actually compiled monthly averages from 1971-2000. For the sake of simplicity however, i fed the date column in a format of ‘Jan-1′ which turns out automatically appends the current year. The proper date should read for example, ‘1/1′ rather than ‘1/1/09′.

In any event, in a data rich GIS environment, we can easily succumb to ‘data overload’. This creates a barrier for decision makers in understanding their situations, and as such can lead to poor planning. Through the careful use of data visualization tools such as this, however, decision makers can be empowered to quickly analyze data in a variety of ways, creating useful information on-the-fly and without the use of specialized GIS technical know-how.

At least I think that’s how it works.

OSM ‘Minutely’ Tile Updates: Thanks CloudMade!

Another great OSM Mapping Party is underway in Tempe Arizona. Brandon of CloudMade came out to run the show, with local hosting by James Fee of RSP Architects. Coffee, pizza, libations, and Open Source GIS provided a great foundation for lively discussion on the practical and philosophical aspects that OpenStreetMaps provides.

Of particular interest was a question posed by a first time OSM’er, who wondered why we had to wait a week for the tiles to render, just to see the results of some type of experiment that we might be trying out in the digitization and attribution of features in the map. Well, no one could really give him a good answer other then, ‘it’s just the way it is’.

It appears though, that CloudMade has provided us with an answer just the other day. The Minutely Updated Tile Server is updated ‘every minute from OpenStreetMap diffs’. The results can be seen below. The first image depicts the standard weekly update view while the bottom image depicts the minutely render. Note the presence of the additional buildings on ASU’s Tempe Campus.

Above: OSM Weekly Update

Above: OSM Weekly Update

Above: CloudMade's Minutely Updated Tile Server for OSM

Above: CloudMade's Minutely Updated Tile Server for OSM

Check out the full post from CloudMade’s blog here: http://blog.cloudmade.com/2009/01/23/nearly-live-tiles/

Geocoding with OpenLayers: A Crash Course In Firebug

The vacation is over. A new job and a new semester are already providing plenty of opportunities to explore those crazy technologies of the geoweb.

A need to incorporate the Yahoo Geocoder into a new OpenLayers app has proved to be a great learning experience in the navigation of the development version of OL. The YahooGeocoder.js addin, created by OL community member, ’sbenthall’, requires two prerequisites to run: A proxy.cgi script as well as a Yahoo Developer API key. Why would this RESTful service require a proxy, you might find yourself asking? Well, because even though you can query it in a RESTful fashion, the data is returned in an XML shell, requiring a proxy to allow complete the XMLHttpRequest. Yahoo has a great article for novice web programmers like myself explaining the role of  Web Proxies, which can be found here.

A quick overview of the primary steps to add the YahooGeocoder.js addin are as follows:

  1. Sign up for a Yahoo APP Key to enable access to their geocoding service.
  2. Add the proxy.cgi script to your webserver’s ‘cgi-bin’. Note: When navigating to the proxy.cgi’s url, you might encounter, ‘access denied’ errors. If you do, make sure that you have the proper permissions set for your cgi-bin directory. This can be done using the terminal command, ‘chmod 755′ targeting cgi-bin directory.
  3. Edit the proxy.cgi script to include ‘local.yahooapis.com’ in the list of ‘allowedHosts’.
    proxy_hosts

    Above: Modified 'proxy.cgi'

  4. Add the YahooGeocoder.js file to the OpenLayers ‘lib/OpenLayers/Control’ folder.
  5. Add “OpenLayers/Control/YahooGeocoder.js” to the variable array, ‘jsfiles’ inside the “lib/OpenLayers.js” library.

    control_add

    Above: Modified 'lib/OpenLayers.js'

  6. Test the geocoder’s functionality using the supplied .HTML file. (Hopefully it should work!)

    good_xml_return1

    Above: Geocoder Result with Properly Formed XML Response.

Six simple steps, but it can be challenging if you haven’t tried to install any addins to the OL library before.

In the above image, the firebug window can be seen returning a properly formed XML Response, having successfully executed the geocoding function. If you enlarge the image, we can compare this to the raw XML-Response using a properly constructed query. Note in both the response captured from firebug (above) as well as the raw XML (below) the presence of the address: 510 High Street, Bellingham WA, broken down into it’s individual units along with the geocoded result.

basic_xml

Above: Basic XML Return

Further diving into the capabilities of firebug, we can use the DOM inspector to ensure that the various parameters required to properly execute the Yahoo Geocoder are in place. Note in the image below the presence of such necessary information as the APP ID Key, Projection, and Class, for the ygc variable. If any of these parameters happened to be incorrectly set, it would be displayed in this view.

dom_inspector

Above: The Firebug DOM Inspector

I’m finally starting to appreciate the power of firebug as a development tool, which just so happens to coincide with my ability to understand it at a basic level. Hopefully as my experience in GIS Web Development grows, so will be ability to use the higher-end functions of this tool.

TMS: Shaded Relief for OSM Data (An Adventure and A 1/2)

After reading Michal Migurski’s excellent post, ‘making friends with hillshading‘ I decided to try my own hand at producing a TMS-compatible hillshade layer for OSM. Motivated by both the high resolution LiDAR data which I happen to have access to, and the lack of access to ArcDesktop during the winter intersession… I set out on the FOSSGIS path.

The resulting overlay

The resulting overlay

The final output was created in a primarily two-step process. The hillshade was created using Matt Perry’s GDAL DEM tools. This command line addition to the GDAL suite of tools negated having to import the DEM into a new GRASS location, and exporting the resulting hillshade back out as another format.

The majority of the work, the tiling that is, was done using GDAL2Tiles. This Google Summer of Code project operates under a command line utility which will take your slick GDAL-compatible raster dataset as input, and export a set of tiles exactly to your specs. What’s more is the capability to automatically generate google maps and OpenLayers viewing pages which can be directly uploaded to a webspace.

Finally, applying a slight transparency to the OSM layer allowed for the shaded relief to appear from behind. Quick-and-Dirty, but it works!

The pain came when trying to compare the local TMS overlay using OSM vs. Google Maps base data. While using Google Maps street data, we clearly see the LiDAR building footprints align with the vector street centerlines. But switching over to OSM data, we see a drastic skew between the two.

Google Street Data: Properly Aligned Local TMS

Google Street Data: Properly Aligned Local TMS

OSM Data: Note Hawaii Belt Rd's location compared to its LiDAR footprint.

OSM Data: Note Hawaii Belt Road's location compared to its LiDAR footprint.

Initially, I assumed that the problem was the result of some projection issues (EPSG: 4326 vs EPSG: 900913), until I realized that all data sets (LiDAR, Google, and OSM) had matching coastlines. Any distortion resulting from the affine transformation between WGS84 and Spherical Mercator would have clearly showed up along the coast as well as the streets.

A quick look at the Potlatch editor revealed the inaccuracy of the TIGER line files used in this section of the map:

Potlatch: TIGER data is a good, but not great.

Potlatch: TIGER data is a good, but not great.

All-in-all I was extremely impressed with the speed and efficency of using the GDAL DEM Tools as well as GDAL2Tiles. Both projects are great representations of what can be done by individuals who choose to build upon existing FOSSGIS projects, and give back to the community.

Additional Note:
A GRASS-based approach to creating a TMS layer is possible, and has been outlined in this paper appearing in the OSGeo Journal.

OSM Recap

What an awesome weekend. There was a good turnout for the Phoenix OSM mapping party, put on by CloudMade and hosted locally by Gangplank. On both days we saw a mixture of GIS professionals and Open Source users come out to support the project. A new neologism to add to the list also came out of the event, ‘crowdsourcing’. The work of many locals who have an inherent expert knowledge of their surrounding environment replacing the work of what would otherwise be performed by a few skilled professionals.

Here is the result of our work:

ASU Mapping Party

Above: ASU Mapping Party

In any event, we decided to focus our efforts on ASU’s campus for the weekend. Before we went out with GPS units, an OSM’er named ‘jfuredy’ had actually taken a stab at some rooftop digitization of buildings along campus. I’ve already noticed that there are inherent conflicts that occur by digitizing from rooftops. Tall buildings of course, will appear oblique, with their roofs offset from their actual footprints. This creates scenarios in which buildings (created from oblique angled rooftops) are overlapping walkways.

Above: Digitization Conflict

Above: Digitization Conflict

I’m excited to continue working on the OSM project, and hope to see more mapping parties in the future. Speaking with Brandon, our contact at CloudMade, I mentioned that the end of January could be a potential date. This would be the first week of the new semester for MAS-GIS, and hopefully more of the students will come out and support the project!

OpenStreetMap.org Mapping Party This Weekend

http://www.meetup.com/OpenStreetMap-Phoenix/ It’s going to rock. It really is.

Having just heard of OSM a little over a month ago, and using their tiles for about two weeks now… I’m really excited to give back to the OSM community. In a way, it’s a chance for those who have never committed code to a FOSS project (like me) to really add some positive input back into a community that we are a part of, if only in a small way. But hey, if we are all contributing small bits to the project-at-large… it’ll add up. A little ‘neogeography‘, anyone?

Additionally, I have put together a small tool to compare the current OSM offerings to that of various commercial mapping vendors (Google, Microsoft VE). Pretty fun exploration of the OpenLayers API!

Check it out here: http://mkgeomatics.com/apps/openlayers/osm_compare.html

Above: Quick and Dirty OSM Tool

Above: Quick and Dirty OSM Tool

Stefan Steinger – FOSS Desktop GIS Overview

Found via the Cascadia Users of Geospatial Open Source (CUGOS) website.

Very recent paper accepted by the International Journal of Geographical Information Science depitcs the current state of the art in FOSS Desktop GIS and much more. The staying power in the article however, IMHO, is the detailed explanation of the various FOSS license agreements which exist and the organisations which play a major role in the development of FOSSGIS as a whole. One of the most interesting tidbits for me was the realization that the Open Geospatial Consortium cannot be equated directly to the FOSSGIS movement.

A couple of people to which we spoke related the activities of  the  Open  Geospatial  Consortium  (OGC,  www.opengeospatial.org)  to  open  source  GIS software.  This  link  is  not entirely  correct,  since  the OGC  is  an  organisation  that  develops standards  for  the processing  and  exchange  of  geo-data  between  different  GIS  platforms. Furthermore a large portion of the member organisations of the OGC, besides universities and authorities, are companies that develop proprietary GIS and Databases.

Link: http://cugos.googlegroups.com/web/sstein_foss_desktop_gis_overview.pdf

Spherical Mercator / New Portfolio

In the continuing adventures of my WFS example, I’ve now added spherical mercator capabilities to the PROJ.4 library, following those instructions Chris Schmidt gave me last week. On the fly reprojection of the data allows for the incorporation of numerous proprietary vendors, such as Google Maps and Microsoft Virtual Earth, as well as open source tile services such as OpenStreetMaps.org.

Above: WFS Spherical Mercator Reprojection with Google Terrain Base Layer

Above: WFS Spherical Mercator Reprojection with Google Terrain Base Layer

Feel free to check it out: http://mkgeomatics.com/apps/openlayers/spherical_merc.html

In other news, I’m also nearly finished moving my portfolio from WWU’s servers to this site. One can click on the Portfolio link to the right, or simply navigate directly to a sub-section if they choose.

ScapeToad – Open Source Cartogram Generator

ScapeToad provides an amazingly easy to use interface for creating cartograms. Accepting shapefiles directly as input, it has a wide variety of fine-tunable settings in addition to its presets. The ability to utilize multiple layers as weights also seems like a pretty exciting feature. Written entirely in java, it’s platform-independent, another plus for mac and / or linux GIS users.

WA_Green

Above: The counties of Washington State.

What is so cool about this program is that not only does it use shapefiles as an input, but it generates shapefiles as output as well. A SizeError attribute is also generated to allow a user to easily control map symbology using GIS software. In addition to this, one can also export as an SVG directly to use in a graphics editor.

The cartogram below was generated using the percentage of a county’s total area covered by state routes. The lighter colors indicate a smaller distortion in size while the darker colors indicate a larger distortion in size. Note the I-5 corridor in Western Washington is heavily affected using this criteria.

WA_Cartogram

Above: The result.

ScapeToad – http://chorogram.choros.ch/scapetoad/index.php

Data (WA DOT) – http://www.wsdot.wa.gov/