Tag Archives: GIS

Moving to GeoDjango

I’ve been creating a simple GeoDjango application for managing environmental sampling metadata, and it’s been a lot of fun so far. I’ve had experience working with many different forms of metadata tracking, from spreadsheets, to wikis, to online project management tools. All of them have their ups and downs, and it seems like there is always a dealbreaker with each organizational method.

Spreadsheets are easy to edit, but lack any form of relational structure (two sets of data for the same report? i guess i’ll just put two entries into the same cell).

sometimes spreadsheets just don't work...

Wikis are cool, allow easy access to information, but are (in certain cases) a pain for folks to edit. Take the experience of table creation. Dokuwiki, a generic wiki software, requires a series of  carefully placed carrots and pipes to delineate headers and columns. A common pain comes when adding a new value to a new row, in which that value exceeds the length of any previous cell. This requires the author to expand the column header, and all previously entered rows, re-aligning the pipes and carrots. Granted as a slightly OCD GIS Analyst, the sight of a well crafted text table fills me with no less wonder then when I saw Albert Bierstadt’s “Puget Sound on the Pacific Coast”, but it’s just darn tedious at times. Additionally, as the log of sampling events grows larger, it gets harder to manage. Dokuwiki, AFAIK provides no ways to automatically resort entire sections of pages or records in tables based on alphabetical order, which would make searching for information on a particular page much faster as content becomes larger and larger.

^ Column One                ^ Column Two                 ^
| Zee 'z' string            | This is a longer string    |
| 2nd longer string of text | 2nd short string           |
| SuperShort                | A string starting with 'A' |

Online project management tools are interesting as well. They allow rapid collaboration between project members, and provide template functionality, allowing for status reports on recurring workflows to be easily generated (e.g., create a template for a report, spawn off an instance of a template for each new project). The downside to these services are that: they cost money, they also may not provide a normalized way to store data, and (of most interest to myself) they probably don’t allow for the storage/visualization of spatial data.

10 megs free for one user? cool!

In comes GeoDjango. Over the next few posts, I think I’ll record my experiences developing an application that allows the storage of metadata, within the context of environmental sampling efforts. The goal is to provide a web application which stores both tabular and spatial data in a normalized fashion, and easily visualize both in an informative way.

Hey look Ma', it's GeoJSON!

cURL’ing to FeatureServer from PostGIS: Easier then I Thought

So I’ve finished cutting a draft tileset using mapnik, depicting bus routes in Bellingham, WA. Now that the cartography is well in progress, I’d like to add some interactivity to the map. My first attempt at this will be to utilize MetaCarta (Chris Schmidt)’s FeatureServer. FeatureServer allows one to use standard HTTP verbs to GET representations of data, POST new data, or DELETE existing data. While querying data you can also pass additional URL parameters like a bounding box or attribute to select out a smaller subset of returned representations. I’ll be POST’ing a bus stop dataset to FeatureServer as GeoJSON. Once the data are stored in FeatureServer, I’ll be able to add popups based on a user’s click of a bus stop.

Getting data stored on my local PostGIS install to my remote FeatureServer instance turned out to be a three step process.

Step One: Convert local PostGIS bus stops layer to GeoJSON via OGR

I had originally planned on writing a pg/plsql function to try and output a bash script. The script would cURL each feature individually to my FeatureServer instance. This proved to be way more work then I had expected. What was the solution? OGR, of course. OGR has read/write drivers for both GeoJSON and PostGIS. This allows one to convert an entire dataset to GeoJSON with a single command (see below).

ogr2ogr -f "GeoJSON" ogrstops.json PG:"host=localhost dbname=routing user=postgres password=*** port=5432" "wtastops(the_geom)"

Step 2: Wrap “coordinate” elements in double brackets

When initially trying to cURL the GeoJSON output to FeatureServer, I was receiving an error stating that a bounding box could not be determined for the first geometry in my dataset. After some trial-and-error, I soon realized that the OGR output FeatureCollection was wrapping each point feature’s geometry in a single set of brackets. This type of behavior follows the GeoJSON specification for a FeatureCollection, as far as I can tell. However, in order for FeatureServer to consume this dataset, each point feature is required to be wrapped in a second set of brackets. I used gedit to run the find/replace. Below is an example of a GeoJSON feature which FeatureServer can consume. This individual feature is part of a larger FeatureCollection.


{ "type": "Feature",
          "properties": {
             "POINT_ID": "1000",
             "POINT_NAME": "Fielding at 32nd",
             "SHELTER": "Yes", "BENCH": "No" },
          "geometry": {
             "type": "Point",
             "coordinates": [[-122.474490,48.730021]]}
}

Step 3: cURL GeoJSON to FeatureServer

The last step is to actually POST the data to FeatureServer. For that, I used cURL.


curl -d @ogrstops.json http://mkgeomatics.com/cgi-bin/featureserver/featureserver.cgi/scribble/create.json

Now that the features have been uploaded, we can view them via FeatureServer as GeoRSS, KML, JSON, GML. Neat!

MetaCarta’s Map Rectifier + ESRI DevSummit Mashup Winner :)

I never knew about the MetaCarta Labs’ Map Rectifier tool, but I’ll expect to be using it more in the future. After uploading an image to the site, a user has full control over the creation of Ground Control Points. The advanced nature of this tool is shown though included RMS error reporting as well as the choice between multiple transformation algorithms. In addition to uploading your own content, a user has the ability to add GCPs for other users’ uploads as well.

Above: The MetaCarta Interface

Above: The MetaCarta Interface

What’s really amazing is the ability to directly access rectified images via WMS overlay. Each image hosted on the site is given a unique URL, we can insert into our favorite web mapping clients.

To try it out, I used the ExtMap – Mashup Framework developed by ArcGIS user alperdincer. This particular application framework was one of the winners at 2009 ESRI DevSummit, with good reason. I was able to quickly pass in the MetaCarta Labs URL, allowing the ExtMap application to consume and display the WMS layer with ease.

Above: Adding a Service to ExtMap

Above: Adding a Service to ExtMap

In addition to WMS layers, we can add in KML/GeoRSS as well as ArcGIS Server Dynamic/Tiled Map Layers.

Above: ExtMap Interface

Above: ExtMap Interface

Easy as pie? Piece of Cake? Yes. It’s innovative projects like these that keep pushing me to learn more about web mapping technology. Big thanks go out to crschmidt (who i assumed was involved w/ the project) at MetaCarta and alperdincer for putting together two great products.

On a final note, the MetaCarta Rectifier has the ability to export out images as geotiffs, allowing us to consume them in our desktop GIS applications. A quick check in ArcCatalog of the Chernobyl sample image I exported out revealed a WGS84 GCS. I can see some really nice workflows combining this tool with tiling programs such as GDAL2Tiles for painless TMS creation.

SpatialLite: My First Look

With such a small footprint (a single file) SpatialLite appears to a novice like myself to be a fantastic niche storage solution for spatial data. In an environment where installing larger FOSS databases such as MySQL or PostGIS/PostgreSQL can be prohibitive, Spatial Lite appears to provide a great solution. Using the provided GUI interface, it’s extremely easy for a first-time user to create an sqllite dbase, load multiple shapefiles, and create spatial indexes against them. Analogous to a FOSSGIS ESRI Geodatabase, I can see a lot of potential uses for GIS developers who require the indexing and searching power of a database as well as the portability of formats such as an ESRI Shapefile or KML.

It looks like a QGIS connection is in the works for the 1.1 release as well.

I’m not sure if it’s already being done, but I’d bet that it would be pretty easy to put together a SpatialLite / FeatureServer combination, considering its native support for so many spatial-backends.

Some tutorials I’ve found to be very helpful have come from: BostonGIS and the SpatialLite project site.

OSM ‘Minutely’ Tile Updates: Thanks CloudMade!

Another great OSM Mapping Party is underway in Tempe Arizona. Brandon of CloudMade came out to run the show, with local hosting by James Fee of RSP Architects. Coffee, pizza, libations, and Open Source GIS provided a great foundation for lively discussion on the practical and philosophical aspects that OpenStreetMaps provides.

Of particular interest was a question posed by a first time OSM’er, who wondered why we had to wait a week for the tiles to render, just to see the results of some type of experiment that we might be trying out in the digitization and attribution of features in the map. Well, no one could really give him a good answer other then, ‘it’s just the way it is’.

It appears though, that CloudMade has provided us with an answer just the other day. The Minutely Updated Tile Server is updated ‘every minute from OpenStreetMap diffs’. The results can be seen below. The first image depicts the standard weekly update view while the bottom image depicts the minutely render. Note the presence of the additional buildings on ASU’s Tempe Campus.

Above: OSM Weekly Update

Above: OSM Weekly Update

Above: CloudMade's Minutely Updated Tile Server for OSM

Above: CloudMade's Minutely Updated Tile Server for OSM

Check out the full post from CloudMade’s blog here: http://blog.cloudmade.com/2009/01/23/nearly-live-tiles/

Geocoding with OpenLayers: A Crash Course In Firebug

The vacation is over. A new job and a new semester are already providing plenty of opportunities to explore those crazy technologies of the geoweb.

A need to incorporate the Yahoo Geocoder into a new OpenLayers app has proved to be a great learning experience in the navigation of the development version of OL. The YahooGeocoder.js addin, created by OL community member, ‘sbenthall’, requires two prerequisites to run: A proxy.cgi script as well as a Yahoo Developer API key. Why would this RESTful service require a proxy, you might find yourself asking? Well, because even though you can query it in a RESTful fashion, the data is returned in an XML shell, requiring a proxy to allow complete the XMLHttpRequest. Yahoo has a great article for novice web programmers like myself explaining the role of  Web Proxies, which can be found here.

A quick overview of the primary steps to add the YahooGeocoder.js addin are as follows:

  1. Sign up for a Yahoo APP Key to enable access to their geocoding service.
  2. Add the proxy.cgi script to your webserver’s ‘cgi-bin’. Note: When navigating to the proxy.cgi’s url, you might encounter, ‘access denied’ errors. If you do, make sure that you have the proper permissions set for your cgi-bin directory. This can be done using the terminal command, ‘chmod 755′ targeting cgi-bin directory.
  3. Edit the proxy.cgi script to include ‘local.yahooapis.com’ in the list of ‘allowedHosts’.

    proxy_hosts

    Above: Modified 'proxy.cgi'

  4. Add the YahooGeocoder.js file to the OpenLayers ‘lib/OpenLayers/Control’ folder.
  5. Add “OpenLayers/Control/YahooGeocoder.js” to the variable array, ‘jsfiles’ inside the “lib/OpenLayers.js” library.

    control_add

    Above: Modified 'lib/OpenLayers.js'

  6. Test the geocoder’s functionality using the supplied .HTML file. (Hopefully it should work!)

    good_xml_return1

    Above: Geocoder Result with Properly Formed XML Response.

Six simple steps, but it can be challenging if you haven’t tried to install any addins to the OL library before.

In the above image, the firebug window can be seen returning a properly formed XML Response, having successfully executed the geocoding function. If you enlarge the image, we can compare this to the raw XML-Response using a properly constructed query. Note in both the response captured from firebug (above) as well as the raw XML (below) the presence of the address: 510 High Street, Bellingham WA, broken down into it’s individual units along with the geocoded result.

basic_xml

Above: Basic XML Return

Further diving into the capabilities of firebug, we can use the DOM inspector to ensure that the various parameters required to properly execute the Yahoo Geocoder are in place. Note in the image below the presence of such necessary information as the APP ID Key, Projection, and Class, for the ygc variable. If any of these parameters happened to be incorrectly set, it would be displayed in this view.

dom_inspector

Above: The Firebug DOM Inspector

I’m finally starting to appreciate the power of firebug as a development tool, which just so happens to coincide with my ability to understand it at a basic level. Hopefully as my experience in GIS Web Development grows, so will be ability to use the higher-end functions of this tool.

OSM Recap

What an awesome weekend. There was a good turnout for the Phoenix OSM mapping party, put on by CloudMade and hosted locally by Gangplank. On both days we saw a mixture of GIS professionals and Open Source users come out to support the project. A new neologism to add to the list also came out of the event, ‘crowdsourcing’. The work of many locals who have an inherent expert knowledge of their surrounding environment replacing the work of what would otherwise be performed by a few skilled professionals.

Here is the result of our work:

ASU Mapping Party

Above: ASU Mapping Party

In any event, we decided to focus our efforts on ASU’s campus for the weekend. Before we went out with GPS units, an OSM’er named ‘jfuredy’ had actually taken a stab at some rooftop digitization of buildings along campus. I’ve already noticed that there are inherent conflicts that occur by digitizing from rooftops. Tall buildings of course, will appear oblique, with their roofs offset from their actual footprints. This creates scenarios in which buildings (created from oblique angled rooftops) are overlapping walkways.

Above: Digitization Conflict

Above: Digitization Conflict

I’m excited to continue working on the OSM project, and hope to see more mapping parties in the future. Speaking with Brandon, our contact at CloudMade, I mentioned that the end of January could be a potential date. This would be the first week of the new semester for MAS-GIS, and hopefully more of the students will come out and support the project!

ScapeToad – Open Source Cartogram Generator

ScapeToad provides an amazingly easy to use interface for creating cartograms. Accepting shapefiles directly as input, it has a wide variety of fine-tunable settings in addition to its presets. The ability to utilize multiple layers as weights also seems like a pretty exciting feature. Written entirely in java, it’s platform-independent, another plus for mac and / or linux GIS users.

WA_Green

Above: The counties of Washington State.

What is so cool about this program is that not only does it use shapefiles as an input, but it generates shapefiles as output as well. A SizeError attribute is also generated to allow a user to easily control map symbology using GIS software. In addition to this, one can also export as an SVG directly to use in a graphics editor.

The cartogram below was generated using the percentage of a county’s total area covered by state routes. The lighter colors indicate a smaller distortion in size while the darker colors indicate a larger distortion in size. Note the I-5 corridor in Western Washington is heavily affected using this criteria.

WA_Cartogram

Above: The result.

ScapeToad – http://chorogram.choros.ch/scapetoad/index.php

Data (WA DOT) – http://www.wsdot.wa.gov/

Attribution, CSS, and IRC

I’ve built up the WFS OpenLayers example I’ve been working on to now include attribution (by hovering over a feature) as well as a pointless select feature capability. Click away! It won’t do anything but, that’s fine.

A different baselayer is also available, openstreetmaps.org (via telascience.org tiles). You may notice that some of these tiles fail to reload upon panning and zooming operations. I asked why this was on the OpenLayers IRC, and was told that these tiles basically suck. I’m stuck using them however, as they are Lat/Long WGS84. OSM Mapnik and Google maps require a spherical mercator projection, and as such PROJ.4 needs to be amended to support this EPSG code so that Mapserver and render properly.

Screenshot of simple WFS example

Screenshot of simple WFS example

It looks pretty straightforward, but another project for another day.

In any event, I’ve kept the OpenLayers WMS as the primary base layer, which doesn’t have display issues.

whew…

This took a ‘lot’ longer than i expected.

http://www.mkgeomatics.com/apps/openlayers/WA_WFS_WGS84.html

Calling Mapserver as a WFS layer, displaying with OpenLayers. Many thanks to Chris Schmidt for a quick response on the OL mailing list!

Currently set to extract attributes in the OL html, but still need to build in the attribute display. A good example of the necessary javascript can be found in the OL Cookbook.

I also set a quick thank you to Jeff McKenna at Gateway Geomatics for putting together such a great set of UMN Mapserver WFS and WMS tutorials. He actually wrote back! oh FOSS4G, what a nice community.