Skip to content

cURL’ing to FeatureServer from PostGIS: Easier then I Thought

So I’ve finished cutting a draft tileset using mapnik, depicting bus routes in Bellingham, WA. Now that the cartography is well in progress, I’d like to add some interactivity to the map. My first attempt at this will be to utilize MetaCarta (Chris Schmidt)’s FeatureServer. FeatureServer allows one to use standard HTTP verbs to GET representations of data, POST new data, or DELETE existing data. While querying data you can also pass additional URL parameters like a bounding box or attribute to select out a smaller subset of returned representations. I’ll be POST’ing a bus stop dataset to FeatureServer as GeoJSON. Once the data are stored in FeatureServer, I’ll be able to add popups based on a user’s click of a bus stop.

Getting data stored on my local PostGIS install to my remote FeatureServer instance turned out to be a three step process.

Step One: Convert local PostGIS bus stops layer to GeoJSON via OGR

I had originally planned on writing a pg/plsql function to try and output a bash script. The script would cURL each feature individually to my FeatureServer instance. This proved to be way more work then I had expected. What was the solution? OGR, of course. OGR has read/write drivers for both GeoJSON and PostGIS. This allows one to convert an entire dataset to GeoJSON with a single command (see below).

ogr2ogr -f "GeoJSON" ogrstops.json PG:"host=localhost dbname=routing user=postgres password=*** port=5432" "wtastops(the_geom)"

Step 2: Wrap “coordinate” elements in double brackets

When initially trying to cURL the GeoJSON output to FeatureServer, I was receiving an error stating that a bounding box could not be determined for the first geometry in my dataset. After some trial-and-error, I soon realized that the OGR output FeatureCollection was wrapping each point feature’s geometry in a single set of brackets. This type of behavior follows the GeoJSON specification for a FeatureCollection, as far as I can tell. However, in order for FeatureServer to consume this dataset, each point feature is required to be wrapped in a second set of brackets. I used gedit to run the find/replace. Below is an example of a GeoJSON feature which FeatureServer can consume. This individual feature is part of a larger FeatureCollection.


{ "type": "Feature",
          "properties": {
             "POINT_ID": "1000",
             "POINT_NAME": "Fielding at 32nd",
             "SHELTER": "Yes", "BENCH": "No" },
          "geometry": {
             "type": "Point",
             "coordinates": [[-122.474490,48.730021]]}
}

Step 3: cURL GeoJSON to FeatureServer

The last step is to actually POST the data to FeatureServer. For that, I used cURL.


curl -d @ogrstops.json http://mkgeomatics.com/cgi-bin/featureserver/featureserver.cgi/scribble/create.json

Now that the features have been uploaded, we can view them via FeatureServer as GeoRSS, KML, JSON, GML. Neat!

PL/pgSQL function to Iterate pgRouting

I’ve been working on a side project using pgRouting to determine the least-cost path across a street network from a given building to the nearest bus stop within Bellingham, WA. It’s one thing to execute pgRouting’s built-in functions for a single vertex (building) to another vertex (bus stop)… but another to have the function iterate through all buildings and their closest bus stop.

So that began my first experience with using PL/pgSQL. The benefit for using the procedural language for PostgreSQL lies in its ability to loop through collections of records easily. I’ve posted my function below. It’s not pretty, but it’s filled with enough notices to let me know where an error occurs, which helped me understand how things were acting each step of the way. Here is the basic idea:

  • Loop through a table in which each row has a source and destination vertex
  • Execute the pgRouting function using these two vertices, determining the length of the least-cost path.
  • Populate a field, ‘dist_calc’ with the distance.
CREATE OR REPLACE FUNCTION bulk_route_generate() RETURNS VOID AS $$
DECLARE
 bld_row bld_wtastops_staging5%ROWTYPE;
 dist_calc RECORD;
BEGIN
 RAISE NOTICE 'Beginning Function';
 FOR bld_row IN SELECT * FROM bld_wtastops_staging5 WHERE bld_wtastops_staging5.bld_vert_node IS NOT NULL
 AND bld_wtastops_staging5.wtastops_vert_node IS NOT NULL
 -- BEGIN ADDING BUM NODES TO SKIP OVER
 AND bld_wtastops_staging5.bld_vert_node <> 2915
 AND bld_wtastops_staging5.wtastops_vert_node <> 293
 -- ADD START GID
 -- USED ONLY IF BUM NODES EXIST
 --AND bld_wtastops_staging5.bld_gid >= 29200
 ORDER BY bld_wtastops_staging5.bld_gid LOOP
 RAISE NOTICE 'Value of wtastops_vert_node is %. The value of bld_vert_node is %',bld_row.wtastops_vert_node, bld_row.bld_vert_node;
 RAISE NOTICE 'Value of wtastops_gid is %. The value of bld_gid is %',bld_row.wtastops_gid, bld_row.bld_gid;
 -- BEGIN STANDARD pgRouting A*Star FUNCTION
 SELECT SUM(cost) INTO dist_calc FROM shortest_path_astar('
 SELECT gid as id,
 source::integer,
 target::integer,
 length::double precision as cost,
 x1, y1, x2, y2
 FROM streets_9102748',
 bld_row.bld_vert_node, bld_row.wtastops_vert_node, false, false);
 RAISE NOTICE 'Value of dist_calc is %.',dist_calc;
 EXECUTE 'UPDATE bld_wtastops_staging5
 SET route_dist = ' ||dist_calc|| '
 WHERE ' ||bld_row.bld_gid|| ' = bld_wtastops_staging5.bld_gid';
 END LOOP;
 -- BAIL OUT ON ERRORS
 EXCEPTION
 WHEN CONTAINING_SQL_NOT_PERMITTED THEN
 RAISE NOTICE ' EXECPTION Value of wtastops_vert_node is %. The value of bld_vert_node is %',bld_row.wtastops_vert_node, bld_row.bld_vert_node;
END;
$$ LANGUAGE 'plpgsql';
-- EXECUTE FUNCTION
SELECT bulk_route_generate();

I’m excited at the possibilities that using PL/pgSQL offers in terms of manipulating data. I’m sure that the above function can be cleaned up quite a bit, too. If I ever have the need to re-visit this or similar problems, I’ll be sure to do some serious head-scratching to think about a better approach!

Here is an image of the resulting data generated using mapnik. Areas from dark green to light-green are within a 1/4 mile distance, while areas from yellow-to-red represent distances increasingly greater then a 1/4 mile. The large checkered areas are where the dataset failed to route. More on that at another time.

bld_sym_diverging_web

the result as seen in mapnik

pgRouting III: PHP + OpenLayers Interface

With the routing database configured and populated, and with geoserver rendering the WMS, now the focus can shift on designing the actual display and functionality.

The conceptual plan is as follows:

  • Extract the geometry of a user’s click on the map.
  • Pass the extracted geometry to a PHP script, via an HTTP GET request.
  • Use the PHP script to pass the geometry as part of an SQL query against the PostGIS/pgRouting database.
  • Return the geometry from the database as GeoJSON, and deserialize it into an OpenLayers vector layer feature.

The code to extract a user’s clicked coordinates was taken from this OpenLayers example. It was then modified to pass the xy coordinates to a second function, designed to create a URL which will execute a PHP script.

trigger: function(e) {
 var xy = map.getLonLatFromViewPortPx(e.xy);
 executeSQL(xy);
 }

Passing the XY variable to the executeSQL() function, we are able to now seperate out the individual X and Y coordinates, and apply them to their respective parameters in our URL string.

// Build the URL
 var json_url = "http://localhost/near_vertex_astar.php?";
 json_url += "x=" + escape(xy.lon);
 json_url += "&y=" + escape(xy.lat);

Having constructed the URL, we are now ready to use it to populate an OpenLayers vector layer with data.

// Make a fresh vector layer, pulling features from our script URL
 json_layer = new OpenLayers.Layer.Vector("GeoJSON", {
 styleMap: myStyles,
 strategies: [new OpenLayers.Strategy.Fixed()],
 protocol: new OpenLayers.Protocol.HTTP({
 url: json_url,
 format: new OpenLayers.Format.GeoJSON()
 })
 });

Alright! So where are we at right now? A user has clicked the map, and that click’s geometry has been extracted and sent to a PHP script on the server for further work. The PHP script will execute SQL in the PostGIS/pgRouting data base to do the following:

  • Find the closest vertex in our routing network to the user’s map click. This will be used as a source vertex.
  • Find all firestations within 5km of the vertex (which have been pre-attributed with the closest vertex on the routing network to their location).
  • Calculate the cost (as defined by total length of the route) from the source vertex to each fire station (really the routing network vertex).
  • Return back as GeoJSON only the geometry for the route with the lowest cost.

Why all the hassle with determining the cost? Can’t you just use PostGIS’ ST_DWithin() function to find the closet firestation to our user’s click and create the route? Well you could, but it might not always be the shortest route.

Euclidean distance versus Manhattan. Which one is shorter?

Euclidean distance versus Manhattan. Which one is shorter?

This behavior can be respresented in the routing network with the example below. Two different routes are generated from the same source vertex based on the combination of routing algorithm and account for route cost. On the left, the dijkstra algorithm is used to return the route to the closest fire station as the result of an ST_DWithin() query. On the right, the A-Star algorithm is used, and the route costs of all fire stations within a buffer are taken into account. As we can see, a different route and a different station are returned.

Comparing the two search algorithms and cost relationships.

A link to the JS and PHP scripts can be found at the end of this post. This definitely is not the most elegant solution to working with routing, but in terms of an experiment it was a great learning exercise. I’m really excited to dive deeper into PostGIS and pgRouting. The next step in the process will be incorporating OSM data, and adding in addition attributes which affect cost (speed limits, one-way streets, etc).

View the PHP.

View the OL JS.

pgRouting Part II: PostGIS + Geoserver

Since compiling Orkney’s pgRouting extension to PostgreSQL/PostGIS, I’ve decided to try my hand at creating a simple web interface to poke into the database. The current setup is as follows:

  • Display: OpenLayers
  • Renderer: Geoserver (via non-cached WMS)
  • Spatial Backend: PostGIS/pgRouting enabled PostgreSQL
  • Data: Public GIS data from the city of Bellingham, Washington’s GIS department.

For the sake of brevity, (but really because both TOPP has created some fantastic guides) I won’t go into the specifics of installing all the pieces. Just as an FYI, remember to set your ‘JAVA_HOME’ environment variable and make sure that you don’t have things trying to use the same port!

The Bellingham data is currently stored in NAD83 State Plane WA North Feet, a typical projection for this area. This projection however, is not part of the EPSG projection set, and as such is not included in a vanilla install of PostGIS.

In order to add this to the collection of spatial reference systems used by my PostGIS install, I went with the ridiculously cool spatialreference.org site (A crschmidt, dbsgeo, hobu, and umbrella joint, hah). Navigating to the projection’s page gives me the option to generate an INSERT statement, adding the projection’s info into my database.

To load shapefiles into the PostGIS database, I chose to use the SPIT plugin for QGIS. Loading the data was fairly straightforward. I had an issue with a datefield that was present in the source shapefile, and had to delete the column manually using Open Office Database. I haven’t found a way to delete fields from a shapefile using QGIS.

spit

The SPIT Interface

After uploading the streets data into my PostGIS database, the next step was to transform the geometry into the Web Mercator 900913 Projection. This was done using standard PostGIS functions, adding a new, second, geometry column to the existing streets table. This reprojected data was then exported from my staging PostGIS database as a shapefile using the QGIS, ‘Save As Shapefile’ tool, and re-imported into my production database (with the routing functions).

With data stored in the web mercator projection, inside of our PostGIS/pgRouting database, the next step was to add the layers to Geoserver. Using Geoserver 2.x, the process included the following steps (all done through the web-admin).

  • Add the new data store pointing the PostGIS database.
  • Add new layers (resources) which point to the tables of interest in our PostGIS database.

After creating the connections between PostGIS and Geoserver, the creation of WMS services is taken care of, allowing us to roll them into OpenLayers with relative ease.

I guess this got a little off-topic from what I originally wanted to write about. I think that I’ll save the actual breakdown of my OL code (taking a user’s map click to and using it to calculate a route to the nearest fire-station as determined by manhattan distance, as opposed to euclidean distance) for another day.

pgRouting On Ubuntu Netbook Remix 9.10

While working through Regina Obe and Leo Hsu’s PostGIS In Action I thought that I’d jump into the world of routing. My plan was to develop a sample application that could be used to plan bicycle routes throughout the city of Seattle. A quick google search proved that someone has already done it, and done it very well! http://www.ridethecity.com/ provides cycling routes using OSM data for many major cities, Seattle included.

Undeterred and inspired, i decided to compile the pgRouting set of tools for PostGIS and give them a whirl.

My primary tutorial for moving through the install and execution of functions came from the 2009 FOSS4G Tokyo & Osaka workshop entitled, “FOSS4G routing with pgRouting tools and OpenStreetMap road data.” Although my installation on Ubuntu Netbook Remix (UNR) 9.10 required a little different setup, this guide definitely got me 99% of the way there.

The majority of my installation woes were caused by the different pathways used on my UNR install of PostgreSQL vs. what are apparently the standard paths.

After attempting to execute cmake to compile pgRouting, I’d be presented with an error stating that the ‘POSTGRESQL_INCLUDE_DIR’ was not found. A locate command pointed me to the correct path for my PostgreSQL installation. By modifying the FindPostgreSQL.cmake file to search for the correct path, I was back in business.

Following the workshop instructions, I then attempted to create the database directly from the terminal, which yielded the following result.

matt@matt-netbook:~$ createdb -U postgres routing
createdb: could not connect to database postgres: could not connect to server: No such file or directory
 Is the server running locally and accepting
 connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?

After reading the documentation associated with “createdb”, i tried adding the “-h” flag pointing to “localhost”, which solved the problem.

The final error which I ran into had to do with the “$libdir” environment variable. While trying to register the pgRouting functions in my new database, I’d be presented with the following:

psql:/usr/share/postlbs/routing_core.sql:32: ERROR:  could not access file "$libdir/librouting": No such file or directory
psql:/usr/share/postlbs/routing_core.sql:43: ERROR:  could not access file "$libdir/librouting": No such file or directory
psql:/usr/share/postlbs/routing_core.sql:53: ERROR:  could not access file "$libdir/librouting": No such file or directory

Getting impatient at this point (i wanted to route!) I modified the SQL files to reference the explicit path of my PostgreSQL lib directory. Once that was done, I had a working routing database!

Loading the sample data, creating the indexes, and executing the queries was amazingly straightforward. To test visualizing the data, I exported one of the tutorial queries directly into a new table.

SELECT * INTO export
 FROM dijkstra_sp('ways', 10, 20);
qgis_routing

The route depicted in red as seen in QGIS.

Just for kicks, I tried exporting the data as GeoJSON and visualzing it via OpenLayers.

The following SQL query aggregates the exported line segments into a single GeoJSON object:

SELECT ST_AsGeoJSON(ST_UNION(the_geom)) AS geom_union
FROM export;

Using the vector-formats OL example, which displays GeoJSON in either EPSG 4326 or 102113, I was able to visualize the line segment with no problem.

GeoJSON representation of line segment generated using pgRouting, displayed in OpenLayers

Well that’s all for one day. So it looks like the bike riding app is out, but I’m sure that there will be many more interesting ideas for pgRouting that will come to mind as I continue to explore PostGIS.

ESRI UC Student Assistant, Sweet!

Just found out that I’ll be participating in the 2009 ESRI UC as a student assistant. Big thanks go out to Dr. Robert Balling and James Fee, who both wrote letters of recommendation for me.

Now to get back to more pressing matters, like refactoring this giant wad o’ javascript.

See you in San Deigo!

Visualizing An Existing MySQL Database

So I’ve been working for about a month with a fairly-normalized (53-table) database in which I draw out all kinds of tabular information, and display it in a spatial context. This has required the numerous multiple table joins, with all kinds of weird relationships… you know, the kind that usually don’t work out very well?

In any event, my SOP for handling these queries was to submit sample data through the codeigniter site that our project’s web developer has been courageously firing away at. In this sense, I’d sort-of trace the flow of new information through the various tables of the database, monitoring the information stream as best as I could. I thought to myself, that there has to be a better way to handle this stuff! In comes the MySQL Workbench. This handy tool from the MySQL Dev Zone apparently comes in two flavors: FOSS and commercial.

The free version served my visualization needs perfectly. The layout of the program is very solid. I was easily able to take an SQL export of the existing database and import it into the Workbench, through a tool they call ‘Reverse Engineer MySQL Create Script’. Once the schema has been injected into the program, a model can be automatically created containing all of the tables as well as relationships. The auto-layout feature however, leaves a lot to be desired.

Above: Automatic Layout Results, Snazzy!

Above: Automatic Layout Results, Snazzy!

After about twenty-minutes of fooling with the table graphics, a usable layout can be produced. One feature that I think is really convenient, but will never use, is the automatic setting of the diagram width and height based on numbers of pages. This is useful for those who need a quick-print out of their database for whatever reason.

Above: Workbench w/ Completed Diagram

Above: Workbench w/ Completed Diagram

The real benefit for me however, is the automatic highlighting of key values linking tables together. I’m now able to quickly work my way from the table I need to get to, drilling backwards until I see the table I need to start with.

Above: Automatically Highlight The Key Fields Between Two Tables.

Above: Automatically Highlight The Key Fields Between Two Tables.

MetaCarta’s Map Rectifier + ESRI DevSummit Mashup Winner :)

I never knew about the MetaCarta Labs’ Map Rectifier tool, but I’ll expect to be using it more in the future. After uploading an image to the site, a user has full control over the creation of Ground Control Points. The advanced nature of this tool is shown though included RMS error reporting as well as the choice between multiple transformation algorithms. In addition to uploading your own content, a user has the ability to add GCPs for other users’ uploads as well.

Above: The MetaCarta Interface

Above: The MetaCarta Interface

What’s really amazing is the ability to directly access rectified images via WMS overlay. Each image hosted on the site is given a unique URL, we can insert into our favorite web mapping clients.

To try it out, I used the ExtMap – Mashup Framework developed by ArcGIS user alperdincer. This particular application framework was one of the winners at 2009 ESRI DevSummit, with good reason. I was able to quickly pass in the MetaCarta Labs URL, allowing the ExtMap application to consume and display the WMS layer with ease.

Above: Adding a Service to ExtMap

Above: Adding a Service to ExtMap

In addition to WMS layers, we can add in KML/GeoRSS as well as ArcGIS Server Dynamic/Tiled Map Layers.

Above: ExtMap Interface

Above: ExtMap Interface

Easy as pie? Piece of Cake? Yes. It’s innovative projects like these that keep pushing me to learn more about web mapping technology. Big thanks go out to crschmidt (who i assumed was involved w/ the project) at MetaCarta and alperdincer for putting together two great products.

On a final note, the MetaCarta Rectifier has the ability to export out images as geotiffs, allowing us to consume them in our desktop GIS applications. A quick check in ArcCatalog of the Chernobyl sample image I exported out revealed a WGS84 GCS. I can see some really nice workflows combining this tool with tiling programs such as GDAL2Tiles for painless TMS creation.

CloudMade Tile Request Graphics

I just found a neat feature from CloudMade, a heat map showing intensity of their tile requests at each zoom scale. As can be expected, Europe and North America are the definite zones of high activity. It’s also interesting to note the high activity in other regions such as Chile and the Philippines.

Above: CloudMade's Tile Request Overlay

Above: CloudMade's Tile Request Graphic

Following the link to the stats JavaScript, it looks like they are using a custom OpenLayers layer class, OpenLayers.Layer.Cloudmade. Sweet!

SpatialLite: My First Look

With such a small footprint (a single file) SpatialLite appears to a novice like myself to be a fantastic niche storage solution for spatial data. In an environment where installing larger FOSS databases such as MySQL or PostGIS/PostgreSQL can be prohibitive, Spatial Lite appears to provide a great solution. Using the provided GUI interface, it’s extremely easy for a first-time user to create an sqllite dbase, load multiple shapefiles, and create spatial indexes against them. Analogous to a FOSSGIS ESRI Geodatabase, I can see a lot of potential uses for GIS developers who require the indexing and searching power of a database as well as the portability of formats such as an ESRI Shapefile or KML.

It looks like a QGIS connection is in the works for the 1.1 release as well.

I’m not sure if it’s already being done, but I’d bet that it would be pretty easy to put together a SpatialLite / FeatureServer combination, considering its native support for so many spatial-backends.

Some tutorials I’ve found to be very helpful have come from: BostonGIS and the SpatialLite project site.