I’ve just updated the SF Bay African Music Calendar with recently-announced concerts in the SF Jazz series. There are some great shows planned for this spring, including Tinariwen, Salif Keita, and Ladysmith Black Mambazo.
Here’s a sample of the desert blues of Tinariwen.
Great music, true to its desert roots. I’ve heard their songs in the most unexpected places lately, including the film The End of Poverty.
I’ve been interested in delving into the world of Open Source GIS, and the bike planner project is a good reason to learn more. I have a huge amount of appreciation for ESRI‘s software — I use it at work all the time, and it does many things extremely well. (The sheer number of commands and options can make it intimidating to new users.)
ESRI has been around seemingly forever (founded 1969), has a large and generous user community, conferences, customer support, annual conferences… and their software has a big price tag. They do have a grants program, and it’s great. I know many nonprofits who received ESRI software for free or at a big discount.
But choice is good. And there are increasingly more choices that are free and open source. It’s hard to keep tabs on what is out there, what the various packages do, and their strengths and limitations. I’ve compiled a few notes on a few of the more well-known open-source GIS programs below.
GDAL stands for Geospatial Data Abstraction Library. When you download it, you also get a copy of OGR, which stands for itself, OGR Simple Features Library. (Apparently, it used to stand for something else years ago…)
GDAL is a translator library for raster geospatial data formats that is released under an X/MIT style Open Source license by the Open Source Geospatial Foundation. As a library, it presents a single abstract data model to the calling application for all supported formats. It also comes with a variety of useful commandline utilities for data translation and processing. The NEWS page describes the November 2009 GDAL/OGR 1.6.3 release.
The related OGR library (which lives within the GDAL source tree) provides a similar capability for simple features vector data. I’ve used OGR in the past to convert GML files to shapefiles when I was experimenting with OpenStreetMap data, and it worked like a charm.
If you want to use these tools on Windows or Linux, you grab the package called FWTools, “a set of Open Source GIS binaries for Windows (win32) and Linux (x86 32bit) systems produced by Frank Warmerdam (FW).” With it you get a bunch of useful utilities for working with different GIS files. In my experience, these are useful *if* you already know what you are doing, and have a particular task in mind, such as: I need to re-project this satellite image from UTM coordinates into state plane coordinates so I can use it with the rest of my project data.
FWTools for Windows is distributed with Python 2.3.4, which is already quite old, released in May 27th, 2004. This makes me wonder how up-to-date this package is.
Also, a word of caution: If you have ESRI’s ArcGIS installed and use Python for scripting, do NOT upgrade or change your version of Python, or your ability to run ArcGIS tools via scripts will be broken.
GDAL looks especially useful to programmers, as there are bindings available in a number of different languages. My recommendation: If you are planning to write code to manipulate your geodata, look here first so you don’t have to reinvent the wheel!
GRASS (Geographic Resources Analysis Support System) is a Software for performing spatial analysis. It consists of more than 350 modules for processing vector (2D/3D), raster and vector data. Many interfaces to other programs in related domains like geostatistics, databases, mapserver and even other GIS software exist. It can serve as a Desktop GIS and as the backbone of a complete GIS infrastructure.
GRASS GIS
I know that this is mature software that has been around for a long time, and is used by lots of serious scientists. My impression was that it is mostly suited for raster analysis, and that its vector capabilities (for working with points, lines, and polygons) were not as well-developed.
GRASS is run from the command line, which means you have to type in text commands to do most things. There are a few different GUIs available, JGrass, and qGIS. It doesn’t look like JGrass is very well-developed or has a large userbase. qGIS may be the way to go here.
Here is a 34-page document called GRASS GIS in a Nutshell from 2005.
The latest manual (version 7) has the page, Intro vector map processing and network analysis, which describes working with lines. The plus: it seems fully capable of managing network topology, intelligently dealing with edges and nodes. The minus: the manual text is dense, and difficult to understand.
Some information about working with vectors from section 6.1 that makes it sound like a potentially good tool (ability to bring in *many* formats, and ability to create topologies:
Vector maps can be imported from various sources such as ArcInfo-Coverages, CSV, DGN, SHAPEfiles, GML, MapInfo, MySQL, ODBC, OGDI, PostgreSQL/PostGIS, S57, SDTS, TIGER, UK .NTF,and VRT. The module for importing vector maps is v.in.ogr. The input ?dsn? (data source name) parameter can be a file, a directory or a database connection, depending on the data format. As GRASS is a topological GIS, non-topological Simple Feature data such as SHAPE files are transformed into a topological representation upon import. Data quality is verified during the import, and vector features which violate topological conditions are stored in a separate layer for later inspection. For more details please refer to the manual page of v.in.ogr.
It looks like the easiest way to learn about GRASS is to buy a copy of the GRASS book. (Doesn’t look like you can read it online anywhere…)
Quantum GIS
Quantum GIS or qQIS, looks like it’s under very active development, with revisions being pushed out frequently. As of December 2009, it’s on version 1.30. (I was hesitant to install it on my office desktop when the version number was less than 1!)
Quantum GIS (QGIS) is a user friendly Open Source Geographic Information System (GIS) licensed under the GNU General Public License. QGIS is an official project of the Open Source Geospatial Foundation (OSGeo). It runs on Linux, Unix, Mac OSX, and Windows and supports numerous vector, raster, and database formats and functionalities.
One of its strenghts is that it upports a number of different data formats (and can do on-the-fly projection!):
Does it support the creation and editing of topologies? Seems like the answer is no, or at least I can’t find any references to the word “topology” in the documentation. But perhaps, because of the linkage with GRASS, it can do this? I need to investigate further to find out.
OSGeo is the Open Source Geospatial Foundation, and their website is a portal of information and links to many different applications, a number of which are listed here. The foundation “is a not-for-profit organization whose mission is to support and promote the collaborative development of open geospatial technologies and data. The foundation provides financial, organizational and legal support to the broader open source geospatial community.”
What we’re interested in is the software OSGeo4W. They maintain a “binary distribution”, which is a bundle of programs from the OSGeo family. For Windows users, there is OSGeo4W, “a windows based stack of GeoFOSS which is based upon cygwin.”
I tried OSGeo4W, and my experience was a disaster. I tried installing it back in August, but qGIS did not work. I kept getting cryptic error messages. Spent a few hours trying again, looking for workarounds, etc. Eventually decided to throw in the towel, but was unable to un-install it. It ended up messing up my installation of Python, and leaving a few hundred megabytes of files. Not sure what will happen if I just delete all of these… Oh well, you can not set your expectations too high for free software developed by volunteers. I hand it to the developers for their work on this important, and no doubt challenging project. But it’s not quite ready for prime time.
Update: I’ve since tried installing cygwin, and never got it to work on my computer. Perhaps that is the reason why OSGeo4W never worked either.
For Linux users, the project to compile all the installers in one place looks a little less well-focused. If you’re using the Debian flavor of Linux, there is the DebianGIS project, which enables Debian users to install GRASS, GDAL, and Proj4.
PostGIS is software that lets you turn a PostgreSQL database into a GIS database. A whole bunch of different programs can use PostGIS as a “backend”, i.e. a place to store the geodata. The list includes open-source packages like qGIS, GRASS, and uDig, but also commercial packages like ArcGIS and MapInfo.
In fact, the wikipedia article on PostGIS is a nice list of different current open-source GIS programs.
Two reasons for us to be interested in this package. First, PostgreSQL is commonly used as a web database, perhaps second to MySQL, so it is ready to use for a web project. Second, there is an open-source module called pgRouting for calculating directions.
From the wikipedia article on uDig:
uDig is a GIS software program produced by a community led by Canadian-based consulting company Refractions Research. It is based on the Eclipse platform and features full layered Open Source GIS. It is written in Java and released under GNU Lesser General Public License.
uDig can use GRASS for complex vector operations and also embeds JGRASS and specialized hydrology tools from the Horton Machine. It supports shapefiles, PostGIS, WMS, and many other data sources natively.
The MapWindow application is a free, extensible, geographic information system (GIS) that can be used as an alternative desktop GIS, to distribute data to others, or to develop and distribute custom spatial data analyses.
Based on the documentation here, it looks like it is only capable of handling shapefiles, but it looks like it has decent capabilities for editing vector geometry and attribute tables. There is no mention of more advanced capabilities like topology or networks, or linking to external data.
Conclusion: MapWindow might be worthwhile for simple editing of shapefiles on your local hard drive. Probably not capable enough for our needs.
MapServer is a software package for map rendering. It lets you create your own web mapping service. Of limited use for the bike mapper project, as we are simply displaying limited amounts of route data on top of Google Maps, and don’t need to render our own basemaps.
There is much more out there. I’ve tried to highlight the projects that are under active development, and likely to stay useful in the future, or powerful enough to handle real tasks (rather than just being prototypes or proofs-of-concept. Drop me a line or leave a comment if you know of other programs/resources that might be helpful in the Bike Mapper project.
Others worth looking in to?
One of the many good uses of wikipedia: looking up things you think you know, and finding out they are totally wrong. Or in some cases apocryphal or only partly true. These both have to do with NASA. Where did I get these ideas?
I was under the mistaken impression that James Hansen, the outspoken climate scientist, was either fired or resigned from from NASA during the dark, anti-science days of the G.W. Bush administration. I even said as much during a short radio interview on KPFA, while discussing climate change impacts to California on the morning of December 7th. Thanks to an alert colleague for pointing out my error. Let this serve as my correction.
Dr. Hansen continues to serve as the director of NASA’s Goddard Institute for Space Studies. Dr. Hansen went public with accusations that the administration “tried to influence his public statements about the causes of climate change” and tried to distort and suppress science that showed human’s role in global warming. He’s been embroiled in other controversies too: being arrested at a protest of mountaintop removal, and encouraging protesters to march on the Capitol Power Plant in Washington, D.C.
He has been an outspoken critic of the fossil fuel industry, and has written a number of pieces in the lead-up to the Copenhagen Climate Summit, including an editorial in the New York Times criticizing Cap and Trade emissions-reduction schemes.
I remember shortly after the launch of the Hubble Space Telescope in 1990 that astronomers were dismayed (and NASA was embarrassed) to find out that its mirror was flawed, making its images blurry. I seem to remember that this was because someone accidentally did an incorrect units conversion, using inches where it should have been millimeters.
Hubble Space Telescope
Not so. It was an error with a testing instrument that resulted in a slightly misshapen mirror, meaning it couldn’t focus properly.
The units problem was the $125 million Mars Climate Orbiter, which crash-landed on September 23, 1999. The problem?
Project teams in Colorado and California were using different measurement systems — one in feet and pounds, and the other in metric units — to measure “critical” information for the spacecraft’s maneuvers.
The Hubble story has a happy ending, as NASA was able to repair the telescope. It continues to do valuable science, and return amazing images. Related: NASA’s Astronomy Picture of the Day is well worth visiting daily, or a subscribing to its RSS feed.
Here’s an alarming and important piece by George Monbiot, an environment journalist in the UK, published in the British newspaper The Guardian on November 20th. The Urgent Threat to World Peace is….Canada. He concludes, “The harm this country could do in the next two weeks will outweigh all the good it has done in a century.”
Canada is slipping down the development ladder, retreating from a complex, diverse economy towards dependence on a single primary resource, which happens to be the dirtiest commodity known to man. The price of this transition is the brutalisation of the country, and a government campaign against multilateralism as savage as any waged by George Bush.
BikeMetro is a bike route planning website for Southern California. When I looked at it a year ago, it was down most of the time, so I wrote it off. Now that it’s back up and seems fairly stable, I see a lot of good stuff there.
The grand-pappy of bike route planners in the US, it was developed by a called GeoPerception for the South Coast Air Quality Management District. It covers four counties in Southern California: Los Angeles, Orange, San Bernadino, and Riverside. GeoPerception went out of business, and the hosting responsibilities were taken over by a company called Surf Consulting, which appears to be a one-man shop in San Diego.
bikemetro.com
Despite the fact that the user interface looks dated, there are lots of really great features. It’s not surprising that it seems outdated — the site was developed in 2002, before the advent of the click-and-drag “slippy maps” that Google pioneered. However, most of what you’d want in a bike mapper are here in some form:
Aside from the map, I really like some of the additional outputs:
bikemetro.com
But the application is far from perfect. Here are the elements that are not so good:
My critiques obviously come from the standpoint of a year-2009 web map user. Back in 2002, web maps weren’t that interactive, nor was anyone using Ajax.
And that’s the problem. A complex application like this can’t just be launched. It needs to be maintained and updated. There must be hundreds of miles of new roads in the four-county area in the last 7 years. Have any of these made it into the map? What about changes to the bike network? What happens to user feedback? Are peoples comments and suggestions used to improve the underlying geodata?
Lastly, it is definitely not open source. Their terms of use state that the entire thing is copyrighted, and the license agreement is long and restrictive. Is this right? Information developed by government agencies in the US is by definition public domain. Is this not so for information developed by private contractors using public money?
The minutes from a May 2008 board meeting of the Air Quality Management District give some information concerning the history and future of the bikemetro mapper. Bottom line: The District sounds disappointed that “the website has not been periodically upgraded as originally envisioned” and plans to spend $200,000 to move it over to CommuteSmart.info, a website from the 4-county transportation agencies with traffic and transit info (basically a SoCal equivalent of the Bay Area’s 511.org).
As an element of their FY 2001-02 Work Program, the MSRC [Mobile Source Air Pollution Reduction Review Committee] co-funded the development of a web-based bicycle routing and mapping system. GeoPerception, Inc., the developer of the BikeMetro website, was required to provide website hosting and maintenance for a period of five years. This contractual obligation ended in March 2008. Recently, the MSRC was contacted by Surf Consulting, which has been performing BikeMetro hosting and maintenance on behalf of GeoPerception since October 2005. Surf Consulting sought direction concerning the MSRC’s desires relative to the possible continued availability of BikeMetro to the South Coast’s cycling community. The cost for follow-on website hosting services has been quoted by Surf Consulting at $400.00 per month. This cost would include website hosting and minor maintenance only.
In a separate recommended action in this letter, the bicycle routing and mapping capability included in BikeMetro is currently recommended to be migrated to the regional CommuteSmart.info website under a contract to the Riverside County Transportation Commission (RCTC); however, this new capability is not anticipated to be available for approximately 12 months. On April 17, 2008, the MSRC considered options for continued operation of the BikeMetro website in the interim. Further details are described below in the Proposals section.
[...]
As mentioned above, the MSRC co-funded the development of a web-based bicycle routing and mapping system as an element of their FY 2001-02 Work Program. GeoPerception, Inc., the developer of the BikeMetro website, is no longer an operating business entity. While GeoPerception did make arrangements for Surf Consulting to fulfill GeoPerception’s contractual hosting requirements through March 2008, the website has not been periodically upgraded as originally envisioned. Also, higher than normal site downtime was noted in the first few months of 2006. On August 17, 2006, the MSRC considered various options for long-term viability of the BikeMetro website. Further details are described below in the Proposals section.
[...]
As stated in the Background section, it came to the attention of the MSRC that GeoPerception had ceased to function as a business entity, the site was being maintained by a third party, and the BikeMetro website had been experiencing higher than normal downtime incidences. On August 17, 2006, the MSRC considered long-term options for BikeMetro. Alternatives included terminating operation, maintaining the site in its existing form, or upgrading. The MSRC determined that an opportunity existed, not only to upgrade the site with additional functionalities and updated mapping information, but also to bring BikeMetro to the attention of a wider audience, by re-hosting the site as an element of the broader CommuteSmart.info regional website. CommuteSmart.info is administered by RCTC on behalf of the transportation commissions for Los Angeles, Orange, Riverside, San Bernardino, and Ventura Counties. As an element of Phase I of their FY 2006-07 Work Program, the MSRC unanimously approved a sole-source contract with RCTC, in an amount not to exceed $200,000, to migrate BikeMetro into their CommuteSmart.info site. RCTC would be responsible for performing website upgrades, updates, hosting and maintenance.
ridesf.com bike mapper
In my first bike mapper roundup, I failed to mention John Roark’s work-in-progress, ridesf.com. Hey, nice favicon! According to John, “It is mostly useless right now since I don’t have a conflated data set that includes bike lanes and non bike lanes. It really is just a work in progress/hobby.” He’s off to a good start though. I like the clean, uncluttered design. Perhaps the biggest strength, in my opinion, is that it’s based on the open-source PostGIS, which is an extension to the PostGRE database system.
At last night’s SF Bike Planner meeting, I found out that you can get directions based on OpenStreetMap data. The site, launched in May 2008, is the project by the OSM user Lambertus (real name unknown) in the Netherlands. I was under the impression that the OSM data were not suitable for routing, because they don’t have enough information about turn restrictions, etc. Apparently not.
Bike Directions from yournavigation.org
On the site, you can choose different routing options like “Car”, “Heavy Goods”, or “Bicycle”. No idea what “Mofa” means… The site gives reasonable directions in the East Bay. On this route from Oakland to Richmond it routes me up the Ohlone Greenway for much of the route (good!), but up and over the hill near the Kaiser campus (bad! steep!). Not surprising as it is based on two-dimensional data. Still, it covers the whole planet (at least the parts mapped by OSM, which extends its reach daily). Impressive, but clearly still a work in progress. For example, there are no text directions yet.
There is information about YOURS (Yet Another OpenStreetMap Routing Sytem) here, including their API. Worth looking tabs on this project.