A couple dozen people were gathered at the SoMa offices of Granicus, a cloud vendor that provides streaming online video services to cities and legislative bodies throughout the U.S. It was after 9 p.m. and the crowd had dwindled as we realized that some of the main challenges of the event - CityCampSF Hackathon 2011 - could not be completed due to lack of access to key government data.
CitiReport publisher Larry Bush had sponsored a $1,000 prize for the best “Open Ethics” application, to help highlight influence trends in the City and relationships between lobbying activity, financial contributions to City leaders, development projects, vendor contracts and legislative actions. However, despite millions in spending on the Ethics Commission and online transparency measures, access to data in these areas is woefully lacking. Here’s how I summarized the quality of the data going into the December 10 event:
“SF Planning: The Planning Department has several data sets on DataSF (the central repository for City data created by Gavin Newsom and the Department of Technology) but the project information on key developments active right now does not seem to be there. Instead, we get this nicely designed “Complete List of Projects” with no structured data (such as a simple CSV file with headers that include the project description, location, lead planner and developers) in sight.
“SF Ethics: The Ethics Commission has the legal duty of collecting huge amounts of campaign finance and good government data. But their datasets on OpenSF are horribly out of date, and their online list of lobbyists would take a master coder or many, many man hours to parse into an open format for analysis (again, simple CSV files with headers would do – name of lobbyist, who they work for, how much they got paid, who they contacted, when, what project they were trying to influence). This lack of functional transparency is glossed over with a great data display, but none of the underlying data is easily available on the site. This means you get the results that the display tells you are important, while independent analysis is nearly impossible.
“SF Controller: The Controller’s Office is responsible for keeping track of city salaries and city contracts. They don’t have a fancy data display (which I bet saved them a lot of money), but they do allow you to search their contractor database and download data in structured open formats. Thank you!
“SF Board of Supervisors: The Clerk of the Board publishes huge volumes of minutes of board agendas and votes. However, they are PDFs with no structure. Many of the attachments for Board items end up as PDF-formatted scans of documents without optical character recognition, rendering them unsearchable by people viewing them on the web, and keeping them out of open web search results. This lack of functional transparency makes it very difficult to know who’s voting on which planning projects for example.
“In summary, if we want to make connections between who is lobbying for a contract, which developers are giving money to politicians, and who’s voting on what, it is very, very difficult. Lack of functional transparency means that if you want to evaluate your government in San Francisco, you have to know almost exactly what piece of straw in a data haystack you’re looking for before you even start.”
Recently, in response to a records request we made during the civic hacking event, the Ethics Commission took a significant step in releasing a public application programming interface (API) for accessing the lobbyist database administered by vendor NetFile (this is what an API “call” for a list of registered lobbyists in 2010 looks like). It’s one of the first City data APIs, and a model for the kind of transparency the City can create by requiring access to APIs and/or complete data sets when vendors administer city data. It’s important enough that we need legislation providing guidelines for technical openness for City data. APIs and source databases mean we don’t have to get a story in the design of a website, report or search application, we can analyze the data for ourselves.
Beware of tech coalitions that say the City needs tax breaks for more tech jobs. The industry is doing fine. What the City needs is reform in how it manages its data and in how it procures its IT resources. Coalitions of technologists and open government advocates can have a big impact when we look at the systems behind our local government and ask the right questions.
Adriel Hampton has organized three “CityCamp” events in SF in the past two years to bring together government and neighborhood leaders, technologists and journalists for the public good, and is co-founder of the San Francisco Technology Democrats.