Important Tools for Visualising and Communicating Data

This list of resources represent an ongoing and growing series of blog posts presenting the most inspiring collection of important, effective, useful and practical data visualisation tools. You can also view these resources via a publicly accessible Google Spreadsheet.
** This series of posts will be undergoing a thorough update during January and February 2013! **
Part 1: Tools for Analysis, Graphing and Enterprise
Part 2: Visual Programming Languages and Environments
Part 3: Google’s Charting and Visualisation Tools
Part 4: Tools for Mapping
Part 5: Specialist Tools and Visualisation Communities
Part 6: Visualisation Presentation and Publishing Tools

The Most Influential Data Visualisation Books

Part 7: A Personal Collection of Influential Books on Data Visualisation and Other Related Subjects (1)
Part 8: A Personal Collection of Influential Books on Data Visualisation and Other Related Subjects (2)
Part 9: A Personal Collection of Influential Books on Data Visualisation and Other Related Subjects (3)

http://code.google.com/p/google-refine/,

http://www.google.com/fusiontables/Home, http://multimedia.journalism.berkeley.edu/tutorials/google-refine-export-json/json/, http://www.google.com/fusiontables/public/tour/index.html, http://www.google.com/support/fusiontables/bin/answer.py?hl=en&answer=184641, http://www.computerworld.com/s/article/9196283/H_1B_visa_data_Visual_and_interactive_tools, https://sites.google.com/site/fusiontablestalks/stories?ft_source=tour_defaulttab&__utma=1.1922599578.1299797723.1299797723.1299797723.1&__utmb=173272373.2.10.1299797450&__utmc=173272373&__utmx=-&__utmz=1.1299797723.1.1.utmcsr=%28direct%29|utmccn=%28direct%29|utmcmd=%28none%29&__utmv=-&__utmk=27954108

http://www.impure.com/ (http://www.youtube.com/watch?v=4oc47BB374U),  http://www.youtube.com/watch?v=XYVAPfb8k5U, http://www.guardian.co.uk/news/datablog/interactive/2011/mar/08/pay-gap-gender-women-men,

http://vis.stanford.edu/wrangler/,

http://www.tableausoftware.com/public/blog/2011/04/data-shaping, http://www.tableausoftware.com/public/training, http://www.computerworld.com/s/article/9210078/Tech_unemployment_higher_than_white_collar_average#interactive_graph,

http://code.google.com/p/google-refine/wiki/Screencasts,

http://www-958.ibm.com/software/data/cognos/manyeyes/, http://www-958.ibm.com/software/data/cognos/manyeyes/page/Visualization_Options.html, http://www-958.ibm.com/software/data/cognos/manyeyes/page/Data_Format.html,

http://www.dataviz.org/, http://www.w3.org/TR/html4/present/frames.html, http://www.dataviz.org/,

http://simile-widgets.org/exhibit/, http://people.csail.mit.edu/karger/Exhibit/CAR/, http://simile-widgets.org/wiki/Getting_Started_with_Exhibit

http://www.filamentgroup.com/lab/update_to_jquery_visualize_accessible_charts_with_html5_from_designing_with/

http://sixrevisions.com/javascript/20-fresh-javascript-data-visualization-libraries/ (extract)

http://code.google.com/apis/charttools/index.html, http://code.google.com/apis/visualization/documentation/queries.html, http://code.google.com/apis/chart/docs/gallery/dynamic_icons.html, https://chart.googleapis.com/chart?chs=75×50&cht=gom&chd=t:70&chco=FF0000,FF8040,FFFF00,00FF00,00FFFF,0000FF,800080, http://code.google.com/apis/visualization/documentation/using_overview.html, http://code.google.com/apis/chart/docs/making_charts.html, http://code.google.com/apis/visualization/documentation/using_overview.html, http://code.google.com/apis/visualization/documentation/gallery.html

http://code.google.com/p/google-refine/wiki/DocumentationForUsers,

https://www.statwing.com/, http://blogs.computerworld.com/business-intelligenceanalytics/20909/startup-aims-simplify-data-analysis, The idea behind Statwing is to provide some basic, automated statistical analysis on data that users upload to the site — correlations, frequencies, visualizations and so on — without requiring you to know when, say, to use achi-squared distributionversus a z-test.,
statistical functions: http://office.microsoft.com/en-us/excel-help/statistical-functions-HP005203066.aspx

protovis: http://vis.stanford.edu/protovis/ex/

http://vis.stanford.edu/protovis/docs/start.html

sort tools by skill levels

http://www.qgis.org/
http://tbarmann.webfactional.com/nicar/qgis_tutorial/

Learn more: Timothy Barmann of The Providence Journal posted two very useful tutorials for the CAR conference that are still available: Introduction to QGIS and The Latest in Mapping With JavaScript and jQuery. Barmann also offers a sample: Rhode Island’s Ethnic Mosaic. Another resource to help you get started: QGIS Tutorial Labs from Richard E. Plant, professor emeritus at the University of California, Davis.

Note: If you’re interested in GIS and want to consider other free software options, download this PDF listing of Open Source/Non-Commercial GIS Products. And if you’re looking for a free open-source desktop GIS program that might be fairly easy to use, Jacob Fenton, director of computer-assisted reporting at American University’s Investigative Reporting Workshop, recommends taking a look at the System for Automated Geoscientific Analyses (SAGA) site. Finally, if analyzing geographic data in a conventional database sounds interesting, PostGIS ”spatially enables” the PostgreSQLrelational database, according to the site.

http://www.arl.org/sparc/openaccess/

most of these are outdated and people don’t even use, simplifying this to the best tools is the goal, any outdated or irrelevant, deprecated solutions will be deleted.

 Google(which has a number of third-party front ends such as Map A List, an add-on that adds info to a Google Map from a spreadsheet). There’s also Yahoo Maps Web Services and Bing Maps – all with APIs. But there are numerous oth

OpenHeatMap

 “How OpenHeatMap Can Help Journalists“

http://www.computerworld.com/s/article/9215504/22_free_tools_for_data_visualization_and_analysis?taxonomyId=18&pageNumber=8

OpenLayers

http://fuzzytolerance.info/code/openlayers-with-a-google-street-view-widget/
http://www.geoext.org/index.html

 OpenLayers Simple Example. A good sample isUshahidi’s Haiti map.

There are other JavaScript libraries for overlaying information on maps, such as Polymaps. And there are a number of other mapping platforms, such as Google Maps, which offers numerous mapping APIs; Yahoo Maps Web Services, with its own APIs; the Bing Maps platform and APIs; andGeoCommons.

“Links and resources available below may be useful for those interested in pursuing open access publication or advocating for open access to others in the academic community, to grant-making institutions, or even to bodies of government. Resources supplied here include guides, presentation materials, and handbooks produced by SPARC and other organizations. These provide definitions and developments in the field, and point those interested to the growing success of Open Access. Please write to sparc[at]arl[dot]org with additions or corrections.”
http://www.worldbank.org/open/
https://openknowledge.worldbank.org/
http://www.transparency.org/

What is the Open Aid Partnership?
Transparency of development assistance, public budgets and service delivery is critical for citizen engagement. Innovative technologies, such as mapping, provide powerful new tools for strategic planning and for greater transparency and accountability. Recognizing the significant impact that these innovations and an empowered civil society can have on improving development effectiveness, the World Bank Institute and bilateral donor partners, foundations and civil society have formed an Open Aid Partnership. The Partnership will be working in close collaboration with the International Aid Transparency Initiative (IATI) and the Open Government Partnership (OGP). The partnership brings development partners together to enhance the openness and effectiveness of development assistance.

What are the Open Aid Partnership’s main objectives?

• improve aid transparency and coordination by developing an Open Aid Map that visualizes the location of donor-financed programs at the local level;

• better monitor the impact of development programs on citizens;

• enhance the targeting of development programs;

• foster accountability by empowering citizens to provide direct feedback on project results;

• strengthen capacity of civil society and citizens to use open aid data.

http://www.openaidmap.org/

Putting Development on a Map (Mapping for Results)

The Partnership builds on the World Bank’s Mapping for ResultsInitiative, which has mapped 30,000 activities in all 143 of its client countries, and overlays these data with sub-national poverty and human development indicators at the local level. The initiative is based on the premise that the combination of visualization technologies and open data on development assistance can enable a more transparent, inclusive and effective development process.

Main Components of Open Aid Partnership:

1. Map activities supported by development assistance and create a web-based collaborative Open Aid Map that helps improve coordination, efficiency, transparency and accountability of development assistance.

2. Support developing countries in building national mapping platforms.

3. Promote citizen feedback initiatives for better reporting on development assistance and public service provision in order to enhance transparency and accountability.

4. Build capacity of civil society to act as information intermediaries for citizens and make these maps more accessible, as well as the capacity of public service providers to receive and respond to feedback.

5. Evaluate the development impact of national mapping platforms and feedback initiatives on public services and related capacity building.

http://www.open-contracting.org/

he information available through the AidData database serves as a platform for testing new ways to make aid information more relevant for different audiences. For example, recent work on geocoding aidcan help civil society organizations identify the aid-funded activities that are underway in their communities. AidData’s work supports the efforts of the International Aid Transparency Initiative(IATI) by allowing users to download data in IATI format. Additionally, AidData Rawserves as a repository for datasets that have not yet been vetted or that are not appropriate for inclusion in the main AidData database but provide added informational value.  

http://www.aiddata.org/content/index/Services/geocoding

1) Collect, standardize, and organize geo-enabled data. Teams of experienced researchers are available to geocode project data so that it can be used for maps and other visualizations. AidData has partnered with the World Bank Institute, through the Mapping for Results initiative, and works with theAfrican Development Bank, the Kellogg Foundation, and the Malawi Ministry of Financeto geocode information.
2) Prepare visualizations and analytics that leverage the power of geo-enabled data. Once an organization has geo-enabled data, we work in partnership with Esrito visualize this information on state-of-the-art interactive maps. Custom dashboards that combine maps with graphs and charts support monitoring and evaluation efforts, and help analysts and decision-makers identify risks and define next steps.
3) Prepare implementation reports with recommendations. AidData works with organizations to determine challenges and opportunities to geo-enable their data collection and dissemination efforts, and prepare reports with actionable recommendations. Reports can include roadmaps for compliance with international data standards such as the aid information reporting standard developed by the International Aid Transparency Initiative(IATI).
4) Build and implement custom IT solutions. Based on these reports and recommendations,Development Gatewaycan help design, integrate, and implement custom geo-enabled modules and applications that extend current client systems/processes to create sustainability. Mobile applications can be developed to enable real-time data collection and increase accessibility to broader user groups. Seamlessly integrated web and mobile applications offer organizations a comprehensive way to make their work more efficient and effective.

http://openlayers.org/QuickTutorial/ [open streetmap]

TimeFlow

https://github.com/FlowingMedia/TimeFlow/wiki/Top-Tips

https://www14.software.ibm.com/webapp/iwm/web/preLogin.do?source=AW-0VW

http://www.wordle.net/

http://www.simile-widgets.org/timeline/

http://nodexl.codeplex.com/

http://www.analytictech.com/ucinet/

d free NodeXL tutorial (PDF) or these basic step-by-step instructions on analyzing your own Facebook social network(PDF). One Facebook app for downloading your own friend information for use in NodeXL is Name Gen Web.

gephi tutorial: http://gephi.org/tutorials/gephi-tutorial-quick_start.pdf
http://thejit.org/demos/, http://thejit.org/

http://www.spatstat.org/spatstat/,

http://www.peteraldhous.com/CAR/Aldhous_CAR2011_RforStats.pdf,

http://jacobfenton.s3.amazonaws.com/R-handson.pdf,

http://cran.r-project.org/doc/manuals/R-intro.html,

http://www.r-statistics.com/tag/visualization/,
http://csvkit.readthedocs.org/
Source: http://www.computerworld.com/s/article/9215504/22_free_tools_for_data_visualization_and_analysis?taxonomyId=18&pageNumber=2

Data visualization and analysis tools

*Highcharts is free for non-commercial use and $80 for most single-site-wide licenses.

This paper outlines proposals for meeting
the objectives of the International Aid
Transparency Initiative (IATI) without
disproportionate cost, and explains what
value IATI would add to existing systems
for reporting aid.  Detailed work on
implementation issues is scheduled
through the IATI Technical Advisory Group
(TAG) during 2010. Membership of the
TAG is open, and so far, over 100
individuals have contributed to its work,
including representatives of each
stakeholder group.
There are many people and organisations
with diverse, legitimate and important
needs for information about aid.  
Developing country governments need
information about how aid is being spent
in their country.  Parliamentarians in
developing countries and in donor
countries want to hold their government to
account. Communities in developing
countries need to know what resources
are available for their development
priorities and in what way they can
influence how those resources are used. A
village council wants to know what aid is
available  to improve water in its area.  
Researchers need better data to
understand how aid can be more effective.  
Taxpayers want to know how their money
is being spent.
No single database can satisfactorily
meet the needs of all these potential
users.  
These users all want information tailored
to their own needs.  Often they want
information from many different donors,
combined with information from other
sources, such as the government’s
spending, or disease surveillance data.
Yet it is unrealistic to expect donors to
provide information separately to
hundreds of possible information systems.
This then is the dilemma: users need
information presented in ways specific
to their needs, but donors cannot
provide information to each of them
individually.
There are broadly two ways to respond to
this challenge. A limited response is for
those donors who currently report to the
Development Assistance Committee
(DAC)  databases to step up
the information that they already provide,
and for all donors to improve reporting to
country government aid management
systems (AIMS).
This paper sets out a more
comprehensive response and shows how
IATI could improve reporting to existing
systems, and at the same time meet a
much wider range of needs for
information, including documents as well
as data.
Donors would extend their existing
processes for collecting information about
aid, which they currently use to report to
the DAC and other systems. They would
include additional information needed by
other stakeholders, much of which is
currently collected and provided
separately.  As now, donors would choose
their own systems to manage this data
collection. They would put this combined
information into the public domain more
rapidly and in a common format.  They
would register the location of the data in a
“registry”  – a kind of online catalogue
which enables users to find it.
This approach can be summarised as
“publish once, use often”.  
The combination of common, open
formats plus the registry would add huge
value to the information already being
published by donors, and the additional
information they would publish as a result
of IATI, because users would be able to
access information of particular interest to
them, in a format that is useful to them,
without having to trawl round all the donor
websites individually.  This would open up
the information to a wider range of users
and democratise access to information
through services such as mobile phones
or Google.
The information collected and published
under IATI would provide the information
needed for donor reporting to existing
systems, such as DAC and country AIMS
and national budgets. This would reduce
duplicate information collection and
reporting.
To meet their commitments under the
Accra Agenda for Action (AAA), and in the
context of growing calls for government
transparency, donors are increasingly
publishing more information about aid.
Clearly this will involve some costs to
donors. These IATI proposals are
designed to minimise the additional
burden of this greater transparency, and
yet obtain the maximum benefits from their
efforts by ensuring that the information,
once collected, is universally accessible.

Based on extensive stakeholder consultation summarised in Chapter One, aindinfo concludes that the
system to implement the IATI declaration signed in Accra in 2008 should:
1. meet in full the information needs of developing country government  AIMS and budgets without
imposing a burden on developing countries, including complying with local definitions and
classifications;
2. build on the work that has been done through the DAC to develop common definitions and reporting
processes, and avoid the establishment of duplicate or parallel reporting processes;
3. produce information which is easily accessible to parliamentarians, civil society, the media and
citizens as well as to governments (in line with the expanded definition of country ownership agreed
at Accra);
4. provide accurate, high quality and meaningful information, and enable users to distinguish official
statistics, which have been professionally scrutinised, from management information about projects
and programmes;
5. include information about spending by non-DAC donors, multilateral organisations, foundations and
NGOs;
6. be easy to understand, reconcile, compare, add up, read alongside other sources of information,
and be easy to organise and present in ways that are useful to information users;
7. be legally open, with as few barriers to access and reuse as possible;
8. reduce duplicate reporting by donor agencies and minimise additional costs;
9. be electronically accessible in an open format so a wide range of third party intermediaries can
access the information and present it either as comprehensive information or niche analysis;
10. result in access to information about aid which is more timely, more detailed, more forward looking
and more comprehensive than existing data, and which includes wider information on aid, such as
key policy and appraisal documents and the outputs and outcomes it achieves

The International Aid Transparency
Initiative (IATI) was launched at the Accra
High Level Forum on Aid Effectiveness in
September 2008. IATI is a multistakeholder initiative to accelerate access
to aid information to increase
effectiveness of aid in reducing poverty.
The Accra Agenda for Action (AAA)
recognised that increased transparency is
central to the objectives of the Paris
Declaration. Transparency is essential to
meet the five underlying principles of
ownership, alignment, harmonisation,
managing for results, and mutual
accountability. The AAA expanded the
concept of country ownership to include
parliamentarians, civil society
organisations (CSOs), academics, the
media and citizens. Donors agreed to
support efforts to increase the capacity of
all development actors to play an active
role in policy dialogue. The AAA
committed donors to  “disclose regular,
detailed and timely information about our
aid flows” and to  “support information
systems for managing aid”.
IATI provides a way for donors to meet
this commitment in a coherent and
consistent way. IATI has 18 signatories, of
whom 13 are DAC members. These
signatories resolved to “give strong
political direction” and “invest the
necessary resources in accelerating the
availability of aid information”.
IATI also contributes to Cluster C on
Transparent and Responsible Aid, which
sits under the Working Party on Aid
Effectiveness (WP-EFF.) IATI has been
tasked by the Cluster with developing
reporting formats and definitions for
sharing information about aid, drawing on
the expertise of the Working Party on
Statistics (WP-STAT.) Proposals
developed by IATI will be available to
inform the Cluster’s work.
IATI aims to agree a four-part standard
consisting of:
(1) an agreement on what would be
published
(2) common definitions for sharing
information
(3) a common electronic data format  
(4) a code of conduct.
The details of what would be covered by
IATI and how this would be published will
be decided by the IATI members, following
detailed research by the Technical
Advisory Group (TAG) and consultation
with stakeholders. It is intended that the
standard will be adopted at first by IATI
members but it may over time be adopted
by other DAC donors, and by other nonDAC donors, other foundations and  nongovernmental organisations (NGOs).
There is widespread support among
developing country governments for
extending the coverage of aid information
to non-traditional donors.
IATI responds to growing demands from
civil society and citizens for greater
transparency of information about
spending and results, and for access to
key documents as well as data. The
ambitions of IATI are consistent with many
other recent initiatives to increase
transparency, for example President
Obama’s August 2009 memo on
transparency, the World Bank’s new
disclosure policy, which represents a

paradigm shift to proactive disclosure with
limited exceptions, and the development
of online information portals for citizens,
such as in Brazil.  IATI seeks to harness
the power of new technology to deliver
real improvements in the lives of the
world’s poorest people, in the  same way
that email, internet access and mobile
phone networks have revolutionised the
way that aid agencies themselves do their
business.
Since its launch in September 2008, IATI
has focused on consultation with
developing countries and CSOs, factfinding missions to a number of donor
countries, and detailed work by the TAG
on parts 1 and 4 of the proposed IATI
standard, covering an agreement on what
would be published and a code of
conduct.
The IATI Conference, held in The Hague
in October 2009, confirmed widespread
support for the objectives of IATI, and
consensus on the key information needs
of different stakeholders. At the same
time, it was clear during the IATI
conference that a number of stakeholders
would welcome greater clarity on how IATI
might  work in practice, so that they can
consider the full implications of the
initiative for their agencies.
Although detailed work on the precise
practical and technical mechanisms for
implementing IATI is only just beginning,
this paper presents a proposal on how
IATI would work, what this framework
would mean for different stakeholders, and
what added-value it is envisaged that IATI
would offer as a result.
Notes
1. http://www.whitehouse.gov/the_press_
office/TransparencyandOpenGovernm
ent/

The task that needs to be done would be gathering the correct people, agencies, non for profits, and businesses around their area of expertise to execute. From there would be so much so much data to report, analyze and gain insight from that we’ll be busy for a while before we come to general consensus on how people can contribute to the issues in the backlog of their municipal. This is a proposal to change the 2025 vision.

http://selection.datavisualization.ch/

Wolfram|Alpha Pro

[http://tributary.io/] for D3 prototyping and RStudio Server for R (which is amazing if you haven’t tried it)

what matters most is the centralization of this data in an easily scrapable and visualizable format

Geodata- the data that is used to make maps- from location of roads and buildings to topography and boundaries

Culture: data about cultural works and artefacts for example titles and authors- and generally collected and held by galleries, libraries, archives, and musuems.

Science- Data that is produced as part of scientific researcher from astronomy to zoology.

Financial- data such as government accounts (expenditure and revenue) and information on financial markets (stocks, shares, bonds, etc.)

Statistics- data produced by statistical offices such as the census and key socioeconomic indicators

Weather- the many types of information used to understand and predict the weather and climate.

Environment- information related to the natural environment such presence and level of pollutants, the quality and rivers and seas.

Transport- data such as timetables, routes, on-time statistics. (public bus statistics)

Transparency. In a well-functioning, democratic society citizens need to know what their government is doing. To do that, they must be able freely to access government data and information and to share that information with other citizens. Transparency isn’t just about access, it is also about sharing and reuse — often, to understand material it needs to be analyzed and visualized and this requires that the material be open so that it can be freely used and reused.

Releasing social and commercial value. In a digital age, data is a key resource for social and commercial activities. Everything from finding your local post office to building a search engine requires access to data, much of which is created or held by government. By opening up data, government can help drive the creation of innovative business and services that deliver social and commercial value.

Participation and engagement – participatory governance or for business and organizations engaging with your users and audience. Much of the time citizens are only able to engage with their own governance sporadically — maybe just at an election every 4 or 5 years. By opening up data, citizens are enabled to be much more directly informed and involved in decision-making. This is more than transparency: it’s about making a full “read/write” society, not just about knowing what is happening in the process of governance but being able to contribute to it.

DDaattaa AAPPIIss

We provide a number of public APIs that expose the data in our services to developers who want

to re-use it.

okfn / ckan

The Comprehensive Knowledge Archive Network (CKAN) stores metadata and data for datasets.

Various deployments exist, but an API is available for all of them.

Documentation

Python Client

JavaScript Client

Endpoints: demo.ckan.org, datahub.io, data.gov.uk, more…

ooppeennssppeennddiinngg / ooppeennssppeennddiinngg

OpenSpending.org is a datamart for government financial data. It stores budgets and

transactional expenditure and offers search, export and aggregation APIs.

Documentation

JavaScript Client

okfn / bbiibbsseerrvveerr

BibServer is a tool for quickly and easily sharing collections of bibliographic metadata. Most of

the data stored internally can be read through the API.

Documentation

BibSoup.net instance

ppyybboossssaa / ppyybboossssaa

PyBossa is a crowd-sourcing platform where users can help to complete tasks, such as image

analysis or text transcription. The application can be completely controlled via it’s API.1/14/13 Data Sources

okf nlabs.org/data/ 2/2

JavaScript Client

Python Client

Documentation

okfn / aaccttiivviittyyaappii

The activity API collects data about project-related activties on various collaborative platforms,

including GitHub, Twitter and Mailing List feeds.

Endpoint documentation

Frontend source code

pudo / nnoommeennkkllaattuurraa

Nomenklatura is a very simplistic data linking tool. It maintains authoritative lists of names (e.g.

politicians, companies, streets) and offers an API and web-based interactive recon tool to match

variant spellings of these names to the canonical form.

Python client

Endpoint documentation

http://opendatahandbook.org/en/

1/14/13 The Open Data Handbook — Open Data Handbook

opendatahandbook.org/en/ 1/3

Open Data Handbook

The Open Data Handbook

This handbook discusses the legal, social and technical aspects of open data. It

can be used by anyone but is especially designed for those seeking to open up data. It

discusses the why, what and how of open data – why to go open, what open is, and the

how to ‘open’ data.

To get started, you may wish to look at the Introduction. You can navigate through the

report using the Table of Contents (see sidebar or below).

We warmly welcome comments on the text and will incorporate feedback as we go forward.

We also welcome contributions or suggestions for additional sections and areas to examine.

Table of Contents

Introduction

Target Audience

Credits

Credits and Copyright

Why Open Data?

What is Open Data?

What is Open?

What Data are You Talking About?

How to Open up Data

Choose Dataset(s)

Asking the community

Cost basis

Ease of release

Observe peers

Apply an Open License (Legal Openness)

Make Data Available (Technical Openness)

Online methods

Search1/14/13 The Open Data Handbook — Open Data Handbook

opendatahandbook.org/en/ 2/3

Make data discoverable

Existing tools

For government

So I’ve Opened Up Some Data, Now What?

Tell the world!

Understanding your audience

Post your material on third-party sites

Making your communications more social-media friendly

Social media

Getting folks in a room: Unconferences, Meetups and Barcamps

Making things! Hackdays, prizes and prototypes

Examples for Competitions

Conferences, Barcamps, Hackdays

Glossary

Appendices

File Formats

An Overview of File Formats

Open File Formats

How do I use a given format?

What Legal (IP) Rights Are There in Data(bases)

Indices and tables¶

Index

Search Page

An OOppeenn KKnnoowwlleeddggee FFoouunnddaattiioonn pprroojjeecctt..

©© 22001100–22001122,, OOppeenn KKnnoowwlleeddggee FFoouunnddaattiioonn. LLiicceennsseedd uunnddeerr CCrreeaattiivvee CCoommmmoonnss AAttttrriibbuuttiioonn ((UUnnppoorrtteedd)) vv33..00

LLiicceennssee

SSoouurrccee —— IIssssuueess —— MMaaiilliinngg LLiisstt —— TTwwiitttteerr @@OOKKFFNN1/14/13 The Open Data Handbook — Open Data Handbook

opendatahandbook.org/en/ 3/3

RReellaatteedd PPrroojjeeccttss:: OOppeennGGoovveerrnnmmeennttDDaattaa..oorrgg —— TThheeDDaattaaHHuubb..oorrgg —— DDaattaaCCaattaallooggss..oorrgg —— OOppeennSSppeennddiinngg..oorrgg ——

DdaattaaPPaatttteerrnnss..oorrgg

http://www.isitopendata.org/about/

with open data the rise of the citizen scientist, journalist, etc will help us achieve common goals faster  and accomplish tasks that would be too expensive or time consuming to accomplish through other means

http://www.citizencyberscience.net/

http://crowdcrafting.org/

http://opencharities.org/

http://opencorporates.com/

Get this info as JSON, XML, RDF

Data Sources

• Global Data

• International Monetary Fund

• US Securities and Exchange Commission Filings

• Statistics from HMRC (UK)

• Open Corporates

• DueDil

• BP Statistical Review of World Energy

• Revenue Watch

• Transparency International’s Corruption Perception Index

• World Bank

• Annual Reports from individual companies

Tools & Resources

• Excel

• Processing JS

• Visual.ly

• Cartodb

• Raphael JS

• Recline Timeliner

• Evan Raskob’s Intro to Programming

• Data Expeditions Character Sheet

http://datahub.io

Find out more about working with open data by exploring these resources:

• GetTheData.org

• DataPatterns.org

• Open Data Handbook

Inclusive Planning Outreach with Web-based Tools

PlanningPress is a web toolkit for inclusive, responsive, authentic citizen engagement in transportation planning.

The web has opened up new modes of communication between governments and the public, introduced new possibilities for collaborative work, and made dynamic data visualization and analysis possible. PlanningPress makes it straightforward to apply these opportunities to community transportation planning. Everyone involved can review and engage in dialog on ideas and proposals, using maps and a user-friendly interface.

Intended for use by transportation departments and agencies, PlanningPress complements and extends the reach of an existing planning process. It enables regular, non-technical team members to publish updates. The simple content management system is built on WordPress, a widely-used publishing platform.

NYCDOT’s Jackson Heights portal is powered by PlanningPress.  The website introduces the changes proposed for the neighborhood and shows them in detail allowing residents to comment on the plans.  It lays out a timeline for events concerning the project, has an interactive map, news updates and other resources to help people understand the project as it develops.

Open Source

Stamen is an active contributor to and author of multiple open source projects. These collaborative efforts often play a valuable role in our commercial work, and lessons learned from working for clients have a way of making their way into code releases that the public at large can benefit from.

Aight

A collection of tools for making reasonable JavaScript and CSS work in IE8.

Polymaps

Polymaps provides speedy display of multi-zoom datasets over maps, and supports a variety of visual presentations for tiled vector data, in addition to the usual cartography from OpenStreetMap, CloudMade, Bing, and other providers of image-based web maps.

Because Polymaps can load data at a full range of scales, it’s ideal for showing information from country level on down to states, cities, neighborhoods, and individual streets. Because Polymaps uses SVG (Scalable Vector Graphics) to display information, you can use familiar, comfortable CSS rules to define the design of your data. And because Polymaps uses the well known spherical mercator tile format for its imagery and its data, publishing information is a snap.

CityTracking

Dotspotting is the first project Stamen is releasing as part of Citytracking, a project funded by the Knight News Challenge.

We’re making tools to help people gather data about cities and make that data more legible. The code for Dotspotting is available fordownload on Github, and licensed for used under the GNU General Public License.

Modest Maps

Modest Maps is a BSD-licensed display and interaction library for tile-based maps in Flash (ActionScript 2.0 and ActionScript 3.0) and Python.

Our intent is to provide a minimal, extensible, customizable, and free display library for discriminating designers and developers who want to use interactive maps in their own projects. Modest Maps provides a core set of features in a tight, clean package, with plenty of hooks for additional functionality.

Cascadenik

Cascadenik implements cascading stylesheets for Mapnik, a Free Toolkit for developing mapping applications.

It’s an abstraction layer and preprocessor that converts special, CSS-like syntax into Mapnik-compatible style definitions. It’s easier to write complex style rules using the alternative syntax, because it allows for separation of symbolizers and provides a mechanism for inheritance.

Tile Drawer

Tile Drawer makes designing and hosting custom maps simple and straightforward. The project lets anyone run their ownOpenStreetMap server in the cloud with one-step configuration and zero administration. You can use the rendered map tiles in a number of ways: with other GIS data in OpenLayers, in a Flash application built on Modest Maps, or layered into a Google Map as a custom map tile overlay.

Walking Papers

OpenStreetMap, the wiki-style map of the world that anyone can edit, is in need of a new way to add content. Walking Papers is a way to “round trip” map data through paper, to make it easier to perform the kinds of eyes-on-the-street edits that OSM needs now the most, as well as distributing the load by making it possible for legible, easy notes to be shared and turned into real geographical data.

TileStache

TileStache is a Python-based server application that can serve up map tiles based on rendered geographic data.

You might be familiar with TileCache, the venerable open source WMS server from MetaCarta. TileStache is similar, but we hope simpler and better-suited to the needs of designers and cartographers.

http://geodjango.org/

http://postgis.refractions.net/

https://docs.google.com/spreadsheet/ccc?key=0Aon3JiuouxLUdFZPM25HN2pHUk1XSXl0RFg5YkFId0E#gid=0

http://opendatahandbook.org/

http://datawrapper.de/docs/tutorial

Computerworld - Reporters wrangle all sorts of data, from analyzing property tax valuations to mapping fatal accidents — and, here at Computerworld, for stories about IT salaries and H-1B visas. In fact, tools used by data-crunching journalists are generally useful for a wide range of other, non-journalistic tasks — and that includes software that’s been specifically designed for newsroom use. And, given the generally thrifty culture of your average newsroom, these tools often have the added appeal of little or no cost.

I came back from last year’s National Institute for Computer-Assisted Reporting (NICAR) conference with 22 free tools for data visualization and analysis – most of which are still popular and worth a look. At this year’s conference, I learned about other free (or at least inexpensive) tools for data analysis and presentation.

Like that previous group of 22 tools, these range from easy enough for a beginner (i.e., anyone who can do rudimentary spreadsheet data entry) to expert (requiring hands-on coding). Here are eight of the best:

CSVKit

What it does: This utility suite available from Christopher Groskopf’s GitHub account has a host of Unix-like command-line tools for importing, analyzing and reformatting comma-separated data files.

What’s cool: Sure, you could pull your file into Excel to examine it, but CSVKit makes it quick and easy to preview, slice and summarize.

For example, you can see all your column headers in a list — which is handy for super-wide, many-column files — and then just pull data from a few of those columns. In addition to inputting CSV files, it can import several fixed-width file formats — for example, there are libraries available for the specific fixed-width formats used by the Census Bureau and Federal Elections Commission.

Two simple commands will generate a data structure that can, in turn, be used by several SQL database formats (Mr. Data Converter handles only MySQL). The SQL code will create a table, inferring the proper data type for each field as well as the insert commands for adding data to the table.

The Unix-like interface will be familiar to anyone who has worked on a *nix system, and makes it easy to save multiple frequently used commands in a batch file.

Drawbacks: Working on a command line means learning new text commands (not to mention the likely risk of typing errors), which might not be worthwhile unless you work with CSV files fairly often. Also, be advised that this tool suite is written in Python, so Windows users will need that installed on their system as well.

Skill level: Expert

Runs on: Any Windows, Mac or Linux system with Python installed.

Learn more: The documentation includes an easy-to-follow tutorial. There’s also a brief introductory slide presentation that was given at the NICAR conference last month.

Related tools: Google Refine is a desktop application that can do some rudimentary file analysis as well as its core task of data cleaning; and The R Project for Statistical Computing can do more powerful statistical analysis on CSV and other files.

DataTables

What it does: This popular jQuery plug-in (which was designed and created by Allan Jardine) creates sortable, searchable HTML tables from a variety of data sources — say, an existing, static HTML table, a JavaScript array, JSON or server-side SQL.

Source: Apple earnings statements

What’s cool: In addition to sortable tables, results can be searched in real time (results are narrowed further with each search-entry keystroke).

Drawbacks: Search capability is fairly basic and cannot be narrowed by column or by using wildcard or Boolean searches.

Skill level: Expert

Runs on: JavaScript-enabled Web browsers

Learn more: Numerous examples on the DataTables site show many ways to use this plug-in.

FreeDive

What it does: This alpha project from the Knight Digital Media Center at UC Berkeley turns a Google Docs spreadsheet into an interactive, sortable database that can be posted on the Web.

What’s cool: In addition to text searching, you can include numerical range-based sliders. Usage is free. End users can easily create their own databases from spreadsheets without writing code.

FreeDive turns a Google Docs spreadsheet into an interactive, sortable database

FreeDive’s chief current attraction is the ability to create databases without programming; however, freeDive source code will be posted and available for use once the project is more mature. That could appeal to IT departments seeking a way to offer this type of service in-house, allowing end users to turn a Google Doc into a filterable, sortable Web database using the Google Visualization API, Google Query Language, JavaScript and jQuery — without needing to manually generate that code.

Drawbacks: My test application ran into some intermittent problems; for example, it wouldn’t display my data list when using the “show all records” button. This is an alpha project, and should be treated as such.

In addition, the current iteration limits spreadsheets to 10 columns and a single sheet. One column must have numbers, so this won’t work for text-only information. The search widget is currently limited to a few specific choices of fields to search, although this might increase as the project matures. (A paid service like Caspio would offer more customization.) The nine-step wizard might get cumbersome after frequent use.

Skill level: Advanced beginner.

Runs on: Current Web browsers

Learn more: The freeDive site includes several video tutorials at the bottom of the home page as well as test data to try out the wizard.

Related tools: Caspio is a well-established commercial alternative. For a JavaScript alternative with more control over the table created from a Google Docs spreadsheet, you might want to investigate Tabletop, which makes a Google Docs spreadsheet accessible to JavaScript code.

Highcharts JS

What it does: This JavaScript library from Highsoft Solutions provides an easy way to create professional-looking interactive charts for the Web. JQuery,Mootools or Prototype required.

What’s cool: With Highcharts, users can mouse over items for more details; they can also click on items in the chart legend to turn them on and off. There are many different chart types available, from basic line, bar, column and area charts to zoomable time series; each comes with six stylesheet options. Little customization is needed to get a sleek-looking chart — and charts will display on iOS and Android devices as well as on desktop browsers.

Drawbacks:Highcharts, like GoogleMaps, does have a distinctive look, so you may want to customize the Highcharts stylesheets so your visualizations don’t look like numerous other Highcharts on the Web. While charts displayed fine for me on an Android phone, they weren’t interactive (they were on an iPad).

And unlike most JavaScript/jQuery libraries, Highcharts is free only for non-commercial use, although a site-wide license for many companies costs only $80. (The cost jumps to $300 per developer seat in some cases — for example, if charts are customized for individual users.) Rendering can be slow in some older browsers (notably Internet Explorer 6 and 7).

Skill level: Intermediate to Expert.

Runs on: Web browsers

Learn more: The Highcharts demo gallery includes easy-to-view source code; the documentation explains other options.

Related tools: Google Chart Tools create static image charts and graphs or more interactive JavaScript-based visualizations; there are also JavaScript libraries such as Protovis and the JavaScript InfoVis Toolkit. Exhibit is an MIT Simile Project spinoff designed for presenting data on the Web with filtering, sorting and interactive capabilities.

Mr. Data Converter

What it does: How often do you have data in one format — while your application needs it in another? New York Times interactive graphics editor Shan Carter ran into this situation often enough that he coded a tool that converts comma- or tab-delimited data into nine different formats. It’s available as either a service on the Web or an open source tool.