Data Formats

Data comes in a thousand and one formats, some friendlier than others. Let’s review a few!

API

APIs – application programming interface – is a way for computers to communicate to one another. For us, this generally means sharing data. We’ll be coding up Python scripts to talk to and request data from machines around the world, from Twitter to the United States government.

CSV

Comma-separated values are the most common format for data. It’s a quick export away from Excel or Google Spreadsheets, and you’ll find yourself working from CSV’s more often than not.

Although “comma-separated” is in the name, a CSV can arguably also use tabs, pipes, or any other character as a field delimiter (although the tab-separated one can also be a TSV).

JSON

JSON stands for JavaScript Object Notation, and it’s a slightly more complicated format than a CSV. It can contain lists, numbers, sub-items, and all sorts of complexities that are great for expressing the nuance of real-world data. Data from APIs is often formatted as JSON.

Shapefiles

Shapefiles are by far the most common format for geographic data. City council districts, state boundaries, and the nearest wifi spots can often be found as shapefiles. You can import them into software like QGIS or convert them to geography-friendly JSON.

GeoJSON and TopoJSON

GeoJSON and TopoJSON are both specially-formatted JSON files that contain geographic information.

SQL

SQL is the language used to talk to databases, and you’ll sometimes find datasets in SQL format, ready to be imported into your database of choice.