MapBox OSM Development

Gps And Gpx

2013-04-05T00:00:00-04:00

iD alpha3, with Radically Better Feature Editing

2013-03-11T00:00:00-04:00

We just made iD, our new OpenStreetMap editor, simpler and friendlier for new users. This is part of today’s Alpha3 launch that also includes performance improvements, some friendly UI adjustments, and bug fixes. Try editing OpenStreetMap with iD alpha3!

Feature Interface

OpenStreetMap uses tags – pairs of keys and values such as shop=butcher – to represent features and their attributes. Knowing how to tag a feature is one of the hardest parts of editing OpenStreetMap, because there are thousands of different tags and values, and most features require a combination of different tags. For instance, the highway tag to identify basic road type, but details like speed limits, access, and surface require additional tags.

This idea of tags is essential to handle the complexity and detail of mapping the world, but it’s intimidating for new users.

With iD’s new interface, users can create & edit features without being tagging wizards. After drawing a new feature, you can search for a feature type, or select one from the grid. The search is smart and includes synonyms, so you find a feature without having to know its exact name - searching for ‘coffee’ brings up the ‘café’ preset.

For active mappers, we keep recently used options in the grid, so it’s easy to add many features of the same type.

The feature editor then shows relevant attributes with suggested values. Specific inputs, such as the address input which suggests nearby roads, make adding values even easier.

The interface builds on the ideas of existing OSM editors like Potlatch 2 and JOSM and integrates them further into the default editing workflow.

There is still plenty to do – we plan on improving search suggestions, adding more feature types, and customizing more form elements. This is another area where iD must strike the right balance between flexibility and simplicity.

Operations

You can now rotate areas, merge entities of different types, and move multiple objects at once.

Performance

We started using R-Trees for performance tuning iD’s fast clientside labeling and are now using them for much more. In alpha3, all entities are stored in an R-Tree so finding them is super fast.

One-way markers now use a technique by Mike Bostock that improves their appearance and performance.

Design

Point features now use icons from Maki.

Some land use and natural features now have a patterned fill to help identify them.

Translations

Thanks to all the translators, iD is now available in Danish, German, Spanish, French, Italian, Japanese, Latvian, Dutch, Polish, Russian, Turkish, Ukrainian, and Vietnamese!

Thanks to Yohan and Porjo for code contributions.

Try editing OpenStreetMap width iD alpha3!

iD Architecture: Map Rendering and Other UI

2013-02-28T00:00:00-05:00

The third in a three-part series on the technical architecture of the iD editor. Previously we looked at iD’s core data types and actions and modes, behaviors, and operations.

Let’s look at how iD renders entities on the map and its pattern for abstracting other UI components.

Map Rendering

The core of iD’s map rendering system is iD.Map, which takes care of the map projection, zooming and panning, and states like center and zoom level. iD.Map also coordinates a system of smaller pieces that render a vector map.

For rendering entities on screen, iD adopts a geometric vocabulary that’s a higher-level representation than OSM’s simple data model:

A point is a node that is not a member of any way.
A vertex is a node that is a member of at least one way.
A line is a way that is not an area.
An area is a way that is circular and has certain tags, or a series of one or more ways grouped in a multipolygon relation.

For each of these geometric types, iD.svg has a corresponding module: iD.svg.Points, iD.svg.Vertices, iD.svg.Lines, and iD.svg.Areas. iD.Map delegates to these modules to render entities on screen.

Each piece of data is connected to the rendering with d3’s join system. When iD loads and renders an entity for the first time, it is part of the enter selection and the SVG elements needed to represent it are created. When a user modifies an entity, it is part of the update selection and the appropriate attributes of the SVG element, like those that specify the location on screen, are updated. And when a user deletes an entity or moves it offscreen, the corresponding SVG element is in the exit selection, and will be removed.

The iD.svg modules apply classes to SVG elements based on tags, via iD.svg.TagClasses. For example, an entity tagged with highway=residential gets two classes: .tag-highway and .tag-highway-residential. This allows distinct visual styles to be applied via CSS at either the key or key-value levels. Elements also receive a class corresponding to their entity type (node, way, or relation) and one corresponding to their geometry type (point, line, or area).

The iD.svg namespace has a few other modules that don’t have a one-to-one correspondence with entities:

iD.svg.Midpoints renders small “virtual nodes” at the midpoints between two vertices.
iD.svg.Labels renders textual labels.
iD.svg.Surface sets up a stack of SVG groups that ensure that map elements appear in an appropriate z-order.

Other UI

iD provides many user interface elements beyond than the map, including the page footer, an interface for saving changes, and geocoding controls.

The implementations for all non-map UI components live in the iD.ui namespace. These modules follow a pattern for reusable d3 components Mike Bostock suggested in the context of charts. The entry point to a UI element is a constructor function; for example iD.ui.Geocoder(). The constructor function may require a set of mandatory arguments. Most UI components require exactly one argument: a context object produced by the top-level iD() function.

A component needs some way to be rendered on screen by creating new DOM elements or manipulating existing elements. This is done by calling the component as a function, and passing a d3 selection where the component should render itself:

var container = d3.select('body').append('div')
    .attr('class', 'map-control geocode-control');

var geocoder = iD.ui.Geocoder(context)(container);

Alternatively, and more commonly, the same result is accomplished with d3.selection#call:

d3.select('body').append('div')
    .attr('class', 'map-control geocode-control')
    .call(iD.ui.Geocoder(context));

Some components are reconfigurable and provide functionality beyond basic rendering, by providing methods off of the function:

var inspector = iD.ui.Inspector();
inspector(container); // render the inspector
inspector.tags(); // retrieve the current tags
inspector.on('change', callback); // get notified when a tag change is made

Future Direction

We hope you enjoyed the deep dive on iD’s technical internals. As iD evolves, no doubt some of these details will change, and new architectural pieces will emerge as we build out remaining features like presets. We’ve captured the text of these posts in the ARCHITECTURE.md file in the iD repository, and will keep it up to date as the project matures.

iD Architecture: Modes, Behavior, and Operations

2013-02-27T00:00:00-05:00

The second in a three-part series on the technical architecture of the iD editor. Yesterday we looked at iD’s core data types and actions.

Closer to the user interface of iD lie three abstractions for a user’s interaction with the system: modes, behaviors, and operations. Modes contain the code behind the ‘Browse’, ‘Point’, ‘Line’ and ‘Area’ buttons at the top left, behaviors provide a means of reusing common bits of code across multiple modes, and operations provide the user interface for actions. Let’s look at each in detail.

Modes

Modes are settings that change the behavior of an application: for instance, drawing applications tends to have a ‘move’ mode, which moves the picture, as well as a ‘brush’ mode, in which the same clicks on the picture draw lines.

Modes are useful because they allow user interfaces to be customized for certain narrow uses, and for common actions, like clicks and drags, to have multiple meanings. However, they pose a challenge for new users, who might not discover them and have access to functionality, or will not realize what mode they’re currently in.

In iD modes are manifested in the interface by the four buttons at the top left:

The modality of existing OSM editors runs the gamut from Potlatch 2, which is nearly modeless, to JOSM, which sports half a dozen modes out of the box and has many more provided by plugins. iD seeks a middle ground: too few modes can leave new users unsure where to start, whereas too many can be overwhelming.

iD’s initial mode is ‘Browse’: one can drag the map and select entities to edit. From there, users can enter three geometrically-oriented drawing modes, Point, Line, and Area, through the mode buttons or key-shortcuts.

The geometric modes are also split into a mode for the initial point drawn object and another for continuing an existing object. The exception to this rule is points, which have a single step.

Selection is handled by a specific ‘Select’ mode, which displays editing tools for tags and operations.

The API for each mode consists of two methods: enter and exit.

The enter method sets up all the behavior that should be present when that mode is active. This typically means adding listeners to DOM events that will be triggered on map elements, installing keybindings, and showing certain parts of the interface like the inspector in Select mode.

The exit method does the opposite, removing the behavior installed by the enter method. Together the two methods ensure that modes are self-contained and exclusive: each mode knows exactly the behavior that is specific to that mode, and exactly one mode’s behavior is active at any time.

Behavior

Modes share functionality, which we abstract into behaviors. For example, in both the Browse and Draw modes, iD indicates interactive map elements by drawing a halo around them on mouse hover. Instead of duplicating the code to implement this behavior in all these modes, we extract it to iD.behavior.Hover.

Behaviors are inspired by d3’s behaviors. Like d3’s d3.behavior.zoom and d3.behavior.drag, each iD behavior is a function that takes as input a d3 selection (assumed to consist of a single element) and installs DOM event bindings necessary to implement the behavior. iD.behavior.Hover, for example, installs bindings for the mouseover and mouseout events that add and remove a hover class from map elements.

Because certain behaviors are appropriate to some but not all modes, we need the ability to remove a behavior when entering a mode where it is not appropriate. d3’s own behaviors don’t offer this functionality yet. Each behavior implements an off function that uninstalls the behavior. This is very similar to the exit method of a mode, and in fact many modes do little else but uninstall behaviors in their exit methods.

To make modes and behaviors more concrete, here’s an annotated extract from iD.modes.Browse:

// Each mode is constructed with a context. This is a singleton object which
// encapsulates some global state. It's constructed via the top-level iD()
// function.
iD.modes.Browse = function(context) {
    // Using the module pattern here: the constructor creates an object,
    // sets various properties, some of which are functions, and then returns
    // it.
    var mode = {
        id: 'browse'
    };

    // The browse mode is just a collection of behaviors.
    var behaviors = [                    // In browse mode, you can:
        iD.behavior.Hover(),             // * Highlight entities on hover
        iD.behavior.Select(context),     // * Select entities via clicks
        iD.behavior.Lasso(context),      // * Select entities via a shift-drag lasso
        iD.behavior.DragNode(context)];  // * Drag nodes around the map

    // When entering the mode, install each of the behaviors.
    mode.enter = function() {
        behaviors.forEach(function(behavior) {
            context.install(behavior);
        });
    };

    // When exiting the mode, uninstall each of the behaviors.
    mode.exit = function() {
        behaviors.forEach(function(behavior) {
            context.uninstall(behavior);
        });
    };

    return mode;
};

Operations

Operations wrap actions, providing their user-interface: tooltips, key bindings, and the logic that determines whether an action can be performed given the current map state and selection. Each operation is constructed with the list of IDs which are currently selected and a context object which provides access to the history and other important parts of iD’s internal state. After being constructed, an operation can be queried as to whether or not it should be made available (i.e., show up in the context menu) and if so, if it should be enabled.

We make a distinction between availability and enabled state for the sake of learnability: most operations are available so long as an entity of the appropriate type is selected. Even if it remains disabled for other reasons (e.g., because you can’t split a way on its start or end vertex), a new user can still learn that “this is something I can do to this type of thing”, and a tooltip can provide an explanation of what that operation does and the conditions under which it is enabled.

To execute an operation, call it as a function, with no arguments. The typical operation will perform the appropriate action, creating a new undo state in the history, and then enter the appropriate mode. For example, iD.operations.Split performs iD.actions.Split, then enters iD.modes.Select with the resulting ways selected.

// An array of the ids of the currently selected entities.
var selection = ['w12345'];

// Construct the operation. `context` is a context object created by calling `iD()`.
var operation = iD.operations.Split(selection, context);

// Can the operation be performed given the current selection
// and state of the map?
if (operation.available() && operation.enabled()) {
    // If so, do it!
    operation();
}

Next time

We wrap up our architecture series with a discussion of map rendering and other UI components.

iD Architecture: Core and Actions

2013-02-26T00:00:00-05:00

The first in a three-part series on the technical architecture of the iD

The launch of iD, the new web-based editor that we’re building for OpenStreetMap, generated interest from technical users interested in contributing to development or using parts of iD in other tools. If that describes you, or if you’re just curious about the technical details of building an editor, this series is for you.

Let’s start by looking at the code at the center of iD: the core and actions. While iD has an impressive display layer which we’ve improved and refined, the core is independent of map rendering or any UI.

This core implements the very basics of the OpenStreetMap data model and the datastructures needed to track changes. Actions build on this core, providing the “business logic” for making changes to the map data. We’ve aimed to make both highly reusable, hoping that they will prove useful in other OpenStreetMap-related tools.

iD is a relatively traditional editor built on this platform, but the possibilities are endless - purpose-specific editors, more advanced user interfaces, or even using iD’s core purely for querying and displaying OSM data.

Core

The iD core implements OSM data types, a graph of OSM objects’ relationships to one another, an undo/redo history for changes made during editing, and a couple of important auxiliary classes. It eventually aims to be a reusable, modular library to kickstart other JavaScript-based tools for OpenStreetMap.

The OSM data model includes three basic data types: nodes, ways, and relations.

A node is a point type, having a single geographic coordinate.
A way is an ordered list of nodes.
A relation groups together nodes, ways, and other relations to provide free-form higher-level structures.

Each of these three types has tags: an associative array of key-value pairs which describe the object.

In iD, these three types are implemented by iD.Node, iD.Way and iD.Relation. These three classes inherit from a common base, iD.Entity. This is the only use of classical inheritance in iD, but it’s justified by the common functionality of the types. Generically, we refer to a node, way or relation as an entity.

Every entity has an ID either assigned by the OSM database or a negative, local identifier assigned by iD for newly-created objects. IDs from the OSM database as treated as opaque strings; no assumptions are made of them other than that they can be compared for identity and do not begin with a minus sign (and thus will not conflict with proxy IDs). The three types of entities have separate ID spaces: a node can have the same numeric ID as a way or a relation. Instead of segregating ways, nodes, and other entities into different datastructures, iD internally uses fully-unique IDs generated by prefixing each OSM ID with the first letter of the entity type. For example, a way with OSM ID 123456 is represented as ‘w123456’ within iD.

iD entities are immutable: once constructed, an Entity object cannot change. Tags cannot be updated; nodes cannot be added or removed from ways, and so on. Immutability makes it easier to reason about the behavior of an entity: if your code has a reference to one, it is safe to store it and use it later, knowing that it cannot have been changed outside of your control. It also makes it possible to implement the entity graph (described below) as an efficient persistent data structure.

Since iD is an editor, it must allow for new versions of entities. The solution is that all edits produce new copies of anything that changes. At the entity level, this takes the form of methods such as iD.Node#move, which returns a new node object that has the same ID and tags as the original, but a different coordinate. More generically, iD.Entity#update returns a new entity of the same type and ID as the original but with specified properties such as nodes, tags, or members replaced.

Entities are related to one another: ways have many nodes and relations have many members. To render a map of a certain area, iD needs a datastructure to hold all the entities in that area and traverse these relationships. iD.Graph provides this functionality. The core of a graph is a map between IDs and the associated entities; given an ID, the graph can give you the entity. Like entities, a graph is immutable: adding, replacing, or removing an entity produces a new graph, and the original is unchanged. Because entities are immutable, the original and new graphs can minimize memory use by sharing references to entities that have not changed instead of copying the entire graph. This persistent data structure approach is similar to the internals of the git revision control system.

The final major component of the core is iD.History, which tracks the changes made in an editing session and provides undo/redo capabilities. Here, the immutable nature of the core types really pays off: the history is a simple stack of graphs, each representing the state of the data at a particular point in editing. The graph at the top of the stack is the current state, off which all rendering is based. To undo the last change, this graph is popped off the stack, and the map is re-rendered based on the new top of the stack.

This approach constitutes one of the main differences between iD’s approach to data and that of JOSM and Potlatch 2. Instead of changing a single copy of local data and having to implement an ‘undo’ for each specific action, actions in iD do not need to be aware of history and the undo system.

Finally, we have the auxiliary classes iD.Difference and iD.Tree.

iD.Difference encapsulates the difference between two graphs, and knows how to calculate the set of entities that were created, modified, or deleted, and need to be redrawn.

var a = iD.Graph(), b = iD.Graph();
// (fill a & b with data)
var difference = iD.Difference(a, b);

// returns entities created between and b
difference.created();

iD.Tree calculates the set of downloaded entities that are visible in the current map view. To calculate this quickly during map interaction, it uses an R-tree.

var graph = iD.Graph();
// (load OSM data into graph)

// this tree indexes the contents of the graph
var tree = iD.Tree(graph);

// quickly pull all features that intersect with an extent
var features = tree.intersects(
    iD.geo.Extent([0, 0], [2, 2]), tree.graph());

Actions

In iD, an action is a function that accepts a graph as input and returns a new, modified graph as output. Actions typically need other inputs as well; for example, iD.actions.DeleteNode also requires the ID of a node to delete. The additional input is passed to the action’s constructor:

// construct the action: this returns a function that remembers the
// value `n123456` in a closure so that when it's called, it runs
// the specified action on the graph
var action = iD.actions.DeleteNode('n123456');

// apply the action, yielding a new graph. oldGraph is untouched.
newGraph = action(oldGraph);

iD provides actions for all the typical things an editor needs to do: add a new entity, split a way in two, connect the vertices of two ways together, and so on. In addition to performing the basic work needed to accomplish these things, an action typically contains a significant amount of logic for keeping the relationships between entities logical and consistent. For example, an action as apparently simple as DeleteNode, in addition to removing the node from the graph, needs to do two other things: remove the node from any ways in which it is a member (which in turn requires deleting parent ways that are left with just a single node), and removing it from any relations of which it is a member.

As you can imagine, implementing all these details requires an expert knowledge of the OpenStreetMap data model. It is our hope that JavaScript based tools for OpenStreetMap can reuse the iD’s core implementation, significantly reducing the work necessary to create a robust tool.

Next time

Up next, we shift gears from abstract data types and algorithms to the parts of the architecture that implement the user interface core of iD: modes, behavior, and operations.

iD alpha2

2013-02-14T00:00:00-05:00

The iD editor for OpenStreetMap has reached an alpha2 release, with bug fixes, new features, and improvements to performance and usability.

You can try the alpha2 release now, as well as the the latest development version thanks to OpenStreetMap US.

Live By Default

iD now displays and edits live, official OpenStreetMap data by default. In previous releases, we’ve relied on the development server for a safe sandbox for iD, but the editor is now stable enough to be trusted with real data, and the iD team uses it as a daily editor.

This also means that you don’t need a dev server account - your default OSM account will work on the iD testing instance.

Now Available in 日本語!

iD is now available in English, Latvian, Turkish, Spanish, Japanese, German, and French!

This is all thanks to people who graciously put in the time and energy to contribute translations: Satoshi, Martin, Iván, Christian, m0rix, LuisGC, Martins Brunenieks, Alper, and others. Thanks!

Want to help translate iD? We now have a translator’s guide and a very simple system for handling translations.

Performance

We’ve been constantly focused on making iD fast in every way possible, whether through smart architectural choices, like Ansis’s usage of RTrees for labeling, or micro optimizations that ensure iD’s ‘hot’ functions are incredibly fast. We’ve also started profiling iD in Firefox and paying attention to reducing repaints and reflows wherever possible.

Design Improvements

We’ve tweaked road styles to make them more visible, consistent, and beautiful.

The radial menu that was introduced in alpha1 is now a semi-circle so it stays out of the way of the mouse.

Multiple Selections

iD now supports selecting multiple features, so you can quickly remove many points or ways. Holding down shift, clicking, and dragging now produces a ‘lasso’ tool for multiple selections that should be familiar to those who have used vector graphics editors like Illustrator.

This also enabled us to add more operations to iD: you can select multiple items and merge them into one, or delete multiple things at once.

Region-Specific Layers

iD now provides region-specific layers that automatically update as you move the map to different locations. For instance, when editing in the US, you’ll have access to US Census TIGER layers, but if you pan over to London, you can reference Ordinance Survey Open Data layers.

This functionality is thanks to Potlatch 2’s excellent layer resources, which we’re converting to JSON and using in iD.

Usability Improvements

Some of the biggest changes to iD are very subtle usability improvements. You can now connect lines by dragging the end of one onto another - an incredibly useful tweak that addresses the common case in OSM data of data that is poorly connected.

Dragging points and lines to the edges of the map causes the map to scroll, so you don’t have to ‘drop’ the point and scroll the map manually.

Click events now have more tolerance so drawing lines is radically faster and less error-prone.

Recovering Changes

Sometimes you can’t finish your edits in one session and need to close the browser window. iD now uses localStorage to save intermediate work, and offers to recover changes when you load the page again.

Labeling in OpenStreetMap's iD Editor

2013-02-12T00:00:00-05:00

One of the key features in the alpha1 release of the iD map editor was its powerful support for labels. Labels are essential for orienting users and enabling them to find and fix misspellings and omissions. The technical challenge is to place labels quickly and avoid problems like overlapping text and visual clutter.

The basic goal of labeling is identify important features while avoiding overlapping labels. Because iD is an editor for a dynamic dataset, it also needs to handle labels which are updated by the user, hide labels so they don’t interfere with editing, and prioritize labeling features in the currently visible area. Traditional algorithms for automatic label placement use relatively complex simulated forces or annealing to find optimal label placements. In contrast, iD’s approach is simple, but effective for its editing-centric use.

First, iD collects and sorts all points, lines, areas in order of importance, based on their tagged attributes. For each of these potentially labeled features, the algorithm calculates possible positions for the label, iterating through different options to avoid conflicts with previously placed labels. In this way, it’s a fast greedy algorithm, and does run into scenarios in which it can’t place a label.

The set of possible label positions that are calculated for a feature depends on its type. For lines, labels must be shorter than the line and within the visible area. Line labels are flipped if necessary to ensure they do not appear upside-down. Area labels are placed only when they fit in the area’s bounding box. Point labels are currently simple: they’re always placed to the right of the point.

The labels are rendered using SVG. Point and area labels use text elements, whereas lines use textPath. Each label has a rect or path halo placed behind it to make the text readable on top of imagery and features. Labels are rendered on top of all other map features.

To keep editing fast and clear, labels are intelligently shown and hidden based on mouse position - so hovering near a road will fade out the label to reveal any hidden details.

Because labels are updated with each redraw of the map, they need to be extremely fast. iD calculates label bounding boxes based on estimated text length, since it’s slow to use browser-calculated bounds. We use RTree to optimize collision detection and proximity searches, a technique inspired by kothic.js.

Thanks to Vladimir and Tom for their suggestions.

iD’s labeling is robust and fast, but there’s still more work to do: we have a roadmap for further improvements that will be part of ongoing iD improvements.

Try it for yourself and take iD for a spin! If you find an area with poor labeling, please report it in an issue.

iD alpha1

2013-01-29T00:00:00-05:00

Today iD reached a new level of stability with an alpha1 release.

Since releasing iD alpha0 in late December, we’ve made over 500 commits to the project, including a redesign, many bugfixes, usability improvements, and features.

John, Ansis, Saman, and I have started to use iD on the production OSM system with real data, and started our first real new-user testing at the OpenStreetMap editathon at MapBox. We’re confident that iD is reliable enough for real use, so you can now switch to editing openstreetmap.org instead of api06.dev.openstreetmap.org by clicking the dev button in the bottom-right of the interface.

Labels

Ansis wrote a sophisticated labeling system in iD, which makes it easier to identify unnamed and misnamed features and quickly orient yourself in new areas. It uses a fast RTree implementation and takes inspiration from Kothic.js and other javascript-based renderers.

Performance

We’re still pushing hard on performance - we improved the way that iD stores, updates, and searches its database, which is implemented as a persistent data structure. Navigating the map is also faster thanks to increased usage of CSS transforms.

Multipolyons and Relations

John wrote a vastly improved system for relations and multipolygons in iD, which means that multipolygon relations are now displayed and edited correctly. This was one of the largest final issues towards iD editing real map data, especially data in cities with complex buildings and atriums.

Try it Out

We need testers and input now more than ever. The testing instance is now fixed to the alpha1 release, so try it out! Click ‘dev’ in the bottom-right to switch to the official OpenStreetMap.org server and your own OSM account.

File tickets on GitHub for anything amiss, or ask us (jfire, tmcw, ansis, and samanbb) in the #ideditor room on irc.oftc.net. We’d love to know what’s confusing about iD and how we can improve it.

On to alpha2

This is just the beginning: some of the most interesting functionality in iD is still to come. Our big goals for an alpha2 release will be presets support, translations, and operations for more advanced editing actions and tools.

Taginfo

2013-01-17T00:00:00-05:00

Taginfo, a project by Jochen Topf, has been a huge element in making iD great for editing and learning about OpenStreetMap’s tag system.

Many of the most important features in iD are here because of the hard work of others in the community: the OSM API, OpenStreetMap US’s TIGER 2012 layer, algorithms pioneered in Potlatch 2 and JOSM, and much more.

Taginfo is written in a smart style - a C++ parser for wiki & OSM Planet using Jochen’s own osmium framework, combined with a simple sinatra-based web frontend. This means that it’s fast in practice, and a service that we’ve grown to rely on.

The Taginfo API is extensive: it even generates maps of worldwide tag usage, to find region-specific patterns. Here’s a map of the distribution of the highway= key.

We first started using Taginfo to power autocomplete functionality in iD: given a key, we can show popular values, and even filter through these values so that keys with many different values, like name, don’t have confusing autocomplete suggestions.

To take this concept even further, we’re using Taginfo’s wiki_pages API to bring in deeper documentation for specific tag combinations, informing users of particulars like whether a specific road type matches up with highway=motorway_link or how to tag a cycleway.

We’re able to include images from wiki pages because of a new version of the API which Jochen kindly developed with iD in mind.

Taginfo has been a great resource and enables iD to provide a critical improvement to conventional tagging interfaces: tags are still entered through an unlimited freeform, but more useful context around common choices is provided as guidance for both, new and experienced mappers.

OAuth in Javascript

2013-01-15T00:00:00-05:00

With iD, you can securely log in to OpenStreetMap, make changes, and save them to the OpenStreetMap.org database. This is made possible by iD’s implementation of the OAuth specification, a standard system for applications to authenticate and interact. We’ve developed a tiny library called ohauth which provides the basic components of the process, and includes a trusted implementation of sha1 written by Paul Johnston.

iD is an unusual and interesting implementation of OAuth because it’s a purely browser-based Javascript application: as opposed to conventional implementations, there’s no server application in between it and the OSM API.

To make this work, we needed to write much of the fundamental code for authenticating: while JOSM and Potlatch 2 use libraries for Java and Actionscript, respectively, and TileMill uses a node.js OAuth implementation, there’s no established implementation for OAuth 1.0 in purely client-side applications. OAuth 2.0 supports a simplified client-side flow and thus has a well-written Javascript library, but is not supported on OpenStreetMap.org.

The OpenStreetMap website is powered by a Ruby on Rails application called the Rails Port, which luckily uses a well-tested oauth-plugin for its side of the authentication exchange.

Cross Origin Resource Sharing

The biggest challenge for this implementation is the same origin policy that applies to browser Javascript which uses AJAX across domains. iD is built to be hostable anywhere, with implementations on geowiki.com and elsewhere, which can all access the OSM API. So we can’t guarantee that requests for resources, like authentication calls, will be on the same domain.

In the past, we’ve used JSONP in this case: a hack that allows one to request JSON resources across domains. JSONP has many known flaws: potential security flaws, an inability to read and set headers, no support for methods other than GET, and the response body format is traditionally limited to JSON (though you can stringify XML in JSONP).

Thus cross-domain OAuth can’t be done with JSONP, and requires actual cross-domain requests with CORS. Luckily, OpenStreetMap supports the standard, which provides a simple way to jump across the border.

The downside is that support for CORS in Internet Explorer is nearly non-existent: only in IE10 is it supported in standard form. iD does not support IE before version 10 at the moment (with a distant possibility of IE9 support in the future).

Javascript Implementations

As mentioned, OAuth.org links to a javascript library for OAuth 1.

That library unfortunately leaves much to be desired, both in terms of style and design, and isn’t immediately usable in a project like iD. We do, however, use the same implementation of SHA1 by Paul Johnston.

Thus iD uses a somewhat fixed-up SHA1 library and a very simple library titled ohauth, which inherits most strategy from the official implementation. This basic functionality is then wrapped with UI elements and logic in iD.

The Authorize Step

Besides a range of AJAX requests, the OAuth specification requires a step in which users are directed towards the ‘provider’ and choose what permissions they want to give iD. For instance, they can prevent iD from having the ability to view or edit their user profile.

iD initially tried to redirect the entire page to OSM’s /oauth/authorize page, which required a cookie with the value of the request_token so that it can be reused for the access_token step. This isn’t just hackish, but it has a very bad failure case: if that request fails, the user is stranded on a broken page.

The second technique is to use an <iframe> to do authorization. This works much better with the flow of authentication, but interacts poorly with browser security controls: Safari doesn’t permit cookie access in iframes, and all browsers will complain about the necessary step of extracting the iframe’s URL for OAuth: when the iframe is on openstreetmap.org instead of localhost:3000 (or the other domain name/port of your testing page), trying to access its location is against your browser’s rules.

iframe.onload = function() {
    if (iframe.contentWindow.location.search) {

Thus, iD now uses a popup window in order to execute the authorize step of OAuth. This has quite a few advantages: it works in Safari, users are able to easily see what site they’re authenticating against, and bailing on the process is easy: they can simply close the window.

However, this includes yet another wrinkle: the Javascript API for opening windows, window.open, has been abused and overused by many advertising sites, so many browsers limit its use: the only reliable way to open a popup in WebKit-based browsers is to call window.open within the scope of an event handler that is responding to a user interaction like a mouse click. This is a viable approach even if it’s easy to lose that scope like shown in the example below.

a.onclick = function() {
    window.open('foo.html'); // this window will open
};

b.onclick = function() {
    window.setTimeout(function() {
        window.open('foo.html'); // this window will not open
    }, 100);
};

Designing iD

2013-01-11T00:00:00-05:00

We launched the Alpha0 version of iD two weeks ago, and with it, came a host of design changes. At this point, the basics are in place and the project is in a good place to start further refinement.

There’s now a simple layout with a comprehensive icon set and consistent button styles. I tried to keep things clean and straightforward for now. We can refine things later where we need to:

We added custom cursors to make mouse interactions feel more intuitive:

We implemented tooltips for inline help:

Map features, which are rendered in SVG, have been refined. I tried to make sure the markers, nodes, and ways were big enough to easily click and modify, but not so intrusive as to obscure the map. We added special hover and active states to make it clear when an element is selected or can be interacted with:

There’s a consistent style for modals that we use when we need to provide the user with more information or when an action is required:

We approached the project collaboratively: I started with a full mock-up of the application:

Which allowed me to easily try ideas out before implementing them:

Once I had a concept down I was happy with, I made issues in the issue tracker. Tom, John and I would refine the concepts, then we’d start on implementation. If the task was simple, like restyling buttons, I did most of the work myself. It took a while to get the hang of editing the DOM with d3.js, but now I can work with it more or less as though it’s HTML.

Shortcomings of the current design and plans for the future

iD is a functioning editor, but it’s far from finished.

For starters, the layout isn’t quite polished yet. Some parts of the site are flexible, others are fixed width. I’d love to implement a more cohesive system so elements of the UI fit together on a stronger grid. Right now the footer links feel a little too tacked on, and the tag inspector needs to be more carefully positioned and needs to have its behavior refined.

The built-in tag reference still has a long way to go before its as useful as it could be. There’s a issue on the topic open right now.

The SVG styles for map data are close, but still not complete. I’d love to have more consistent styles for selection across buildings, areas, and streets.

The most important & challenging design task that is yet to be tackled, is an easy to use preset interface for users who aren’t familiar with OSM’s tagging system. Discussion on the topic has already begun.

Lastly, we need a better onboarding experience for new users. There are a few ways to approach this: Do we want to create a pop-based step-by-step walkthrough? A help page? Should we simply try to make the interface more intuitive and the existing tooltips stronger so we don’t even need a walkthrough?

Over the course of the next few weeks, I will be working together with other iD contributors to refine existing concepts, propose new ideas, and help make sure everything’s implemented properly.

Join the design conversation by creating issues or providing input on the issue tracker. Or, if you’re feeling ambitious, fork the project and start playing around with the SVG source files! For questions you can always find me and other contributors on #ideditor on irc.oftc.net.

iD Alpha0

2012-12-22T00:00:00-05:00

Today we’re tagging the very first version of the iD editor for OpenStreetMap. It’s come a very long way in the past few months since kicking off after Richard’s talk at the State of the Map US conference: design improvements by Saman, testing infrastructure and core data strategy with John, and a lot more.

It also has a long way to go, but we’re going to shift development into feature & bug branches so that you can start tinkering with it and giving input.

Here’s a quick idea of what’s there to try out:

Points, Lines, Areas, and Tags

iD now supports creating, modifying, and deleting all of the fundamental datatypes of OpenStreetMap: you can draw, tag, and increase the accuracy of roads, buildings, and places on the map.

The tag editor takes advantage of Taginfo for inline autocomplete and documentation. Popular tag combinations are shown as you type.

iD’s background layer system already supports Bing, TIGER, OSM, and custom layers which use XYZ and quadtile schemes. You can dim layers to 50% opacity or turn off the background layer entirely.

iD shows you the usernames of local mappers so you can get in contact & ask about changes that you’re making - the local community.

What’s to Come

There are still many issues to fix and implement. Besides just bugs and visual tweaks, there are some major features that we’ve yet to crack the surface on:

A powerful, customizable set of presets which guide the editing process for typical features like buildings and roads
Relations editing, including user-friendly tools for adding and fixing turn restrictions, so that applications like OSRM have a richer dataset
A plugin system to allow for advanced features like those in JOSM without making iD itself unwieldy
Full translatability to allow iD to be used in many native languages

How to Help

Try editing with the testing instance!. We’d love to have more help developing and testing iD.

To help test, edit the map, find bugs, and report them on GitHub after searching for existing issues.

To help code, check out the issues and find one that sounds interesting. We idle in #ideditor on irc.oftc.net and would love to help out.

Javascript Browser Testing and Continuous Integration with iD

2012-12-06T00:00:00-05:00

Guest blogger John Firebaugh is an avid Ruby and Javascript developer, an OpenStreetMap mapper, and a contributor to the OpenStreetMap.org web site. He joined us for a sprint this week in the MapBox office in Washington DC where, among other things, he focused on testing.

It is extremely important that the iD editor is stable and tested: when deployed, it will be used to modify the singular OpenStreetMap database. To ensure that iD’s growth is accompanied by stability and reliability, we spent time this week refining the choice of tools we use for testing, improving test coverage, and setting up a continuous integration system. You can now check out iD’s testing status at any time on Travis-CI.

Automated testing helps us improve iD’s architecture and technical design (through test-driven development) and reduces the chances of inadvertently breaking existing code as we add new features and fix bugs. In this post I’ll outline our test infrastructure and explain how to test your iD instance when developing.

Testing Tools

iD runs in a browser, as client-side Javascript that relies on a DOM. We’re using a combination of testing tools that allow us to test both interactively in a number of browsers, and automatically, whenever a change is submitted to the repository.

Mocha provides the framework for defining tests, grouping them logically into suites, and running them. We use it in the style of behavior-driven development, whose main constructs are describe, to group a set of related test specifications, and it, to specify a facet of behavior. We use a top-level describe to name the module under test, second-level describes for each of its methods, and as many its as are necessary to fully specify each method’s behavior. For a good example, have a look at the tests for iD.Entity.
Chai provides the language for test assertions, with which we declare exactly how we expect the modules and functions that make up iD to behave. For instance, with chai’s assertions we can say that we expect(iD.Node({tags: {foo: 'bar'}}).tags).to.eql({foo: 'bar'}).
Sinon provides test spies, stubs, and mocks. Some parts of iD lend themselves to being tested in isolation, some parts require that their direct collaborators be mocked or stubbed, and some are best integration tested, with only the connection to the OSM API stubbed out. In all of these scenarios, we can take advantage of the test doubles provided by Sinon (even in isolation testing, spies are handy for verifying that callbacks are triggered at appropriate times). And with sinon-chai, they are all nicely integrated with chai’s assertion language.

All Together

Here’s how we confirm that the DeleteNode action actually removes a node from iD’s graph database:

// The group of tests
describe("iD.actions.DeleteNode", function () {
    // A specific desired behavior
    it("removes the node from the graph", function () {
        var node   = iD.Node(),
            action = iD.actions.DeleteNode(node),
            graph  = action(iD.Graph([node]));
        // The assertion: if graph.entity(node.id) is defined,
        // a test failure is reported
        expect(graph.entity(node.id)).to.be.undefined;
    });
});

iD tests run within the browser, so we can easily open different browsers and confirm there are no platform-specific bugs. In-browser testing also makes it possible to use Chrome’s Web Developer tools to pinpoint the cause of a failure.

Testing Your Changes

How do you run the tests locally, after making a change to the code? It’s easy: just open your repository’s copy of test/index.html in a browser. This file includes each of the individual script files that comprise iD, so rerunning the tests after a change is as simple as refreshing the page.

A companion file, index_packaged.html, tests the concatenated and minified iD.min.js package. It catches some problems that index.html doesn’t, but it requires running make after any change.

We encourage you to run the tests before submitting a pull request, and to add tests for any changes or additions you make.

Continuous Integration

To ensure that the tests pass after every commit, we rely on Travis CI, an open source, sponsor-supported project that provides free continuous-integration testing for projects on GitHub. When a change is pushed to GitHub, Travis is notified via post-receive hook and runs and reports tests, emailing the maintainers on test failures.

The tests run in-browser, so how do they run automatically? We use phantomjs, a headless WebKit runner, which integrates with Mocha via mocha-phantomjs providing a console-driven test runner and reporter. Luckily, travis-ci has phantomjs installed by default, so it runs perfectly on their servers.

Let’s Build (and Test)

The second ingredient for our testing is people - try it out and report issues and bugs! Combining continuous integration testing with reporting and management of bugs will make iD a rock-solid editor for OpenStreetMap.

Designing a New OpenStreetMap Editor

2012-12-06T00:00:00-05:00

Richard Fairhurst set forth this vision for iD, a new OpenStreetMap editor, in a blog post a few months ago:

The principle for admittance to OSM should be “I know stuff that can go on this map”, not “I’m good at using an OSM editor”. The more local knowledge that can be added to OSM, the better the map is.

He proposed a redesigned editor that is modal, seamless, and fast - well suited for both beginners and experienced users. Now working on the iD project with redoubled efforts with funding from the Knight Foundation this continues to be our goal.

At this point, much of the basic editing functionality is in place. Still, no actual designer has been involved, and no thorough review of UI details has taken place. The visual design needs work, but that’s the easy part: Icons, button styles, layout details, those things will fall into place. The hard part is laying the groundwork, creating the paradigms for an intuitive, yet powerful user interface.

Some of the challenging questions include: How do we improve the first time experience for new users? How do we progressively expose more complexity while keeping the tool easy to use? How do we balance map legibility with showing as much data as possible? Should map panning and zooming use standard conventions, or should clicking, double-clicking, and the scroll-wheel be reserved for editing mechanics? What should the user flow look like for adding new nodes? These are some of the harder questions that I hope to be able to make informed decisions about as I get deeper into the design process.

Many of these design problems are being tracked & discussed right now on iD’s GitHub project and I welcome additional input. Designing iD will be an iterative process, and if the first take on a feature of user flow isn’t quite right we should make sure to revisit it.

A first review of existing editors and literature

Right now, I am just starting to work on the iD user interface directly, with mock-ups and commits. Before I got my hands dirty with design work, I researched existing literature and experimented with other map editors out there. Here is a short list of the most interesting editors and posts I’ve taken a look at. My notes here reflect less the full result of my reviews but more a quick listing of what I’m aware of in this space. If you have any additional pointers in this field for me, I would love to hear about them.

Potlatch 2

Potlatch 2 is the go-to editor for in-browser editing on OpenStreetMap.org. One of the best parts of Potlatch is the attention put into keybindings. This makes Potlatch the fastest web-based editor for experienced users. As a designer who spent a lot of time with Inkscape and Illustrator, I can quickly make, connect, and manipulate paths, without constantly searching around for which buttons I need to click or moving through slow step-by-step walkthroughs. Because Potlatch supports the entire spectrum of data manipulations on OSM, it’s more flexible and complex than comparable editors from other organizations.

JOSM

The Java OpenStreetMap editor (JOSM) is a desktop application and the most popular editor in OSM. JOSM can do more and is faster than any of the other editors on this list, but it has a steeper learning curve. It allows users direct access to the data model and doesn’t force users to use defaults. This is the editor that you can use to map the world from scratch.

OSM is open to describing any geographic feature in the world and its data structure is developed in an organic, community driven process. Hence editors for OSM need to provide a large degree of freedom. JOSM, Potlatch and also iD need to be designed with that kind of flexibility in mind. This is different from the editors I reviewed next like Waze, or the heavily moderated Google MapMaker.

note: In the original version of this blogpost I innacurately said JOSM has “no abstraction”, when in truth users can use presets and bypass the free tagging interface if they’d like to.

Waze editor

Of all the existing editors, the waze map editor impressed me most. It has the advantage of serving a more niche purpose: it’s an editor for a map for car navigation, so a lot of the complexity of an OSM editor can instantly be dropped. Another interesting design decision is the turn restriction UI: It allows for a kind of direct manipulation by clicking icons that would be reserved for sidebars and menus in other editors. Very intuitive and easy to use.

Google Map Maker

Google MapMaker has one goal in mind: accessibility. The whole tool is designed to encourage someone new to start making a change and commit it. Some ways it accomplishes this goal is through its edit-first, sign-in-later approach. Adding ways and nodes is a guided experience, with many small steps, with one choice to make at each step, so on the first run users know what to do.

Nokia’s Here Map Creator

I like the way the details panel is hidden, it appears only when you it needs to be filled out. This intuitively pulls users through the process of adding to the map. I know when I should be focusing on manipulating things on the map, and when I should be describing the things I’ve added to the map. The UI also gradually introduces complexity with expandable menus. Even upon expansion, the choices are clear and the UI is attractive.

This is the only editor that takes control of the right click menu. I don’t know if this is the right approach for iD, but it does make it easy to quickly add new nodes and ways. Good keyboard shortcuts can accomplish the same thing.

Reading List

This list covers some of the reading I’ve done to get up to speed on iD. If you’re interested in the project, you might find these resources helpful.

iD project Notes

This is the document Tom and John put together to outline iD’s design & development philosophy.

CloudMade blogposts about Mapzen

This link should almost go in the editors section, because it’s about Mapzen, an editor CloudMade began developing a couple years ago. Unfortunately, the mapzen project is no longer actively maintained, and it’s completely offline. Still, a few years ago this was an exciting and promising effort and the developers did a great job of blogging their work. Some interesting features of Mapzen included very tight integration between the editor and a social platform, and deliberate design of the editable data overlays in the editor.

It all starts with an editor – the blog edit

This is Richard’s blog post that I quote above. It’s the starting point for much of the ongoing iD work.

OSM wiki page for Potlatch 2, OSM wiki page for good practices in contributing to the map and OSM wiki page for editing standards and conventions

These Wiki pages are essential reading. It’s hard to design an editor when you don’t understand what the results need to look like.

Correction: A previous version of the post suggested that JOSM does not abstract the data model and doesn’t provide a lot of defaults while presets cover actually a large portion of OSM tag configurations.

Design Improvements to OpenStreetMap.org

2012-11-21T00:00:00-05:00

Part of our Knight funded work on OpenStreetMap aims to improve the social experience for mappers. While concrete goals in this realm remain to be fleshed out as we focus on editor work, we know that UI improvements to OpenStreetMap.org will be essential.

To get started, I recently wrapped up a round of design and markup clean-ups to OpenStreetMap.org. This work is available as a ready-to-merge pull request to the openstreetmap-website project and is pending a final review. At the end of this post you will find a series of screenshots showing current progress.

This first batch of improvements is fairly straightforward. Before introducing new features or UI concepts, I decided to start by making a concrete impact through fixing existing problems, modernizing outdated markup, and nudging the site towards better typography, hierarchy, and user-friendliness one step at a time. As a designer starting to work with the OpenStreetMap.org web site I see these improvements as an important step towards understanding the code base and the OpenStreetMap community so I can make informed suggestions and participate in more profound improvements to the project in the coming months.

Concrete improvements include:

A clearer, more harmonious typgraphic scale.
A persistent header and subnav pattern across most pages on the website.
Table-based layouts on the user profile page, the changelist page, the data browser pages, and elsewhere have been replaced with modern HTML markup.
A number of utility classes, like button and a series of padding classes reduce CSS repetition & make it easier to style new content or features if they get added in the future.
Link lists are now properly marked up as <ul> elements.
Most of the icons on the site have been replaced with cleaner, clearer, more consistent alternatives.
A standard pattern of 20px & 10px padding and margins has been introduced, improving the rhythm of the pages and giving the content some much-needed breathing room.
Many smaller, more insignificant fixes that I ran across in the process of tackling some of the larger goals.

One of the most important reasons for modernizing the site with more semantic markup and reusable classes is to give more flexibility to myself or other designers for styling and arranging content in the future. For example, there was no way to create the visual separation between the header and the body of the content area before I introduced separate header and content divs to every page, and now that those divs are in place and consistent, it’s easy to make large scale changes to layout and heirachy by adjusting those container classes.

Next Steps

First and foremost, the next thing I plan to do is work with Tom Hughes and other maintainers of the OpenStreetMap web site to make sure this work can be merged in safely, with all the kinks fixed.

After that, with some of the more interdependent fixes complete, I will be able to approach remaining tasks one at a time. The rest of the incremental improvements on my todo list include:

A more inviting signup page
Better styling for markers and popups. I am waiting on the port of the OpenLayers maps to Leaflet before I dig into this.
Revamped button styles across the site
Replace table layouts with more semantic markup for all forms.
Better 404 page

While working on the rest of these small changes, I’ll start to spend time concepting a broader vision for the OSM project. Expect to hear more from me soon with details.

For now, have a look at the screen shots below or the pull request to see what has been accomplished so far.

Screenshots

Here are some screenshots showing my progress so far. Hover over the images to see the old version, or click for a full size image.

Home page:

Changeset page:

User page:

Diary page:

Data browser panel:

Key panel:

Getting serious about SVG

2012-11-20T00:00:00-05:00

iD depends on SVG for drawing map features, displaying tiles, and a model upon which to build complex interactions.

In the process of building it, we’ve learned a lot about SVG’s performance equation - and it’s time to share some of this. A lot of this is taken from NOTES.md, a sort of developer-journal which has grown over the last few weeks.

note that the examples in this post are designed to run around ~30fps in Google Chrome on a fast computer. They’re made to push the limits, so may crash slower browsers on slower machines.

First Steps with SVG

We’re generally using d3 for graphics in iD. It’s a very, very light abstraction layer over SVG - it eliminates the need for manual namespacing, gives a jQuery-like chainable attr setter, and so on.

SVG elements are elements on-page just like HTML elements, with a few differences:

There’s no z-index - order in-page is the rendering model
SVG has its own transform property, which is different and has difference performance implications than CSS’s transform property.
Many CSS styles work with SVG elements, but many do not. Some things we’d think of as styling variables, like the radii of circles, are actually expressed as attributes rather than style rules.
SVG has a powerful and possibly awful system of doing inter-references: a <defs> element for definitions which are used by other elements by xlink:ref attributes. For instance, you need to define a <path> in <defs> in order to reference it from a textPath element in your main drawing canvas if you wish to draw text on a path.

Principles

There are a few basic goals of any performance tuning task:

Reducing repaints & reflows
Triggering code paths that your browser can optimize, like using rounded pixels in Canvas or using specific forms of functions to avoid V8 deoptimization

SVG’s special moves: g and defs

The basic structure of the map canvas for iD is as follows:

div (supersurface)
  svg (surface)
    defs
      rect#clip
      path (textPath data)
    g (tilegroup)
    r (vector root)
      g (fill, casing, stroke, text, hit, temp)
        (path, g, marker, etc)

Besides just having more shapes, SVG has different concepts - one is the <g> element, that represents a group. The groups can be very useful, since they can be transformed with the transform attribute: for instance, markers are set up like

<g transform='translate(10, 20)'>
    <image />
    <circle />
</g>

SVG’s defs element is another oddball: it defines content that doesn’t show up on screen, but can be used via reference: I’ll mention this later on with textPath as a example. But another usage of defs for this project is making a rect element which is used to clip the vector and tile layers.

CSS Transforms are a big boost - sometimes.

For panning the map, transforms seemed to be the main solution - they allow simple pan behaviors to be carried out on a single element, and in a way that doesn’t require a browser to recalculate individual element positions.

However, early results were not encouraging - CSS transforms, even of the ‘hardware accelerated’ 3D variety - are not accelerated in Google Chrome when used with SVG elements - they have the same performance profile as SVG attribute transforms.

The answer is to transform HTML elements, which does trigger fast code in Google Chrome. In iD we do this by transforming a ‘surface’ element (the SVG parent) of the map instead of the group (<g>) elements that make up the tile surface and vector surface.

(update: these now compensate for different vendor prefixes for the transform CSS property)

Rounded pixels give a boost

Vladimir brought up this one and I didn’t expect it to be a significant bump, since SVG is vectorized, I assumed that it wouldn’t have raster-pixel bumps like Canvas does. However, it is a boost, and not just because of the shorter string length of d attributes - rounding and outputting long fixed-point numbers is also faster than non-rounded values. This is about a 20% improvement in testing.

Markers along a path

One of the features we need in iD is an indication of roads that are one-way, and which way they are going.

SVG has some built-in functionality for this purpose - you can define a marker in a defs element and use it on path elements with marker-mid and so on. Unfortunately, this doesn’t cut it, since OSM paths have an extremely variable number of vertices - some straight lines have hundreds of vertices, some have only a few, and would be very sparsely labeled.

Our current solution to this is to use a textPath with a glyph - a right-facing triangle. Here’s what it looks like in a standalone demo.

First, determine the size of an arrow - how many pixels it occupies from left to right on a path:

var arrow = svg.append('text').text('►');
var alength = arrow.node().getComputedTextLength();
arrow.remove();

Then getTotalLength lets you get the pixel-length of a path, and you can then use multiplication & letter spacing to distribute just enough markers for the path:

tp.attr('letter-spacing', alength * 2)
    .attr('xlink:href', '#p').text(function() {
        return (new Array(Math.floor(
            path.node().getTotalLength() / alength / 2)))
            .join('►');
    });

Update: I noticed that this example is broken in Firefox. This is due to the following (also noted on the example page):

The original version of this failed because Firefox does not support letter-spacing on SVG elements. The second version failed because it has a bug in getComputedTextLength. The third failed because it does not preserve space according to xml:space.

Needless to say, Firefox is not the main target for iD testing because of a buggy and incomplete implementation of SVG relative to WebKit.

iD Updates

2012-11-14T00:00:00-05:00

For the last few weeks, I’ve been working on building iD, along with John Firebaugh and with lots of help from Richard Fairhurst.

We’re all learning a lot: here are some specifics.

Making iD Fast

iD has to be fast. If the experience of moving the map, drawing roads, and so forth isn’t satisfying, it won’t be adopted.

There’s no single ingredient that makes maps fast: it’s essentially a combination of:

Javascript performance, like the speed of projecting data to screen coordinates, grabbing data from the local datastore, and assorted code like styling and processing mouse and touch events. In the case of canvas, maintaining a scenegraph is a big javascript performance task.
Network performance, or how quickly you can grab data from an API to display on-page
DOM Performance, like drawing to a Canvas element or updating attributes of an SVG element.

The clearest ‘first load’ issue for iD is network performance: data is coming down from OSM’s servers in untiled WMS-style queries and the servers we’re dealing with are not blazingly fast. We plan to resolve this partially by using tiled requests with smart caching, and otherwise handle expectations by making it clear when map data is loading.

Javascript performance isn’t a big bottleneck at this point, though there are certain inefficiencies that would be great to address, mainly in the finer points of d3.

The biggest bottleneck that I’m currently seeing is the DOM. iD uses SVG for rendering maps because it gives a clear path to styling data and handling interactivity. While it’s possible to use Canvas with a scenegraph, it’s likely to be more error-prone as far as hit-detection with multi-layered maps.

And so the biggest performance problem with, for instance, dragging the map, is the cost of reassigning the d attribute to all svg paths in order to move them onscreen.

There are ways around this:

Project vector features to a certain ‘zero origin’ and transform them into place onscreen
Handle pan behaviors by setting a transform on the entire map object

Both of these have tradeoffs: data on the map is not static, and so d3 needs to handle the case of a user dragging a line segment extremely well. And thus a heuristic like setting transforms on the entire map element will require a fallback for correctly calculating the position of transformed paths.

To fill in some background information: this is referring to CSS transforms, which have the positive effect of limiting browser repaints and reflows, the biggest performance hit of changing elements onscreen.

SVG’s performance relative to Canvas is a popular topic: canvas is generally faster for drawing many objects while SVG is better for large canvases. But the main consideration in this project is not raw drawing performance, but the ability of the rendering layer to support interactivity and updates: existing map renderers like KothicJS do fast canvas rendering, but static rendering like Mapnik.

Measuring Fast

I’m mostly developing in Chrome - there doesn’t seem to be a very good profiler for Firefox, unfortunately. The main metric for debugging performance problems is the Javascript CPU Profile option of the WebKit debugger.

Setting attributes, pulling data, and projecting points (‘mercator’) all contribute to the performance profile of iD.

Another dimension we’re paying close attention to is memory usage, or heap size. While much of iD uses the module pattern - for instance, the main iD.Map object - for highly allocated objects, we’re using Javascript prototype-based objects, which cut down on the size of closures in the heap.

Heap size is and should be dominated by the actual data of the map, stored in coordinate arrays and so on.

The Size of iD

The size of Javascript code has varying importance - often, it is cached after first load and doesn’t effect performance in any real way. However, given non-shared caches and large libraries (OpenLayers comes to mind) code size is a real performance consideration.

We’re using uglifyjs for code compression, and a simple Makefile & cat system for building a big iD.js file.

Right now d3 is the majority of the iD payload. This is relatively acceptable, since iD itself is very small, and would likely be much larger if it didn’t depend on d3 for specifics like map projection, event binding, and so on.

Making iD Friendly

iD has to be friendly. Meaning that it needs to be as self-explanatory as possible, and then fall back to help quickly.

First this is a question of interaction paradigms. The map should be some combination of an Adobe Illustrator experience and a Google Maps experience, without being too confusing. This means that, unlike Potlatch 2, we’re planning on having, essentially, modes.

One big difference will also be the separation of drawing ‘roads’ versus drawing ‘areas’. In the OSM data model, ways can represent both buildings - closed loops with area - and streets. There’s very little distinction on a technical level, but a very large distinction in most users minds, so we’re going to mirror the mental model.

The next big ‘what’ in OSM is the tagging model: while it’s flexible, it’s not clear.

iD is going to have a system of presets which has solid visual cues and good descriptions. Behind this is strong integration with TagInfo or similar: an index of commonly used tag combinations which leads users to make correct choices.

In order to make that work, we’re going to need to help out a lot on editing OSM wiki help - improving the descriptions of tag combinations to make them both distinct and descriptive.

Making iD Correct

OpenStreetMap is filled with corner cases. A few pathological conditions I’ve listed in the NOTES.md file which has been developed along with iD:

Ways with one node
Relations which contain themselves (circular references)
Nodes with no tags and no way attached
Ways which contain only nodes that are subsets of the nodes of other ways
Paths with intersecting boundaries (invalid geometries)

Handling each of these cases correctly is essential to iD: this editor cannot be destructive, or make design decisions that puts data integrity at risk.

What’s Next

We’re working on the meat of the problem currently: the ability to draw, edit, and remove map features, have those operations be reversable as undo operations and create changesets from that.

The data model has progressed greatly: an immutable graph, a simple history representation, and lightweight data objects are complete. There’s a lot around the edges, like entering and exiting drawing states, that needs work.

And the issue of iterating through the graph and generating a changeset is just about to be started.

If you’re interested in contributing, take a look at the repository on github, the current notes and open issues. We’re keeping a very open contributing process - pull requests are merged quickly, everything happens in the master branch, and issues are discussed in GitHub issues or in the #osm-dev IRC channel. Join us!

First Steps With iD Editor

2012-10-25T00:00:00-04:00

The OSM Editor

The iD editor is a new Javascript map editor initiated by Richard Fairhurst, the author of Potlatch 2, the current default editing experience for OpenStreetMap. One of OSM’s clearest needs is a good map editor - it’s the difference between new users becoming consistent contributors or being scared away by the complexity of mapping.

The idea of a new editor is a much-discussed topic - Potlatch 2 is built on Flash, and the advanced editor option, JOSM, can be intimidating even to the tech-savvy.

Architecture

iD is exciting because it’s a smart approach to building the future - for the most part, it’s taking knowledge from Richard’s and others’ work with Potlatch to make a stable, modular layer for OSM data model. From there, we’re working on building a great user interface that’s intuitive and simple, but not limiting.

The initial work on iD used the dojo framework as a library and source of structure. I’ve been working in the past week to limit this dependency - it now uses underscore.js for functional programming mechanisms and jQuery for DOM & AJAX.

The biggest change recently is moving to d3 for data-binding and graphics. We’re using d3.geo.mercator as the map projection, d3.behavior.zoom for navigating the map, and d3.behavior.drag for dragging handle points for ways and points for POIs.

The map tiling component is extremely minimal - less than 100 lines of javascript. The emphasis of the project will really be on keeping rendering fast & light and then focusing on the functionality of the editor, which will require extensive work with the OSM object model, an undo system, and the various APIs.

Currently the MapCSS backend dependency has been removed, and replaced with a simpler system that operates by simple CSS classes added to path elements and styling functions for deriving icons from points. This seems to work well, and basic d3 functionality like sorting selections works well to replicate OSM’s layer and bridge rendering priorities.

For instance, here’s the current sorting function:

function waystack(a, b) {
    if (!a || !b) return 0;
    if (a.tags.layer !== undefined && b.tags.layer !== undefined) {
        return a.tags.layer - b.tags.layer;
    }
    if (a.tags.bridge) return 1;
    if (b.tags.bridge) return -1;
    var as = 0, bs = 0;
    if (a.tags.highway && b.tags.highway) {
        as -= highway_stack.indexOf(a.tags.highway);
        bs -= highway_stack.indexOf(b.tags.highway);
    }
    return as - bs;
}

I’ve started on basic testing using Jasmine, though test coverage needs a lot of improvement.

The really cool thing is that iD has iD.Node, iD.Way, and iD.Relation - Javascript wrappers for basic bits of OSM that we can reason about. So, in the course of building an editor it’s handy to have shortcuts like activeWay.isClosed(), and combine these with ‘actions’ - essentially stock operations, like adding nodes to ways.

Immediate Future Plans

Creating an editor can be fraught: you can make a system in which people unknowingly duplicate, delete, or corrupt OpenStreetMap data. So it’s important that iD is being built with previous experience and lots of tests - and ideally this process will lead to thorough documentation on how to do things the right way, as well.

Right now text labels are turned off due to an apparent limitation of SVG’s textPath element. This is a top priority to get back. On the bright side, the refactor allows us to use use and defs elements to reduce the duplication of paths for lines.

I’ve also started to rebuild UI elements, like a tag editor with the new system and with lightweight HTML+CSS that can be styled nicely. These bits are a bit tricky, and are intertwined with the issue of whether to keep multiple versions of objects or to store transformations between versions.

Contribute

If you’d like to help out, or just git clone and kick the tires, see the GitHub repository for iD. Right now there’s a lot moving - so pardon the dust - but it’s a fun early look at a really big change for OSM.

Hacking on OWL and APIs On October 27 and 28

2012-10-23T11:00:00-04:00

Photo heatherontravels

Matt Amos, one of the maintainers of the OpenStreetMap Watch List (OWL) is in town for the weekend. Tom and I will take this opportunity and join him hacking on OWL and on OpenStreetMap APIs.

Join us: We’ll get together on Saturday October 27th and Sunday October 28th at the MapBox office at 11AM and we’ll go until about 6PM. There will be plenty of coffee and if we’re hungry we’ll order in. If you’re not in DC: we’ll also hang out on IRC - #osm-dev on irc.oftc.net.

Welcome to the MapBox OSM Development Blog

2012-10-23T10:45:00-04:00

World wide OpenStreetMap data coverage 2012

After kicking off Knight Foundation funded OpenStreetMap work two weeks ago we have gotten tons of feedback from the community on mailing lists, the openstreetmap-website repository and at the State of the Map in Portland. This was extremely valuable input and it’s fantastic to see so many people we can engage with on improving OpenStreetMap. There is much more work to be done than what we can hope to tackle in the next months, but there are incredibly clear next actions to follow now. Specifically, Tom has started to contribute to the iD project. Here is a note by Richard Fairhurst on how to get involved in development.

We’re starting this blog as a casual space to talk about our ongoing Knight-funded work. This is the place to stay in the loop of what we’re up to. Expect regular updates of where work is happening and bigger picture posts. We will continue to post key developments to talk and hash out conversations on dev, while we’re keeping our heads down on work in issue queues like iD’s.