On Android Compatibility

[This post is by Dan Morrill, Open Source & Compatibility Program Manager. — Tim Bray]

At Google I/O 2010, we announced that there are over 60 Android models now, selling 100,000 units a day. When I wear my open-source hat, this is exciting: every day the equivalent of the entire population of my old home city starts using open-source software, possibly for the first time. When I put on my hat for Android Compatibility, this is humbling: that’s a whole lotta phones that can all share the same apps.

Another thing we launched at Google I/O was an upgraded and expanded source.android.com. The new version has refreshed info on the Android Open-Source Project, and some new tips and tools for device manufacturers — useful if you’re an OEM. However, it also has details on the Android compatibility program, now. This is also aimed mostly at OEMs, but Tim Bray suggested that developers might be interested in a peek at how we keep those 100,000 devices marching to the same beat, every day. So here I am, back on the blog.

The F-word, or, Remember remember the fifth of November

I remember sitting with my colleagues in a conference room in Building 44 on November 5, 2007, listening to Andy Rubin and Eric Schmidt announce Android to the world. I remember a lot of the press stories, too. For instance, Android was “just words on paper” which was especially entertaining since I knew we were getting ready to launch the first early-look SDK a mere week later.

Another meme I remember is... yes, “fragmentation”. Literally before the close of business on the same day we announced Android (4:46pm to be precise), I saw the first article about Android “fragmentation.” The first day wasn’t even over yet, and the press had already decided that Android would have a “fragmentation” problem.

The thing is, nobody ever defined “fragmentation” — or rather, everybody has a different definition. Some people use it to mean too many mobile operating systems; others to refer to optional APIs causing inconsistent platform implementations; still others use it to refer to “locked down” devices, or even to the existence of multiple versions of the software at the same time. I’ve even seen it used to refer to the existence of different UI skins. Most of these definitions don’t even have any impact on whether apps can run!

Because it means everything, it actually means nothing, so the term is useless. Stories on “fragmentation” are dramatic and they drive traffic to pundits’ blogs, but they have little to do with reality. “Fragmentation” is a bogeyman, a red herring, a story you tell to frighten junior developers. Yawn.

Compatibility

Now, that’s not to say that there aren’t real challenges in making sure that Android devices are compatible with each other, or that there aren’t very real concerns that keep app developers awake at night. There definitely are, and I spend a great deal of time indeed thinking about them and addressing them. The trick is to define them clearly.

We define “Android compatibility” to be the ability of a device to properly run apps written with the Android SDK. This is a deceptively simple way to frame it, because there are a number of things that can go wrong. Here are a few:

• Bugs - devices might simply have bugs, such as a buggy Bluetooth driver or an incorrectly implemented GPS API.

• Missing components - devices might omit hardware (such as a camera) that apps expect, and attempt to “fake” or stub out the corresponding API.

• Added or altered APIs - devices might add or alter APIs that aren’t part of standard Android. Done correctly this is innovation; done poorly and it’s “embrace and extend”.

Each of these is an example of something that can make an app not run properly on a device. They might run, but they won’t run properly. These are the things that I spend my time preventing.

How It Works

As stewards of the platform we realize that it’s vital to allow only compatible devices to participate in the Android ecosystem. So, we make compatibility a strict prerequisite for access to Android Market and the right to use the Android name. This means that developers can rely on the fact that Android Market — the keystone of the Android ecosystem — will only allow their apps to run on compatible devices. It’s pretty self-evident that a single app ecosystem is better than many small ones, so OEMs are generally pretty motivated to ship compatible devices.

But motivation alone doesn’t get us very far without tools to actually ensure compatibility, which is where the Android compatibility program comes in. This program is like a stool with three legs: the Android source code, the Compatibility Definition Document, and the Compatibility Test Suite.

It all starts with the Android source code. Android is not a specification, or a distribution in the traditional Linux sense. It’s not a collection of replaceable components. Android is a chunk of software that you port to a device. For the most part, Android devices are running the same code. The fact that all Android devices run the same core Android code goes a long way toward making sure those devices all work the same way.

However, this doesn’t solve the problems of missing components or altered APIs, because the source code can always be tweaked. This is where the Compatibility Definition Document (or CDD) comes in. The CDD defines in gory detail exactly what is expected of Android devices. It clearly states, for example, that devices may not omit most components, and that the official Android APIs may not be altered. In a nutshell, the CDD exists to remove ambiguities around what’s required of an Android device.

Of course, none of that overcomes the simple reality of human error — bugs. This is where the Compatibility Test Suite comes in. The CTS is a collection of more than 20,000 test cases that check Android device implementations for known issues. Device makers run the CTS on their devices throughout the development process, and use it to identify and fix bugs early. This helps ensure that the builds they finally ship are as bug-free as possible.

Keeping Up with the Times

We’ve been operating this compatibility process with our OEM partners for over a year now, and it’s largely responsible for those 60+ device models being interoperable. However no process is ever perfect and no test suite is ever 100% comprehensive, and sometimes bugs get through. What happens then?

Well, we have great relationships with our OEMs, and like I said, they’re motivated to be compatible. Whenever we hear about a serious bug affecting apps, we report it to our partners, and they typically prepare a bugfix release and get it out to end users. We will also typically add a test case to the CTS to catch that problem for future devices. It’s an ongoing process, but generally our partners are as interested as we are in the user experience of the devices they sell.

The mobile industry today is “very exciting”, which is code for “changes painfully fast”. We believe that the only way Android will be a success is to keep up with that change, and ultimately drive that change. This means that over time, the CDD will also change. We’ll add new text to handle problem cases we encounter, and the actual requirements will change to accommodate the innovations happening in the field. For example, in the 2.1/Eclair CDD, we tweaked the CDD slightly to make telephony optional, which allows Android to ship compatibly on non-phone handheld devices. Whenever we do this, of course, we’ll make corresponding changes to the Android APIs and Android Market to make sure that your apps are protected from ill effects.

On a somewhat related note, a lot of ink has been spilled on the fact that there are multiple versions of Android out there in users’ hands at the same time. While it’s true that devices without the latest software can’t run some of the latest apps, Android is 100% forward compatible — apps written properly for older versions also run on the newest versions. The choice is in app developers’ hands as to whether they want to live on the bleeding edge for the flashiest features, or stay on older versions for the largest possible audience. And in the long term, as the mobile industry gets more accustomed to the idea of upgradeable phone software, more and more devices will be be upgraded.

What It Means for You

All that is great — but what does it mean for developers? Well, we put together a page in the SDK Documentation to explain this, so you should take a look there. But really it boils down to this:

1. As a developer, you simply decide what features your app requires, and list them in your app’s AndroidManifest.xml.

2. The Android compatibility program ensures that only compatible devices have access to Android Market.

3. Android Market makes sure your app is only visible to those devices where it will run correctly, by filtering your app from devices which don’t have the features you listed.

That’s all!

There almost certainly will be devices that have access to Android Market that probably weren’t quite what you had in mind when you wrote your app. But this is a very good thing — it increases the size of the potential audience for your app. As long as you accurately list your app’s requirements, we’ll do the rest and make sure that your app won’t be accessible to a device where it won’t run properly. After all, we’re app developers ourselves, and we know how painful it is to deal with users upset about an app not working on a device it wasn’t designed for.

Now, that’s not to say that we think our current solution is perfect — no solution is. But we’re continuously working on improvements to the SDK tools and Android Market to make your life as an Android developer even easier. Keep an eye on this blog and on the Android Market itself for the latest info.

Thanks for reading, and happy coding!

Geo-Fencing, Location Privacy & Other Animals

Privacy has been, is and will remain one of the most hotly debated issues within both the web and mobile domains for time to come. Only last week, a probe by German, French, Spanish and Italian authorities gathered steam as fears of Google's Streetview data-gathering exercise appeared to have breached several national privacy rules.

The debate has many dimensions: what should be in the 'public' domain? Where does the trade-off lie between the benefits of increased sharing of information and the risks of over-sharing?

Erick Schonfeld writing in TechCrunch today provides an interesting summary of what Google's, Foursquare's and Facebook's take on the location privacy issue is.

I completely agree with Facebook's VP of Product, Chris Cox, who described a future where phones are "contextually aware" so that they can "check into flights, find deals at gorcery stores and do other things for us at that right place, at the right time". Importantly “The phone should know what we want.”

In my book on Building Location-Aware Applications (with co-author Murat Aktihanoglu), I describe this as the 'Contextual Holy Grail', with context providing the 'intelligent' added element to Presence and Location.

As handsets get smarter, the whole area of 'sensory perception' will also allow predictive services to be delivered to the handset -for example, a handset that can detect your stress levels and location, will be able to recommend an Aromatherapy store nearby.

Geo-fencing allows users to draw virtual fences around neighbourhoods or locations that either you want to maintain as private or public. This helps give users control over the location service and set their desired privacy levels. As greater automation of location services come on stream (checking-in your location really is too cumbersome for most people), geo-fencing will provide a useful way to strike the right privacy balance in a smarter way.

Android Cloud To Device Messaging

[This post is by Wei Huang, who helped implement this feature. — Tim Bray]

In the just-launched Android 2.2, we’ve added a new service to help developers send data from servers to their applications on Android phones. Android Cloud to Device Messaging (C2DM) makes it easier for mobile applications to sync data with servers.

Most of the useful applications on your mobile phone use the Internet to keep users connected. Traditionally, many apps use polling to fetch data periodically. POP mail clients, for example, connect to the server every 15 minutes or so. Polling is fairly easy to implement, and works well in many situations. It’s tricky, though, to select the frequency: Poll too often, you may not see any new data, and create unnecessary stress on the server and network. Poll too rarely and the data on the device may become stale. Polling is especially problematic on mobile devices, because it consumes precious network bandwidth and battery life.

Having the server push messages to the client asynchronously may be a superior choice for getting the latest data to your applications, resulting in fresher data and more efficient use of the network and your battery. However, it’s also tricky to implement a good push solution, and it isn’t free as there is some overhead in maintaining the required connection. On a mobile device like an Android phone, implementing applications to receive these messages is tricky; you have to worry about patchy network coverage and zombie connections created when the wireless carrier’s routers time out connections that are idle for too long.

Many of the Google applications on Android already use push to keep their data fresh, for example Gmail, Contacts, and Calendar. Starting with Android 2.2, C2DM allows third-party developers to use the same service the Google apps do.

Here are a few basic things to know about C2DM:

• It requires Android 2.2; C2DM uses Google services which are present on any device running the Android Market.

• It uses existing connections for Google services. This requires the users to sign into their Google account on Android.

• It allows 3rd party servers to send lightweight data messages to their apps. The C2DM service is not designed for pushing a lot of user content; rather it should be used like a “tickle”, to tell the app that there is new data on the server, so the app can fetch it.

• An application doesn’t need to be running to receive data messages. The system will wake up the app via an Intent broadcast when the the data message arrives, so long as the app is set up with the proper Intent Receiver and permissions.

• No user interface is required for receiving the data messages. The app can post a notification (or display other UI) if it desires.

It’s easy to use the C2DM API. Here is how it works:

• To enable C2DM, an application on the device registers with Google and get a registration ID, and sends the ID to its server.

• When the server needs to push a message to the app on the device, it posts the message via HTTP to Google’s C2DM servers.

• The C2DM servers route the message to the device, and an Intent broadcast is sent to the app.

• The app is woken up to process the message in its Intent Receiver.

• The app can unregister with C2DM when the user no longer wants messages to be pushed to it.

That’s about it! All you need is a server that knows to talk HTTP, and an Android app that knows how to use the Intent APIs. Below are some code samples:

// Use the Intent API to get a registration ID
// Registration ID is compartmentalized per app/device
Intent regIntent = new Intent(
        "com.google.android.c2dm.intent.REGISTER");
// Identify your app
regIntent.putExtra("app",
        PendingIntent.getBroadcast(this /* your activity */, 
            0, new Intent(), 0);
// Identify role account server will use to send
regIntent.putExtra("sender", emailOfSender);
// Start the registration process
startService(regIntent);

The registration ID will be delivered back to your app via an intent broadcast with the Intent Action com.google.android.c2dm.intent.REGISTRATION. Here is a code sample to receive the registration ID.

// Registration ID received via an Intent
public void onReceive(Context context, Intent intent) {
  String action = intent.getAction();
  if (“com.google.android.c2dm.intent.REGISTRATION”.equals(action)) {
    handleRegistration(context, intent);
  }
}

public void handleRegistration(Context context, Intent intent) {
  String id = intent.getExtra(“registration_id”);
  if ((intent.getExtra(“error”) != null) {
    // Registration failed.  Try again later, with backoff.
  } else if (id != null) {
    // Send the registration ID to the app’s server.
    // Be sure to do this in a separate thread.
  }
}

On the server side, your server needs to get a ClientLogin Auth token in order to talk to the C2DM servers. When it wants to push a message to the device, it can send an authenticated http POST with:

• Authorization: GoogleLogin auth=<auth token>

• URL encoded parameters including the registration id, the data key/value pairs, a “collapse key” used for overriding old messages with the same key on the Google C2DM servers, and a few other optional params.

When you use the C2DM service, you no longer need to worry about dealing with flaky mobile data connections, or when the user isn’t connected to the internet (i.e. Airplane mode). C2DM keeps the messages in the server store, and delivers them when the device comes back online. Basically, you can leave all the hard work of designing a robust push service to Google. Your application can take advantage of push infrastructure we’ve already built and tested, and stay more connected to the internet. Best of all, you won’t ruin the users’ battery life.

Information about how to build C2DM enabled applications on Android is online at the code lab, and more will be coming as we approach general release.