AndrewPearson.org

Andrew Pearson's Little Corner of the Internet...

Tuesday, July 20, 2010

Android: Why To Use JSON And How To Use It

It is a pretty common task in smartphone application development to query a website for some information, do a bit of processing, and present it to the user in a different form. There are two different main ways of going about this:

1.) Just download a whole webpage and use an HTML or XML parser to try and extract the information which you want.
2.) Some websites have API's which allow queries that instead of returning webpages, return XML, JSON, or some other way of presenting data.

Clearly, option 2 (if available) makes a lot more sense. Instead of downloading a large webpage (wasting data), parsing the entire thing (wasting battery), and then trying to analyze it (wasting your time going through often improperly written HTML), you can download much smaller, easier to manage text which conforms to XML, JSON, or whatever it is that the API provides. (I will be posting another tutorial about option 1 soon).

You might wonder why you would want to use JSON instead of XML for your smartphone application. After all, XML has been a much-ballyhooed technology buzzword for many years. There is, however, a good (and simple) reason. XML is (usually) bigger. The closing tags in XML do not exist in JSON, and therefore save you a few bytes for each tag that you don't need. JSON can generally express the same data using fewer characters, thus saving the phone from having to transfer more data every time there is a query to a website. This makes JSON a natural choice for quick and efficient website querying (for websites which offer it).

One such website is www.archive.org. Among many other things, archive.org allows people to upload recordings from concerts for other people to download for free. It's pretty awesome. They also have an API which allows you to query their system which will return results in XML, JSON, or a variety of other formats.

I am currently writing an application for browsing archive.org from your phone to find shows and then either download or stream the tracks. I'll show you how I do the first part (finding shows) using JSON and just a few lines of code.

First, you need your JSON query. I am going to query archive.org for "Lotus," asking for a JSON result containing 10 items with their respective date, format, identifier, mediatype, and title. According to the archive.org search API, my query should look like this:

String archiveQuery = "http://www.archive.org/advancedsearch.php?q=Lotus&fl[]=date&fl[]=format&fl[]=identifier&fl[]=mediatype&fl[]=title&sort[]=createdate+desc&sort[]=&sort[]=&rows=10&page=1&output=json&callback=callback&save=yes";

Now that we have our query, we simly open an HTTP connection using the query, grab an input stream of bytes and turn it into a JSON object. On a side note, notice that I am using a BufferedInputStream because its read() call can grab many bytes at once and put them into an internal buffer. A regular InputStream grabs one byte per read() so it has to pester the OS more and is slower and wastes more processing power (which in turn wastes battery life).

InputStream in = null;
String queryResult = "";
try {
URL url = new URL(archiveQuery);
HttpURLConnection urlConn = (HttpURLConnection) url.openConnection();
HttpURLConnection httpConn = (HttpURLConnection) urlConn;
httpConn.setAllowUserInteraction(false);
httpConn.connect();
in = httpConn.getInputStream();
BufferedInputStream bis = new BufferedInputStream(in);
ByteArrayBuffer baf = new ByteArrayBuffer(50);
int read = 0;
int bufSize = 512;
byte[] buffer = new byte[bufSize];
while(true){
read = bis.read(buffer);
if(read==-1){
break;
}
baf.append(buffer, 0, read);
}
queryResult = new String(baf.toByteArray());
} catch (MalformedURLException e) {
// DEBUG
Log.e("DEBUG: ", e.toString());
} catch (IOException e) {
// DEBUG
Log.e("DEBUG: ", e.toString());
}

At this point, our JSON response is stored in the String queryResult. It looks kind of like this:

callback({
"responseHeader": {
... *snip* ...
}, "response": {
"numFound": 1496,
"start": 0,
"docs": [{
"mediatype": "audio",
"title": "The Disco Biscuits At Starscape 2010",
"identifier": "TheDiscoBiscuitsAtStarscape2010",
"format": ["Metadata", "Ogg Vorbis", "VBR MP3"]
}, {
"title": "Lotus Live at Bonnaroo Music & Arts Festival on 2010-06-10",
"mediatype": "etree",
"date": "2010-06-10T00:00:00Z",
"identifier": "Lotus2010-06-10TheOtherStageBonnarooMusicArtsFestivalManchester",
"format": ["Checksums", "Flac", "Flac FingerPrint", "Metadata", "Ogg Vorbis", "Text", "VBR MP3"]
}, {
"title": "Lotus Live at Mr. Smalls Theatre on 2006-02-17",
"mediatype": "etree",
"date": "2006-02-17T00:00:00Z",
"identifier": "lotus2006-02-17.matrix",
"format": ["64Kbps M3U", "64Kbps MP3", "64Kbps MP3 ZIP", "Checksums", "Flac", "Flac FingerPrint", "Metadata", "Ogg Vorbis", "Text", "VBR M3U", "VBR MP3", "VBR ZIP"]
}, {
... *snip* ...

We see that the information we want is stored in an array whose key is "docs" and is contained in an item called "response". We can grab this information VERY easily using the JSONObject class provided by Android as shown below:

JSONObject jObject;
try {
jObject = new JSONObject(queryResult.replace("callback(", "")).getJSONObject("response");
JSONArray docsArray = jObject.getJSONArray("docs");
for (int i = 0; i < 10; i++) {
if (docsArray.getJSONObject(i).optString("mediatype").equals("etree")) {
String title = docsArray.getJSONObject(i).optString("title");
String identifier = docsArray.getJSONObject(i).optString("identifier");
String date = docsArray.getJSONObject(i).optString("date");
System.out.println(title + " " + identifier + " " + date);
}
}
} catch (JSONException e) {
// DEBUG
Log.e("DEBUG: ", JSONString);
Log.e("DEBUG: ", e.toString());
}

The first thing that I do is create a JSONObject from "queryResult", which is the JSON response from archive.org. Note that I remove "callback(" from the JSON string because, even though archive.org returns it, it should not actually be part of the JSON string (I realized this when I was catching JSONException errors).

After that, we are ready to do some JSON parsing. Since this is just a tutorial, I hardcode "10" into the for loop because I requested 10 items. This would be a bad idea in production code (if you don't know why you are a huge noob and should not be writing production code). I only want items whose mediatype is "etree", and for each of these items I print the title, identifier, and date.

Voila, you now know how to use JSON in Android.

23 comments:

  1. Can you also share the server side of the code that a server has to include to provide this service to android clients?

    ReplyDelete
  2. The server side code does not matter to the phone, as long as the server presents valid JSON (or XML or whatever you want to parse). For example, in the tutorial I am parsing JSON from archive.org. Where does that come from? Basically, you send a query to archive.org, who in turn responds to you with valid JSON describing the results of your query. A query to archive.org looks like: http://www.archive.org/advancedsearch.php?q=Lotus&fl[]=date&fl[]=format&fl[]=identifier&fl[]=mediatype&fl[]=title&sort[]=createdate+desc&sort[]=&sort[]=&rows=10&page=1&output=json&callback=callback&save=yes
    In terms of how archive.org generates the JSON, that is outside of the scope of this tutorial. This tutorial deals with using Android to parse data in common format types that web API's often generate for you. It does not deal with the API's themselves generating the formatted data. If you have a more specific question, maybe I can help you?

    ReplyDelete
  3. Thanks for posting this tutorial. I did have a question for you. You state "The first thing that I do is create a JSONObject from queryResult", however when you create the JSONObject I don't see queryObject being passed to it. jObject = new JSONObject(JSONString.replace("callback(", "")).getJSONObject("response");
    Maybe I'm just going man blind but it feels like I'm missing something. Thanks!

    ReplyDelete
  4. You are more than welcome. That was my fault and I just fixed it on the post. Thanks for catching it. JSONString should have been QueryResult. Any other questions?

    ReplyDelete
  5. Instead of removing "callback(" in the result I think you can skip the "callback=callback" part of the archive.org url.

    ReplyDelete
  6. 谢谢!以后你知道一些中文吧!^^

    ReplyDelete
  7. Could you post the complete final code? Thanks for the explanation.

    ReplyDelete
  8. The final and complete code is embedded in a significantly larger and more complicated program (it's currently in alpha stage). Do you just want the relevant parts of the code, or do you want the whole thing? Sorry for the late response, by the way.

    ReplyDelete
  9. Thanks a lot Andrew, I had been looking exactly for this. I was hunting for a way to deal with the input stream buffer for a long time and it always threw an exception
    I wanted to ask how did you come to the size of ByteArrayBuffer(50) and bufsize = 512;
    Will it have trouble with larger json files.
    Is there a way to make it dynamic according to the json file like using getContentLength() or something.

    ReplyDelete
  10. Hey Ankit, I'm happy that you found this helpful. The initial size of the ByteArrayBuffer is not a big deal. It is dynamically resized to accomodate more data than you initially instantiate it for. I figured that starting small might be a good idea, so as to save some of the limited memory that a device running the code might have. I have no particular reason for setting bufsize to 512, other than that it seemed like a reasonable value. I don't think that this code should have any trouble with larger json files. If you are really interested in the performance of different sized buffers, I would encourage you to experiment. I would definitely be interested to see your results.

    ReplyDelete
  11. Hi Andrews,,
    I hav started development in Android..and I am very new to this JSON programming, your code is worth reading and quite helpful but I want you additional favor...can you give me the complete code....to send a particular table stored in sqlite3 database(in my mobile database) to the server and save it there in the table...and also I want to fetch the same (or other table) again and replace the mobile table(sqlite3) content by the table data fetched from the server....

    hope to get your response soon..

    Thanks..

    Tushar

    ReplyDelete
  12. Hey Tushar,

    My tutorial is about taking JSON data and parsing it in Android. You're asking me to tell you how to convert your sqlite database to JSON to send to a server, to be parsed and stored there, so that it can be fetched once again by the client to be stored again in your sqlite database. That is outside of the scope of this tutorial. I would be more than happy to help you figure out how to parse JSON that you are receiving from a server, but your question is asking a lot more than that.

    Regrettably, I cannot write your code for you! You aren't going to learn that way. When you say that you want "the complete code," what do you mean? What exactly are you asking for? Maybe I am misunderstanding you and I can help you out anyway.

    ReplyDelete
  13. Hi Andrews,

    Thanks a lot for your response..
    I feel sorry if any of my words hurt you..

    I am on a learning stage and working on a project that has the feature that a user can save his phone contacts on the server and if the user loses his mobile, the one can restore the data(contacts exported to the server) in his new mobile....so I want to do it...

    there is no one to help me out, and am highly grateful to you....

    you said you can help me to figure out how to parse JSON that i receive from the server....so please...

    I don't know the steps involved in this complete process, whether it is related to Java(Android) code, PHP or any API...

    Kindly help me or give me any reference for the same...

    Thanks a lot...

    Tushar

    ReplyDelete
  14. Andrew,

    Can you explain why this page was rated so low in WOT (Web of Trust) for the uninitiated. It says the site is not to be trusted! Also, why does it appear that Archive.org has blocked access to returning structured queries being formed this way now? I went to their advanced search page and I can't seem to query anything remotely dealing with JSON or XML now.

    RSS still works though

    Did you step on their toes or something?

    I'm definitely smelling something fishy going on.

    ReplyDelete
  15. I'm not sure what the "Web of Trust" is, or why my site is rated low on it. I've owned the domain name for under half of a year, so I really have no idea.

    A few times I have seen the JSON/XML portions of the archive.org advanced search engine stop working for a little while, with no notice or obvious reason why. Maybe maintenance? Who knows. I doubt it is because my app is so successful (though that would be awesome). I'm pretty sure that archive.org would be happy with me getting more of the music that they host out there, anyway.

    I've been meaning to add a few more tutorials based on my Vibe Vault app. You should check out the source if you are interested.

    www.andrewpearson.org/vibevault .

    ReplyDelete
  16. Web of Trust is a popular browser plugin that attempts to shield you from stuff like malicious sites that are known to carry virii and/or drive-by-download exploits. It allows users also to rate pages based on aspects like trustworthiness, privacy, child safety, etc and from those ratings the plugin then shields you from visiting "bad" sites.. or at least it supposed to.

    For example, before being allowed in this page, I had to specifically tell Web of Trust that I understood the risks in doing so because it insisted on blocking your page from loading anything.. that's why I wanted to give you a heads up on it.

    Getting back on topic, I actually prefer the Live Music for Android app a little more simply because it returns back a list of artists rather than having to know it beforehand like your app does (but I do have them both installed on my phone). Each has their own weaknesses and advantages.

    Speaking of your app, any plans to expand to the rest of Archive.org's audio pages perhaps the netlabel content?

    Thanks for your response, I was worried for a sec that you bit the hand that feeds too hard ;)

    ReplyDelete
  17. We are thinking about it, but we want to finish implementing the features which we have envisioned for the live shows before moving on to expand to other types of archive.org's audio. It was a conscious design decision to not list every artist. To be honest, we felt that it was sort of silly to have users scroll through a list of over 4,000 artists (many of which they probably don't know) to find something. In the future, we might implement an autocomplete for the artist field so that the user can see who is on archive.org.

    As far as the "Web of Trust" goes, I don't know what to say. This part of my site is hosted through blogger, so I don't understand why the "Web of Trust" would be concerned with its security. Hopefully it stops listing this site as malicious!

    ReplyDelete
  18. Different strokes for different folks.. I'm definitely not going to come on here and dictate the way you should program.. especially to someone who was very generous releasing a free app that the public has access to the source to.

    You sound like someone that is very reasonable Andrew... that's very refreshing to find on the web.

    As you probably have guessed, I'm not at all a programmer and don't claim to be but I do consider myself to know enough of the nuts and bolts (mostly from a few computer science courses) to be a little dangerous.

    I guess what I want to say is that if a person like me DID have the skills to modify your source, and I stress that these are only ideas I'm tossing out here, I would probably try to solve that artist problem you mentioned in your last post by splitting the whole shebang into the letters of the alphabet.

    Android's plain vanilla music player doesn't even do this... which I always have found kind of odd (especially if you are someone like me that likes to download a slew of multiple artist albums)

    Anyway, I've always thought (at least from a UI perspective) that this was counter-intuitive for the user.

    Here's even more theoretical stuff to throw out there (you have my blessing or anyone else out there that is reading this to take it if it is useful to you)

    I wonder if it would be possible to hold off on any data extraction until the user clicked on a letter which would then return those artists starting with that letter? That way (at least in my mind) the program wouldn't have to have so much listings to wade through and store in ram? Then, I suppose you could do the same kind of idea with other interesting fields like years, genres, dates, etc.

    The advanced search engine at Archive.org offers a lot. Also, how cool would it be to see an app populate the new shows from Live Music or from another section (I'm just saying.. haha) sorted by genre, location, maybe even lineage quality! Ah, I'm dreaming now... wake me up someone.

    Like I hinted at before, I do wish programming came naturally to me like it does for you. Alas, my brain just doesn't allow for much mathematical thinking. I'm more of a hands-on kind of person which I guess is better than nothing :P

    Take care,
    David

    ReplyDelete
  19. Different strokes for different folks.. I'm definitely not going to come on here and dictate the way you should program.. especially to someone who is very generous releasing a free app that the public has access to the source to.

    You sound like someone that is very reasonable... that's very refreshing to find on the web.

    As you probably have guessed, I'm not at all a programmer and don't claim to be but I do consider myself to know enough of the nuts and bolts (mostly from a few computer science courses) to be a little dangerous.

    The advanced search engine at Archive.org offers a lot. Another idea I had was if an app could populate the new shows from Live Music or from another section (I'm just saying.. haha) but it being sorted by genre, location, maybe even lineage quality! Ah, I'm dreaming now... wake me up someone.

    Oh the last thing I was going to mention about WOT. Here's the profile of your page over there: http://www.mywot.com/en/scorecard/blog.andrewpearson.org it looks like it has been raised. I rated your site favorably yesterday so maybe they took that in consideration? Anyway, it looks like you are out of the danger zone now.

    ReplyDelete
  20. Please disregard one of my posts.. I didn't mean to double-post. I shortened my reply the second time since I thought Blogger didn't accept my first one. (I got a weird out of memory error or something.)

    Thanks,
    David

    ReplyDelete
  21. Thank you for posting this tutorial for JSON on android. Its not exactly the best well documented topic that I've seen. And many people that want developers want them to know a slew of different types of Acronyms.. Ant, JSON, SVC, XML, etc... Can be overwhelming at times when they throw like 20 different acronyms at you. Anyways... Thanks!

    ReplyDelete
  22. Great work and Good Explanation..........Very helpful stuff.

    ReplyDelete
  23. Thanks for sharing such valuable information. Keep posting such great info for us, thanks.

    ReplyDelete