DevSlant - Obviously Obfuscated Originality

One of the things that started happening @work in the last year is that more developers within the company began looking to use a wider (and more recent) selection of technology. @work has been running for years on a LAMP (Linux, Apache 1.3x, MySQL, Perl) stack.

With a move to JavaScript, HTML5, and CSS3 for front-end development, and node.js for some of the back-end infrastructure, there have been opportunities to try new languages, databases, caches, etc.

Like any group of people picking new technology, sometimes you pick wisely. Other times…

When you don’t know what your choice means

About 8 months ago, one project team was facing a high volume of reads (and a fairly high volume of writes). They decided to try using MongoDB for storing their data. They went through some rough outages, but managed to persevere and got it working (and it still works). However, sometimes success makes everything look like it can be solved the same way.

There were some features that were meant only for paying users. This is where things quickly got painful.

They used MongoDB to also store whether or not the user had paid, and should therefore be granted access to additional content and features. There was already a billing system that stored this information, but they wanted to cache the billing access in MongoDB. Some of you will already be thinking of the ways this can go wrong, but for now I’m not going to focus on the risks of having data and responsibilities being duplicated across different platforms. Instead I’ll focus on the disadvantage of using MongoDB for this.

In this particular use case, with MongoDB’s eventual consistency, users could have successfully paid, but depending on which MongoDB node the requests came to immediately after purchasing, the node may or may not have the updated data.

MongoDB isn’t where you want to store this kind of information. You need your view of billing data to be consistent 100% of the time. Imagine if you occasionally lost access to your TV channels because the system storing your billing state wasn’t consistent.

People don’t like it when you mess up the access they have paid for.

Speed and consistency

Since it was now impacting users for something they had paid for, I became involved from the billing side of @work. Myself and another developer (from the Integrations team) volunteered to take care of. We were faced with a few problems:

1) The technical - data needed to be accessed frequently (about 4,000 - 5,000 queries per second with higher volumes expected), and had to be consistent each time

2) There was no documentation on what had been done or how the product was supposed to work. Argh.

Fortunately, the actual amount of data to store was small, and could easily fit into memory. This is where we could have used memcache. We already had memcache in use in other areas of the company infrastructure, so it would be easy to spin up another instance.

After a bit of research though, we decided to use Redis. Aside from being ridiculously fast (testing showed it capable of 100,000 queries per second on our servers), it supports storing data structures such as hashes, sets, sorted sets, and lists.

Additionally, Redis was easy to work with. Compared to the rest of the code we needed to change, suddenly the data store and it’s performance was no longer a concern. Redis just worked.

Since moving the data over to Redis and ensuring that the billing system was responsible for updating it, I can say that Redis has been completely solid for us. Billions and billions of hits, no downtime, reasonably easy to slave and support for fail-over with persistence to disk. Our customers now have a correct, and consistent experience when using the product.

Some technology can really help you across a range of problem domains (I’ll blog about Cassandra for that in the future). But you should always be thinking (and testing!) about your choices, lest you end up using a shiny new hammer to solve every nail.

2011 was a mixed bag for me. I still work for the same company (7.5 years now, which is…forever in tech), but the company seems to be changing for the better. Didn’t get nearly as much into android and iPhone programming as I thought I was going to. Began teaching myself using online courses that you see from places like MIT’s open courseware and soon some of the Stanford courses. And waiting for our 2nd child to arrive in April 2012.

Books

2011 was a much slower pace for me reading wise. I read 31 books (60 in 2010).

Favourite Books in 2011:

• Robopocalypse: A Novel
• Ready Player One
• Reamde
• Science Fair Season

Tech

I was far more restrained in my usual tech binges :-). Main addition in 2011 was finally getting a smartphone (HTC Dezire Z). Working with mobile phones for so long, you realize most of them are crap - and by that I mean the software running on them in non-standard, poorly implemented, thrown-together junk done by a hardware manufacturer that has no clue how to developer software that people actually want to use.

The advent of iPhone and Android phones seems to have really improved the quality of what you get now. Not that they are perfect, but compare a Samsung, HTC, LG or iPhone to the phones made without Android/iPhone, and you’ll hate everything about the software.

Even so, it’s been a year and I’m itching to upgrade the phone. The phone seemed fast enough when it first came out, but at this point I’ve unlocked it, rooted it, replaced the system on it with a Cyanogenmod (replaced the version of Android that HTC put on it with one that has far less junk). And I’ve overclocked it to nearly double it’s original speed.

I read more on my iPad and my Kobo than ever before. Nearly every book I buy now is an eBook - I’d have no room for them all if I didn’t!

Photography

I just didn’t keep up with practising my photography in 2011. I have lots of shots of Riley, some of which I think go beyond just a family photograph, but I really need to spend some time on some photo projects. 

I did produce a photo album of Riley that I thought turned out really well, and I attended a Kelby training seminar on lighting, shooting and retouching which was excellent, but I need to apply it more often.

2012 Expectations?

Well, aside from baby #2, I’m starting a company - Gameflow Studios - to work on some side projects I’ve had in mind for a long time, and the timing and technology (and my knowledge) have finally come together. I’m also eager to see how the MITx courses turn out. It’s not an MIT degree, but there will be some kind of testing and certification process.

On May 26, 2011 Google announced the Google Wallet, which, along with support from merchants and banks, will allow you to pay for anything using your phone.

This is hardly the first shot fired in the mobile payment war, but it is an interesting one that has potential to disrupt (but also improve) existing billing solutions.

Take a look at what the traditional mobile billing providers have been doing. By traditional I’m talking about carriers (or some 3rd party they have outsourced to) that perform the billing anyone with a mobile phone is used to. It is the service that the carriers use to bill you, what the content providers use to charge you for those ring tones you buy, or the extra pig you buy in Big Barn World :-). It’s billing for  usage and services you access through your mobile provider. 

But that’s pretty limited when you compare it to what you could be using your phone to buy…

In Vancouver, we already have the simple things like paying for parking using your mobile, but the real goal is to get you buying anything and everything through your phone.

What’s so exciting about that you may ask? From a users perspective, it’s really just the ease with which you can buy anything, whether you’re buying lunch, renting a canoe, paying for a taxi, anything you can think of, you will be able to pay for with just a simple wave/click/something of your phone. Wanna bet how much impulse buying that can lead to? And the coupon offers!

The mobile payment space is getting a lot of interest lately, Square allows you to perform credit card payments with a free credit card reader they will send you, allowing you to pay for anything, anywhere your mobile has connectivity to a network. Visa just invested big in them. If you can’t beat ‘em, invest in ‘em. Google Wallet has support from MasterCard.

The carriers have been somewhat slow <cough> when it comes to billing. In the US, most of the carriers have swallowed up lots of smaller carriers over the decades, all with their own billing systems that had to either be integrated as-is, or migrated to another system. 

Heck, this kind of thing is still taking place, just look at Alltel. Part of it is now owned by Verizon and users will be switched over to Motricity (who bought InfoSpace). The other part of Alltel (to be renamed to, wait for it; Alltel Wireless Communications Corporation) is being transitioned to wmode.

This, along with the walled-garden approach by most carriers, has led to little innovation. Using Telefonica as an example, right now, chances are if you are a content provider, and say you have a deal with Telefonica O2, you will still need to go to each individual Telefonica brand, get a deal, and integrate with a possibly different billing back-end.

What’s changing for the carriers? Some are trying to create billing APIs that allow anyone to perform billing with any of their related companies using one single interface, regardless of which individual carrier you signed up with. In other words, you could be a Telefonica O2 subscriber, or a Telefonica Vivo user, but it would no longer matter, because there would be one simplified API to use for billing, so users can be charged seamlessly, and you only integrate once.

Carriers, credit card companies, and tech companies are all gearing up for the payment processing land grab. Some will make money off transaction fees, others will mine the data - oh yes, it will be mined, make no mistake, every transaction you make, the store you made it, even if it’s a lemon aid stand (GPS will still show the location :-) ), whether or not you used a coupon, how long you had the coupon for before using etc, etc, and others will provide services around all of that, creating a whole new ecosystem of companies and services.

Billing – it’s not glamourous, but it is where all the $$ pass through. (Did I mention I’m the Billing Architect @ airG ;-))

I’ve added the Disqus commenting system to the site, let’s see how this goes.

I’ve started to play with OpenGL for Android, and like most people I ended up at the NeHe (Neon Helium) tutorials for OpenGL. There are an extensive number of tutorials that you can go through there, absolutely worth your time if your interested in OpenGL regardless of your target platform. The tutorials have been ported to most languages.

For an Android specific version of some of the tutorials, you should take a look at InsanityDesign. The author has done a great job of converting some of the tutorials into Android, and it’s impressive how much simpler the code becomes.

One of the early tutorials (#6) focused on mapping a texture onto all the faces of a cube. Works well, no problem. If you happen to have read through the original tutorial, you’ll see at the end that it casually mentions that you should try mapping six different textures (one for each face). Seems like a good idea.

Well, that’s where things went down the rabbit hole a bit :-). I tried a few things, didn’t make much progress, so I decided to google around and see how others had solved this. It turns out very few have - or more specifically, very few solved it using the techniques that were used to map one texture onto all six faces.

Everyone recommends using atlas texturing, which essentially means putting all your textures together into one large texture (shaped like an unfolded cube). So I was trying to develop a solution using techniques that were not recommended. And yet…it seems like it should have been a simple thing to do. I wasn’t striving for performance, just an understanding of the techniques involved.

So I started hacking up the code…

It turns out, it’s not that hard, but since I was unable to find an example of this, and others have been searching for the same thing, here is the complete code (for all classes).

Here’s a screenshot of the program running on my phone:

The program is broken up into 3 source files, the manifest, and 6 images (in the res/drawable/ folder). Note that the image dimensions have to be a power of 2. The images I used were resized to 256x256, 24-bit, using the GIMP. I haven’t included the strings.xml - it’s likely your IDE will auto-create it for you when you start a project anyways.

AndroidManifest.xml

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
      package="ca.gameflow.textureMap"
      android:versionCode="1"
      android:versionName="1.0">
    <application android:label="@string/app_name" android:icon="@drawable/icon"
        android:theme="@android:style/Theme.NoTitleBar.Fullscreen">
        <activity android:name=".Run"
                  android:label="@string/app_name">
            <intent-filter>
                <action android:name="android.intent.action.MAIN" />
                <category android:name="android.intent.category.LAUNCHER" />
            </intent-filter>
        </activity>
    </application>
    <uses-sdk android:minSdkVersion="3" />
</manifest> 

Run.java - This is the starting activity.

package ca.gameflow.textureMap;
import android.app.Activity;
import android.opengl.GLSurfaceView;
import android.os.Bundle;
/**
 * The initial Android activity, setting and initiating
 * the OpenGL ES Renderer class (in textureMap)
 */
public class Run extends Activity {
    // The OpenGL View
    private GLSurfaceView glSurfaceView;
    // Initialize the OpenGL View and set our own Renderer
    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        // create an instance with this activity
        glSurfaceView = new GLSurfaceView(this);
        // set our own Renderer
        glSurfaceView.setRenderer(new textureMap(this));
        // set the GLSurface as View to this Activity
        setContentView(glSurfaceView);
    }
    /**
     * Remember to resume the glSurfaceView
     */
    @Override
    protected void onResume() {
        super.onResume();
        glSurfaceView.onResume();
    }
    @Override
    protected void onPause() {
        super.onPause();
        glSurfaceView.onPause();
    }
}

textureMap.java - creates the cube instance, sets OpenGL flags, and calls the cube instance to draw itself.

package ca.gameflow.textureMap;
import javax.microedition.khronos.egl.EGLConfig;
import javax.microedition.khronos.opengles.GL10;
import android.content.Context;
import android.opengl.GLU;
import android.opengl.GLSurfaceView.Renderer;
public class textureMap implements Renderer
{
    private Cube cube;
    private float xRot;
    private float yRot;
    private float zRot;
    private Context context;
    public textureMap(Context context) {
        this.context = context;
        cube = new Cube(context);
    }
    public void onSurfaceCreated(GL10 gl, EGLConfig config) {
        cube.loadGLTexture(gl, this.context);
        gl.glEnable(GL10.GL_TEXTURE_2D);
        gl.glShadeModel(GL10.GL_SMOOTH);    // enable smooth shading
        gl.glClearColor(0.0f, 0.0f, 0.0f, 0.5f);    // black background
        gl.glClearDepthf(1.0f); // depth buffer setup
        gl.glEnable(GL10.GL_DEPTH_TEST);    // enables depth testing
        gl.glDepthFunc(GL10.GL_LEQUAL); // the type of depth testing to do
        // really nice perspective calculations
        gl.glHint(GL10.GL_PERSPECTIVE_CORRECTION_HINT, GL10.GL_NICEST);
    }
    /**
     * Here we do our drawing
     */
    public void onDrawFrame(GL10 gl) {
        // clear screen and depth buffer
        gl.glClear(GL10.GL_COLOR_BUFFER_BIT | GL10.GL_DEPTH_BUFFER_BIT);
        gl.glLoadIdentity();    // reset the current modelview matrix
        /**
         * Call the draw method in the object to draw itself
         */
        gl.glTranslatef(0.0f, 0.0f, -5.0f);
        gl.glScalef(0.8f, 0.8f, 0.8f);
        gl.glRotatef(xRot, 1.0f, 0.0f, 0.0f);
        gl.glRotatef(yRot, 0.0f, 1.0f, 0.0f);
        gl.glRotatef(zRot, 0.0f, 0.0f, 1.0f);
        cube.draw(gl);
        // Rotation speed - change these to speed up / slow down the rotation
        xRot -= 4.9f;
        yRot += 2.1f;
        zRot += 2.9f;
    }
    /**
     * If the surface changes, reset the view
     */
    public void onSurfaceChanged(GL10 gl, int width, int height) {
        if (height == 0) {  // prevent divide by zero
            height = 1;
        }
        gl.glViewport(0, 0, width, height); // reset the current viewport
        gl.glMatrixMode(GL10.GL_PROJECTION);    // select the projection matrix
        gl.glLoadIdentity();    // reset the projection matrix
        // calculate the aspect ratio of the window
        GLU.gluPerspective(gl, 45.0f, (float)width / (float)height, 0.1f, 100.0f);
        gl.glMatrixMode(GL10.GL_MODELVIEW); // select the modelview matrix
        gl.glLoadIdentity();    // reset the modelview matrix
    }
}

Cube.java - this is where the real work is done. This loads up the individual textures for each face, sets up the buffers for each face, and draws them. I have no doubt that someone more experienced in Java, Android and OpenGL will find many things to cringe about :-).

package ca.gameflow.textureMap;
import android.content.Context;
import android.graphics.Bitmap;
import android.graphics.BitmapFactory;
import android.opengl.GLUtils;
import javax.microedition.khronos.opengles.GL10;
import java.io.IOException;
import java.io.InputStream;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
import java.nio.FloatBuffer;
public class Cube {
    private FloatBuffer textureBuffer;
    private FloatBuffer vertexBuffer0;
    private FloatBuffer vertexBuffer1;
    private FloatBuffer vertexBuffer2;
    private FloatBuffer vertexBuffer3;
    private FloatBuffer vertexBuffer4;
    private FloatBuffer vertexBuffer5;
    private ByteBuffer indexBuffer;
    // textures pointer
    private int[] textures = new int[6];
    private float vertices0[] = {
        -1.0f, -1.0f, 1.0f, // vertex 0
        1.0f, -1.0f, 1.0f,
        -1.0f, 1.0f, 1.0f,
        1.0f, 1.0f, 1.0f,
    };
    private float vertices1[] = {
        1.0f, -1.0f, 1.0f,
        1.0f, -1.0f, -1.0f,
        1.0f, 1.0f, 1.0f,
        1.0f, 1.0f, -1.0f,
    };
    private float vertices2[] = {
        1.0f, -1.0f, -1.0f,
        -1.0f, -1.0f, -1.0f,
        1.0f, 1.0f, -1.0f,
        -1.0f, 1.0f, -1.0f,
    };
    private float vertices3[] = {
        -1.0f, -1.0f, -1.0f,
        -1.0f, -1.0f, 1.0f,
        -1.0f, 1.0f, -1.0f,
        -1.0f, 1.0f, 1.0f,
    };
    private float vertices4[] = {
        -1.0f, -1.0f, -1.0f,
        1.0f, -1.0f, -1.0f,
        -1.0f, -1.0f, 1.0f,
        1.0f, -1.0f, 1.0f,
    };
    private float vertices5[] = {
        -1.0f, 1.0f, 1.0f,
        1.0f, 1.0f, 1.0f,
        -1.0f, 1.0f, -1.0f,
        1.0f, 1.0f, -1.0f,
    };
    // initial texture coordinates (u, v)
    private float texture[] = {
        // mapping coordinates for the vertices
        0.0f, 0.0f,
        0.0f, 1.0f,
        1.0f, 0.0f,
        1.0f, 1.0f,
    };
    // Each row defines a cube face (2 triangles per face)
    private byte indices[] = {
        0, 1, 3,  0, 3, 2,
        4, 5, 7,  4, 7, 6,
        8, 9, 11,  8, 11, 10,
        12, 13, 15,  12, 15, 14,
        16, 17, 19,  16, 19, 18,
        20, 21, 23,  20, 23, 22,
    };
    private int[] images = {
        R.drawable.image1,
        R.drawable.image2,
        R.drawable.image3,
        R.drawable.image4,
        R.drawable.image5,
        R.drawable.image6,
    };
    private Context context;
    public Cube(Context context) {
        this.context = context;
        ByteBuffer byteBuffer = ByteBuffer.allocateDirect(vertices0.length * 4);
        byteBuffer.order(ByteOrder.nativeOrder());
        vertexBuffer0 = byteBuffer.asFloatBuffer();
        vertexBuffer0.put(vertices0);
        vertexBuffer0.position(0);
        byteBuffer = ByteBuffer.allocateDirect(texture.length * 4);
        byteBuffer.order(ByteOrder.nativeOrder());
        textureBuffer = byteBuffer.asFloatBuffer();
        textureBuffer.put(texture);
        textureBuffer.position(0);
        byteBuffer = ByteBuffer.allocateDirect(vertices1.length * 4);
        byteBuffer.order(ByteOrder.nativeOrder());
        vertexBuffer1 = byteBuffer.asFloatBuffer();
        vertexBuffer1.put(vertices1);
        vertexBuffer1.position(0);
        byteBuffer = ByteBuffer.allocateDirect(vertices2.length * 4);
        byteBuffer.order(ByteOrder.nativeOrder());
        vertexBuffer2 = byteBuffer.asFloatBuffer();
        vertexBuffer2.put(vertices2);
        vertexBuffer2.position(0);
        byteBuffer = ByteBuffer.allocateDirect(vertices3.length * 4);
        byteBuffer.order(ByteOrder.nativeOrder());
        vertexBuffer3 = byteBuffer.asFloatBuffer();
        vertexBuffer3.put(vertices3);
        vertexBuffer3.position(0);
        byteBuffer = ByteBuffer.allocateDirect(vertices4.length * 4);
        byteBuffer.order(ByteOrder.nativeOrder());
        vertexBuffer4 = byteBuffer.asFloatBuffer();
        vertexBuffer4.put(vertices4);
        vertexBuffer4.position(0);
        byteBuffer = ByteBuffer.allocateDirect(vertices5.length * 4);
        byteBuffer.order(ByteOrder.nativeOrder());
        vertexBuffer5 = byteBuffer.asFloatBuffer();
        vertexBuffer5.put(vertices5);
        vertexBuffer5.position(0);
        indexBuffer = ByteBuffer.allocateDirect(indices.length);
        indexBuffer.put(indices);
        indexBuffer.position(0);
    }
    public void draw(GL10 gl) {
        gl.glActiveTexture(GL10.GL_TEXTURE0);
        gl.glClientActiveTexture(GL10.GL_TEXTURE0);
        // Face 0
        gl.glBindTexture(GL10.GL_TEXTURE_2D, textures[0]);
        gl.glEnableClientState(GL10.GL_VERTEX_ARRAY);
        gl.glEnableClientState(GL10.GL_TEXTURE_COORD_ARRAY);
        gl.glFrontFace(GL10.GL_CCW);
        gl.glVertexPointer(3, GL10.GL_FLOAT, 0, vertexBuffer0);
        gl.glTexCoordPointer(2, GL10.GL_FLOAT, 0, textureBuffer);
        gl.glDrawElements(GL10.GL_TRIANGLES, 6, GL10.GL_UNSIGNED_BYTE, indexBuffer);
        // Face 1
        gl.glBindTexture(GL10.GL_TEXTURE_2D, textures[1]);
        gl.glVertexPointer(3, GL10.GL_FLOAT, 0, vertexBuffer1);
        gl.glTexCoordPointer(2, GL10.GL_FLOAT, 0, textureBuffer);
        gl.glDrawElements(GL10.GL_TRIANGLES, 6, GL10.GL_UNSIGNED_BYTE, indexBuffer);
        // Face 2
        gl.glBindTexture(GL10.GL_TEXTURE_2D, textures[2]);
        gl.glVertexPointer(3, GL10.GL_FLOAT, 0, vertexBuffer2);
        gl.glTexCoordPointer(2, GL10.GL_FLOAT, 0, textureBuffer);
        gl.glDrawElements(GL10.GL_TRIANGLES, 6, GL10.GL_UNSIGNED_BYTE, indexBuffer);
        // Face 3
        gl.glBindTexture(GL10.GL_TEXTURE_2D, textures[3]);
        gl.glVertexPointer(3, GL10.GL_FLOAT, 0, vertexBuffer3);
        gl.glTexCoordPointer(2, GL10.GL_FLOAT, 0, textureBuffer);
        gl.glDrawElements(GL10.GL_TRIANGLES, 6, GL10.GL_UNSIGNED_BYTE, indexBuffer);
        // Face 4
        gl.glBindTexture(GL10.GL_TEXTURE_2D, textures[4]);
        gl.glVertexPointer(3, GL10.GL_FLOAT, 0, vertexBuffer4);
        gl.glTexCoordPointer(2, GL10.GL_FLOAT, 0, textureBuffer);
        gl.glDrawElements(GL10.GL_TRIANGLES, 6, GL10.GL_UNSIGNED_BYTE, indexBuffer);
        // Face 5
        gl.glBindTexture(GL10.GL_TEXTURE_2D, textures[5]);
        gl.glVertexPointer(3, GL10.GL_FLOAT, 0, vertexBuffer5);
        gl.glTexCoordPointer(2, GL10.GL_FLOAT, 0, textureBuffer);
        gl.glDrawElements(GL10.GL_TRIANGLES, 6, GL10.GL_UNSIGNED_BYTE, indexBuffer);
        // cleanup
        gl.glDisableClientState(GL10.GL_VERTEX_ARRAY);
        gl.glDisableClientState(GL10.GL_TEXTURE_COORD_ARRAY);
    }
    public void loadGLTexture(GL10 gl, Context context) {
        // Generate texture pointer
        gl.glGenTextures(images.length, textures, 0);
        for (int image = 0; image < images.length; image++) {
            // bind texture pointer 
            gl.glBindTexture(GL10.GL_TEXTURE_2D, textures[image]);
            // create nearest filtered texture
            gl.glTexParameterf(GL10.GL_TEXTURE_2D, GL10.GL_TEXTURE_MIN_FILTER, GL10.GL_LINEAR);
            gl.glTexParameterf(GL10.GL_TEXTURE_2D, GL10.GL_TEXTURE_MAG_FILTER, GL10.GL_LINEAR);
            // different possible texture parameters, e.g. GL10.GL_CLAMP_TO_EDGE
            gl.glTexParameterf(GL10.GL_TEXTURE_2D, GL10.GL_TEXTURE_WRAP_S, GL10.GL_REPEAT);
            gl.glTexParameterf(GL10.GL_TEXTURE_2D, GL10.GL_TEXTURE_WRAP_T, GL10.GL_REPEAT);
            // get the texture from the Android resource directory
            {
                InputStream is = context.getResources().openRawResource(images[image]);
                Bitmap bitmap = null;
                try {
                    // BitmapFactory is an Android graphics utility for images
                    bitmap = BitmapFactory.decodeStream(is);
                } finally {
                    try {
                        // Always clear and close
                        is.close();
                        is = null;
                    } catch (IOException e) {
                    }
                }
                // use Android GLUtils to specify a 2D texture image from our bitmap
                GLUtils.texImage2D(GL10.GL_TEXTURE_2D, 0, bitmap, 0);
                // cleanup
                bitmap.recycle();
            }
        }
    }
}

I’ve been re-reading the ThoughtWorks Anthology and trying to do more of a deep-dive this time around. The chapter on services and their contracts for interfaces (Chapter 9) has me comparing the authors solution to my own. The problems are obviously not identical, but I think there is similarity.

note: The author, Ian Robinson is a knowledgeable architect, quite familiar with designing service-oriented solutions. I’m just discussion one small part where I feel a different solution meets my needs better. You can read more from Ian Robinson here.

Essentially the problem is how to provide information returned by the service your system provides, that allows you to add or change the information returned (the schema) without breaking what the clients using your service are expecting.

Or as the author puts it, provide a contract that describes the service, the messages it sends and receives, its endpoints, and the accepted ways of communicating with it.

You want to abstract away the internal implementation so that nothing is coupled to it.

The author goes on to relate that there are two way to manage this:

1) Add schema extension points

2) Perform “just enough” validation of received messages

I’ll try to briefly summarize.

Adding schema extension points: support adding fields to what you return (be it XML or JSON, whatever) so that you can define an extension tag, and have within the extension section whatever additional fields you have added. Consumers can choose which extensions they support and ignore the rest.

Perform “just enough” validation: the consumers should only process data that they care about, and when encountering unknown or unused data, just ignore it.

There are a couple of drawbacks to adding an ‘extension’ in this way.

• You are adding data that only some consumers actually use or care about. 
• The performance of the environment it’s running in may be (needlessly) impacted by adding every piece of extended data. 
• You are giving visibility of those extensions to every consumer (both internal and external). 
• The consumers may have to sift through everything you’re giving them for just the data they care about. 

There is mention of choosing to expose a different schema to each new consumer, constraining your changes to each provider, but at the expense of making the service more complex and more costly.

Let’s take a closer look at this. You have a means of identifying the consumer, and you ideally want to provide only the fields and data that are meant for that specific consumer. This seems ripe for a templating system that maps from the consumer to the specific template (or templates) to use. Using templates to provide custom fields and data is easy. Future changes need only be done to the templates rather than in any code (aside from that which is needed to support new data fields in the first place). You can even break down the templates, and use templates that contain many smaller templates for easily customizing what you are returning.

Let me give an example. The project I am currently on needs to provide a response containing product and purchase information, as well as the action for the calling platform to take next. That action is based on some set of rules that may return different actions based on who the consumer is. It’s been fairly easy to map from an internal code, along with the calling platform code, to what specific action to send back along with the product & purchase information. We could even return it in a different format (ie, instead of JSON, it could be XML, or YAML, or anything really). No other consumer has to know or care about different formats, or different data - they all get whatever they’re expecting.

From the standpoint of the amount of work this has required, this is a better solution that what we had started with originally, it’s smaller and cleaner code, and it pushes changes out of the interface layer and into the data mapping and the templates, which are easy to populate - you just fill out whatever the template has support for.

How are you solving how you handle your external interfaces? Drop me a comment.

I’ve noticed a pattern or habit that I have fallen into recently. Trying to ‘keep up’ with many different technologies, yet never really learning anything in depth. What’s funny is that I spend a fairly large amount of time reading books, blogs, documentation of all kinds, and yet feel that I am not making the progress that I want or expect. Between jumping on incoming emails, tweets, FB updates, RSS feeds and reading programming-related material I seem to be consuming vast quantities of information, but how much actually sticks?

This weekend I went back to (yet another) book that I’ve read in the past when I have started to feel the same way; Pragmatic Thinking & Learning by Andy Hunt. Hunt is one of the co-authors of another favourite of mine – The Pragmatic Programmer.

I don’t have a mentor or coach at work to remind of the things that a book like this covers so well. It’s easy to get caught up in the chase of the next thing and ignore getting anywhere with what you’re interested in.

There are two paragraphs in the book that I think stand out for this specifically:

“Simply mastering the syllabus of knowledge doesn’t increase professional effectiveness. It’s useful, certainly, but by itself doesn’t contribute all that much to what you do in the actual, daily practice of your craft.”

“The ‘body of knowledge’ is demonstrably not the important part. The model you build in your mind, the questions you ask to build that model, and your experiences and practices built up along the way and that you use daily are far more relevant to your performance. They’re the things that develop competence and expertise. Mastery of knowledge alone isn’t sufficient.”

There’s certainly far more in the book that those two paragraphs that are worth noting. Overall the material has reminded me to really learn something by applying it. I tend to read as much as I can, sometimes slipping into the pattern of reading and more reading – but not actually applying what I am reading (because there’s always another book to read that will pull it all together of course!).

So, I’m reversing course. I’ve already started by trying git as a replacement for subversion. I am reading about it – but at the same time trying everything the book mentions, and trying things it doesn’t. I’m just using the book (Pragmatic Version Control Using Git) as a guide – but not one I am relying on to learn the ins and outs of git (which, by the way, is excellent – I already prefer it far more than using subversion).

I’m also going to be trying to getting away from the just reading to actually learning. Lately I have rarely made notes on the material I’m reading, or summarize or rewrite it in my own words. Yet doing so would (I think obviously) help increase the depth of my understanding of the subject.

This actually leads me to my next change – less reading – more projects. Making personal projects the center of my attention, and supplementing that with books or blogs or whatever really challenges and forces me to learn something I’m using, rather than just reading about something that may or may not be of use someday.

I know a lot of people will be thinking to themselves that of course this is the only way to learn things! What’s interesting is I used to do this all the time – learn entirely by doing, only reading material afterwards to see if I missed anything in my explorations. But as a fan of reading in general, somewhere along the way, reading about things is where I began to spend more time rather than trying different things out and only reading what was needed to get something working.

Of course, this can lead to missing out on details as to why things work the way they do, so I don’t intend to ignore the details. Instead, I will go deep on some subjects – really reviewing and asking questions, but only where it’s important to me that I do so. It’s time to break away from passive reading to involvement, from no note taking or simple lists to mind maps, discussions and summarizing what I think. Hmm, that sounds like more material for future blog postings :-).

One of the drawbacks to not having an actual Android phone (for now) is that I want to see what apps are available, but Google only has a sampling of what’s available on their web site. I find that rather annoying.

Of course, I do have the emulator, and in theory I should be able to do something with that. The emulator does not come with Market access, but there are system images of the virtual devices for v1.5 and 1.6.

I have the v1.5, and it’s ok, but I wanted to see what was available for v2.2, which now has 36% of all Android users on it.

I ended up using a ROM image from modaco for the nexus one (note: link may require registration).

So here’s how it all comes together (note, I am using Windows below, but it’s nearly identical under Mac or Linux):

1. Create an Android 2.2 AVD
2. Copy the system.img that comes with the Android SDK to your new AVD folder. (ex: D:\android-sdk-windows\platforms\android-8\images\system.img to C:\Users\drodger\.android\avd\AndMarket22.avd\ ).
3. Now you’ll need to start the emulator, but you need to tell it how much internal storage to use (the default isn’t enough). In the AVD manager, select the New… button by the Hardware section, and scroll down and select the Device ram size property, and then edit the value to 128 (just click on the value). Here’s the command line version: D:\android-sdk-windows\tools\emulator -avd AndMarket22.avd -partition-size 128
4. Wait. Wait some more. The emulator is slow to load this one up sometimes. Don’t bother with the next step until the emulator has completely started.
5. Now you need to get the build.prop file from within the emulator image, and make a change, and push the change back, like so:
6. D:\android-sdk-windows\tools\adb pull /system/build.prop
7. Now edit that file (it will now be in your tools\ directory), and comment out the following line (near the bottom for me):
8. ro.config.nocheckin=yes  <— put a   #  in front of this
9. Now you need to push it back:
10. D:\android-sdk-windows\tools\adb remount
11. D:\android-sdk-windows\tools\adb push build.prop /system/build.prop
12. Now you need the ROM you downloaded from modaco
13. Look within the zip, in the \system\app folder
14. Copy GoogleServicesFramework.apk  and  Vending.apk  to your tools\ directory.
15. D:\android-sdk-windows\tools\adb push GoogleServicesFramework.apk /system/app
16. D:\android-sdk-windows\tools\adb push Vending.apk /system.app
17. Remove the SDK setup package:
18. D:\android-sdk-windows\tools\adb shell rm /system/app/SdkSetup.apk
19. Close the emulator
20. Delete these files in your AVD directory: cache.img, userdata.img, userdata-qemu.img
21. Ok, now go start the emulator again, and you should be good to go!

Most of the time, when I am writing software these days, I write perl code. In VIM. Mostly because that’s what’s installed on both the local and remote servers at work, so it tends to be the most available. I actually like VIM (though there was that brief fling with Emacs, but I won’t go there).

However, at home I am delving into programming for Google’s Android system. Writing code for that in Vim…well, let’s just say that’s counter-productive.

So I did what most other people do, I went the recommended (well, free) course, and downloaded Eclipse, the Android SDK etc, and began playing with Android.

Eclipse is pretty cool, another example of what getting various people together and contributing can do. But…some of the ‘features’ in it bug me. It’s always the small stuff…

Auto-completion is my pet-peeve in Eclipse. It rarely seems to do what I want. Even when it does get it right, I find the cursor to be poorly placed when it has auto-completed a method name. I end up hitting various cursor keys far more often because of this, and it actually drains me of just typing in what I’m trying to accomplish.

Then I saw a tweet about JetBrains having a version of IntelliJ IDEA + Android. Oh, this could be interesting. I used it years ago, always thought it was a very nice IDE. It was time to take another look.

I downloaded the Early Access Program version (available for Windows/Linux/Mac) and imported an eclipse project.

This is by far the nicest, most useful IDE I have yet used (granted, I haven’t looked at Visual Studio for a few years now, it may be comparable). Auto-completion rocks with IntelliJ. Not only does it auto-complete any of the methods it’s indexed, but it auto-completes my variables and methods too! And it guesses the correct method more often than not.

As an example, I was working with a MotionEvent. Often I will want to get the X or Y position from the event. IntelliJ will suggest getX() when I first type it in. The next time, it will suggest getY() - it’s keeping track of what I’ve done before. A small feature? Sure, but I use it all the time. I tab-complete almost half my code now. That means fewer syntax errors, less typing, more time for thinking or just having the task at hand done.

There’s a saying that it’s a poor craftsman that blames his tools. But it’s also a good craftsman that makes use of the best tools. When IntelliJ 10 is available for purchase, I will be happily buying a copy (note, they also have a free community version, and educational version).

What tool(s) are you using that make you more productive?