Give your web app international appeal

Building web apps can be a lot of fun - particularly if people like and use them. If you dig through your stats the chances are that people from all over the world are visiting you and they speak lots of different languages. By only serving your app or website in English you’re assuming that everyone can speak it, or at least understand it well enough to get by. Just as learning to speak a new language can open the door to new cultures and ways of thinking, having a website that speaks multiple languages allows you to engage people who might otherwise have left looking for an alternative.

This was the situation we found ourselves in back in March just after we launched Diarised¹. We had people not only visiting us from all over the world but blogging about us in many different languages. We thought it would be a nice idea to get Diarised working in a few.

The process of getting your website ready for multiple languages is known as internationalisation. Internationalisation goes beyond getting the website working in other languages and covers aspects such as dates, times and currency. In this two part article we’re going to look at how you go about preparing a website for internationalisation. Then we’ll look at solutions to some of the real-world problems that can arise, such as dynamic data and plurals.

Associative Arrays

As a first stab at internationalisation we might try using associative arrays. We could store our translated text using the English text as the key, like so:

  //set up our language arrays

  $english["welcome to example.com"] = "welcome to example.com";
  $english["have a nice day"] = "have a nice day";

  $spanish["welcome to example.com"] = "Bienvenidos a example.com";
  $spanish["have a nice day"] = "Ten un día bueno";
  ...

  /*
  *  We'll set the locale based on a variable in the query string
  */

  switch($_GET["lang"]){
    case "es":		            //es turns the page into Spanish
      $messages = $spanish;
      break;

    default:			    //everything else defaults back to English
      $messages = $english;
      break;
  }
  ...

  echo $messages["welcome to example.com"]

Adding new languages should then be a case of adding more arrays and updating the switch statement. Perfect! Well not quite. There are a few issues with this approach:

1. Dynamic data. What if you need to insert someone’s username into a sentence? Breaking the sentence down into lots of little phrases will only make your translator’s life difficult and will probably lead to inaccurate translations.
2. You’re assuming your translator will be able to understand PHP array syntax and avoid breaking everything. What happens when they decide to stick something in quotation marks? The last thing you want to do after receiving a translation is to start debugging syntax errors.
3. This approach won’t fail gracefully. What happens if you accidentally type: echo $messages["wlcome to example.com"] By missing off the “e” your translation will break and leave blank text.

Clearly this method has issues. What we really want is a way to allow non-techies to translate our text for us and a way for us to simply “plug” the translation back into the website.

Introducing Gettext

After ruling out associative arrays to do Diarised translation’s we decided to use gettext². gettext is the GNU internationalization library and it provides an excellent way of separating code from content. If you’re hosting on Linux there’s a good chance it will already be installed on your server.

So, how does it work?

The official Gettext website gives a highly detailed and fairly confusing overview³ but the gist is:

1. You pass every piece of text you need translated through a function called gettext
2. Once everything has been marked up you run the xgettext command to create a PO (Portable Object) file. This is a plain text file containing the source text and a place for the translated text
3. You send this to your translator to open with a PO editor
4. Once your translator has filled out the translations they send the PO file back to you
5. You compile the PO file into a MO (Machine Object) file that gettext can read
6. You set the locale of your site to a language (usually through the query string) and sit back and admire your website in a totally different language

Step 1 is just a case of the following:

  <p>welcome to example.com</p>

becomes

  <p><? echo gettext("welcome to example.com"); ?></p>

Marking up your code to use gettext is probably the most irritating step but fortunately you only have to do this once. It will then work for as many languages as you like. The next stage is to get this information into our translation file.

How to create a PO file

The first step is to set up the directory structure:

1. In your webroot folder create a folder called locale
2. Inside that create a folder for each language you plan on supporting and use the language code as the folder name, e.g. for Spanish use es_ES or if you wanted say a localised Argentine version you could have es_AR
3. Inside each of those folders create a new one called LC_MESSAGES. This is where we will keep our translation files

To create the PO file you’ll need to use the command line but don’t worry we’re here to hold your hand. Fire up a terminal window, connect to your webserver and be brave.
The command we need is called xgettext, this will scan a script looking for calls to gettext then grab the text you’re passing and put it into a PO file. For example:

  # xgettext -o messages.po *.php

This will search every PHP page in the current working directory and stick the results in messages.po. Once this is done open the file with a plain text editor and make the following change:

  "Content-Type: text/plain; charset=CHARSETn"

becomes

  "Content-Type: text/plain; charset=utf-8n"

Now save it and send it to your translator

Although a PO file is a plain text file its contents aren’t particularly friendly. Luckily there are a few pieces of software that make editing PO files a bit easier (especially for your translator). For Windows there’s poedit⁴, and for the Mac there’s LocFactory Editor⁵. Both are free and will make your translators lives much easier.

When your translator sends the PO file back stick it into the appropriate LC_MESSAGES folder created earlier and open a terminal window so we can compile it

In your command window go to the LC_MESSAGES folder with messages.po and issue the following:

  # msgfmt messages.po

Assuming there were no errors this will churn out a file called messages.mo. This is the compiled file gettext will actually read, any changes to your PO file will require you to redo this step to make the changes live. Now all we need to do is tell gettext which language we want our text in.

Binding a locale

This step can be done via a few lines of PHP near the start of your script:

  $locale = $_GET["locale"];

  putenv("LC_ALL=$locale");
  setlocale(LC_ALL, $locale);

  bindtextdomain("messages", "locale/");	//binds the messages domain to the locale folder
  bind_textdomain_codeset("messages","UTF-8"); 	//ensures text returned is utf-8, quite often this is iso-8859-1 by default
  textdomain("messages");	//sets the domain name, this means gettext will be looking for a file called messages.mo

$locale will need to be set to the locale that you want the website appear in. To begin with it’s simplest to set this via the query string as shown above. One of the advantages of gettext is that if it can’t find a folder for the locale you pick it will just go back to English, meaning when someone sticks locale=kl expecting a Klingon version they’ll just get English.

That’s it for part one. At this point you should be able to at least make a start on preparing your websites for internationalisation. In part two we’re going to look at some of the real world issues you’re likely to run into and tell you what we did to solve them on Diarised.

URLs in this post:

1. Diarised: http://www.diarised.com
2. gettext: http://www.gnu.org/software/gettext/
3. overview: http://www.gnu.org/software/gettext/manual/gettext.html#Overview
4. poedit: http://www.poedit.net/
5. LocFactory Editor: http://www.triplespin.com/en/products/locfactoryeditor.html