Alfonso Jiménez

Integrating Yahoo r3 with Phing

December 21st, 2009 0 comments

Phing Yahoo r3 Yahoo r3 is a powerful tool for building websites which could require multiple languages or dimensions. The r3 outputs may be plain text files, web templates, configuration files or even source code. Sometimes maintaining a big site translated into different languages can become a nightmare, but r3 lets you do it on an easy way. It offers a neat and powerful command line interface which will make your life easier. It can also be managed by using a web interface.

I know this could be a somewhat confusing thing to explain, so I’m going to try to give an example of what you can do with it. At Weblogs SL, we have a platform which handles instances of 37 blogs in 2 languages (Spanish and Portuguese), so we have to maintain 37×2 = 74 sites. They almost share the same layout structure, but sometimes we need to make specific changes on certain site (maybe a customized header or sidebar). r3 lets you do all this kind of things by specializing templates for a particular site/language. Coming back to our example, we have two dimensions: blog and language. Once we have created the dimensions, we can generate all the templates. These raw templates are parseable by the template engine included within our platform, so they are processed by the application afterwards. We can also put all the “static variables” for each blog (such as blog titles, URLs, paths, …) during this pre-build process.

So after far too much research, we came up with Yahoo r3, but we still had an itch to scratch: how can we integrate this into our continuous integration workflow? We are using Phing (and Java Ant) for building our sites, so it was nice to have some Phing tasks to work with r3. So I wrote these two tasks:

r3GenerateTask.php

require_once 'phing/Task.php';
/**
* r3 Generate Task
*
* PHP version 5
*
* @category Phing Task
* @package phing
* @author Alfonso Jimenez
* license http://www.gnu.org/licenses/gpl.html GPL
*
*/
class r3GenerateTask extends Task
{
/**
* path to r3 workspace
*
* var string
*/
protected $r3Workspace;

/**
* init method
*
* @return boolean
*/
public function init()
{
return true;
}

/**
* main method
*
* @return void
*/
public function main()
{
$command = 'r3 -c '. $this->Workspace .' generate -av';
$output = array();
$return = null;

try {
exec($command, $output, $return);

foreach ($output as $line) {
$this->log($line, Project::MSG_VERBOSE);
}

if ($return !== 0) {
throw new BuildException('Task exited with code'. $return);
}
} catch (BuildException $e) {

$op = Project::MSG_WARN;

if ($this->quiet === true) {
$op = Project::MSG_VERBOSE;
}

$this->log($e->getMessage(), $op);
}
}
}

r3SetVarTask.php

require_once 'phing/Task.php';

/**
* r3 setVar Task
*
* PHP version 5
*
* category Phing Task
* package phing
* author Alfonso Jimenez <yo@alfonsojimenez.com>
* license http://www.gnu.org/licenses/gpl.html GPL
*
*/
class r3SetVarTask extends Task {
/**
* path to r3 workspace
*
* var string
*/
protected $r3Workspace;

/**
* dimension
*
* @var string
*/
protected $dimension;

/**
* variable key
*
* @var string
*/
protected $key;

/**
* variable value
*
* @var string
*/
protected $value;

/**
* sets the r3 workspace path
*
* param string $r3Workspace
*
* return void
*/
public function setr3Workspace($r3Workspace)
{
$this->r3Workspace = $r3Workspace;
}

/**
* sets dimension
*
* param string $dimension
*
* return void
*/
public function setDimension($dimension)
{
$this->dimension = $dimension;
}

/**
* sets the variable key
*
* param string $key
*
* return void
*/
public function setKey($key)
{
$this->key = $key;
}

/**
* sets the variable value
*
* param string $value
*
* return void
*/
public function setValue($value)
{
$this->value = $value;
}

/**
* init method
*
* @return boolean
*/
public function init()
{
return true;
}

/**
* main method
*
* @return void
*/
public function main()
{
$command = ‘r3 -c ‘. $this‐>r3Workspace .’ var set ‘. $this‐>dimension
.’ ‘. $this‐>key .’ ‘. $this‐>value;
$output = array();
$return = null;

try {
exec($command, $output, $return);

foreach ($output as $line) {
$this->log($line, Project::MSG_VERBOSE);
}

if ($return !== 0) {
throw new BuildException(‘Task exited with code’. $return);
}
} catch (BuildException $e) {

$op = Project::MSG_WARN;

if ($this->quiet === true) {
$op = Project::MSG_VERBOSE;
}

$this->log($e->getMessage(), $op);
}
}
}

If you want to use these tasks in your Phing file, you have to place them in /usr/share/php/phing/extended/tasks/ first. Then you can call them by writing the following tag-commands:

<r3-generate r3Workspace=”${workspace.dir}”></r3>
<r3-setVar r3Workspace=”${workspace.dir}” dimension=”${dimension}” key=”var_key” value=”${var_value}”></r3>

Let’s see an example of a real Phing file. Imagine that we have to write a task which needs to do the following steps:

Get a Yahoo r3 workspace from a SVN repository (http://svn.test.com/trunk/r3)
Build the Yahoo r3 workspace
Move the generated templates into a certain directory
Clean up the temporary files

So we can straightforward write a Phing task in order to follow this sequence of steps:

<?xml version="1.0"?>
<project name="Yahoo r3 Templates Build" basedir="." default="build">

<property name="tmp.dir" value="/tmp/template" />
<property name="templates.dir" value="/home/r3/templates" />
<property name="svn" value="http://svn.test.com/trunk/r3" />

<target name="build" depends="init, generate, clean" />

<target name="init">
<taskdef name="r3-generate" classname="extended.tasks.r3GenerateTask" />

<mkdir dir="${tmp.dir}" />
<svncheckout svnpath="/usr/bin/svn" repositoryurl="${svn}" todir="${tmp.dir}"/>
</target>

<target name="generate">
<r3-generate r3Workspace="${tmp.dir}"></r3-generate>
</target>

<target name="clean">
<copy todir="${templates.dir}">
<fileset dir="${tmp.dir}/htdocs">
<include name="**" />
</fileset>
</copy>

<delete dir="${tmp.dir}" includeemptydirs="true" failonerror="true" />
</target>
</project>

The Ibracadabra Show starts

July 27th, 2009 1 comment

Zlatan Ibrahimovic - FC Barcelona - BarÃ§a

How should a microblog search engine be?

July 15th, 2009 1 comment

Lens Lately people have been talking too much about searches on microblogging services. It’s been said that even Google is launching a new microblog search engine in the next few months. Some services exist already such as Twingly or Twitter Search (Summize), but none of them answer my queries as I would like. As a normal user, I would insert into a microblog search engine two type of queries: What’s happening right now? or What’s being said about ____ at the moment?, since it doesn’t make sense to look for information about the capital of Zimbabwe or if I want to know everything about a certain family of birds. It may be possible to guess one of these questions using a microblog search service, but I would use a classic search engine though.

Imagine that you’re a frustated Manchester United fan and you want to know what’s going on regarding your favourite football team. So you go to Twitter Search and look for the terms “Manchester United”. The results will leave you stunned because of the multiple kind of information found. Despite receiving hundreds of results each minute, 90% of the found messages are crap. You’ll get lots of duplicated messages, spam from the media Twitter accounts using twitterfeed, messages written in languages that you probably don’t understand (Chinese, Spanish, Italian, Arabic, …), messages written by “irrelevant” users and so so so much crap.

How should a microblog search engine be? Are we retrieving useful information from these kind of sources?

phpredis, 2 Fast 2 Furious

June 2nd, 2009 1 comment

phpredis Redis (REmote DIctionary Server) is a persistent key-value database with built-in net interface written in ANSI-C for Posix systems. Whilst it may at first seem like the wheel is being reinvented here, the need for something beyond a simple key-value database is pretty clear. It’s possible to use Redis like a replacement of memcached, with the main difference being the dataset is stored persistently – not volatile data – and Redis introduces new data structures such as list and sets. Furthermore, it implements atomic operations in order to interoperate with these data structures.

I released a PHP extension called phpredis a couple of weeks ago, which works as a PHP client API for Redis. The project is hosted at Google Code at the moment, and you can get the code directly from the SVN repository: http://phpredis.googlecode.com/svn/trunk/.

Despite a vanilla PHP client library already exists, I felt the need to write it since a PHP extension normally performs better and I wanted to make the most of Redis potential.

Let’s see a snippet of how to make a simple operation to Redis using the PHP client:

    $redis = new Redis();
    $redis->connect('127.0.0.1', 6379);

    $redis->set('key', 27);

    echo $redis->get('key'); // it should print 27

    $redis->incrby('key', 3);

    echo $redis->get('key'); // it should print 30

The code above was quite obvious, it stored a value associate to key and it increments its value by 3. Let’s see another snippet a bit more complex, for example using a list:

    $redis = new Redis();
    $redis->connect('127.0.0.1', 6379);

    $redis->lPush('list', 'val');
    $redis->lPush('list', 'val2');
    $redis->lPush('list', 'val3', 1);

    echo $redis->lPop('list', 1); // it should print val3
    echo $redis->lPop('list'); // it should print val2
    echo $this->lPop('list'); //it should print val

Notice that depending on the last optional parameter (0 by default) it’s possible to append/extract an element to/from the tail or to/from the head of the list.

There is a list including all the available methods. At the moment, I’m working on the implementation of more new Redis commands and on the support for complex data structures such as arrays and PHP objects.

Apart from phpredis, there are more client API for languages as Perl, Python, Ruby and Erlang. You can find this and more at the Redis project homepage. Redis has been written by Salvatore Sanfilippo who I’m chuffed to bits with.