How to Set Up Pentaho Community Build Framework

Introduction

Please note that I am still writing on this article. I will update some parts later on. For some parts I have open questions that still have to be clarified. If you have any ideas/suggestions, please let me know and I happily will add it to this article.

Some sections were also copied from other sources, some of which are mentioned in the "Sources" section.

In the following chapters we will have a look at how to set up an easily maintainable environment. This article is not an introduction to Pentaho. I assume that you are familiar with the Pentaho products.

We will look at how to set up the Pentaho BI Server with the Community Build Framework (CBF).

Why would you use CBF?

Easily upgrade to new version of the BI server without going through a long setup process
Easily create environment specific versions of your BI server without going through a long setup process

Pentaho BI Server Setup with CBF

Sources:

(OS)BI Blog

Pentaho Wiki

Introducing Community Build Framework

We are in the lucky position that Pedro Alves from Webdetails offers the freely available CBF (Community Build Framework), which makes setting up and upgrading the BI Server for various environments a relative easy task. I try to talk you through the setup for the most popular operating sytems.

The main folder/file structure of CBF looks like this (simplified):

CBF

Pentaho BI Server source code folder(s): Each version of the source code will be stored in one folder. These folders are left completely untouched. Changes to the original source can be made via patches.
Project folder(s): For each project you can create a separate folder (i.e. project-nike, project-puma, ...) which holds the project specific configurations and files.

config

build.properties file, which tells ant what and where we want to build. A minimal config contains the path to your Java compiler, solutions and web server (among others).
build-env.properties (optional) where env is a placeholder for any environment. So you could name it build-prod.properties, build-test.properties etc. This is very neat as you can have global settings in the build.properties and environment specific ones in the specific build-env.properties. When we are using Ant to build our BI server, ant will load the build.properties and then use the specific build-env.properties.

patches: all files that need some kind of modification are copied here from the source code (maintaining nearly the same folder structure) and replacing constant values with tokens in form @tokenname@. The tokens are specified in the build.properties.
solution: This folder holds your reports, xactions etc. (same as Pentaho solutions folder on standard BI Server)

Target-Build: All source code files will be copied to this folder to build your BI server. All patches will be applied to this BI server.
Target-Dist: The BI Server will be copied to this folder after everything has been built
build.xml: Note that any build.xml doesn't work with all versions of the BI server. When downloading make sure you choose the version of build.xml that accepts the version of your BI server.

Note that the Pentaho BI Server source code folder(s), target-build and target-dist are shared among all projects. target-build and traget-dist are handled by ant, so you don't have to do anything about them.

Note: Currently the main folder structure has to be created mostly manually.

Create following folder structure to start with. This is common for all operating systems. We will discuss soon OS specific steps.

Pentaho

bi-server-source
project-tutorial

config
patches
solution

target-build
target-dist

The folder for the source code will be created automatically once we download the code with SVN.

In general, the below is how a folder structure could look like after completion of the setup. Ours will look slightly different.

CBF

|-- build.xml

|-- pentaho

| |-- bi-platform-api

| |-- bi-platform-appserver

| |-- bi-platform-assembly

| |-- bi-platform-assembly-open

| |-- bi-platform-build

| |-- bi-platform-engine-core

| |-- bi-platform-engine-security

| |-- bi-platform-engine-services

| |-- bi-platform-legacy

| |-- bi-platform-plugin-actions

| |-- bi-platform-plugin-services

| |-- bi-platform-repository

| |-- bi-platform-sample-data

| |-- bi-platform-sample-solution

| |-- bi-platform-scheduler

| |-- bi-platform-test-foundation

| |-- bi-platform-ui-foundation

| |-- bi-platform-util

| |-- bi-platform-web

| |-- bi-platform-web-portlet

| |-- bi-platform-web-servlet

| |-- dummy-jre

| |-- mantle

| `-- test-solutions

|-- project-client

| |-- patches

| | |-- pentaho

| | `-- target-preconfiguredinstall

| `-- solution

|-- target-build

`-- target-dist

The pentaho directory is where we will copy the a recent build of Pentaho BI Server. The idea is to leave this directory untouched and do all changes in the project-client directory. The below is an example structure, which depends on your specific project needs:

project-client/

|-- config

| |-- build.properties

| `-- build-pedro.properties

|-- patches

| |-- target-build

| | `--

| `-- target-dist

| `-- server

| |-- conf

| | |--

| | `-- jboss-service.xml

| |-- webapps

| | |-- pentaho

| | | `-- WEB-INF

| | | `-- web.xml

| `-- lib

| `-- postgresql-8.2-505.jdbc3.jar

`-- solution

|-- Portal.properties

|-- Portal.url

|-- Portal_pt.properties

|-- admin

.... etc

Pedro Alves explains:

"The idea is very simple:

All changes that would normally go to pentaho/* are placed under "patches" directory (project-client/patches/).
The CBF ant script will pick up the files in the project-client/patches/ directory, scan for tokens and replace the tokens with the variables defined inside the project-client/config/build.properties files, and copy the files to the top level directory of the entire project (In this example MyProjectDir [remark: here we change it to CBF]).
It's not recommended to patch anything under pentaho/*; sources changes are patched in to target-build/* and all other changes are made by patching the final directory, target-dist."

Windows setup

Please note that I set up everything in the directory D:\Pentaho. You might want to choose a more convenient folder for your own project.

Java, Tomcat and Ant

If you don't have Java, Tomcat and Ant running already, install them:

Java: JDK [Java Standard Edition; short Java SE] (JRE is not enough as Ant needs JDK to work properly). Read installation instructions. Install it in D:\Pentaho
Tomcat: At the time of writing, v6 was the most recent one. The location of the tomcat directory is not important; it can be placed almost any where. The location of the tomcat will be set inside of the CBF build.properties file. I installed Tomcat in D:\Pentaho
Ant: Download Apache Ant 1.8.1 and place it in D:\Pentaho. Installation notes: here.

Set environment variables

Using the GUI

Windows 7: Control Panel > System > Advanced System Settings > Click on the Environment Variables button

Set the JAVA_HOME environment variable to the directory where you installed JDK:

Click on New under the System variables section.
Type JAVA_HOME in the variable name field.
Type D:\Program Files\Java\jdk1.6.0_21 in the variable value field

Set the ANT_HOME environment variable to the directory where you installed Ant:

Click on New under the System variables section.
Type ANT_HOME in the variable name field.
Type C:\ant in the variable value field.

Set the PATH environment variable to include the directory where you installed the Ant bin directory:

Find the PATH environment variable in the list. If PATH is not listed, click on New under the System variables section.
Type %ANT_HOME%\bin;%JAVA_HOME%\bin;

Important: If there are other variables listed, create a new variable separated by a semicolon. Ensure there are no spaces before or after the semicolon.

Windows Note:

The ant.bat script makes use of three environment variables - ANT_HOME, CLASSPATH and JAVA_HOME. Ensure that ANT_HOME and JAVA_HOME variables are set, and that they do not have quotes (either ' or ") and they do not end with \ or with /. CLASSPATH should be unset or empty.

Using Command Line

Assume Ant is installed in c:\ant\. The following sets up the environment:

set ANT_HOME=c:\ant

set JAVA_HOME=c:\jdk-1.5.0.05

set PATH=%PATH%;%ANT_HOME%\bin;%JAVA_HOME%\bin;

Check Ant and Java are working

You can check the basic installation by opening a new shell and typing ant. You should get a message like this

Buildfile: build.xml does not exist!

Build failed

or something like this:

D:\Pentaho\CBF>ant

Buildfile: D:\Pentaho\CBF\build.xml

BUILD FAILED

D:\Pentaho\CBF\build.xml:7: FATAL: 'project' property not set. Please provide it

the ant command, eg: ant -Dproject=myproject -Denv=dev

So Ant works. This message is there because you need to write an individual buildfile for your project. With a ant -version you should get an output like

D:\Pentaho\CBF>ant -version

Apache Ant version 1.8.1 compiled on April 30 2010

Type in Java, hit Enter and you should get same info back:

D:\Pentaho\CBF>java

Usage: java [-options] class [args...]

(to execute a class)

or java [-options] -jar jarfile [args...]

(to execute a jar file)

where options include:

-client to select the "client" VM

-server to select the "server" VM

-hotspot is a synonym for the "client" VM [deprecate

The default VM is client.

-cp <class search path of directories and zip/jar files>

If you have problems with either Ant or Java, check that your environment variables are properly set.

Creating the .ant directory

For our work we will need an optional ant task that is not available with the default set up. In your ant directory you find a file called build.xml.

This is an Ant script that you can run to install almost all the dependencies the optional Ant tasks need.

To do so, change to the ANT_HOME directory and execute the command:

ant -f fetch.xml -Ddest=user

You should see then something like this (extract):

D:\Pentaho\apache-ant-1.8.1>ant -f fetch.xml -Ddest=user

Buildfile: D:\Pentaho\apache-ant-1.8.1\fetch.xml

pick-dest:

[echo] Downloading to C:\Documents and Settings\diethard.steiner\.ant\lib

probe-m2:

download-m2:

[echo] Downloading to C:\Documents and Settings\diethard.steiner\.ant\lib

[mkdir] Created dir: C:\Documents and Settings\diethard.steiner\.ant\lib

[get] Getting: http://ibiblio.org/maven2//org/apache/maven/maven-artifact-

ant/2.0.4/maven-artifact-ant-2.0.4-dep.jar

[get] To: C:\Documents and Settings\diethard.steiner\.ant\lib\maven-artifa

ct-ant-2.0.4-dep.jar

In general, options for Ddest are:

system - store in Ant's lib directory
user - store in the user's home directory
optional - store in Ant's source code lib/optional directory, used if building Ant source code

If you go to your user directory now, you should see the .ant folder.

XMLTask.jar

Download XMLTask.jar from here and move the jar file to the .ant\lib\ folder.

Subversion

Install on of the distributions. I provide here a list of options, choose one that suits you:

CollabNet Subversion Edge 1.2.2
http://www.sliksvn.com/en/download and a client like http://tortoisesvn.net/downloads or subclipse (an Eclipse plugin). For some additional info have a look here: http://www.codinghorror.com/blog/2008/04/setting-up-subversion-on-windows.html
Hudson is also an option as it includes subversion
Or download http://www.cygwin.com/ and issue the Linux command.

Database

Let's get started with Hypersonic DB, let on you can change to your own choice of DB.

Download HSQL 2.0 http://sourceforge.net/projects/hsqldb/files/ and save it in a convenient folder
Start the DB Server by double clicking on runServer.bat (in D:\Pentaho\hsqldb-2.0.0\hsqldb\bin)

CBF Build XML

Create the directory D:\Pentaho\CBF. Download CBF's build.xml from the Wiki Page and rename it to build.xml. Save it in Create the directory D:\Pentaho\CBF

Download a BI Server Build

Open the command prompt (DOS Client) and go to this folder. Change to D:\Pentaho\CBF\bi-server-source\ and issue following command to download (check out) BI Server 4:

svn co svn://source.pentaho.org/svnroot/bi-platform-v2/branches/4.0

If you are under a proxy you can use http:// instead of svn://

svn co http://source.pentaho.org/svnroot/bi-platform-v2/branches/4.0

You will see if the command is working once you don't get an error message and it starts adding a lot of files (A at the beginning of each line).

If you are not sure which version is the latest BI Server, just type this URL in your web browser:

http://source.pentaho.org/svnroot/bi-platform-v2/branches/

Create build.properties

An example of a build.properties file you can find here. Make sure that for Windows you use double backslashes in your paths!

#####################################

## GENERIC PROPERTIES

######################################

javac.path = D:\\Program Files\\Java\\jdk1.6.0_21\\bin\\

# Solution path. Required for compile time

solution.path = D:\\Pentaho\\CBF\\project-tutorial\\solution\\

# Solution path required for runtime. Defaults to the above value but can be different if we're

# deploying to another place. Should be used in WEB-INF/web.xml in the solution-path parameter

solution.deploy.path = D:\\Pentaho\\CBF\\project-tutorial\\solution\\

#accepted values: tomcat or jboss

server.name = tomcat

# Tomcat 5.5 path:

tomcat.path = D:\\Pentaho\\apache-tomcat-6.0.29\\

# Copy the sample solutions to our project directory? true | false

copy.pentaho.samples = true

BASE_URL = put your URL here

# Java options for the run command

java.opts = -Xmx512m -XX:MaxPermSize=512m -Xrunjdwp:transport=dt_socket,address=8765,server=y,suspend=n

#####################################

## PROJECT DIRECTORIES - The defaults usually work fine

######################################

pentaho.dir = bi-server-source\\3.5\\

pentaho.build.dir = target-build\\

#####################################

## DEPLOY OPTIONS

######################################

deploy.mode = rsync

deploy.args = -avz --exclude '.svn/' --exclude '*.log' --exclude 'work/' --exclude 'temp_user/' --exclude 'temp/' --exclude 'tmp/'

deploy.dest.server = user@host:path

deploy.dest.solution = user@host:path

Create patches

We will only go through one example here. Please prepare all other files that need changes in a similar fashion.

Example:

Copy web.xml from pentaho/bi-platform-appserver/webapps/pentaho/WEB-INF/web.xml to project-client/patches/target-dist/server/webapps/pentaho/WEB-INF/web.xml

Inside of this file, tokens can be placed that will be set by the CBF ant script. See the example snippet of web.xml bellow.

<context-param>

<param-name>solution-path</param-name>

<param-value>@solution.deploy.path@</param-value>

</context-param>

<context-param>

<param-name>base-url</param-name>

<param-value>@BASE_URL@</param-value>

</context-param>

So basically, the files stays exactly the same, you only replace some values by tokens (highlighted in red).

The tokens @solution.deploy.path@ and @BASE_URL@ are defined in the project-client/config/build.properties or project-client/config/build-client.properties files and will be replaced by the CBF ant script and the new revised web.xml with the replaced tokens will be copied to the top level directory (In this example CBF).

Start Hypersonic DB

Go to D:\Pentaho\hsqldb-2.0.0\hsqldb\bin and start runServer.bat.

Create the build

In your command line tool go to the CBF folder and issue ant -Dproject=tutorial -p, which will show you all the parameters that you can pass to the build.xml. Find an extract below:

D:\Pentaho\CBF>ant -Dproject=tutorial -p

Buildfile: D:\Pentaho\CBF\build.xml

[echo] --------------------------------------------------------------

[echo] ------ CBF - Community Build Framework -------

[echo] ------ Version: 3.5.0 -------

[echo] ------ Author: Pedro Alves (pedro.alves@webdetails.pt) -------

[echo] --------------------------------------------------------------

[echo]

[echo] SETTINGS:

[echo]

[echo] Project: ukint

[echo] Environment: build.properties

[echo] Config File: project-tutorial/config/build.properties

[echo] Server: tomcat

[echo] --------------------------------------------------------------

Pentaho BI Platform build helper..

Main targets:

all Compiles and builds the entire project

clean Cleans the solution

copy-finish Copy target files

copy-init Copy project files

Now issue the following:

ant -Dproject=tutorial -Denv=dev dist-clean all run

The command will first load build.properties and then because of the -Denv=dev, the command will load build-dev.properties. As mentioned, you don't have to have a build-env.properties file, it's optional. If you don't you don't have this file, you just run the following:

ant -Dproject=tutorial dist-clean all run

dist-clean will delete any previous buids.

Ubuntu (Linux) Setup

Open terminal (Applications > Accessories > Terminal).
Issue javac -version to see if you have a recent JDK installed.

The next section describes the setup for users who want to avoid working with command line as much as possible. Users familiar with the Terminal, please jump to "Using Terminal".

Using Applications > Ubuntu Software Center

For users that are not familiar with the command line interface, this is the way to go ... (although we still have to use the Terminal a bit)

In Ubuntu Software Center search for "openjdk-6-jdk" and click to install it [OPEN]
search for Ant, "Java based built tool like Make, Ant" will show up. Click the install button.
Search for subversion. "Advanced version control system, subversion" will shop up. Click the install button. This package includes the subversion client (svn), tools to create a Subversion repository (svnadmin) and to make a repository available over a network (svnserve). The fastest way now to progress is to use the Terminal. Follow these instructions, skip the first step as we already installed subversion. Your svn repository should then reside in /usr/local/svn/repos.
Search for subversion and choose one of the clients, like Subcommander
Search for Tomcat. At the time of this writing, Tomcat 6 was the current version. Install it.
Download a hypersonic database from here. Go to Place > Home Folder, click File > Create Folder and name it "Pentaho". Unzip the HSQLDB file in the Downloads folder and move the unzipped folder to the recently created "Pentaho" folder. [/home/diethardsteiner/Pentaho]
You can also download MySQL if you want (some install info you can find here). We will not cover setting up the environment with MySQL here, but you can later on progress to include MySQL in your environment.
Still being in the "Pentaho" folder, go to File > Create Folder and name it "CBF". Download CBF's build.xml form the CBF Wiki page, extract it and move it to the recently created "CBF" folder. Mark the file, hit F2 and rename it to build.xml. [/home/diethardsteiner/Pentaho/CBF]

Using Terminal

Note: If you followed the instructions of "Using Applications > Ubuntu Software Center", then you can ignore this section.

Install Java: sudo apt-get install openjdk-6-jdk

Install ant: sudo apt-get install ant
Install subversion: sudo apt-get install subversion. A good documentation on how to set up subversion on Ubuntu can be found here. Follow these instructions. Your svn repository should then reside in /usr/local/svn/repos. Additional info can be found here.
Install Tomcat: sudo apt-get install tomcat6
Download a hypersonic database from here. Create a folder: mkdir $Home/diethardsteiner/Pentaho. Move the folder in this directory.

Create the .ant Directory

Ant is located in usr/share/ant. Follow the instructions mentioned in the Windows section: click here.

XMLTask.jar

Download XMLTask.jar from here and move the jar file to /home/<username>/.ant/lib/. ".ant" is a hidden folder, hence you normally wont see it if you go to Places > Home Folder. Press "CTRL+H" to temporarily see the hidden folders or go to Edit > Preferences and in the "View" tab tag "Show hidden and backup files" to permanently see the hidden files.

CBF Build XML

See here

Download a BI Server Build

See here

Create build.properties

See here

Create patches

See here

Mac OS X Setup

I won't go through the procedure here again, just some special notes:

Subversion is included in Mac OS X Leopard and Snow Leopard. Have a look at this tutorial on how to get it running.

Mac specific SVN Clients: Cornerstone and SCPluging

Diethard Steiner on Business Intelligence

Saturday, November 27, 2010

How to Set Up Pentaho Community Build Framework

Introduction

Pentaho BI Server Setup with CBF

Java, Tomcat and Ant

Set environment variables

Using the GUI

Using Command Line

Check Ant and Java are working

Creating the .ant directory

XMLTask.jar

Subversion

Database

CBF Build XML

Download a BI Server Build

Create build.properties

Start Hypersonic DB

Create the build

Ubuntu (Linux) Setup

Using Applications > Ubuntu Software Center

Using Terminal

Create the .ant Directory

XMLTask.jar

CBF Build XML

Download a BI Server Build

Create build.properties

Create patches

Mac OS X Setup

2 comments: