The Pessimistic Programmer

Test driven development with SmartGWT and JPA

2011-06-21T17:48:00.003+02:00

SmartGWT is a collection of GWT widgets that can easily be connected to database tables or some other data on the server.

SmartGWT Pro supports Hibernate/JPA on the server, but there are some limitations, and unit testing it is tricky.

This article shows how to implement and test server logic with SmartGWT and JPA. Specifically, it will show how to:

Configure server.properties for unit testing.
Handle JPA relations with SmartGWT.
Implement an EMF provider so the unit tests and the data sources share the same EntityManager.
Use special JPA queries such as <= and string matching in SmartGWT.
Build complete server logic with pagination and sorting for JPA.

Examples

Before we get into the practicalities of SmartGWT, we need to design good tests. Many who start with test driven development find this very hard. My approach is to test examples of application behavior. If you discuss these examples with the customer first, you can implement automatic tests that are directly based on the requirements.

Let's say I wanted to build an online auction application. I would start by asking the client for some examples of what the application should do. The client might answer that sellers may place items for auction, bidders place bids, and the seller then sells to the highest bidder. This is not what I mean with examples, these are abstract requirements. I want concrete examples with realistic input and output values. I mean something like this:

Jane wants to sell an electric guitar. She enters the following information:

Bob wants to buy an electric guitar. He searches for auctions, clicks on Electric guitar, and places a bid.

Jane can then see the bids in her auction:

The advantages of using examples are that they are detailed, they reveal a lot of misunderstandings and they can be tested.

Objects

Another thing that many struggle with is to design a good domain model, be it a class model or a database schema. The approach shown here makes this easy. Instead of starting with a class diagram, I start by looking for objects in the examples. Then I derive the class model from the objects.

An example connected to objects is called an object case.

Jane wants to sell an electric guitar

For each example, I need to determine how to handle the input from the user and how to calculate the output. In this example, I will create a User object and an Auction object. The arrows show that all the input from the user is stored in these objects.

Bob searches for auctions

In this example, the Free_text and Max_price input is used to search for Auctions, but the Category is not used. This means that something is missing in the object model.

I decide to introduce a new object that will be used to search for category. Now, all input values are used and the output in the table is calculated from the Auction objects, so everything is covered.

Bob clicks on Electric guitar

All the output values can be calculated from an Auction object. We should define another example that shows an actual bid.

Bob places a bid

The input values are stored in two new objects: User and Bid. What happens if the user already exists? We should add another example to deal with this.

Classes

All input and output values are now connected to objects. This is a simple and powerful method to design an object model. Based on these objects, I can create the following class diagram:

Find more functionality

We can use the class diagram to find missing functionality. The diagram below shows how objects are created and read by the examples defined so far.

What is missing?

Create Category objects. Should an adminstrator to this or should it happen automatically? This needs to be discussed with the client.
Set link between Category and Auction. This should be done by Create auction.
Read User objects. What do we need the email for?
Read Bid objects. This should be part of Auction details. We should define an example to show this.
Update attributes. This needs to be discussed with the client.
Delete objects. We need to discuss with the client if this should be possible.

Based on this analysis, I add a Category parameter to the first example and use that parameter to find an existing Category object. I create a link between this Category and the new Auction:

Implement JPA classes

It's straight forward to implement the classes as JPA entities.

@Entity @Table(name="tb_auction")
public class Auction {
    @Id @GeneratedValue(strategy=GenerationType.AUTO)
    private Long id;

    private String title;
    private float startingPrice;
    private String description;
 
    @ManyToOne(optional=false)
    private Category category;

    @ManyToOne(optional=false)
    private User seller;

    @OneToMany(mappedBy="auction")
    private Set<Bid> bids = new HashSet<Bid>();
}

@Entity @Table(name="tb_bid")
public class Bid {
    @Id @GeneratedValue(strategy=GenerationType.AUTO)
    private Long id;

    private float bid;

    @ManyToOne(optional=false)
    private Auction auction;
 
    @ManyToOne(optional=false)
    private User bidder;
}

@Entity @Table(name="tb_category")
public class Category {
    @Id @GeneratedValue(strategy=GenerationType.AUTO)
    private Long id;

    private String name;
    
    @OneToMany(mappedBy="category")
    private Set<Auction> auctions = new HashSet<Auction>();
}

@Entity @Table(name="tb_user")
public class User {
    @Id @GeneratedValue (strategy=GenerationType.AUTO)
    private Long id;

    private String email;
    
    @OneToMany(mappedBy="seller")
    private Set<Auction> auctions = new HashSet<Auction>();

    @OneToMany(mappedBy="bidder")
    private Set<Bid> bids = new HashSet<Bid>();
}

(Getters and setters removed for clarity.)

Implement tests

Each example is a test. I divide each test into 4 parts:

Initial objects
Input
Output
Result objects

With SmartGWT and JPA this becomes:

Initial objects: Create objects in the database. The best way to do this in a unit test is to use a HSQLDB in-memory database, which can create the database schema on the fly.
Input: Call a SmartGWT datasource with the specified parameters.
Output: Check the output from the datasource.
Result objects: Check the resulting objects in the database.

Let's start with testing the first example: Jane wants to sell an electric guitar.

public class AuctionTest extends TestCase {
 private DataSource auctionDS;
 private DataSource bidDS;
 
 private HashMap<String,Object> input = new HashMap<String,Object>();

 @Override
 protected void setUp() throws Exception {
  super.setUp();
  auctionDS = DataSourceManager.get("Auction");
  bidDS = DataSourceManager.get("Bid");
 }

 public void testJaneWantsToSellAnElectricGuitar() throws Exception {
  // Initial objects:
  category1 = new Category();
  category1.setName("Musical instruments");
  persist(category1); 

  // Input:
  input.put("title",       "Electric guitar for beginners");
  input.put("startingPrice",  150f);
  input.put("description", "Nice electric guitar with amplifier.");
  input.put("email",       "jane99@abc123.com");
  input.put("categoryId",  category1.getId());
  auctionDS.add(input);

  // Output: None.

  // Result objects:
  List users = getResultObjects(User.class);
  assertEquals(1, users.size());
  assertEquals("jane99@abc123.com", users.get(0).getEmail());

  List auctions = getResultObjects(Auction.class);
  assertEquals(1, auctions.size());
  Auction auction = auctions.get(0);
  assertEquals("Electric guitar for beginners",        auction.getTitle());
  assertEquals(150f,                                   auction.getStartingPrice());
  assertEquals("Nice electric guitar with amplifier.", auction.getDescription());
  assertEquals(category1.getId().intValue(),           auction.getCategoryId());
 }

 private void persist(Object... objects) throws Exception {
  EntityManager em = EMF.getEntityManager();
  Object tx = EMF.getTransaction(em);
  for(Object obj : objects) {
   em.persist(obj);
  }
  EMF.commitTransaction(tx);
 }

 private <T> List getResultObjects(Class<T> clazz) throws Exception {
  EntityManager em = EMF.getEntityManager();
  Object tx = EMF.getTransaction(em);
  List<T> result = em.createQuery("from " + clazz.getName()).getResultList();
  EMF.commitTransaction(tx);
  return result;
 }
}

Here is the second example: Bob wants to buy an electric guitar. This is more complicated with some initial objects and several steps.

public void testBobWantsToBuyAnElectricGuitar() throws Exception {
  initialObjects2Auctions();
  
  // Search for auctions.
  input.put("categoryId", category1.getId());
  input.put("title",      "guitar");
  input.put("maxPrice",   250f);
  
  List<Auction> output = auctionDS.fetch(input);
  assertEquals(2, output.size());
  assertEquals("Guitar",          output.get(0).getTitle());
  assertEquals(200f,              output.get(0).getStartingPrice());
  assertEquals("Electric guitar", output.get(1).getTitle());
  assertEquals(150f,              output.get(1).getStartingPrice());

  // Auction details.
  input.clear();
  input.put("id", auction1.getId());
  
  output = auctionDS.fetch(input);
  assertEquals(1, output.size());
  Auction auction = output.get(0);
  assertEquals("Electric guitar for beginners",        auction.getTitle());
  assertEquals("Nice electric guitar with amplifier.", auction.getDescription());
  assertEquals(150f,                                   auction.getStartingPrice());
  assertEquals("No bids.",                             auction.getHighestBid());

  // Place bid.
  input.clear();
  input.put("auctionId",   auction1.getId());
  input.put("bidderEmail", "bob@abc999.com");
  input.put("bid",         150);
  ErrorReport errors = bidDS.validate(input, true);
  assertEquals(null, errors);
  bidDS.add(input);

  // Result objects:
  List<User> users = getResultObjects(User.class);
  assertEquals(1, users.size());
  User user = users.get(0);
  assertEquals("bob@abc999.com", user.getEmail());
  List<Bid> bids = getResultObjects(Bid.class);
  assertEquals(1, bids.size());
  Bid bid = bids.get(0);
  assertEquals(150,              bid.getBid());
  assertEquals(user.getId(),     bid.getBidder().getId());
  assertEquals(auction1.getId(), bid.getAuction().getId());
 }
 
 private void initialObjects2Auctions() throws Exception {
  category1 = new Category();
  category1.setName("Musical instruments");
  persist(category1);

  auction1 = initialAuction(category1, "Electric guitar for beginners", "Nice electric guitar with amplifier.", 150f);
  auction2 = initialAuction(category1, "Guitar", "Used guitar.", 250f);
  persist(auction1, auction2);
 }

 private Auction initialAuction(Category category, String title, String description, float startingPrice) {
  Auction object = new Auction();
  object.setTitle(title);
  object.setDescription(description);
  object.setStartingPrice(startingPrice);
  category.addAuction(object);  
  return object;
 }

Let's add some more tests while we're at it:

public void testNoSearchCriteria() throws Exception {
  initialObjects2Auctions();
  assertEquals(2, auctionDS.fetch(input).size());
 }

 public void testSearchForUnknownCategory() throws Exception {
  initialObjects2Auctions();
  input.put("categoryId", category1.getId() + 1);
  assertEquals(0, auctionDS.fetch(input).size());
 }

 public void testSearchForMaxPrice150() throws Exception {
  initialObjects2Auctions();
  input.put("maxPrice", 150f);
  assertEquals(1, auctionDS.fetch(input).size());
 }

 public void testSearchForMaxPrice250() throws Exception {
  initialObjects2Auctions();
  input.put("maxPrice", 250f);
  assertEquals(2, auctionDS.fetch(input).size());
 }

 public void testSearchForTitlePiano() throws Exception {
  initialObjects2Auctions();
  input.put("title", "piano");
  assertEquals(0, auctionDS.fetch(input).size());
 }

 public void testSearchForTitleElectricGuitar() throws Exception {
  initialObjects2Auctions();
  input.put("title", "electric guitar");
  assertEquals(1, auctionDS.fetch(input).size());
 }

These tests will actually fail, but that's ok. When I fix them, I will learn much about the behavior.

The first thing I notice, is a compilation error because of a missing method, so let's add it:

public class Auction {
 public String getHighestBid() {
  Float highestBid = null;
  for(Bid bid : getBids()) {
   if(highestBid == null || bid.getBid() > highestBid) {
    highestBid = bid.getBid();
   }
  }
  return (highestBid != null ? highestBid.toString() : "No bids.");
 }
 ...

This can instead be handled by a database query in the "serverObject" that will be implemented later in this article.

To use HSQLDB for unit testing, add hsqldb.jar to the project and create a persistence.xml file in the test/META-INF directory with the following properties:

<property name="hibernate.connection.url" value="jdbc:hsqldb:mem:auction"/>
<property name="hibernate.hbm2ddl.auto" value="create-drop"/>

Implement datasources

The tests use the datasources "Bid" and "Auction" that I haven't defined yet. Here is a starting point:

<DataSource ID="Auction"
    serverConstructor="com.isomorphic.jpa.JPADataSource"
    beanClassName="com.objectgeneration.auctions.Auction">
    <fields>
        <field name="id"          type="sequence" hidden="true"       primaryKey="true" />
        <field name="title"       type="text"     title="Title"       required="true"   />
        <field name="description" type="text"     title="Description" required="true" />
        <field name="categoryId"  type="integer"  title="Category"    canEdit="false" />
    </fields>
</DataSource>

<DataSource ID="Bid"
    serverConstructor="com.isomorphic.jpa.JPADataSource"
    beanClassName="com.objectgeneration.auctions.Bid">
    <fields>
        <field name="id"   type="sequence" hidden="true" primaryKey="true"/>
        <field name="bid"  type="number"   title="Bid"   required="true"/>
    </fields>
</DataSource>

Using SmartGWT and JPA from JUnit

I am getting eager to implement the behavior, but unfortunately I have to fix some issues to get SmartGWT to work with JPA in JUnit first. Let's start:

That didn't go so well. Here is a stack trace:

java.lang.NullPointerException
 at com.isomorphic.io.ISCFile.<init>(ISCFile.java:145)
 ...
 at com.isomorphic.datasource.DataSourceManager.get(DataSourceManager.java:68)
 at com.objectgeneration.auctions.AuctionTest.setUp(AuctionTest.java:31)

The problem here is that SmartGWT thinks it is running as a servlet, but it does not find its web root. This can be configured in server.properties. Create a server.properties file for unit testing in the test folder. The content should be similar to the production version, except for the following:

# This is user specific. Use Ant filtering to generate this.
webRoot: /Users/lars/temp/auctions/war/

jpa.emfProvider: com.isomorphic.jpa.EMFProviderLMT

# Name of the datasource (from persistence.xml)
jpa.persistenceUnitName: ds

Also make sure the test source has a separate output folder (see Eclipse: Project > Properties > Java Build Path > Source). Otherwise the server.properties from src and test will overwrite each other in the output directory with random results.

Running JUnit again, we get the next problem:

java.lang.ClassCastException: java.util.ArrayList cannot be cast to java.util.Map
 at com.isomorphic.datasource.BasicDataSource.buildFieldData(BasicDataSource.java:426)
 ...
 at com.isomorphic.datasource.DataSourceManager.get(DataSourceManager.java:68)
 at com.objectgeneration.auctions.AuctionTest.setUp(AuctionTest.java:31)

This can be fixed by another setting in test/server.properties:

isomorphicPathRootRelative: Auctions_js/sc

(It should point to the sub-directory of war/ where initsc.js is created.)

JPA relations and SmartGWT

Trying again I get this stack trace:

javax.persistence.PersistenceException: org.hibernate.PropertyValueException: not-null property references a null or transient value: com.smartgwt.sample.server.Auction.category
 at org.hibernate.ejb.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:637)
 at org.hibernate.ejb.AbstractEntityManagerImpl.persist(AbstractEntityManagerImpl.java:226)
 at com.isomorphic.jpa.JPADataSource.executeAdd(JPADataSource.java:452)
 at com.isomorphic.datasource.DataSource.execute(DataSource.java:1050)
 at com.isomorphic.jpa.JPADataSource.execute(JPADataSource.java:218)
 …
 at com.isomorphic.datasource.DataSource.add(DataSource.java:1948)
 at com.objectgeneration.auctions.AuctionTest.testJaneWantsToSellAnElectricGuitar(AuctionTest.java:45)
Caused by: org.hibernate.PropertyValueException: not-null property references a null or transient value: com.smartgwt.sample.server.Auction.category

SmartGWT does not handle JPA relations like "Category category" in class Auction, it only handles foreign keys like "long categoryId". But I want to use JPA relations. The workaround is to have both and define only the foreign key in the datasource as above. The Java code is as follows:

public class Auction {
 @ManyToOne(optional=false)
 @JoinColumn(name="category", nullable=false, insertable=false, updatable=false)
 private Category category;
 
 @Column(name="category")
 private long categoryId;
...
}

The category column is mapped to 2 Java variables, but only one of them can be writable. Both of them need to be set when creating the initial objects:

public class Category {
 public void addAuction(Auction auction) {
  auction.setCategory(this);
  getAuctions().add(auction);
  assert getId() != null;
  auction.setCategoryId(getId());
 }
...
}

Trying again:

javax.persistence.EntityExistsException: org.hibernate.exception.ConstraintViolationException: could not insert: [com.smartgwt.sample.server.Auction]
 …
 at com.isomorphic.datasource.DataSource.add(DataSource.java:1948)
 at com.objectgeneration.auctions.AuctionTest.testJaneWantsToSellAnElectricGuitar(AuctionTest.java:45)
Caused by: org.hibernate.exception.ConstraintViolationException: could not insert: [com.smartgwt.sample.server.Auction]
 ...
Caused by: java.sql.SQLException: Integrity constraint violation - no parent FKD896457230F353B7 table: TB_USER in statement [insert into tb_auction (id, category, description, seller, startingPrice, title) values (null, ?, ?, ?, ?, ?)]

Now we are finally getting to the business logic. I cannot create an Auction without a User.

I decide to create a new User datasource and add a User before creating the Auction:

public void testJaneWantsToSellAnElectricGuitar() throws Exception {
  // Initial objects:
  category1 = new Category();
  category1.setName("Musical instruments");
  persist(category1);

  // Input:
  long userId = createUser("jane99@abc123.com");
  input.put("title",       "Electric guitar for beginners");
  input.put("startingPrice",  150f);
  input.put("description", "Nice electric guitar with amplifier.");
  input.put("categoryId",  category1.getId());
  input.put("sellerId",    userId);
  auctionDS.add(input);

  ...
 }

 private long createUser(String email) throws Exception {
  HashMap<String,object> input = new HashMap<String,object>();
  input.put("email", email);
  ErrorReport errors = userDS.validate(input, true);
  assertEquals(null, errors);
  DSRequest dsRequest = new DSRequest(userDS.getName(), DataSource.OP_ADD);
  dsRequest.setValues(input);
  DSResponse dsResponse = dsRequest.execute();
  return ((User)dsResponse.getData()).getId();
 }

I also connect the initial Auction objects in the other unit tests to a User. This makes some of the tests pass:

Many of these failures are because the tests are not isolated. If I run each test separately, many of them will pass. The reason for this is that the database is created when the first test is run, and then the next test uses the same database with the objects that are left. I need to empty the database between each test. This is easy to do in JPA; just call EntityManagerFactory.close(). But I don't have access to the EntityManagerFactory that SmartGWT uses. To fix this, I need to implement my own "EMF provider" and point to it in test/server.properties:

jpa.emfProvider: com.objectgeneration.auctions.server.MyEMFProvider

And here is the implementation:

import javax.persistence.EntityManager;
import javax.persistence.EntityManagerFactory;
import javax.persistence.EntityTransaction;
import javax.persistence.Persistence;

import com.isomorphic.jpa.EMFProviderInterface;

public class MyEMFProvider implements EMFProviderInterface {
 private static EntityManagerFactory factory;
 
 // Support multi-threaded tests.
 private static ThreadLocal<Entitymanager> entityManager = new ThreadLocal<Entitymanager>();
 
 public MyEMFProvider() {
 }
 
 /** Close the factory after each unit test so it starts with an empty database next time. */
 public static void close() {
  if(entityManager.get() != null) {
   entityManager.get().close();
   entityManager.set(null);
  }
  if(factory != null) {
   factory.close();
   factory = null;
  }
 }

 protected EntityManagerFactory createFactory() {
  return Persistence.createEntityManagerFactory("ds");
 }

 public EntityManagerFactory get() {
  if(factory == null) {
   factory = createFactory();
  }
  return factory;
 }

 public EntityManager getEntityManager() {
  if(entityManager.get() == null) {
   entityManager.set(get().createEntityManager());
  }
  return entityManager.get();
 }

 public void returnEntityManager(EntityManager em) {
  // Do nothing.
 }
 
 public Object getTransaction(EntityManager em) {
  EntityTransaction tx = em.getTransaction();
  if(!tx.isActive()) {
   tx.begin();
  }
  return tx;
 }

 public void commitTransaction(Object obj) {
  EntityTransaction tx = (EntityTransaction) obj;
  if(tx.isActive()) {
   tx.commit();
  }
 }
 
 public void rollbackTransaction(Object obj) {
  EntityTransaction tx = (EntityTransaction) obj;
  if(tx.isActive()) {
   tx.rollback();
  }
 }
}

Add the following to the test class to start with an empty database in each unit test:

protected void tearDown() throws Exception {
  MyEMFProvider.close();
  super.tearDown();
}

This takes us much closer to our goal, and we can continue with the business logic:

Implement datasource

There is no wonder that maxPrice is not working, the Auction class does not even have this field. I want to find all Auctions where Auction.startingPrice <= the maxPrice parameter. To implement this, I need to implement special handling for this data source in a "serverObject":

<DataSource ID="Auction"
    serverConstructor="com.isomorphic.jpa.JPADataSource"
    beanClassName="com.smartgwt.sample.server.Auction">
    <serverObject className="com.objectgeneration.auctions.AuctionRequestHandler"/>
    <fields>
        <field name="id"          type="sequence" hidden="true"   primaryKey="true" />
        <field name="title"       type="text"     title="Title"       required="true"   />
        <field name="description" type="text"     title="Description" required="true" />
        <field name="categoryId"  type="integer"  title="Category"    canEdit="false" />
        
        <!-- This field does not exist in the Auction class. It is implemented in AuctionRequestHandler. -->
        <field name="maxPrice"    type="number"   title="Max price" hidden="true" />
    </fields>
</DataSource>

The serverObject in the XML above refers to a class that will be called instead of the default SmartGWT handler, and there I can handle maxPrice:

import java.util.List;
import java.util.Map;

import javax.persistence.EntityManager;
import javax.persistence.Query;

import org.apache.log4j.Logger;

import com.isomorphic.datasource.DSRequest;
import com.isomorphic.datasource.DSResponse;
import com.isomorphic.jpa.EMF;
import com.smartgwt.sample.server.Auction;

public class AuctionRequestHandler {
 private static final Logger log = Logger.getLogger(AuctionRequestHandler.class);
 
 public DSResponse fetch(DSRequest dsRequest) throws Exception {
  Map<String,Object> criteria = dsRequest.getCriteria();
  log.info("fetch(" + criteria + ")");
  
  String queryString = "from Auction";
  String separator = " where ";
  for(String paramName : criteria.keySet()) {
   queryString += separator;
   if(paramName.equals("maxPrice")) {
    queryString += "startingPrice <= :" + paramName;
   } else {
    queryString += paramName + " = :" + paramName;
   }
   separator = " and ";
  }
  
  EntityManager em = EMF.getEntityManager();
  Query query = em.createQuery(queryString);
  for(String paramName : criteria.keySet()) {
   Object paramValue = criteria.get(paramName);
   query.setParameter(paramName, paramValue);
  }
  List<Auction> result = query.getResultList();
  return new DSResponse(result, DSResponse.STATUS_SUCCESS);
 }
}

I want to search for title with substrings and ignore case. Substrings is built in to SmartGWT, but not ignore case. Here is an updated version with special handling for the title field:

public DSResponse fetch(DSRequest dsRequest) throws Exception {
  Map<String,Object> criteria = dsRequest.getCriteria();
  log.info("fetch(" + criteria + ")");
  
  String queryString = "from Auction";
  String separator = " where ";
  for(String paramName : criteria.keySet()) {
   Object paramValue = criteria.get(paramName);
   queryString += separator;
   if(paramName.equals("maxPrice")) {
    queryString += "startingPrice <= :" + paramName;
   } else if(paramName.equals("title")) {
    queryString += "UPPER(" + paramName + ") LIKE '%" + ((String)paramValue).toUpperCase() + "%'";
   } else {
    queryString += paramName + " = :" + paramName;
   }
   separator = " and ";
  }
  
  EntityManager em = EMF.getEntityManager();
  Query query = em.createQuery(queryString);
  for(String paramName : criteria.keySet()) {
   if(!paramName.equals("title")) {
    Object paramValue = criteria.get(paramName);
    query.setParameter(paramName, paramValue);
   }
  }
  List result = query.getResultList();
  return new DSResponse(result, DSResponse.STATUS_SUCCESS);
 }

Success!

Wrapping it up

One improvement could be to move the getHighestBid() logic to a database query. The AuctionRequestHandler could call this query and store the value in a @Transient field in the Auction class to send it to the client, or it could send HashMaps instead of Auction objects.

We also need to handle the things that SmartGWT does by default: Transactions, pagination and sorting. (I believe that if you don't implement pagination and sorting in the server, the client will handle it instead. This may be acceptable if you don't have thousands of rows in that database table. Please refer to the SmartGWT documentation.)

A complete solution with a general superclass and special handling in a subclass looks like this:

import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;

import javax.persistence.EntityManager;
import javax.persistence.EntityTransaction;
import javax.persistence.Query;

import org.apache.log4j.Logger;

import com.isomorphic.datasource.DSRequest;
import com.isomorphic.datasource.DSResponse;
import com.isomorphic.jpa.EMF;

public class RequestHandler {
 private static final Logger log = Logger.getLogger(RequestHandler.class);

 public DSResponse fetch(DSRequest dsRequest) throws Exception {
  return handleRequest(dsRequest);
 }
 
 public DSResponse add(DSRequest dsRequest) throws Exception {
  return handleRequest(dsRequest);
 }

 public DSResponse remove(DSRequest dsRequest) throws Exception {
  return handleRequest(dsRequest);
 }
 
 public DSResponse update(DSRequest dsRequest) throws Exception {
  return handleRequest(dsRequest);
 }

 protected DSResponse handleRequest(DSRequest dsRequest) throws Exception {
  EntityManager em = EMF.getEntityManager();
  EntityTransaction tx = (EntityTransaction) EMF.getTransaction(em);
  try {
   DSResponse dsResponse = handleInsideTransaction(dsRequest);
   tx.commit();
   return dsResponse;
  } catch(Exception e) {
   MyEMFProvider.rollback(tx);
   throw e;
  } finally {
   MyEMFProvider.finish();
  }
 }

 protected DSResponse handleInsideTransaction(DSRequest dsRequest) throws Exception {
  return dsRequest.execute();
 }

 @SuppressWarnings("unchecked")
 protected <T> DSResponse query(EntityManager em, DSRequest dsRequest, Class<T> clazz) {
  String queryString = "from " + clazz.getName();

  LinkedHashMap<String, Object> parameters = new LinkedHashMap<String, Object>();
  queryString = addCriteria(queryString, dsRequest.getCriteria(), parameters);
  
  List<T> result = selectObjects(em, dsRequest, queryString, parameters);
  long totalRows = countObjects(em, dsRequest, queryString, parameters, result);

  DSResponse dsResponse = new DSResponse(result, DSResponse.STATUS_SUCCESS);
  dsResponse.setStartRow(dsRequest.getStartRow());
  dsResponse.setEndRow(dsRequest.getStartRow() + result.size());
  dsResponse.setTotalRows(totalRows);
  return dsResponse;
 }

 private String addCriteria(String queryString, Map<String, Object> criteria, Map<String, Object> parameters) {
  String separator = " where ";
  for(String paramName : criteria.keySet()) {
   Object paramValue = criteria.get(paramName);
   queryString += separator + addCriterium(paramName, paramValue, parameters);
   separator = " and ";
  }
  return queryString;
 }

 /** Override this method for special handling of query parameters. */
 protected String addCriterium(String paramName, Object paramValue, Map<String, Object> parameters) {
  parameters.put(paramName, paramValue);
  return paramName + " = :" + paramName;
 }

 private <T> List<T> selectObjects(EntityManager em, DSRequest dsRequest, String queryString, Map<String, Object> parameters) {
  queryString = addOrderBy(queryString, dsRequest.getSortBy());
  Query query = em.createQuery(queryString);
  addPagination(dsRequest, query);
  setQueryParameters(query, parameters);
  return query.getResultList();
 }

 private String addOrderBy(String queryString, String sortBy) {
  if(sortBy != null) {
   if(sortBy.startsWith("-")) {
    queryString += " ORDER BY " + sortBy.substring(1) + " DESC";
   } else {
    queryString += " ORDER BY " + sortBy + " ASC";
   }
  }
  return queryString;
 }

 private void addPagination(DSRequest dsRequest, Query query) {
  long startRow = dsRequest.getStartRow();
  if(startRow >= 0) {
   query.setFirstResult((int)startRow);
  }
  
  long endRow = dsRequest.getEndRow();
  if(endRow > startRow) {
   query.setMaxResults((int)endRow - (int)startRow);
  }
 }

 private long countObjects(EntityManager em, DSRequest dsRequest, String queryString, Map<String, Object> parameters, List result) {
  long totalRows;
  long startRow = dsRequest.getStartRow();
  long endRow = dsRequest.getEndRow();
  if(startRow < 0 || endRow <= startRow) {
   totalRows = result.size();
   log.debug("no pagination, totalRows=" + totalRows);
  } else if(result.size() < endRow - startRow) {
   totalRows = result.size();
   log.debug("all rows received, totalRows=" + totalRows);
  } else {
   queryString = "select count(*) " + queryString;
   log.debug("countObjects: queryString=" + queryString + ", parameters=" + parameters);
   Query countQuery = em.createQuery(queryString);
   setQueryParameters(countQuery, parameters);
   Object countResult = countQuery.getSingleResult();
   log.debug("countObjects: result=" + countResult);
   totalRows = (Long) countResult;
  }
  return totalRows;
 }

 private void setQueryParameters(Query query, Map<String,Object> parameters) {
  for(String paramName : parameters.keySet()) {
   Object paramValue = parameters.get(paramName);
   query.setParameter(paramName, paramValue);
  }
 }
}


import java.util.Map;

import com.isomorphic.datasource.DSRequest;
import com.isomorphic.datasource.DSResponse;
import com.isomorphic.datasource.DataSource;

public class AuctionRequestHandler extends RequestHandler {
 @Override
 protected DSResponse handleInsideTransaction(DSRequest dsRequest) throws Exception {
  if(dsRequest.getOperationType().equals(DataSource.OP_FETCH)) {
   return query(dsRequest, Auction.class);
  } else {
   return dsRequest.execute();
  }
 }
 
 @Override
 protected String addCriterium(String paramName, Object paramValue, Map<String, Object> parameters) {
  if(paramName.equals("maxPrice")) {
   parameters.put(paramName, (Float) paramValue);
   return "startingPrice <= :" + paramName;
  } else if(paramName.equals("title")) {
   return "UPPER(" + paramName + ") LIKE '%" + ((String)paramValue).toUpperCase() + "%'";
  } else {
   return super.addCriterium(paramName, paramValue, parameters);
  }
 }
}

Implement user interface

When the server logic works, it is relatively straight forward to implement the user interface. The code looks something like this:

    public void onModuleLoad() {
        final DataSource auctionDS = DataSource.get("Auction");
        final DataSource bidDS = DataSource.get("Bid");

        final ListGrid auctionGrid = new ListGrid();
        auctionGrid.setDataSource(auctionDS);
        auctionGrid.setAutoFetchData(true);

        final ListGrid bidGrid = new ListGrid();
        bidGrid.setDataSource(bidDS);
        bidGrid.setAutoFetchData(false);
        ...
    }

That's all SmartGWT needs to fetch data from the server when needed.

Conclusion

Is it really worth all this effort? I would say yes. It is a lot less work than it would be to implement your own server logic, and in many cases you can just use the default SmartGWT data sources. If you like SQL better than JPA, you can use SmartGWT SQL datasources, wich are more flexible than the JPA, but then you cannot use this unit test framework.

This testing methodology has saved us lots of work. It is far cheaper to test the logic in unit tests than to test it manually, fix problems, and restart the application to test again. With this kind of unit tests, most of the server logic just works. The remaining work is to implement and fine-tune the user interface. And of course, the unit tests keep our code working.

It is actually not just a testing methodology but a requirements methodology also. Using examples and objects makes the communication between clients/analysts and developers crystal clear. It reveals misunderstanding and missing details so that the developers get all the information they need to test and implement new functionality.

Object cases make it easy to design tests, and the tests makes it easy to focus development and figure out what to do next.

Back to the agile values

2011-06-02T23:17:00.001+02:00

In recent years the term agile has become overused. Many seem to think that if they have have unit tests, standup meetings and burn-down charts, they're agile. All these practices are good, but they don't necessarily make you agile. Even iterations or some kind of certified master don't necessarily make you agile.

So what is agile? If I were to sum it up with one word it would be communication. Communication is everywhere in the agile manifesto:

[We value] individuals and interactions over processes and tools.
I see this as a reaction against processes like RUP that felt like a software development factory where developers were replacable cog wheels. Agile recognizes that it's individuals with intelligence, creativity and drive that make a project succeed.

But individuals are not working in isolation, they need to interact with others. Interactions means communication. Not one-way communication but interactive dialogues. Misunderstandings are inevitable in communication. When you say or write something, it is almost certain that the receiver will misunderstand something. You can't just send someone a document and think they will understand what you mean. You need to verify what they understood, and the best way to do this is in a face to face conversation.

A key part of agile is to have close communication within the team and between the team and the customers.

[We value] working software over comprehensive documentation.
Documentation is a form of communication. Some teams stop writing documents "because that's not agile", but that's a huge misunderstanding. Agile does value documentation, but it values working software more. Working software demonstrates progress better than completing a number of documents, and it demonstrates the team's understanding of the requirements better than a requirements document. But documentation may be useful to explain what you were thinking when you developed the software.

I said that iterations don't necessarily make you agile, but iterations are definitely needed to be agile. Iterations is not a goal in itself, the purpose of iterations is to improve communication with the customers by getting feedback often. It is inevitable that we misunderstand what the customers need. Iterations help us to discover these misunderstandings early, before they get too expensive to fix.

[We value] customer collaboration over contract negotiation.
Collaboration certainly means communication. The development team needs a positive dialogue with the customers, and not just communicate with formal documents.

[We value] responding to change over following a plan.
This may not seem like to be about communication, but actually it is. Where do the changes come from? From the customers. The customers and the team should communicate often, not just up front.

These values are the basis for practices like on-site customer, iterative development and pair programming. It's the values that make you agile, not various practices. The practices vary depending on the size and complexity of the project.

The problem with use cases

2011-05-16T10:03:00.000+02:00

The greatest benefit I get from use cases is that they focus on the user. Use cases help me to think about what the user wants to do instead of only focusing on implementation details.

The biggest problem I have with use cases is that they are not structured. They are basically free text. For instance, if we have a use case Withdraw money from ATM, we may define that it has a precondition that Open account is performed, but we don't get any help from the method to see that.

What happens if someone later changes the Open account use case or defines a Close account use case? How do we find which other uses cases that need to be modified? We can look through the old use case diagrams and find dependencies, but I can almost guarrantee that these dependencies have not been maintained after they were initially created.

The solution to this is to connect the use cases to an object model. I don't mean a use-case realization with view and controller objects like ATM_Screen and ATM_Session, I mean the model objects from the problem domain, such as Customer, Account and Transaction. These are the terms you use when you discuss the functionality with actual users.

If you model the Withdraw money from ATM use case as objects, it could be something like this:

You have an Account object with balance $1000.
You insert a card in an ATM and enters the pin code 1234.
You withdraw $100 from the ATM.
Afterwards, the Account balance is changed to $900 and a new Transaction is created.

To find dependencies between this use case and other use cases, we ask where the Account object and balance came from. This is something to discuss with the client or end users. They may answer that you can open an account and deposit cash.

The Open account use case can be modeled as follows:

The customer gives name and address and shows id to a teller.
Create an Account object.

The Deposit cash use case can be modeled as follows:

You have an Account with balance $0.
The customer gives the account number 12345678 and $1000 in cash to a teller.
The Account balance is updated to $1000.

There you have the dependencies between Open account, Deposit cash and Withdraw money from ATM: They all use Account objects.

Analyze objects to find more functionality

We can continue with our analysis by asking questions such as:

Can the balance be changed in any other way than Deposit cash and Withdraw money from ATM?
Where did the Account number come from? Can it be changed?
How is the pin code set?
Can Accounts be deleted?
In which ways can you read Account numbers, balances and pin codes?
How can Transactions be created, updated, read and deleted?

The questions above deal with creation, deletion, update and reading of domain objects. They will help to discover more functionality, which can be analyzed further, and so on, until you have a very complete functional specification.

There are a few more questions that can be helpful to discover more functionality. They are based on 4 kinds of classes: People, places, things and events.

Ask which people will perform the various functionality. This will help us to model users, roles and privileges.
Finally, ask which information to store about user actions (events). Transaction is such a class that stores information about changes of Account balance.

This method is a fast way to come up with lots of requirements which would otherwise take days or weeks of brainstorming. It is similar to use cases in that focuses on what the users can do. But it is different in that it models this as objects. Therefore, I am calling this method object cases.

What exactly is an object case?

An object case is a use case scenario connected to domain objects. It consists of 4 parts:

Initial objects
Input
Output
Result objects

The input, output and objects are specified with concrete values. In my experience, this improves communication with clients and end users, because it reduces misunderstandings and help to find missing details.

Object cases are also a great help for developers; they have all the information they need in the object case and don't need to ask for clarifications so much.

Testers can also benefit from object cases. Instead of writing test specificiations based on use cases or some other abstract requirements, the object cases are ready to test.

Objects can also be used for project planning: Develop the functionality around a domain class together. Start with the function that creates objects of that class.

Getting the requirements right with object cases

2011-05-10T16:52:00.001+02:00

One of the biggest problems in software projects is poor requirements. We always seem to misunderstand what the user is really trying to do or discover some missing functionality at the end of the project. Agile methods reduce this problem with short feedback cycles to discover misunderstandings early, but it would be even better to get rid of the misunderstandings before implementation.

Two things that really help me in this area are examples and objects. This article describes how to combine these with a method I am calling object cases.

An object case is an example of a user function connected to domain objects. It consists of 4 parts:

Initial objects
Input
Output
Result objects

A typical week using object cases can be as follows:

Monday morning: Go through new functionality with the client.
Monday afternoon: Planning and estimation.
Tuesday: Implement automatic tests.
Wednesday - Friday: Implement the actual functionality.

My experience is that object cases help us a lot in all this activities. When discussing requirements, the objects generate questions such as "How do I find this object?" or "When is this object created?" These kinds of questions helps us to discover missing functionality early and saves us lots of rework.

A consequence of this is that the plans are really good. It doesn't happen often that we forget tasks in the plan.

Another experience is that after Monday, I hardly need to discuss anything with the client. The object cases give me all the details I need to implement automatic tests and production code without guessing.

Examples

I always use some examples to make things concrete. This helps to clear out the inevitable misunderstandings and reveal details we did not think of.

By example, I mean a concrete example of something a user can do. For instance, here is an example of how to use a system for enrollment in university seminars:

Jane Enrolls in Seminar

Jane enters her name: “Jane Jackson” and student number: "111-222".
System displays the available seminars: ENG 103, COM 402.
Jane chooses seminar ENG 103.
System calculates and displays fees: $800
Jane indicates she wants to enroll.
System enrolls Jane in the seminar.
System sends a bill to Jane Jackson with the following content: ENG 103, fall 2011, $800
System prints an enrollment receipt: Jane Jackson, ENG 103, fall 2011.

Note that the example has a concrete user and concrete input and output values everywhere.

I have never studied in America, so I am sure there are some mistakes and missing details. But that is one of the main points of using concrete examples: It is easy to spot mistakes.

The alternative is an abstract requirement, such as a use case:

Enroll in Seminar
Basic Course of Action:

Student inputs her name and student number.
System verifies the student is eligible to enroll in seminars. If not eligible then the student is informed and use case ends.
System displays list of available seminars.
Student chooses a seminar or decides not to enroll at all.
System validates the student is eligible to enroll in the chosen seminar. If not eligible, the student is asked to choose another.
System validates the seminar fits into the student’s schedule.
System calculates and displays fees.
Student verifies the cost and either indicates she wants to enroll or not.
System enrolls the student in the seminar and bills them for it.
The system prints enrollment receipt.

(Source: Agile Requirement Modeling by Scott W. Ambler.)

The use case above is actually very good, but the example has more information:

A student number is in the format 111-222.
The University has seminars that are called for instance ENG 103 and COM 402.
A typical fee for a seminar can be $800.
The bill includes information about the student name, seminars, semester and total price.

When I discuss the example with the stakeholders, they are likely to find errors in my assumptions and point them out. This gives me better quality of the requirements before I start development.

More importantly, examples enable further discussions with the customer, such as:

Which information should be displayed in the list of seminars?
Which seminars shall be listed?
How is the fee of $800 calculated?

The answers to these questions will likely lead to requirements for more functionality that the system needs to support. For instance, functionality is needed to edit course fees.

Objects

The next step is to model these examples as objects.

In the example above, we need the following initial objects:

In the resulting objects, a Bill object should be created and a Seminar object updated:

The objects give structure to the examples. This makes it easy to find missing pieces in the requirements. They inspire more questions, such as when is the Student object created, what happens when the Bill is paid or not paid?

Just using examples and objects informally will greatly improve the communication between stakeholders and programmers. The stakeholders understand the examples and the programmers get all the information they need. Another benefit is that the examples can be used as functional tests.

Completeness

The examples can be used to find objects, which can be used to find more examples, and so on. In this way, we can build a complete specification.

To ensure that the model is complete, we ask these questions:

Can all output be calculated from the objects?
Is all input used to find or modify objects?

In the example above, this becomes:

Jane Enrolls in Seminar

Input name: “Jane Jackson” and student number: "111-222". (Used to find a Student object.)
Output available seminars: ENG 103, COM 402. (Calculated from the Seminar objects.)
Input seminar ENG 103. (Used to find a Seminar object.)
Output fees: $800. (Calculated from the Bill object.)
-
-
Output bill: ENG 103, fall 2011, $800.
(How is 'fall 2011' calculated? We need to add this to the object model, perhaps in a new Term object.)
Output enrollment receipt: Jane Jackson, ENG 103, fall 2011. (Same as above.)

This results in the following class model:

We can use the class model to find more examples. We need examples that create, delete, update and read all the classes.

Our example creates a Bill object. But when are Student, Seminar and Term objects created? When are these objects updated and can they be deleted?

If we ask the client, they will tell us more functionality that the system needs to have, for instance:

Register student
Plan seminar
(and some more examples to update and delete objects)

Finally, we can ask the client for examples of how this information is used (read). Our example reads information from Student and Seminar objects. But when are the Bill objects read? Again, this needs to be discussed with the client. Instead of just creating a simplistic user interface with CRUD operations that displays tables of seminars, students etc, we should find out what the users really want to do. This will help us to design a good user interface.

In this case, the client might want the following functionality:

Print a list of all unpaid bills.
Print a list of all students in a seminar.
Send email to all students in a seminar.
Calculate the total seminar fees for a term.

When we make concrete examples with input and output for all this, we will probably need even more objects, and so the requirements grow. Connecting objects and examples is a great way to discover missing functionalilty.

How does this help?

Examples improve the communication. They give clients, analysts, developers and testers a common understanding of what the system needs to do. They are sufficiently detailed so people don't have to guess.

Objects give structure to the examples. This helps us to make the examples consistent and complete.

Objects and examples is a great starting point for developers. Firstly, they have very detailed requirements. They don't need to guess or ask the client all the time. Secondly, the developers can use the object model as a basis for implementation. And thirdly, they can write automatic tests from the examples.

The examples are testable, so the testers can basically test the examples, but they will need to write more examples than the "sunny day" scenarios that clients and developers come up with. It is a good idea to involve the testers in defining the examples to discuss these details early.

Character encoding with MercurialEclipse

2011-02-18T13:41:00.001+01:00

I'm posting this here since it took me some time to find it.

In Eclipse with Mercurial and MercurialEclipse, I have some files with UTF-8 encoding. When I compared these files, I would see differences at every line with an international character.

Solution: Change the project encoding to UTF-8 (Project Properties > Resource) and restart Eclipse.

Reading FireFox bookmarks with JPA

2011-02-09T16:56:00.023+01:00

One of the primary goals of DB Importer is simplicity. To make this concrete, the goal is to be able to start using an existing database with JPA in 5 minutes.

Now it is time to test this. I choose the FireFox SQLite bookmarks database as an example. Not because it is easy, but because it is hard ;-)

Preconditions: Eclipse and DB Importer are installed and a Java project is created.

1. Reverse Engineer Database
Right-click on a Java package and select DB Importer > Import from Database.
Select the org.sqlite.JDBC driver in the wizard:

2. Locate the Database
I need to replace <sqlite-file> in the URL with the actual database file. But where does FireFox store its bookmarks?

To find out this, I turn to my favourite programming tool: Google. Google "firefox bookmarks database" and click on the top link gives me this site, which explains that you should look for a file called "places.sqlite" under C:\Users\<user name>\AppData\Roaming\Mozilla\Firefox\Profiles\<some random characters>.default.

To find this file on my Mac, I open a terminal and enter:

ls $HOME/Library/Application\ Support/Firefox/Profiles/*/places.sqlite

This gives me the file, and i enter the full path in the wizard:

3. Generate Entity Classes
When I push Next, a new editor is opened:

This shows the tables and columns in the database and a preview of the Java code that will be generated.

Two of the tables are marked with errors. Clicking on them reveals that this is because they lack a primary key, so they cannot be used with JPA.

I Save the editor. This generates the Java files, but they have compilation errors because I have not yet added the JPA libraries.

4. Create a Test Application
To use JPA, I need to create a configuration file (persistence.xml) and add the JPA libraries and the JDBC driver to the project. This can be done with Maven, or you can use a wizard in DB Importer to quickly create a test application: Right-click on the Java package and select DB Importer > New Test Application. This opens another wizard:

I enter a class name and push Finish. This adds all the necessary JAR files to the project, creates SQLiteDialect.java, persistence.xml and PrintBookmarks.java and opens the two last files.

The generated PrintBookmarks class is just a template:

public class PrintBookmarks {
public static void main(String[] args) {
 javax.persistence.EntityManagerFactory factory =
    javax.persistence.Persistence.createEntityManagerFactory("unit1");
 javax.persistence.EntityManager em = factory.createEntityManager();
 javax.persistence.EntityTransaction tx = em.getTransaction();
 tx.begin();
 try {
    // TODO Replace User with a real @Entity class.
    java.util.List result = em.createQuery("from User").
      getResultList();
    System.out.println("Found " + result.size() + " objects:");
    for(User obj : result) {
       // TODO Change to some real method.
       System.out.println(obj.getName());
    }
    tx.commit();
 } catch(RuntimeException e) {
    tx.rollback();
    throw e;
 } finally {
    em.close();
    factory.close();
 }
}
}

But just by replacing "User" with "MozBookmarks" and "obj.getName()" with "obj.getTitle()" we have a working application.

Let's try: Run As > Java Application.

Success! All my bookmarks are printed. And it went really quickly, so I think I am satisfied with the simplicity.

Bonus: Modify the Generated Code
Simplicity is one thing, but how about flexibility? DB Importer gives you all the flexibility you need with a Groovy script that configures the generated code. Don't worry if you don't know Groovy, the script uses Java syntax.

(Note that configuring this script is only available in the commercial edition of DB Importer. You get simplicity for free but flexibility is commercial.)

First, I want to change the generated class names. For instance, I want the table MOZ_BOOKMARKS to be reverse engineered to a Java class Bookmark.

I select a table in the tree and double-click on the className in the table to the right. This opens an editor with ImportDB.groovy. This script allows you to modify anything you want in the generated Java code. I modify the className method and Save the editor. This changes the className in the preview from MozBookmarks to Bookmark.

I would also like to generate relations for foreign keys. The referenced table is missing in SQLite, so I need to calculate the referenced table based on the column name. I edit the getReferencedTable method as follows and Save the import script.

I can then modify the test app to use the Bookmarks - Place relation. I also modify it to print the bookmark hierarchy:

/**
 * List all FireFox bookmarks.
 * This depends on correct configuration in
 * persistence.xml. Also, FireFox must not be running,
 * since it locks the database.
 */
public class PrintBookmarks {
    private EntityManagerFactory factory;
    private EntityManager em;
    private EntityTransaction tx;

    public static void main(String[] args) {
        new PrintBookmarks().run();
    }

    public void run() {
        factory = Persistence.createEntityManagerFactory("unit1");
        em = factory.createEntityManager();
        tx = em.getTransaction();
        tx.begin();
        try {
            printBookmarks(getTopBookmarks(), "");
            tx.commit();
        } catch (RuntimeException e) {
            tx.rollback();
            throw e;
        } finally {
            em.close();
            factory.close();
        }
    }

    private void printBookmarks(Collection list, String mrg) {
        for (Bookmark bm : list) {
            System.out.print(mrg + bm.getTitle());
            Place place = bm.getFk();
            if (place != null) {
                System.out.print(" - " + place.getTitle() + ": "
                        + place.getUrl());
            }
            System.out.println();
            printBookmarks(getChildBookmarks(bm), mrg + "   ");
        }
    }

    private List getTopBookmarks() {
        return getBookmarksByParent("0");
    }

    private List getChildBookmarks(Bookmark parent) {
        return getBookmarksByParent(parent.getId());
    }

    @SuppressWarnings("unchecked")
    private List getBookmarksByParent(String parent) {
        Query query = em.createQuery(
            "from Bookmark where parent=:parent");
        query.setParameter("parent", parent);
        return query.getResultList();
    }
}

The complete source with the reverse engineered classes and this test app is available at bitbucket.

Just one more feature...

2011-01-30T22:05:00.007+01:00

DB Importer is ready to launch, but I just want to add one more feature, so I have decided to extend the beta period 2 weeks. I know I should realease early and all that, but key feature of this product is simplicity, and I think I can really make it simpler than it is now.

My goal is that someone with little or no experience with JPA can create a small Java application that connects to an existing database using JPA in 5 minutes. To reach this goal, I want to bundle a JPA implementation with the product so you can get started without downloading and integrating those JAR files.

If you don't care about simplicity, there are other free tools out there. DB Importer is for those who would rather pay a little than using their precious time figuring out how to use a new tool.

DB Importer

2010-12-03T13:52:00.006+01:00

I often need to use an existing database from Java. I like JPA, so I usually want to generate JPA @Entity classes for the existing tables. There are some tools out there that does this, but they are not exactly easy to use. I tried several, and I could not get them to work in a reasonable time. (I have more important things to do than to fight a complicated tool.)

I also want to configure the import, for instance to generate @Version and @GeneratedValue annotations, to tweak the generated class names, to generate @ManyToMany relations from connection tables, etc.

So, I decided to write a tool myself. (Why spend a day getting a tool to work when you can make your own in a few weeks?)

I have already made one tool like this: HiberObjects Import DB Pro, but I realize that most programmers don't want to use UML. So, I made a similar tool but without UML.

The code generation can be configured with a Groovy script. To keep this simple, I use plain Java syntax in the script.

The tool is an Eclipse plug-in called DB Importer. It will be available in a free edition and a commercial edition. DB Importer is currently in Beta. Please try it out and let me know what you think.

Make it or break it

2010-07-20T14:27:00.010+02:00

Some times I am working on a project alone doing both design, planning, programming and testing. I have found it useful to separate these different kinds of work as much as possible. That's because programmers are not good testers. Programming and testing requires totally opposite mindsets:

Success as a programmer is to get something to work (make it)
Success as a tester is to find something that doesn't work (break it)

If I start testing while I'm in programming mode, I will not try very hard to break it, so lots of errors will slip through.

But on the other hand, I can easily get into programming from testing. When I am testing, I am working with an example, and the example helps me to focus my programming. But I try to avoid jumping from programming to testing anyway, because it is hard to get back to effective testing again.

1. Design
I get the best results if I start with designing a few examples of how the functionality will be used. The examples help me to focus and remember all details. I will also use them as my test specification. Then I create automatic tests based on these examples.

2. Implementation
The automatic tests will remind me what I need to do, so it's no problem to take lunch or end the work day now. But often I am so focused that I just continue with implementation until all the automatic tests pass. After this I take lunch or a long break before I start manual testing.

3. Manual testing
I don't write a test specification for myself, I just run through the examples I designed in the beginning. If I find any errors, I don't fix them right away, but write them down in a test log. When I have tested everything, I can get back to programming again. Since I have some automatic tests in place, it's usually easy to add some more tests that captures the errors.

Selecting columns or rows but not cells in JTable

2010-04-21T13:53:00.007+02:00

Here is a snippet to enable selection of either columns or rows, but not cells, in a JTable:

   public Integer findColumn(JTable table, Point p)
   {
      JTableHeader tableHeader = table.getTableHeader();
      p.translate(-tableHeader.getX(), -tableHeader.getY());
      if(p.x >= 0 && p.x < tableHeader.getSize().width &&
            p.y >= 0 && p.y < tableHeader.getSize().height)
      {
         int x = p.x;
         for(int i = 0; i < table.getColumnCount(); i++)
         {
            String columnName = table.getColumnName(i);
            TableColumn column = table.getColumn(columnName);
            x -= column.getWidth();
            if(x < 0)
            {
               return column.getModelIndex();
            }
         }
      }
      
      return null;
   }
   
   public void enableRowSelection(final JTable table)
   {
      table.getTableHeader().addMouseListener(new MouseAdapter()
      {
         @Override
         public void mouseClicked(MouseEvent e)
         {
            table.getSelectionModel().clearSelection();
            table.setColumnSelectionAllowed(true);
            table.setRowSelectionAllowed(false);
            int column = findColumn(table, e.getPoint());
            table.getColumnModel().getSelectionModel().
               setSelectionInterval(column, column);
         }
      });
      table.addMouseListener(new MouseAdapter()
      {
         @Override
         public void mousePressed(MouseEvent e)
         {
            table.setColumnSelectionAllowed(false);
            table.setRowSelectionAllowed(true);
            table.getColumnModel().getSelectionModel().
               clearSelection();
         }
      });
   }

Some adjustment may be needed to enable selection of single/multiple columns or rows.

Refactoring can be easier in Groovy than in Java

2008-12-24T13:59:00.005+01:00

I love Groovy for the productivity, but I am sceptical to maintenance and refactoring of large code bases in Groovy. Without type safety, you cannot detect refactoring errors at compile time, so you need very high coverage with unit tests.

But the other day, I experienced something that got me thinking: I have a few Java classes and some Groovy scripts that call the Java code something like this:
def var = javaObject.func()

I changed the return type of the Java method, and the Groovy code just kept on working. I didn't have to change it at all. Why? Because the Groovy code did not know the type I was returning. The dynamic nature of Groovy made refactoring easier, not harder!

How can type safety make refactoring harder? Because type safety leads to duplication of information. In Java, the code above would look like this:
String var = javaObject.func()

The return type "String" would be duplicated in calls to this method, in interfaces that specified this method, in subclasses, in variables, and so on. Not so in Groovy. The return type is declared only in one place.

I always wondered why we designed programming languages different than databases. In databases, we try to avoid duplication of information. Duplication is a maintenance nightmare. But in programming, we duplicate information all the time. Not only with return types, but with boilerplate code, design patterns and class, variable and method names. In databases, we define immutable primary keys instead of refering to rows by values that could be modified. But in programming languages, we refer to everything by name. So when we want to change a name, we need a complicated "refactoring" tool to replace this name everywhere it occurs.

I dream of a programming tool that let my type code like: "x = myObject.func1()" but under the hood it would create references to "myObject" and "func1" instead of resolving that at compile time. Then, there would be no "rename refactoring", just change the name and the reference would still be valid. The text should then be updated automatically.

How to test Java Persistence API in 5 minutes

2008-12-04T22:25:00.014+01:00

I am the author of HiberObjects and want to give an example of how useful it can be.

Yesterday, someone asked me a Java Persistence API (JPA) question: If an object is removed from a bidirectional relation, do both objects have to be saved to update the database correctly?

I thought I knew the answer, but better test it first. This is how I did it:

1) Design a class diagram with 2 persistent classes User and Membership and 1 unit test UserPersistentTest:

2) Initialize the database with some objects that I can test against: Right-click on the UserPersistentTest class and select Design Objects, then design objects like this:

3) Save the diagram. This will generate JPA classes, a unit test and a helper class. The unit test will initialize the database with objects like this:

   private void initObjects()
   {
      EntityManager entityManager = persistenceHelper.getEntityManager();
      EntityTransaction tx = entityManager.getTransaction();
      tx.begin();
      user = new User();
      membership = new Membership();
      user.setName("Lars");
      membership.setX(100);
      entityManager.persist(user);
      entityManager.persist(membership);
      user.getMembership().add(membership);
      membership.getUser().add(user);
      tx.commit();
      entityManager.close();
   }

4) Implement the test method:

   public void test1() throws Exception
   {
      EntityManager em = persistenceHelper.getEntityManager();
      em.getTransaction().begin();
      User user1 = (User) em.createQuery("from User").getSingleResult();
      assertEquals(1, user1.getMembership().size());
      
      Membership membership1 = (Membership) em.createQuery("from Membership").getSingleResult();
      assertEquals(1, membership1.getUser().size());
      
      assertSame(membership1, user1.getMembership().iterator().next());
      assertSame(user1, membership1.getUser().iterator().next());
      
      user1.removeMembership(membership1);
      assertEquals(0, user1.getMembership().size());
      assertEquals(0, membership1.getUser().size());
      em.getTransaction().commit();
      
      em.getTransaction().begin();
      User user2 = (User) em.createQuery("from User").getSingleResult();
      assertEquals(0, user2.getMembership().size());
      Membership membership2 = (Membership) em.createQuery("from Membership").getSingleResult();
      assertEquals(0, membership2.getUser().size());
      em.getTransaction().commit();
   }

5) Run as JUnit test:

This demonstrates the answer to the question: You don't need to save any of the objects explicitly if you change them in a transaction.

My favorite Eclipse view

2008-07-19T22:27:00.007+02:00

Have you discovered the Display view in the Eclipse debugger? It is my personal favorite. In it, you can execute any Java code in the context of a running debugger.

To open this view, select Window > Show View > Display.

When the debugger hits a break point, you can type some Java code in the Display view. The Java code can use the local variables in the currently selected frame in the Debug view.

To execute the code and display the returned value, push the button with a "J":

If you just want to execute some code that doesn't return a value, push the button with an arrow ">" and a "J":

The standard output will be printed to the Console view:

This is incredibly useful! You can for instance run some code in a debugger and run some complex debugging code, or prototype some code for a new method.

World War 2, Rationalism and Agile Development

2008-07-13T21:00:00.010+02:00

One of the central values of the Agile Manifesto is Responding to change over following a plan. This doesn't mean that it's not agile to make a plan. You should have a plan, but you should realize the limitations of the plan.

Are there any parallels to this in history?

The Soviet Union had 5-year plans. Stalin was quite succesful in industrializing the rural country in a short span of time. But his methods did not work well in the battlefield. In the 2nd world war, the Soviet suffered severe losses in the beginning.

You can plan how to build and manage a factory, because the complexity is limited. But a war is far more complex. You cannot predict what the enemy is going to do. This doesn't mean that you should go to war without a plan. Far from it, you should plan as much as possible, but be prepared to improvize when something unexpected happens. As Dwight D. Eisenhower, a successful American general in World War 2, said: "Plans are nothing; planning is everything."

Building software is also complex. But some managers try to run a software development organization as a factory. They treat their employees like cog-wheels. They think that Employee #1093 can take over the responsibilities of Employee #835 with just some training. Or that 4 programmers can do in 3 months what one programmer can do in 1 year. It doesn't work like that. If software development was so predictable, then robots could do it. But they can't.

I don't like being treated like a cog-wheel. I am not Employee #1093, I am me. I am unique with my own weaknesses and strengths. If I am treated like a replaceable part, I may start to behave like one. Then, I will only do as I'm told, nothing more, nothing less. If nobody listens to my ideas, I will stop thinking creatively, and just do things the way we always did.

Since I don't want to become mediocre, I avoid environments like that. I seek challenging environments where my ideas are heard and I am given responsibility to work as I think is best.

I think that Churchill, the Prime Minister of Great Britain during WW2, was agile too. He was not afraid to change. He said: "To improve is to change; to be perfect is to change often.

He also said, "However beautiful the strategy, you should occasionally look at the results." That sounds like iterative development to me. You make a plan, and then review the results now and then to see how it works. (See more fantastic quotes of Churchill here.)

So how about Hitler? How agile was he? Not agile at all. He had a great vision and a plan that seemed to work very well in the beginning. But as we all know, he failed. I think this is similar to a waterfall project: He was 80% finished with conquering Western Europe, and 75% finished with Eastern Europe. If he had run the war more iteratively, he should have completed Western Europe first, and then started on Eastern Europe. Thank God, he didn't!

What did Hitler and Stalin have in common? Both nazis and communists were rationalists with a strong belief in their own enlightened minds. I dare say that Churchill and Eisenhower had a more humble attitude, accepting that they could not make perfect plans. I would also say that Churchill was empiric in that he wanted to review the results of his plan.

Rationalists think they can plan everything. Empirists believes in experimental evidence. I think that waterfall development is a result of rationalist thinking, but it doesn't work well, because our minds are limited. We cannot predict everything, but we are capable of planning to a large degree. Therefore, I believe iterative development where you make a plan but also review the results regularly is the most effective way.

PS: I am not a historian nor a philosopher, so please, enlighten me if I got something wrong.

Use examples to make your code easier to understand

2008-05-17T23:24:00.004+02:00

Programmers are used to abstract thinking. To program is to generalize: A method is a general specification of what to do during execution. A class is a general specification of objects. A superclass is a generalization of several classes.

Altough our minds are capable of abstract thinking, concrete thinking is much easier, and concrete examples are the foundation for abstractions. For instance, when we were children, our parents didn't try to teach us about cars by explaining to us cars are and what they can do. Instead, they just pointed at a car that was driving by and said ”Look, a car!” When they had done that a number of times, we knew what a car was.

Another example is prejudice. We all have prejudices, because this is the way our minds work. If we have met a few people from Denmark in our lives, and those people were friendly, we ”know” that Danes are friendly. And this works even stronger for negative prejudices.

My point is that we learn by examples. Einstein said that examples is not another way to teach, it is the only way.

As programmers, we are highly skilled in abstract thinking and expressing ourselves in programming languages. But when the code becomes too complex, we try to comprehend it by making examples in our minds of what the code would do in different circumstances. We imagine or write down the values of different variables and what the behavior of different tests and loops will be with those values.

And when it becomes too hard to imagine what is actually going on by reading the abstract code, we use debuggers for help. A debugger is not abstract. It gives a concrete description of what is actually happening at runtime. It gives information on how methods execute in specific instances and which objects exist at a specific moment. This concrete information is so much easier to understand than the abstract code.

Therefore, it's a good idea to design concrete examples of objects and scenarios before implementing the abstract classes.

Unit tests are exactly such examples. They make it easier to understand what happens at runtime. And a well designed unit test is the best way to explain to somebody what the code does. If you ever had to take over somebody else's code, you know how hard it can be to understand it. Where do you start? Unit tests is a good place to start.

So, my suggestion is to think of unit tests as documentation. Write unit tests that are easy to read and that explain the intent of the code.

I believe in Java for future web applications

2008-03-08T15:16:00.002+01:00

Will Java become the dominant platform for web applications? Actually, I think it has a good chance.

How will the future web applications be? I think we can expect the following:
* Highly interactive
* Complex, more work will be done on the client
* Collaborative, not only for games, but also for other applications
* Mobile
* Location aware

AJAX is fantastic compared to "web 1.0", and technology like GWT (Java) makes it relatively easy to develop complex user interfaces. But HTML and HTTP still sucks as an application platform, something they were never meant to be. I believe that if Microsoft hadn't managed to kill the Applet, Java would be the dominant platform for more complex web applications today.

I think Consumer JRE can be that platform. Java can be used to make complex, interactive applications that communicate with each other client-to-client.

This doesn't mean we have to implement in Java. When consumers have a JRE ready, there is nothing stopping us from developing applications in Groovy, JRuby and Scala and deliver them over the web.

I don't know much about Flash, but I am sure it is at least as good as Java for making flashy applications. But when it comes to complex collaborative and mobile applications, I think Java has the upper hand.

And Java has one more powerful force behind it: The community. The Java community has developed hundreds of libraries and frameworks. Too many, perhaps. But this community is capable of revolutionizing the way web applications are developed in ways we cannot predict now.

Passwords in phpBB 3

2008-02-26T01:18:00.002+01:00

Here is a port of phpBB3's password handling to Java.

import java.io.UnsupportedEncodingException;
import java.security.GeneralSecurityException;
import java.security.MessageDigest;

/**
 * Port of phpBB3 password handling to Java. 
 * See phpBB3/includes/functions.php
 * 
 * @author lars
 */
public class PHPBB3Password {
  private static final int PHP_VERSION = 4;
  private String itoa64 = 
"./0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz";

  public String phpbb_hash(String password) {
    String random_state = unique_id();
    String random = "";
    int count = 6;

    if (random.length() < count) {
      random = "";

      for (int i = 0; i < count; i += 16) {
        random_state = md5(unique_id() + random_state);
        random += pack(md5(random_state));
      }
      random = random.substring(0, count);
    }

    String hash = _hash_crypt_private(
      password, _hash_gensalt_private(random, itoa64));
    if (hash.length() == 34)
      return hash;

    return md5(password);
  }

  private String unique_id() {
    return unique_id("c");
  }

  // global $config;
  // private boolean dss_seeded = false;

  private String unique_id(String extra) {
    // TODO Generate something random here.
    return "1234567890abcdef";
  }

  private String _hash_gensalt_private(String input, String itoa64) {
    return _hash_gensalt_private(input, itoa64, 6);
  }

  private String _hash_gensalt_private(
    String input, String itoa64, int iteration_count_log2) {
    if (iteration_count_log2 < 4 || iteration_count_log2 > 31) {
      iteration_count_log2 = 8;
    }

    String output = "$H$";
    output += itoa64.charAt(
      Math.min(iteration_count_log2 +
      ((PHP_VERSION >= 5) ? 5 : 3), 30));
    output += _hash_encode64(input, 6);

    return output;
  }

  /**
   * Encode hash
   */
  private String _hash_encode64(String input, int count) {
    String output = "";
    int i = 0;

    do {
      int value = input.charAt(i++);
      output += itoa64.charAt(value & 0x3f);
 
      if (i < count)
        value |= input.charAt(i) << 8;
 
      output += itoa64.charAt((value >> 6) & 0x3f);
 
      if (i++ >= count)
        break;
 
      if (i < count)
        value |= input.charAt(i) << 16;
 
      output += itoa64.charAt((value >> 12) & 0x3f);
 
      if (i++ >= count)
        break;
 
      output += itoa64.charAt((value >> 18) & 0x3f);
    } while (i < count);

    return output;
  }

  String _hash_crypt_private(String password, String setting) {
      String output = "*";

      // Check for correct hash
      if (!setting.substring(0, 3).equals("$H$"))
        return output;

      int count_log2 = itoa64.indexOf(setting.charAt(3));
      if (count_log2 < 7 || count_log2 > 30)
        return output;

      int count = 1 << count_log2;
      String salt = setting.substring(4, 12);
      if (salt.length() != 8)
        return output;

      String m1 = md5(salt + password);
      String hash = pack(m1);
      do {
        hash = pack(md5(hash + password));
      } while (--count > 0);

      output = setting.substring(0, 12);
      output += _hash_encode64(hash, 16);

      return output;
  }

  public boolean phpbb_check_hash(
    String password, String hash) {
      if (hash.length() == 34)
        return _hash_crypt_private(password, hash).equals(hash);
      else
        return md5(password).equals(hash);
      }

  public static String md5(String data) {
    try {
      byte[] bytes = data.getBytes("ISO-8859-1");
      MessageDigest md5er = MessageDigest.getInstance("MD5");
      byte[] hash = md5er.digest(bytes);
      return bytes2hex(hash);
    } catch (GeneralSecurityException e) {
      throw new RuntimeException(e);
    } catch (UnsupportedEncodingException e) {
      throw new RuntimeException(e);
    }
  }
  
  static int hexToInt(char ch) {
    if(ch >= '0' && ch <= '9')
      return ch - '0';
  
    ch = Character.toUpperCase(ch);
    if(ch >= 'A' && ch <= 'F')
      return ch - 'A' + 0xA;
  
    throw new IllegalArgumentException("Not a hex character: " + ch);
  }
 
  private static String bytes2hex(byte[] bytes) {
    StringBuffer r = new StringBuffer(32);
    for (int i = 0; i < bytes.length; i++) {
      String x = Integer.toHexString(bytes[i] & 0xff);
      if (x.length() < 2)
        r.append("0");
      r.append(x);
    }
    return r.toString();
  }

  static String pack(String hex) {
    StringBuffer buf = new StringBuffer();
    for(int i = 0; i < hex.length(); i += 2) {
      char c1 = hex.charAt(i);
      char c2 = hex.charAt(i+1);
      char packed = (char) (hexToInt(c1) * 16 + hexToInt(c2));
      buf.append(packed);
    }
    return buf.toString();
  }
}

Agile practices don't primarily solve problems

2008-02-22T12:08:00.005+01:00

But they sure help detecting problems early. And that's extremely valuable.

For instance, iterative development with deliveries early in the project will test the technical feasibility and the development speed. But perhaps most important, we will get feedback on the functionality from the clients, to verify if we have understood their problem correctly.

Failing in any of these areas will lead to project failure. And it is better to discover this when 20% of the budget is used than when 90% of the budget is used. First, you reduce your losses. And even better, you have a fair chance of fixing the project and turning it into success.

Many agile projects succeed not because they are more productive, but because they discover problems in early iteations and then reduce the requirements.

Unit testing is another practice that makes you aware of problems early. This makes it cheaper to fix problems, but it also gives confidence, so you dear restructure code when necessary.

Another agile practice is close communication with business experts, preferably face to face. The non-agile alternative is written requirements. The problem is that developers misunderstand what the business experts want. That's inevitable. What we need, are mechanisms to discover misunderstandings. That mechanism is called feedback. The business expert explain what they want, and the developer explain how they understood that to the business expert, which confirms that this was correct. That's easier to do face to face than with written documents.

Hungarian notation and Groovy

2008-02-11T21:57:00.001+01:00

Comments and types are metadata and strictly not necessary to make a program, according to a popular post. Yes, they are not necessary to communicate with the computer, but that's not what we are trying to do here. For all but the smallest projects, we are communicating with people. The code should be understandable to others. And it doesn't hurt if it's understandable to ourselves either.

Hungarian notation was a popular in C to show the type of a variable in its name, for instance:

int nSize;
char cChar;

I never liked this, because if I change the type of a variable I would also have to change its name everywhere. But with dynamically typed languages, I some times get in trouble because I forget what type a variable is. So, I started to think if maybe Hungarian notation could be useful.

For example, consider the following dynamically typed Groovy code:

(1) def x = "100"

the following statically typed Java code:

(2) String x = "100";

and this statically typed Java code:

(3) int x = "100";

Which is most easy to understand?
(3) is easily recognized by both humans and the compiler as an error.
(1) and (2) are valid code. But 20 lines down the code, you have forgotten what type x is and write:

x++

Then (2) will give you a compiler error, but (1) won't be detected until runtime.

Does this mean I'm against dynamically typing? No. But if we remove the static type checking, I think we should use other means to communicate that information. For instance, it is a bad idea to call a String x. This would be much better:

def s = "100"

In Groovy, I often use maps instead of creating a new class:

def map = [:]
map.x = 100
map.y = 200

How am I supposed to remember what the map contains down the road?

I don't want to remember, I want the map to remind me! I have enough things to think about, I don't want to remember the type of every variable too!

I have a lame technique I use to not forget important things: I try to make them remind me instead of having to remember them. For instance, if I take off the lid to fill gas in the car, I put the car keys by the lid. That way, it's impossible for me to forget it. I cannot drive off without seeing the lid and remember to put it on. I use similar techniques at work. This reduces stress and helps me to concentrate, because I don't need to remember so much.

Applying this technique to Groovy gives this code:

def point = [:]
point.x = 100
point.y = 200

So, I am not advocating Hungarian notation, but good naming conventions that communicate what we need to know.

Programming is more about communication than how fast you can crank out code.

The Pessimistic Programmer

2008-02-06T17:29:00.000+01:00

I decided to change the title of this blog to "The Pessimistic Programmer". Why? Am I a depressed person that thinks nothing will work? No, I am an optimist in life. Something good is going to happen today :-) But in programming, something will surely go wrong.

I don't actually view this as pessimism, but as realism. I want to be prepared for the worst that can possibly happen. Hope for the best, prepare for the worst. But my wife explained to me that pessimists always say that they are just being realistic. So, I might as well face it: I am a pessimist.

I think a good programmer needs to be pessimistic; always thinking about what can go wrong and how to prevent it. I don't say that I am a good programmer myself. No, I make far too many mistakes for that. But I have learnt how to manage my mistakes with testing and double checking.

Über-programmers can manage well without being pessimistic. They have total overview of the code and all consequences of changes. But I'm talking about us mere mortals. But if you are an über-programmer, you should be pessimistic about what the next guy will do to your code. Place some comments and unit tests in there, to keep him out of trouble!

An optimistic programmer doesn't see the need for unit tests. He will run the program and be satisfied when it runs one time without errors. He will say things like "What can possibly go wrong?" Or if something goes wrong anyway, "It must be an error in the input from that other module".

Haven't we all been there? Suddenly, I feel so old...

Passwords in SMF 1.1.4

2008-02-06T08:22:00.000+01:00

I'm posting this in case someone else are struggling with SMF password encrypting like I did.

SMF 1.1.4 uses SHA-1 with a salt. You would think that the passwordSalt in the database is used as the salt, but it isn't. That's probably a field that was used in old versions. Instead, the membername is used as the salt, but *before* the password, not *after*, as most search results indicated.

I finally found this link to a php file that is *not* in the distribution.

function smf_registerMember(
[...]
        $register_vars = array(
                'memberName' => "'$username'",
                'realName' => "'$username'",
                'passwd' => '\'' . sha1(strtolower($username) . $password) . '\'',
                'passwordSalt' => '\'' . substr(md5(rand()), 0, 4) . '\'',

So I tried this Java code:

sha1(username.toLowerCase() + password);

That worked!

Here is the source for the sha1 method:

   public static String sha1(String data)
   {
      byte[] bytes = data.getBytes();
      try
      {
         MessageDigest md5er = MessageDigest.getInstance("SHA-1");
         byte[] hash = md5er.digest(bytes);
         return bytes2hex(hash);
      }
      catch (GeneralSecurityException e)
      {
         throw new RuntimeException(e);
      }
   }

   private static String bytes2hex(byte[] bytes)
   {
      StringBuffer r = new StringBuffer(32);
      for (int i = 0; i < bytes.length; i++)
      {
         String x = Integer.toHexString(bytes[i] & 0xff);
         if (x.length() < 2)
            r.append("0");
         r.append(x);
      }
      return r.toString();
   }

Example Driven Development and Unit testing

2008-01-25T00:14:00.001+01:00

In a project I once worked, we were required by company standards to write formal test specifications for manual testing. It was a lot of overhead to write these Word documents and get them through the bureaucracy. Finally, we proposed to write the test specifications as comments inside functional unit tests. That way, we could maintain the test documents easily. We saved a lot of work, and the documents got higher quality because it was easier to update them. And it really helped us to write good unit tests that covered the functionality.

In this post, I will take this approach one step further to show how unit tests and manual tests can be unified.

In an earlier post Example driven development, I argued that a few simple examples can be used for requirements, manual testing and unit testing. I don't say that a few examples are sufficient as a requirements specification, but they may be in relatively simple projects. And it is far better than nothing. Examples help you to think clearly, and to communicate accurately with others. Don't we all use examples when we try to explain something? That's the best way to explain something anyway.

The problem is that it is hard to keep the examples up to date, and then they lose the value they had for communication with the client and manual testing.

But if we implement the examples as unit tests, we completely avoid this problem! As long as the unit tests pass, the examples will be in sync with the code. And if the client changes the requirements, we modify the unit tests, and then implement the changes until the unit tests pass.

Unit tests are hard to read for non-programmers, but if we put a lot of effort into it, we can make them readable. They don't need to be writable, as programmers will write them.

Here is an example of a test of a servlet that generates licenses:

   public void testGenerateLicense() throws Exception
   {
      // Call the servlet.
      InputParams params = new InputParams();
      params.productId = "123456";
      params.quantity = 1;
      params.firstName = "Lars";
      params.lastName = "Høidahl";
      params.email = "lars@mycompany.com";
      params.company = "Object Generation";
      params.country = "Sweden";
      File licenseFile = servlet.generateLicense(params);
      
      // Check the returned file.
      assertTrue("Attachment is a license file",
            licenseFile.getName().endsWith(".lic"));
      
      // Check that 1 user was created in the database.
      List users = userDatabase.getAllUsers();
      assertEquals("Users", 1, users.size());
      User user = users.get(users.size()-1);
      assertEquals("User name", "Lars", user.getUsername());
      assertEquals("Email", "lars@mycompany.com", user.getEmail());
      assertEquals("Country", "Sweden", user.getCountry());
   }

   public void testGenerateMultipleLicenses() throws Exception
   {
      // Call the servlet.
      InputParams params = new InputParams();
      params.productId = "123456";
      params.quantity = 3;
      params.firstName = "Lars";
      params.lastName = "Høidahl";
      params.email = "lars@mycompany.com";
      params.company = "Object Generation";
      params.country = "Sweden";
      File licenseFile = servlet.generateLicense(params);
      
      // Check the returned file.
      assertTrue("Attachment is a zip file",
            licenseFile.getName().endsWith(".zip"));
      ZipFile zipFile = new ZipFile(licenseFile);
      assertEquals("Number of entries", 3, zipFile.size());
      
      // Check that 3 users were created in the database.
      List users = userDatabase.getAllUsers();
      assertEquals("Users", 3, users.size());
      User user = users.get(users.size()-1);
      assertEquals("User name", "Lars", user.getUsername());
      assertEquals("Email", "lars@mycompany.com", user.getEmail());
      assertEquals("Country", "Sweden", user.getCountry());
   }

Database dump with Java

2008-01-23T22:18:00.001+01:00

I need to update a database that is created by PHP. The problem is that I am not a PHP coder, but a Java coder, and I need to use some other Java libraries to get the job done. So how can find out exactly which tables to update and how? It would take me weeks to search the PHP code, and I still wouldn't be sure if I got it right.

The first step is to install a clean application on my computer. There is no user data in the database, so if I perform commands like creating a user etc in the web application, I can look at what changed in the database. I'm sure that could be done in MySQL, but I'm not an expert on that either. When the only tool you have is a hammer, everything looks like a nail. So, I'll use Java for that to.

So, I wrote a small Java application that produces exactly the output that I need. It reads metadata from the database to find all tables and columns, lists that metadata and the content of all the rows.

Here it is:

import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;
import java.io.PrintWriter;
import java.sql.Connection;
import java.sql.DatabaseMetaData;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.ResultSetMetaData;
import java.sql.SQLException;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.Properties;

public class DumpDatabase {
   private Connection conn;
   private PrintWriter out;

   public static void main(String[] args) {
      try {
         DumpDatabase app = new DumpDatabase();
         String timestamp = new SimpleDateFormat("MMdd-hhmmss")
.format(new Date());
         String fileName = "dumpdatabase-" + timestamp + ".txt";
         System.out.println("Creating " + fileName);
         app.dumpDatabase(fileName);
         System.out.println("Done.");
      } catch(Exception e) {
         e.printStackTrace();
         System.exit(1);
      }
   }

   public DumpDatabase() throws IOException, 
ClassNotFoundException, SQLException {
      // Get database connection from hibernate.properties.
      // Or hard-code your own JDBC connection if desired.
      InputStream in = getClass().getResourceAsStream(
"/hibernate.properties");
      Properties properties = new Properties();
      properties.load(in);
      String driver = properties.getProperty(
"hibernate.connection.driver_class");
      String url = properties.getProperty(
"hibernate.connection.url");
      String user = properties.getProperty(
"hibernate.connection.username");
      String password = properties.getProperty(
"hibernate.connection.password");

      Class.forName(driver);
      conn = DriverManager.getConnection(url, user, password);
   }
   
   public void dumpDatabase(String fileName) 
throws FileNotFoundException, SQLException {
      out = new PrintWriter(fileName);
      listAll();
      out.close();
      conn.close();
   }

   public void listAll() throws SQLException {
      DatabaseMetaData metadata = conn.getMetaData();
      String[] types = { "TABLE" };
      ResultSet rs = metadata.getTables(
null, null, null, types);
      while(rs.next()) {
         String tableName = rs.getString("TABLE_NAME");
         listTable(tableName);
      }
   }

   private void listTable(String tableName) throws SQLException  {
      PreparedStatement statement = conn.prepareStatement(
            "select * from " + tableName + " a");
      out.println("----" + tableName + "----");
      int rowNo = 0;
      ResultSet rs = statement.executeQuery();
      while(rs.next()) {
         if (rowNo == 0)
            printTableColumns(rs);
         printResultRow(rs);
         rowNo++;
      }
   }

   private void printTableColumns(ResultSet rs) 
throws SQLException {
      ResultSetMetaData metaData = rs.getMetaData();
      for(int i = 0; i < metaData.getColumnCount(); i++) {
         int col = i + 1;
         out.println(metaData.getColumnName(col) + " "
               + metaData.getColumnTypeName(col) + " " + "("
               + metaData.getPrecision(col) + ")");
      }
      out.println("");
   }

   private void printResultRow(ResultSet rs) throws SQLException {
      ResultSetMetaData metaData = rs.getMetaData();
      for(int i = 0; i < metaData.getColumnCount(); i++) {
         String column = metaData.getColumnName(i + 1);
         try {
            String value = rs.getString(column);
            if (value != null && !value.equals("null") 
                  && !value.equals("") && !value.equals("0"))
               out.print(column + ": " + value + ", ");
         }
         catch(SQLException e) {
            out.print(column + ": " + e.getMessage());
         }
      }
      out.println("");
   }
}

The application reads the JDBC properties from an hibernate.properties file, for example like this:

hibernate.dialect=org.hibernate.dialect.MySQL5Dialect
hibernate.connection.driver_class=com.mysql.jdbc.Driver
hibernate.connection.url=jdbc:mysql://localhost/users
hibernate.connection.username=<username>
hibernate.connection.password=<password>
hibernate.show_sql=true

It creates a file like dumpdatabase-01-23-10-30-38.txt:

----tb_role----
----tb_user----
user_id BIGINT (20)
username VARCHAR (255)
password VARCHAR (255)
first_name VARCHAR (255)
last_name VARCHAR (255)
email VARCHAR (255)

----tb_user_role----

Then, I create a user and run the program again to get a new dump dumpdatabase-01-23-10-32-05.txt:

----tb_role----
----tb_user----
user_id BIGINT (20)
username VARCHAR (255)
password VARCHAR (255)
first_name VARCHAR (255)
last_name VARCHAR (255)
email VARCHAR (255)

user_id: 1, username: albert, password: pw, first_name: albert, 
last_name: albertson, email: albert@dot.com, 
----tb_user_role----

Of course, in the real application the dumps are thousands of lines, but with Eclipse "Compare with each other", it' still easy to see what happened, so I know exactly what my Java code needs to do.

Refactor Groovy code?

2008-01-21T01:22:00.001+01:00

I love Groovy! It's great for getting things done quickly. From Groovy, I can call Ant and XSLT easily. I can build and parse XML. I can call external applications, read file content and traverse a file hierarchy with very little code. And I can modify my program at runtime, without compiling and deploying it again.

But this dynamics come at a cost. For instance, I really miss code completion. Some times, I go to a Java editor and create an object of some class and push CTRL-SPACE to find out the parameters for a method. Also, I miss that the editor doesn't indicate errors if I misspell a function or variable name.

And I always wondered how well a dynamic language would scale. Now, I think I know the answer: It doesn't scale well! I have a Groovy template of 500 lines, and it's almost impossible to modify it. And today, I have been trying to refactor a Groovy class of 800 lines, and it's really, really hard.

How hard would it be to modify a Java class of 500-800 lines? Not hard at all. Unfortunately, the Java version of these classes would probably be several thousand lines long, so there is no perfect solution that I know of. And I can probably blame myself for hacking these unmaintainable scripts together. But that's why I loved Groovy in the first place! The ability to produce something quickly.

Actually, I write some code quickly in Java too some times, but I never have any problems refactoring it later. Maybe the problem is that there is no refactoring support in Groovy. I am so spoiled by Eclipse for Java in this area. I really hope this is the problem, and that refactoring support and auto completion comes to my editor soon. Because I love the productivity that Groovy gives.

Until then, my advice is this: Use Groovy for relatively small scripts or to replace several other tools with one. For larger programs, I think it's better to stick with Java for now. Unless you know the domain, so you write code right the first time, and never need to refactor it.

Defensive Programming

2008-01-20T13:39:00.000+01:00

After thinking some more about what I wrote in Small trick with Copy & Paste, I saw that that's just an example of an attitude to programming. We can call it defensive programming.

When I program, I always try to think "What can possibly go wrong here? And how can I make sure that doesn't happen?". One example is that I mess up variable names.

Another example is that copy and paste are a source of problems. If I duplicate code, and I need to change it, I may forget to change it everywhere. So, it's usually better to generalize it.

Another is that I change how a method is supposed to be used. For example, let's say I have a function:

    void sendEmail(String to, String cc,
                   String subject, String body)

But then I change it to

    void sendEmail(String to, String subject,
                   String body, String attachment)

So all callers have to change their parameters for this to work properly.

To be sure this doesn't break, it's better to rename the method:

    void sendEmailWithAttacment(String to, String subject,
                                String body, String attachment)

Then, the compiler will make sure it works. At least in a static language like Java. In Groovy or something you wouldn't detect this until runtime. That's why I'm not totally sold into dynamic languages, although I like Groovy a lot. If we use a dynamic language, we have to be defensive by using more unit tests.

A final example is if you fix a bug. There is always a risk that you or someone else will revert the change. How can we be sure this doesn't happen? The best way is to reproduce the bug with a unit test before fixing it. If this is too much work, at least write a comment explaining the fix.

It all comes down to admitting that I do mistakes and trying to prevent them.

Example Driven Development

2008-01-18T23:26:00.001+01:00

This article is the first in a series about Test Driven Development. A test is an example of what a program should do. Examples make the requirements concrete, so they become easier to understand. I will show how examples can be used for requirements, manual testing and unit testing.

I am currently working on automatic order handling for a client. I will implement a servlet with the following tasks:
1. Decode input parameters from a web page.
2. Generate an encrypted license with some of those parameters.
3. Store the license in a database. I can model this database as I want.
4. Store customer data in another database. This database schema is "carved in stone".
5. Send the license as an email to the user.

This is a small project, where I will implement, test and deploy the servlet myself. If it doesn't work, my client will lose customers, and I will be responsible.

So how can I make sure this doesn't happen?

I don't trust myself to make so good quality that there won't be any problems. So, I need to do thorough testing. I will make unit tests, but how do I make good ones?

Unit testing is important, but it's not enough. There are always some issues that don't show up until manual testing. I also need to test the integration between my program and other components manually. But how do I make good manual tests? Do I need to write a formal test specification for this simple program?

I think I do. I need to discuss the requirements with my client, and some examples of what the program is supposed to do is the best way to do that. By making a few concrete examples, I can make sure there are no misunderstandings, and that no requirements are missing. I'm sure there will be some misunderstandings anyway, but that's why I'm going to test it.

I'll use this test specification both for manual testing and as basis for the unit tests. That way, I'll have good unit tests that cover the functionality well, and there hopefully shouldn't bee too many surprises when it comes to integration testing.

I start by writing down 2 examples that the servlet should handle. For each example, I will list the input parameters, output values and any changes to the databases. It doesn't take long to write these down on a piece of paper, and it will be very much worth the effort.

Example 1: A customer orders 1 license of type "Enterprise"

Input

product_id   = (ask the client what the product id is)
quantity     = 1
company_name = Dot Com Inc
email        = john@dotcom.com

I have already identified a very important question I need to ask my client: What are the product ids that the servlet will receive?

License Content

type         = Enterprise
company name = Dot Com Inc
email        = john@dotcom.com
expires      = 3/25/2009

License Database
Insert a license with the following values:

type         = Enterprise
company name = Dot Com Inc
email        = john@dotcom.com
date         = 3/25/2008
expires      = 3/25/2009

Customer Database
(I need to ask the client about this.)

Email

to         = john@dotcom.com
subject    = License
content    = Please find attached an Enterprise license to
             Dot Com Inc and email john@dotcom.com.
attachment = enterprise.lic

Example 2: A customer orders 1 license of type "Basic"

Input

product_id   = (Ask the client)
quantity     = 1
company_name = Dot Com Inc
email        = john@dotcom.com

License Content

type         = Basic
company name = Dot Com Inc
email        = john@dotcom.com
expires      = 3/25/2009

License Database
Insert a license with the following values:

type         = Basic
company name = Dot Com Inc
email        = john@dotcom.com
date         = 3/25/2008
expires      = 3/25/2009

Customer Database
(Ask the client.)

Email

to           = john@dotcom.com
subject      = License
content      = Please find attached a Basic license to
               Dot Com Inc and email john@dotcom.com.
attachment   = basic.lic

Now, I need to go through these examples with my client.

When I talked to the client, the he pointed out that it is possible to order multiple licenses of the "Enterprise" product at once. I hadn't though about that. Together, we discussed how to handle this.

In the next article, I will write about how to implement and unit test the servlet.

Small Java trick with copy - paste

2008-01-18T12:54:00.000+01:00

Here is a small trick I use to avoid getting into problems.

Some times, I need to copy a block with a variable, for instance:

      Address addr = list.get(0);
      assertEquals("city", "Uppsala", addr.getCity());

and paste it to create something similar like this:

      Address addr2 = person.getAddress()
      assertEquals("city", "Uppsala", addr.getCity());

Did you see the bug?

The trick I use to avoid bugs like this, is to change the variable name in the original block:

      Address addr1 = list.get(0);
      assertEquals("city", "Uppsala", addr1.getCity());
      Address addr2 = person.getAddress()
      assertEquals("city", "Uppsala", addr.getCity());

Then the bug becomes obvious.

I can't tell you how many times this has saved me.

Use cases and robustness analysis

2008-01-12T00:55:00.001+01:00

Use cases or user stories? Use cases are bigger and require more work. User stories are quicker and easier, more agile. That may be ok if you have other agile practices in place. If you have a customer or functional expert available to the team at all times, user stories may work quite well. But if you don't, you may need something heavier, like use cases.

It also depends on the complexity of the requirements. If the requirements are simple, use cases are not necessary. But for complex requirements, I think use cases are better suited than user stories. They help the customer to think through the requirements earlier, and they give the developers better understanding of what the system is supposed to do.

But use cases are not agile! Yes, they may be. Agile doesn't mean lightweight, it means rightweight. It means you adapt the process to the project. A large, complex project needs a heavier process than small and simple ones.

RUP had an activity called robustness analysis to analyze model, view and controller objects from use cases. I never really understood the purpose of this, until recently I read a book called Use Case Driven Object Modeling with UML. I strongly recommend this book for everyone using use cases and/or object oriented analysis. It argues against many popular opinions about use cases that they should be abstract and not deal with the user interface. These kind of abstract use cases don't give the developers the information they need, leaving them to make their own assumptions about how the system should work.

This doesn't mean that the use cases should include user interface design, but some specification of the user interface is necessary. Not only is this necessary to keep the developers on track, it also helps to make the use cases better.

The book explains how drive the object design from the use cases, resulting in a good object model that is consistent with the requirements.

Why Agile Methods Reduce Cost

2008-01-07T10:17:00.001+01:00

The greatest cost of software development is complexity:

The problems we are trying to solve are complex. The customers often don't understand the requirements completely before the software development starts.
The mapping between the requirements and the software is also complex, so that misunderstandings between the customers and the developers are the rule and not exceptions.
The software itself is complex. It is hard for the developers to get an overview of what it does, and the consequences of modifying the software are hard to predict.

This complexity adds risk to software development projects, so it is actually uncommon for software projects to complete in time and on budget. Complexity also results in poor quality, since errors occur all the way in the development process; in the requirements, while mapping the requirements to the implementation, and in the implementation.

The risk and the low quality obviously increases the development cost. The cost is also increased by the the need for more developers and experts who know how to handle the complexity, and the need for substantial testing to find and fix errors.

So how can we deal with this complexity?

Requirements
We cannot reduce the complexity of the requirements, but I beleive a good domain model is important to understand complex requirements. Make the model concrete by using examples. Model the object instances for a use case instance, and go through the variations and changes that can happen with the customer.

Communication
The number one success factor in development projects is communication. We need to improve communication between customers and developers, and within the development team.

Agile methods does this through a number of key practices:

Iterative development gives rapid feedback on misunderstandings between customers and developers.
Ideally, a customer should be available to the developers all the time.
Daily stand-up meetings improve the communication within the team.
Pair programming.

Also, the domain model is very valuable as a means for communicating within the team.

Software Complexity
By complexity of software I mean how many thoughts the developer has to hold in his head at once. Early Java frameworks separated the code into lots of losely connected classes and XML files. The purpose was reuse and easy configuration, but it nevertheless made the program fragmented and hard to understand.

I personally hate it when I have to jump from a Java class to struts-config.xml to find the name of a form Java class. I just want to push F3. I find it ironic that software developers that are designing user interfaces for a living can come up with something as awkard as XML configuration files.

It is said that the human mind can hold between 5 and 9 pieces of information at once. If you need to hold the 1 action class, 1 form, 1 DAO, 1 JSP page, 2 XML files and 1 Javascript file, that doesn't leave much space for the feature you are trying to develop.

Luckily, this situation is beeing addressed by a new generation of programming languages and frameworks such as Ruby on Rails.

I beleive the cost of switching to a simpler framework will pay back several times by the improved speed that comes by being able to focus on what you are actually trying to do.