Thursday, December 6, 2012

Examples and Objects

The most successful and fun project I have been part of developed a network simulator for operator training. We had a great team and freedom to work how we wanted, but another key to success was the domain model. The analyst drew object diagrams to explain to the programmers and testers how a network topology could be, for instance like this:


This may look simple, but these diagrams explained a lot to us. We used them to discuss what should happen in different scenarios, such as what should happen when a Trunk is connected to another Port. Then we implemented these scenarios as unit tests. This resulted in a robust implementation with few bugs.

But in most projects, we don't have anything like this. In one project I spent several days trying to understand how a trading system worked. I had the domain model and database schema, but it was not clear when these objects were created, updated and deleted in the database. I read thousands of lines of code and run the program in a debugger, but this didn't tell me when the behavior was executed. Finally, I found out, and I drew some diagrams. When I presented this to the project manager, analyst, testers and other developers, it was a revelation to people that had worked on that system for years. They finally understood the internal behavior of their application. This behavior was certainly clear to the original designers of the application, but it was not documented anywhere. The result was that new programmers like myself struggled to understand the application logic. This happens all the time in our industry. It is frustrating for developers and costly for companies.

Developers need some documentation; the code is not enough. But what is the best format for this documentation? We don't need thick Word documents. A few examples of domain objects can take us very far. And the great thing about domain objects is that they can be discussed with users, analysts and testers, so that the whole team get a shared understanding of the behavior of the application.

We can take a few key examples of the system behavior and show which domain objects are needed in those examples and which objects are created, deleted and updated. This gives a high-level view of the internal behavior of the application.

The example below shows one way to do this.

Given the following objects:


When the user enters the following:


Then the application should give the following output:
  • Total hours: 39

And the following objects should be created:



Examples connected to domain objects give a high-level view of what the application is doing. It is suitable for discussions with users, analysts, developers and testers, and it is great documentation for future developers.

The example is ready to test and the format is well suited for test automation. By using this specification as input to a test automation tool, you get high-quality tests that focus on the domain logic instead of brittle user interface testing. This kind of tests are great input to programmers, because they are directly connected to the specification and they contain the details that the programmers need. (I have written about specification by example here.)

In the example above, I used the terms Given, When and Then to show how this method can be compared to Behavior Driven Development (BDD). The difference is that BDD uses 3 steps (Given, When and Then), while this method uses 4: Initial objects, Input, Output and Result objects. More importantly, the present method uses domain objects more explicitly than BDD. This gives a strong connection between the specification (examples) and the object model. When this is combined with automatic tests, the specification, code and tests will be tightly integrated and kept consistent with each other. The domain objects become the glue that binds everything together. 

Tool Support

I am working on a tool where a specification by example is connected to objects and used for automatic testing. The examples and objects are written a wiki-like text format. This is used to generate HTML documentation. The domain logic is then implemented in Groovy and tested against the specification with JUnit.

Thursday, November 29, 2012

Writing Better Requirements with Examples and Screen Sketches

We were agile, we had a Scrum master, we had standup-meetings, we had unit tests, we worked iteratively and met the product owner regularly. We did everything right, except the requirements. When we were almost ready to launch, we suddenly understood that we had missed a critical piece of functionality; namely the complex pricing model. The product owner thought we knew how this should work, but we didn't. This was not a feature that could just be patched onto the application in the end, it took several weeks of restructuring. We might blame the product owner for not communicating this clearly, but we were the software professionals. It's our responsibility to find out what our customers want.

Examples

What could we have done to avoid this embarrassment? Should we have spent the first month of the project writing requirements? No, I don't think that's the solution. That might have helped, but it would have cost too much.

There is a much simpler thing we could have done. In one word: Examples. If we had given the product owner an example of how we would calculate the price early in the project, the product owner would immediately have pointed out the misunderstanding and explained how to calculate the price correctly. This would have enabled us to make a better plan and implement that feature effectively.

Examples is a way to write useful requirements without too much effort. They are easy to understand by all parties involved: Users, project managers, analysts, developers, UI designers and testers. Examples reduce misunderstanding and thereby reduce risk and save cost.

Another advantage of examples is that they can be tested automatically. This is not possible with traditional requirements specifications. Then, you first write the general requirements, then test specifications and perhaps try to automate the tests. But examples are tests, and if they are written in an adequate format, they can be used by testing tools. This saves manual work with testing, ensures that the implementation and specification are consistent, and helps developers to focus.

Screen sketches

What is the best format to write examples? What does the user understand best? The user interface, of course! That's why prototypes are so useful. To quote Jakob Nielsen:

Whatever you do, at least promise me this: Don't just implement feature requests from "user representatives" or "business analysts." The most common way to get usability wrong is to listen to what users say rather than actually watching what they do. Requirement specifications are always wrong. You must prototype the requirements quickly and show users something concrete to find out what they really need. (www.useit.com)

A problem with prototypes is that they take a while to build. Writing requirements should be quick and easy so you can do it together in a workshop. Also, prototypes are not suitable for automatic testing.

Instead, we can specify examples as user interface sketches. This is not user interface design, but a user interface specification. The sketches don't need fancy widgets, it suffices with labels, text fields, tables, check boxes and selection boxes, so that input and output values can be displayed. At this stage, we only need to specify what the user can do, not how.

For instance, an abstract requirement like "The employees report worked hours every week," may seem obvious to the customer, but it leaves lot of room for interpretation. Instead, we can give a concrete example like this:


This example is much easier to understand than the abstract requirement. The customer can easily spot mistakes and fix them. For instance, the customer may point out that they can work on multiple projects at the same time and give a better example like this:


In my opinion, this is as clear as it gets. It reveals a lot of questions for further discussion, like who defines the projects, can the employees work on weekends, what happens if the employee works more or less than 40 hours in a week, etc. Discussing these questions with the customers before implementation can easily save an iteration or two in the project.

Tool support

Screen sketches are easy to read, but are they also easy to write and suitable for test automation? It is fairly easy to make a tool that lets you design screen sketches; either a GUI builder, or simply writing them in a text format. This data can then be used by a tool to test the application. I have started on such a tool where you can write wiki-like text and generate HTML pages with screen sketches. The examples can then be used for automatic testing. You can read more about the tool and try it here.

Conclusion

Examples are useful for communicating with customers, as specification for programmers and UI designers and as test specification for testers. Ideally, all these groups should discuss the examples in detail together with the customer and analyst before an iteration. The programmers will then have all the details they need, the testers will know what to test, the customers will get the right product, and everybody's happier.

Friday, November 23, 2012

The Future of Programming

The problem with abstractions

Programmers are experts in abstract thinking. To program is to generalize: A method is a general specification of different execution paths. A class is a general specification of objects. A superclass is a generalization of several classes.

Although our minds are capable of abstract thinking, concrete thinking is more natural, and concrete examples are the foundation for abstractions. For instance, how do you teach children what a car is? You don't give a definition like: 'A car is a wheeled motor vehicle used for transporting passengers.' Instead, you point at a car and say: 'Look, a car!' After seeing a few cars, the child understands what a car is.


Notice what I just did! I started with a general principle and then gave an example. Which part of the paragraph above is easier to understand, the first sentence (general) or the rest (example)?

Einstein said that examples is not another way to teach, it is the only way to teach. It is easier to start with a few examples and then generalize than the other way around. If you start with an abstract description, you will often try to comprehend it by visualizing an example in your head.

The problem with text

Another point is that children understand better what they see than what they hear. Or as they say, children do what you do, not what you say. This is not only true for children. We understand what we sense in fractions of a second, but words must be analyzed before they give meaning. This doesn't take long for simple sentences, but it can take hours, days or months to understand source code of any significant size. Some source code is actually incomprehensible and nobody dares to touch it.

What does this have to do with programming?

As programmers, we are highly skilled in abstract thinking and textual programming languages, but when the code becomes too complex, we try to comprehend it by imagining its behavior. And when it becomes too hard to imagine the actual execution by reading the abstract code, we log what happens or run the code in a debugger. Logs and debuggers are not abstract. They give concrete descriptions of what is actually happening. They give information on method calls, object instances and variable values. This concrete information is much easier to understand than the abstract code.

We can move programming closer to how the mind works by using examples as part of the implementation. Unit testing is one way to do this, because unit tests are examples of what the code can do. There is also a growing movement for using examples as requirements. This improves the communication between users, analysts, programmers and testers. These examples can be automatically tested to connect them to the code.

Objects are examples of classes

So what is the best way to specify examples? I think that the most effective formats are user interface sketches and objects. The user interface is the best way to explain the external behavior of a program, and objects can be used to describe the internal behavior. I am not talking about technical objects that is part of the infrastructure, but domain objects that the end user understands.

Objects can be specified as text, in tables or in diagrams. Diagrams are easiest to understand, but they can be cumbersome to draw and edit.

Here is an example of an object diagram:
It is pretty obvious what this means.

If Lars sells his car to John, the objects should be changed like this:
Alternatively, the same information can be specified in tables or as text.

Modeling examples as objects is an effective way to build a domain model. This makes the examples consistent with each other and helps to reveal details and missing functionality. They are great as specification for programmers, because they are specific and contain a lot of details. And they can be automatically tested easily.

Tool Support

I believe that using examples and objects is the future of programming because it is closer to how our mind works.

I have started on a tool where you can write a specification by examples with screen sketches and objects. The application is then implemented in Groovy; a dynamic language that runs on the Java platform. The tool can test the examples to verify the program. Read more about the tool and try it out here!

Stay posted!

Thursday, November 22, 2012

Object cases tool released

I just released our new tool. It supports specification by example, object modeling, automatic testing and implementation in Groovy. The tool is open source. Here it is.

Stay tuned!

Tuesday, June 21, 2011

Test driven development with SmartGWT and JPA

SmartGWT is a collection of GWT widgets that can easily be connected to database tables or some other data on the server.

SmartGWT Pro supports Hibernate/JPA on the server, but there are some limitations, and unit testing it is tricky.

This article shows how to implement and test server logic with SmartGWT and JPA. Specifically, it will show how to:
  • Configure server.properties for unit testing.
  • Handle JPA relations with SmartGWT.
  • Implement an EMF provider so the unit tests and the data sources share the same EntityManager.
  • Use special JPA queries such as <= and string matching in SmartGWT.
  • Build complete server logic with pagination and sorting for JPA.

Examples

Before we get into the practicalities of SmartGWT, we need to design good tests. Many who start with test driven development find this very hard. My approach is to test examples of application behavior. If you discuss these examples with the customer first, you can implement automatic tests that are directly based on the requirements.

Let's say I wanted to build an online auction application. I would start by asking the client for some examples of what the application should do. The client might answer that sellers may place items for auction, bidders place bids, and the seller then sells to the highest bidder. This is not what I mean with examples, these are abstract requirements. I want concrete examples with realistic input and output values. I mean something like this:

Jane wants to sell an electric guitar. She enters the following information:


Bob wants to buy an electric guitar. He searches for auctions, clicks on Electric guitar, and places a bid.





Jane can then see the bids in her auction:



The advantages of using examples are that they are detailed, they reveal a lot of misunderstandings and they can be tested.

Objects

Another thing that many struggle with is to design a good domain model, be it a class model or a database schema. The approach shown here makes this easy. Instead of starting with a class diagram, I start by looking for objects in the examples. Then I derive the class model from the objects.

An example connected to objects is called an object case.

Jane wants to sell an electric guitar

For each example, I need to determine how to handle the input from the user and how to calculate the output. In this example, I will create a User object and an Auction object. The arrows show that all the input from the user is stored in these objects.

Bob searches for auctions

In this example, the Free_text and Max_price input is used to search for Auctions, but the Category is not used. This means that something is missing in the object model.

I decide to introduce a new object that will be used to search for category. Now, all input values are used and the output in the table is calculated from the Auction objects, so everything is covered.

Bob clicks on Electric guitar

All the output values can be calculated from an Auction object. We should define another example that shows an actual bid.


Bob places a bid

The input values are stored in two new objects: User and Bid. What happens if the user already exists? We should add another example to deal with this.


Classes

All input and output values are now connected to objects. This is a simple and powerful method to design an object model. Based on these objects, I can create the following class diagram:


Find more functionality

We can use the class diagram to find missing functionality. The diagram below shows how objects are created and read by the examples defined so far.


What is missing?
  • Create Category objects. Should an adminstrator to this or should it happen automatically? This needs to be discussed with the client.
  • Set link between Category and Auction. This should be done by Create auction.
  • Read User objects. What do we need the email for?
  • Read Bid objects. This should be part of Auction details. We should define an example to show this.
  • Update attributes. This needs to be discussed with the client.
  • Delete objects. We need to discuss with the client if this should be possible.

Based on this analysis, I add a Category parameter to the first example and use that parameter to find an existing Category object. I create a link between this Category and the new Auction:

Implement JPA classes

It's straight forward to implement the classes as JPA entities.
@Entity @Table(name="tb_auction")
public class Auction {
    @Id @GeneratedValue(strategy=GenerationType.AUTO)
    private Long id;

    private String title;
    private float startingPrice;
    private String description;
 
    @ManyToOne(optional=false)
    private Category category;

    @ManyToOne(optional=false)
    private User seller;

    @OneToMany(mappedBy="auction")
    private Set<Bid> bids = new HashSet<Bid>();
}

@Entity @Table(name="tb_bid")
public class Bid {
    @Id @GeneratedValue(strategy=GenerationType.AUTO)
    private Long id;

    private float bid;

    @ManyToOne(optional=false)
    private Auction auction;
 
    @ManyToOne(optional=false)
    private User bidder;
}

@Entity @Table(name="tb_category")
public class Category {
    @Id @GeneratedValue(strategy=GenerationType.AUTO)
    private Long id;

    private String name;
    
    @OneToMany(mappedBy="category")
    private Set<Auction> auctions = new HashSet<Auction>();
}

@Entity @Table(name="tb_user")
public class User {
    @Id @GeneratedValue (strategy=GenerationType.AUTO)
    private Long id;

    private String email;
    
    @OneToMany(mappedBy="seller")
    private Set<Auction> auctions = new HashSet<Auction>();

    @OneToMany(mappedBy="bidder")
    private Set<Bid> bids = new HashSet<Bid>();
}
(Getters and setters removed for clarity.)

Implement tests

Each example is a test. I divide each test into 4 parts:
  1. Initial objects
  2. Input
  3. Output
  4. Result objects

With SmartGWT and JPA this becomes:
  1. Initial objects: Create objects in the database. The best way to do this in a unit test is to use a HSQLDB in-memory database, which can create the database schema on the fly.
  2. Input: Call a SmartGWT datasource with the specified parameters.
  3. Output: Check the output from the datasource.
  4. Result objects: Check the resulting objects in the database.

Let's start with testing the first example: Jane wants to sell an electric guitar.


public class AuctionTest extends TestCase {
 private DataSource auctionDS;
 private DataSource bidDS;
 
 private HashMap<String,Object> input = new HashMap<String,Object>();

 @Override
 protected void setUp() throws Exception {
  super.setUp();
  auctionDS = DataSourceManager.get("Auction");
  bidDS = DataSourceManager.get("Bid");
 }

 public void testJaneWantsToSellAnElectricGuitar() throws Exception {
  // Initial objects:
  category1 = new Category();
  category1.setName("Musical instruments");
  persist(category1); 

  // Input:
  input.put("title",       "Electric guitar for beginners");
  input.put("startingPrice",  150f);
  input.put("description", "Nice electric guitar with amplifier.");
  input.put("email",       "jane99@abc123.com");
  input.put("categoryId",  category1.getId());
  auctionDS.add(input);

  // Output: None.

  // Result objects:
  List users = getResultObjects(User.class);
  assertEquals(1, users.size());
  assertEquals("jane99@abc123.com", users.get(0).getEmail());

  List auctions = getResultObjects(Auction.class);
  assertEquals(1, auctions.size());
  Auction auction = auctions.get(0);
  assertEquals("Electric guitar for beginners",        auction.getTitle());
  assertEquals(150f,                                   auction.getStartingPrice());
  assertEquals("Nice electric guitar with amplifier.", auction.getDescription());
  assertEquals(category1.getId().intValue(),           auction.getCategoryId());
 }

 private void persist(Object... objects) throws Exception {
  EntityManager em = EMF.getEntityManager();
  Object tx = EMF.getTransaction(em);
  for(Object obj : objects) {
   em.persist(obj);
  }
  EMF.commitTransaction(tx);
 }

 private <T> List getResultObjects(Class<T> clazz) throws Exception {
  EntityManager em = EMF.getEntityManager();
  Object tx = EMF.getTransaction(em);
  List<T> result = em.createQuery("from " + clazz.getName()).getResultList();
  EMF.commitTransaction(tx);
  return result;
 }
}

Here is the second example: Bob wants to buy an electric guitar. This is more complicated with some initial objects and several steps.




public void testBobWantsToBuyAnElectricGuitar() throws Exception {
  initialObjects2Auctions();
  
  // Search for auctions.
  input.put("categoryId", category1.getId());
  input.put("title",      "guitar");
  input.put("maxPrice",   250f);
  
  List<Auction> output = auctionDS.fetch(input);
  assertEquals(2, output.size());
  assertEquals("Guitar",          output.get(0).getTitle());
  assertEquals(200f,              output.get(0).getStartingPrice());
  assertEquals("Electric guitar", output.get(1).getTitle());
  assertEquals(150f,              output.get(1).getStartingPrice());

  // Auction details.
  input.clear();
  input.put("id", auction1.getId());
  
  output = auctionDS.fetch(input);
  assertEquals(1, output.size());
  Auction auction = output.get(0);
  assertEquals("Electric guitar for beginners",        auction.getTitle());
  assertEquals("Nice electric guitar with amplifier.", auction.getDescription());
  assertEquals(150f,                                   auction.getStartingPrice());
  assertEquals("No bids.",                             auction.getHighestBid());

  // Place bid.
  input.clear();
  input.put("auctionId",   auction1.getId());
  input.put("bidderEmail", "bob@abc999.com");
  input.put("bid",         150);
  ErrorReport errors = bidDS.validate(input, true);
  assertEquals(null, errors);
  bidDS.add(input);

  // Result objects:
  List<User> users = getResultObjects(User.class);
  assertEquals(1, users.size());
  User user = users.get(0);
  assertEquals("bob@abc999.com", user.getEmail());
  List<Bid> bids = getResultObjects(Bid.class);
  assertEquals(1, bids.size());
  Bid bid = bids.get(0);
  assertEquals(150,              bid.getBid());
  assertEquals(user.getId(),     bid.getBidder().getId());
  assertEquals(auction1.getId(), bid.getAuction().getId());
 }
 
 private void initialObjects2Auctions() throws Exception {
  category1 = new Category();
  category1.setName("Musical instruments");
  persist(category1);

  auction1 = initialAuction(category1, "Electric guitar for beginners", "Nice electric guitar with amplifier.", 150f);
  auction2 = initialAuction(category1, "Guitar", "Used guitar.", 250f);
  persist(auction1, auction2);
 }

 private Auction initialAuction(Category category, String title, String description, float startingPrice) {
  Auction object = new Auction();
  object.setTitle(title);
  object.setDescription(description);
  object.setStartingPrice(startingPrice);
  category.addAuction(object);  
  return object;
 }

Let's add some more tests while we're at it:

public void testNoSearchCriteria() throws Exception {
  initialObjects2Auctions();
  assertEquals(2, auctionDS.fetch(input).size());
 }

 public void testSearchForUnknownCategory() throws Exception {
  initialObjects2Auctions();
  input.put("categoryId", category1.getId() + 1);
  assertEquals(0, auctionDS.fetch(input).size());
 }

 public void testSearchForMaxPrice150() throws Exception {
  initialObjects2Auctions();
  input.put("maxPrice", 150f);
  assertEquals(1, auctionDS.fetch(input).size());
 }

 public void testSearchForMaxPrice250() throws Exception {
  initialObjects2Auctions();
  input.put("maxPrice", 250f);
  assertEquals(2, auctionDS.fetch(input).size());
 }

 public void testSearchForTitlePiano() throws Exception {
  initialObjects2Auctions();
  input.put("title", "piano");
  assertEquals(0, auctionDS.fetch(input).size());
 }

 public void testSearchForTitleElectricGuitar() throws Exception {
  initialObjects2Auctions();
  input.put("title", "electric guitar");
  assertEquals(1, auctionDS.fetch(input).size());
 }

These tests will actually fail, but that's ok. When I fix them, I will learn much about the behavior.

The first thing I notice, is a compilation error because of a missing method, so let's add it:
public class Auction {
 public String getHighestBid() {
  Float highestBid = null;
  for(Bid bid : getBids()) {
   if(highestBid == null || bid.getBid() > highestBid) {
    highestBid = bid.getBid();
   }
  }
  return (highestBid != null ? highestBid.toString() : "No bids.");
 }
 ...

This can instead be handled by a database query in the "serverObject" that will be implemented later in this article.

To use HSQLDB for unit testing, add hsqldb.jar to the project and create a persistence.xml file in the test/META-INF directory with the following properties:
<property name="hibernate.connection.url" value="jdbc:hsqldb:mem:auction"/>
<property name="hibernate.hbm2ddl.auto" value="create-drop"/>

Implement datasources

The tests use the datasources "Bid" and "Auction" that I haven't defined yet. Here is a starting point:
<DataSource ID="Auction"
    serverConstructor="com.isomorphic.jpa.JPADataSource"
    beanClassName="com.objectgeneration.auctions.Auction">
    <fields>
        <field name="id"          type="sequence" hidden="true"       primaryKey="true" />
        <field name="title"       type="text"     title="Title"       required="true"   />
        <field name="description" type="text"     title="Description" required="true" />
        <field name="categoryId"  type="integer"  title="Category"    canEdit="false" />
    </fields>
</DataSource>

<DataSource ID="Bid"
    serverConstructor="com.isomorphic.jpa.JPADataSource"
    beanClassName="com.objectgeneration.auctions.Bid">
    <fields>
        <field name="id"   type="sequence" hidden="true" primaryKey="true"/>
        <field name="bid"  type="number"   title="Bid"   required="true"/>
    </fields>
</DataSource>

Using SmartGWT and JPA from JUnit

I am getting eager to implement the behavior, but unfortunately I have to fix some issues to get SmartGWT to work with JPA in JUnit first. Let's start:


That didn't go so well. Here is a stack trace:
java.lang.NullPointerException
 at com.isomorphic.io.ISCFile.<init>(ISCFile.java:145)
 ...
 at com.isomorphic.datasource.DataSourceManager.get(DataSourceManager.java:68)
 at com.objectgeneration.auctions.AuctionTest.setUp(AuctionTest.java:31)

The problem here is that SmartGWT thinks it is running as a servlet, but it does not find its web root. This can be configured in server.properties. Create a server.properties file for unit testing in the test folder. The content should be similar to the production version, except for the following:
# This is user specific. Use Ant filtering to generate this.
webRoot: /Users/lars/temp/auctions/war/

jpa.emfProvider: com.isomorphic.jpa.EMFProviderLMT

# Name of the datasource (from persistence.xml)
jpa.persistenceUnitName: ds
Also make sure the test source has a separate output folder (see Eclipse: Project > Properties > Java Build Path > Source). Otherwise the server.properties from src and test will overwrite each other in the output directory with random results.

Running JUnit again, we get the next problem:
java.lang.ClassCastException: java.util.ArrayList cannot be cast to java.util.Map
 at com.isomorphic.datasource.BasicDataSource.buildFieldData(BasicDataSource.java:426)
 ...
 at com.isomorphic.datasource.DataSourceManager.get(DataSourceManager.java:68)
 at com.objectgeneration.auctions.AuctionTest.setUp(AuctionTest.java:31)
This can be fixed by another setting in test/server.properties:
isomorphicPathRootRelative: Auctions_js/sc
(It should point to the sub-directory of war/ where initsc.js is created.)

JPA relations and SmartGWT


Trying again I get this stack trace:
javax.persistence.PersistenceException: org.hibernate.PropertyValueException: not-null property references a null or transient value: com.smartgwt.sample.server.Auction.category
 at org.hibernate.ejb.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:637)
 at org.hibernate.ejb.AbstractEntityManagerImpl.persist(AbstractEntityManagerImpl.java:226)
 at com.isomorphic.jpa.JPADataSource.executeAdd(JPADataSource.java:452)
 at com.isomorphic.datasource.DataSource.execute(DataSource.java:1050)
 at com.isomorphic.jpa.JPADataSource.execute(JPADataSource.java:218)
 …
 at com.isomorphic.datasource.DataSource.add(DataSource.java:1948)
 at com.objectgeneration.auctions.AuctionTest.testJaneWantsToSellAnElectricGuitar(AuctionTest.java:45)
Caused by: org.hibernate.PropertyValueException: not-null property references a null or transient value: com.smartgwt.sample.server.Auction.category
SmartGWT does not handle JPA relations like "Category category" in class Auction, it only handles foreign keys like "long categoryId". But I want to use JPA relations. The workaround is to have both and define only the foreign key in the datasource as above. The Java code is as follows:
public class Auction {
 @ManyToOne(optional=false)
 @JoinColumn(name="category", nullable=false, insertable=false, updatable=false)
 private Category category;
 
 @Column(name="category")
 private long categoryId;
...
}
The category column is mapped to 2 Java variables, but only one of them can be writable. Both of them need to be set when creating the initial objects:
public class Category {
 public void addAuction(Auction auction) {
  auction.setCategory(this);
  getAuctions().add(auction);
  assert getId() != null;
  auction.setCategoryId(getId());
 }
...
}
Trying again:
javax.persistence.EntityExistsException: org.hibernate.exception.ConstraintViolationException: could not insert: [com.smartgwt.sample.server.Auction]
 …
 at com.isomorphic.datasource.DataSource.add(DataSource.java:1948)
 at com.objectgeneration.auctions.AuctionTest.testJaneWantsToSellAnElectricGuitar(AuctionTest.java:45)
Caused by: org.hibernate.exception.ConstraintViolationException: could not insert: [com.smartgwt.sample.server.Auction]
 ...
Caused by: java.sql.SQLException: Integrity constraint violation - no parent FKD896457230F353B7 table: TB_USER in statement [insert into tb_auction (id, category, description, seller, startingPrice, title) values (null, ?, ?, ?, ?, ?)]
Now we are finally getting to the business logic. I cannot create an Auction without a User.

I decide to create a new User datasource and add a User before creating the Auction:
public void testJaneWantsToSellAnElectricGuitar() throws Exception {
  // Initial objects:
  category1 = new Category();
  category1.setName("Musical instruments");
  persist(category1);

  // Input:
  long userId = createUser("jane99@abc123.com");
  input.put("title",       "Electric guitar for beginners");
  input.put("startingPrice",  150f);
  input.put("description", "Nice electric guitar with amplifier.");
  input.put("categoryId",  category1.getId());
  input.put("sellerId",    userId);
  auctionDS.add(input);

  ...
 }

 private long createUser(String email) throws Exception {
  HashMap<String,object> input = new HashMap<String,object>();
  input.put("email", email);
  ErrorReport errors = userDS.validate(input, true);
  assertEquals(null, errors);
  DSRequest dsRequest = new DSRequest(userDS.getName(), DataSource.OP_ADD);
  dsRequest.setValues(input);
  DSResponse dsResponse = dsRequest.execute();
  return ((User)dsResponse.getData()).getId();
 }
I also connect the initial Auction objects in the other unit tests to a User. This makes some of the tests pass:


Many of these failures are because the tests are not isolated. If I run each test separately, many of them will pass. The reason for this is that the database is created when the first test is run, and then the next test uses the same database with the objects that are left. I need to empty the database between each test. This is easy to do in JPA; just call EntityManagerFactory.close(). But I don't have access to the EntityManagerFactory that SmartGWT uses. To fix this, I need to implement my own "EMF provider" and point to it in test/server.properties:
jpa.emfProvider: com.objectgeneration.auctions.server.MyEMFProvider

And here is the implementation:
import javax.persistence.EntityManager;
import javax.persistence.EntityManagerFactory;
import javax.persistence.EntityTransaction;
import javax.persistence.Persistence;

import com.isomorphic.jpa.EMFProviderInterface;

public class MyEMFProvider implements EMFProviderInterface {
 private static EntityManagerFactory factory;
 
 // Support multi-threaded tests.
 private static ThreadLocal<Entitymanager> entityManager = new ThreadLocal<Entitymanager>();
 
 public MyEMFProvider() {
 }
 
 /** Close the factory after each unit test so it starts with an empty database next time. */
 public static void close() {
  if(entityManager.get() != null) {
   entityManager.get().close();
   entityManager.set(null);
  }
  if(factory != null) {
   factory.close();
   factory = null;
  }
 }

 protected EntityManagerFactory createFactory() {
  return Persistence.createEntityManagerFactory("ds");
 }

 public EntityManagerFactory get() {
  if(factory == null) {
   factory = createFactory();
  }
  return factory;
 }

 public EntityManager getEntityManager() {
  if(entityManager.get() == null) {
   entityManager.set(get().createEntityManager());
  }
  return entityManager.get();
 }

 public void returnEntityManager(EntityManager em) {
  // Do nothing.
 }
 
 public Object getTransaction(EntityManager em) {
  EntityTransaction tx = em.getTransaction();
  if(!tx.isActive()) {
   tx.begin();
  }
  return tx;
 }

 public void commitTransaction(Object obj) {
  EntityTransaction tx = (EntityTransaction) obj;
  if(tx.isActive()) {
   tx.commit();
  }
 }
 
 public void rollbackTransaction(Object obj) {
  EntityTransaction tx = (EntityTransaction) obj;
  if(tx.isActive()) {
   tx.rollback();
  }
 }
}
Add the following to the test class to start with an empty database in each unit test:
protected void tearDown() throws Exception {
  MyEMFProvider.close();
  super.tearDown();
}
This takes us much closer to our goal, and we can continue with the business logic:


Implement datasource

There is no wonder that maxPrice is not working, the Auction class does not even have this field. I want to find all Auctions where Auction.startingPrice <= the maxPrice parameter. To implement this, I need to implement special handling for this data source in a "serverObject":
<DataSource ID="Auction"
    serverConstructor="com.isomorphic.jpa.JPADataSource"
    beanClassName="com.smartgwt.sample.server.Auction">
    <serverObject className="com.objectgeneration.auctions.AuctionRequestHandler"/>
    <fields>
        <field name="id"          type="sequence" hidden="true"   primaryKey="true" />
        <field name="title"       type="text"     title="Title"       required="true"   />
        <field name="description" type="text"     title="Description" required="true" />
        <field name="categoryId"  type="integer"  title="Category"    canEdit="false" />
        
        <!-- This field does not exist in the Auction class. It is implemented in AuctionRequestHandler. -->
        <field name="maxPrice"    type="number"   title="Max price" hidden="true" />
    </fields>
</DataSource>

The serverObject in the XML above refers to a class that will be called instead of the default SmartGWT handler, and there I can handle maxPrice:
import java.util.List;
import java.util.Map;

import javax.persistence.EntityManager;
import javax.persistence.Query;

import org.apache.log4j.Logger;

import com.isomorphic.datasource.DSRequest;
import com.isomorphic.datasource.DSResponse;
import com.isomorphic.jpa.EMF;
import com.smartgwt.sample.server.Auction;

public class AuctionRequestHandler {
 private static final Logger log = Logger.getLogger(AuctionRequestHandler.class);
 
 public DSResponse fetch(DSRequest dsRequest) throws Exception {
  Map<String,Object> criteria = dsRequest.getCriteria();
  log.info("fetch(" + criteria + ")");
  
  String queryString = "from Auction";
  String separator = " where ";
  for(String paramName : criteria.keySet()) {
   queryString += separator;
   if(paramName.equals("maxPrice")) {
    queryString += "startingPrice <= :" + paramName;
   } else {
    queryString += paramName + " = :" + paramName;
   }
   separator = " and ";
  }
  
  EntityManager em = EMF.getEntityManager();
  Query query = em.createQuery(queryString);
  for(String paramName : criteria.keySet()) {
   Object paramValue = criteria.get(paramName);
   query.setParameter(paramName, paramValue);
  }
  List<Auction> result = query.getResultList();
  return new DSResponse(result, DSResponse.STATUS_SUCCESS);
 }
}

I want to search for title with substrings and ignore case. Substrings is built in to SmartGWT, but not ignore case. Here is an updated version with special handling for the title field:
public DSResponse fetch(DSRequest dsRequest) throws Exception {
  Map<String,Object> criteria = dsRequest.getCriteria();
  log.info("fetch(" + criteria + ")");
  
  String queryString = "from Auction";
  String separator = " where ";
  for(String paramName : criteria.keySet()) {
   Object paramValue = criteria.get(paramName);
   queryString += separator;
   if(paramName.equals("maxPrice")) {
    queryString += "startingPrice <= :" + paramName;
   } else if(paramName.equals("title")) {
    queryString += "UPPER(" + paramName + ") LIKE '%" + ((String)paramValue).toUpperCase() + "%'";
   } else {
    queryString += paramName + " = :" + paramName;
   }
   separator = " and ";
  }
  
  EntityManager em = EMF.getEntityManager();
  Query query = em.createQuery(queryString);
  for(String paramName : criteria.keySet()) {
   if(!paramName.equals("title")) {
    Object paramValue = criteria.get(paramName);
    query.setParameter(paramName, paramValue);
   }
  }
  List result = query.getResultList();
  return new DSResponse(result, DSResponse.STATUS_SUCCESS);
 }

Success!

Wrapping it up

One improvement could be to move the getHighestBid() logic to a database query. The AuctionRequestHandler could call this query and store the value in a @Transient field in the Auction class to send it to the client, or it could send HashMaps instead of Auction objects.

We also need to handle the things that SmartGWT does by default: Transactions, pagination and sorting. (I believe that if you don't implement pagination and sorting in the server, the client will handle it instead. This may be acceptable if you don't have thousands of rows in that database table. Please refer to the SmartGWT documentation.)

A complete solution with a general superclass and special handling in a subclass looks like this:
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;

import javax.persistence.EntityManager;
import javax.persistence.EntityTransaction;
import javax.persistence.Query;

import org.apache.log4j.Logger;

import com.isomorphic.datasource.DSRequest;
import com.isomorphic.datasource.DSResponse;
import com.isomorphic.jpa.EMF;

public class RequestHandler {
 private static final Logger log = Logger.getLogger(RequestHandler.class);

 public DSResponse fetch(DSRequest dsRequest) throws Exception {
  return handleRequest(dsRequest);
 }
 
 public DSResponse add(DSRequest dsRequest) throws Exception {
  return handleRequest(dsRequest);
 }

 public DSResponse remove(DSRequest dsRequest) throws Exception {
  return handleRequest(dsRequest);
 }
 
 public DSResponse update(DSRequest dsRequest) throws Exception {
  return handleRequest(dsRequest);
 }

 protected DSResponse handleRequest(DSRequest dsRequest) throws Exception {
  EntityManager em = EMF.getEntityManager();
  EntityTransaction tx = (EntityTransaction) EMF.getTransaction(em);
  try {
   DSResponse dsResponse = handleInsideTransaction(dsRequest);
   tx.commit();
   return dsResponse;
  } catch(Exception e) {
   MyEMFProvider.rollback(tx);
   throw e;
  } finally {
   MyEMFProvider.finish();
  }
 }

 protected DSResponse handleInsideTransaction(DSRequest dsRequest) throws Exception {
  return dsRequest.execute();
 }

 @SuppressWarnings("unchecked")
 protected <T> DSResponse query(EntityManager em, DSRequest dsRequest, Class<T> clazz) {
  String queryString = "from " + clazz.getName();

  LinkedHashMap<String, Object> parameters = new LinkedHashMap<String, Object>();
  queryString = addCriteria(queryString, dsRequest.getCriteria(), parameters);
  
  List<T> result = selectObjects(em, dsRequest, queryString, parameters);
  long totalRows = countObjects(em, dsRequest, queryString, parameters, result);

  DSResponse dsResponse = new DSResponse(result, DSResponse.STATUS_SUCCESS);
  dsResponse.setStartRow(dsRequest.getStartRow());
  dsResponse.setEndRow(dsRequest.getStartRow() + result.size());
  dsResponse.setTotalRows(totalRows);
  return dsResponse;
 }

 private String addCriteria(String queryString, Map<String, Object> criteria, Map<String, Object> parameters) {
  String separator = " where ";
  for(String paramName : criteria.keySet()) {
   Object paramValue = criteria.get(paramName);
   queryString += separator + addCriterium(paramName, paramValue, parameters);
   separator = " and ";
  }
  return queryString;
 }

 /** Override this method for special handling of query parameters. */
 protected String addCriterium(String paramName, Object paramValue, Map<String, Object> parameters) {
  parameters.put(paramName, paramValue);
  return paramName + " = :" + paramName;
 }

 private <T> List<T> selectObjects(EntityManager em, DSRequest dsRequest, String queryString, Map<String, Object> parameters) {
  queryString = addOrderBy(queryString, dsRequest.getSortBy());
  Query query = em.createQuery(queryString);
  addPagination(dsRequest, query);
  setQueryParameters(query, parameters);
  return query.getResultList();
 }

 private String addOrderBy(String queryString, String sortBy) {
  if(sortBy != null) {
   if(sortBy.startsWith("-")) {
    queryString += " ORDER BY " + sortBy.substring(1) + " DESC";
   } else {
    queryString += " ORDER BY " + sortBy + " ASC";
   }
  }
  return queryString;
 }

 private void addPagination(DSRequest dsRequest, Query query) {
  long startRow = dsRequest.getStartRow();
  if(startRow >= 0) {
   query.setFirstResult((int)startRow);
  }
  
  long endRow = dsRequest.getEndRow();
  if(endRow > startRow) {
   query.setMaxResults((int)endRow - (int)startRow);
  }
 }

 private long countObjects(EntityManager em, DSRequest dsRequest, String queryString, Map<String, Object> parameters, List result) {
  long totalRows;
  long startRow = dsRequest.getStartRow();
  long endRow = dsRequest.getEndRow();
  if(startRow < 0 || endRow <= startRow) {
   totalRows = result.size();
   log.debug("no pagination, totalRows=" + totalRows);
  } else if(result.size() < endRow - startRow) {
   totalRows = result.size();
   log.debug("all rows received, totalRows=" + totalRows);
  } else {
   queryString = "select count(*) " + queryString;
   log.debug("countObjects: queryString=" + queryString + ", parameters=" + parameters);
   Query countQuery = em.createQuery(queryString);
   setQueryParameters(countQuery, parameters);
   Object countResult = countQuery.getSingleResult();
   log.debug("countObjects: result=" + countResult);
   totalRows = (Long) countResult;
  }
  return totalRows;
 }

 private void setQueryParameters(Query query, Map<String,Object> parameters) {
  for(String paramName : parameters.keySet()) {
   Object paramValue = parameters.get(paramName);
   query.setParameter(paramName, paramValue);
  }
 }
}


import java.util.Map;

import com.isomorphic.datasource.DSRequest;
import com.isomorphic.datasource.DSResponse;
import com.isomorphic.datasource.DataSource;

public class AuctionRequestHandler extends RequestHandler {
 @Override
 protected DSResponse handleInsideTransaction(DSRequest dsRequest) throws Exception {
  if(dsRequest.getOperationType().equals(DataSource.OP_FETCH)) {
   return query(dsRequest, Auction.class);
  } else {
   return dsRequest.execute();
  }
 }
 
 @Override
 protected String addCriterium(String paramName, Object paramValue, Map<String, Object> parameters) {
  if(paramName.equals("maxPrice")) {
   parameters.put(paramName, (Float) paramValue);
   return "startingPrice <= :" + paramName;
  } else if(paramName.equals("title")) {
   return "UPPER(" + paramName + ") LIKE '%" + ((String)paramValue).toUpperCase() + "%'";
  } else {
   return super.addCriterium(paramName, paramValue, parameters);
  }
 }
}

Implement user interface

When the server logic works, it is relatively straight forward to implement the user interface. The code looks something like this:
    public void onModuleLoad() {
        final DataSource auctionDS = DataSource.get("Auction");
        final DataSource bidDS = DataSource.get("Bid");

        final ListGrid auctionGrid = new ListGrid();
        auctionGrid.setDataSource(auctionDS);
        auctionGrid.setAutoFetchData(true);

        final ListGrid bidGrid = new ListGrid();
        bidGrid.setDataSource(bidDS);
        bidGrid.setAutoFetchData(false);
        ...
    }
That's all SmartGWT needs to fetch data from the server when needed.

Conclusion

Is it really worth all this effort? I would say yes. It is a lot less work than it would be to implement your own server logic, and in many cases you can just use the default SmartGWT data sources. If you like SQL better than JPA, you can use SmartGWT SQL datasources, wich are more flexible than the JPA, but then you cannot use this unit test framework.

This testing methodology has saved us lots of work. It is far cheaper to test the logic in unit tests than to test it manually, fix problems, and restart the application to test again. With this kind of unit tests, most of the server logic just works. The remaining work is to implement and fine-tune the user interface. And of course, the unit tests keep our code working.

It is actually not just a testing methodology but a requirements methodology also. Using examples and objects makes the communication between clients/analysts and developers crystal clear. It reveals misunderstanding and missing details so that the developers get all the information they need to test and implement new functionality.

Object cases make it easy to design tests, and the tests makes it easy to focus development and figure out what to do next.

Thursday, June 2, 2011

Back to the agile values

In recent years the term agile has become overused. Many seem to think that if they have have unit tests, standup meetings and burn-down charts, they're agile. All these practices are good, but they don't necessarily make you agile. Even iterations or some kind of certified master don't necessarily make you agile.

So what is agile? If I were to sum it up with one word it would be communication. Communication is everywhere in the agile manifesto:

[We value] individuals and interactions over processes and tools.
I see this as a reaction against processes like RUP that felt like a software development factory where developers were replacable cog wheels. Agile recognizes that it's individuals with intelligence, creativity and drive that make a project succeed.

But individuals are not working in isolation, they need to interact with others. Interactions means communication. Not one-way communication but interactive dialogues. Misunderstandings are inevitable in communication. When you say or write something, it is almost certain that the receiver will misunderstand something. You can't just send someone a document and think they will understand what you mean. You need to verify what they understood, and the best way to do this is in a face to face conversation.

A key part of agile is to have close communication within the team and between the team and the customers.

[We value] working software over comprehensive documentation.
Documentation is a form of communication. Some teams stop writing documents "because that's not agile", but that's a huge misunderstanding. Agile does value documentation, but it values working software more. Working software demonstrates progress better than completing a number of documents, and it demonstrates the team's understanding of the requirements better than a requirements document. But documentation may be useful to explain what you were thinking when you developed the software.

I said that iterations don't necessarily make you agile, but iterations are definitely needed to be agile. Iterations is not a goal in itself, the purpose of iterations is to improve communication with the customers by getting feedback often. It is inevitable that we misunderstand what the customers need. Iterations help us to discover these misunderstandings early, before they get too expensive to fix.

[We value] customer collaboration over contract negotiation.
Collaboration certainly means communication. The development team needs a positive dialogue with the customers, and not just communicate with formal documents.

[We value] responding to change over following a plan.
This may not seem like to be about communication, but actually it is. Where do the changes come from? From the customers. The customers and the team should communicate often, not just up front.

These values are the basis for practices like on-site customer, iterative development and pair programming. It's the values that make you agile, not various practices. The practices vary depending on the size and complexity of the project.

Monday, May 16, 2011

The problem with use cases

The greatest benefit I get from use cases is that they focus on the user. Use cases help me to think about what the user wants to do instead of only focusing on implementation details.

The biggest problem I have with use cases is that they are not structured. They are basically free text. For instance, if we have a use case Withdraw money from ATM, we may define that it has a precondition that Open account is performed, but we don't get any help from the method to see that.

What happens if someone later changes the Open account use case or defines a Close account use case? How do we find which other uses cases that need to be modified? We can look through the old use case diagrams and find dependencies, but I can almost guarrantee that these dependencies have not been maintained after they were initially created.

The solution to this is to connect the use cases to an object model. I don't mean a use-case realization with view and controller objects like ATM_Screen and ATM_Session, I mean the model objects from the problem domain, such as Customer, Account and Transaction. These are the terms you use when you discuss the functionality with actual users.

If you model the Withdraw money from ATM use case as objects, it could be something like this:
  1. You have an Account object with balance $1000.
  2. You insert a card in an ATM and enters the pin code 1234.
  3. You withdraw $100 from the ATM.
  4. Afterwards, the Account balance is changed to $900 and a new Transaction is created.


To find dependencies between this use case and other use cases, we ask where the Account object and balance came from. This is something to discuss with the client or end users. They may answer that you can open an account and deposit cash.

The Open account use case can be modeled as follows:
  1. The customer gives name and address and shows id to a teller.
  2. Create an Account object.

The Deposit cash use case can be modeled as follows:
  1. You have an Account with balance $0.
  2. The customer gives the account number 12345678 and $1000 in cash to a teller.
  3. The Account balance is updated to $1000.

There you have the dependencies between Open account, Deposit cash and Withdraw money from ATM: They all use Account objects.

Analyze objects to find more functionality

We can continue with our analysis by asking questions such as:
  • Can the balance be changed in any other way than Deposit cash and Withdraw money from ATM?
  • Where did the Account number come from? Can it be changed?
  • How is the pin code set?
  • Can Accounts be deleted?
  • In which ways can you read Account numbers, balances and pin codes?
  • How can Transactions be created, updated, read and deleted?

The questions above deal with creation, deletion, update and reading of domain objects. They will help to discover more functionality, which can be analyzed further, and so on, until you have a very complete functional specification.

There are a few more questions that can be helpful to discover more functionality. They are based on 4 kinds of classes: People, places, things and events.
  • Ask which people will perform the various functionality. This will help us to model users, roles and privileges.
  • Finally, ask which information to store about user actions (events). Transaction is such a class that stores information about changes of Account balance.

This method is a fast way to come up with lots of requirements which would otherwise take days or weeks of brainstorming. It is similar to use cases in that focuses on what the users can do. But it is different in that it models this as objects. Therefore, I am calling this method object cases.

What exactly is an object case?

An object case is a use case scenario connected to domain objects. It consists of 4 parts:
  1. Initial objects
  2. Input
  3. Output
  4. Result objects

The input, output and objects are specified with concrete values. In my experience, this improves communication with clients and end users, because it reduces misunderstandings and help to find missing details.

Object cases are also a great help for developers; they have all the information they need in the object case and don't need to ask for clarifications so much.

Testers can also benefit from object cases. Instead of writing test specificiations based on use cases or some other abstract requirements, the object cases are ready to test.

Objects can also be used for project planning: Develop the functionality around a domain class together. Start with the function that creates objects of that class.