Refactoring can be easier in Groovy than in Java

I love Groovy for the productivity, but I am sceptical to maintenance and refactoring of large code bases in Groovy. Without type safety, you cannot detect refactoring errors at compile time, so you need very high coverage with unit tests.

But the other day, I experienced something that got me thinking: I have a few Java classes and some Groovy scripts that call the Java code something like this:
def var = javaObject.func()

I changed the return type of the Java method, and the Groovy code just kept on working. I didn't have to change it at all. Why? Because the Groovy code did not know the type I was returning. The dynamic nature of Groovy made refactoring easier, not harder!

How can type safety make refactoring harder? Because type safety leads to duplication of information. In Java, the code above would look like this:
String var = javaObject.func()

The return type "String" would be duplicated in calls to this method, in interfaces that specified this method, in subclasses, in variables, and so on. Not so in Groovy. The return type is declared only in one place.

I always wondered why we designed programming languages different than databases. In databases, we try to avoid duplication of information. Duplication is a maintenance nightmare. But in programming, we duplicate information all the time. Not only with return types, but with boilerplate code, design patterns and class, variable and method names. In databases, we define immutable primary keys instead of refering to rows by values that could be modified. But in programming languages, we refer to everything by name. So when we want to change a name, we need a complicated "refactoring" tool to replace this name everywhere it occurs.

I dream of a programming tool that let my type code like: "x = myObject.func1()" but under the hood it would create references to "myObject" and "func1" instead of resolving that at compile time. Then, there would be no "rename refactoring", just change the name and the reference would still be valid. The text should then be updated automatically.

How to test Java Persistence API in 5 minutes

I am the author of HiberObjects and want to give an example of how useful it can be.

Yesterday, someone asked me a Java Persistence API (JPA) question: If an object is removed from a bidirectional relation, do both objects have to be saved to update the database correctly?

I thought I knew the answer, but better test it first. This is how I did it:

1) Design a class diagram with 2 persistent classes User and Membership and 1 unit test UserPersistentTest:



2) Initialize the database with some objects that I can test against: Right-click on the UserPersistentTest class and select Design Objects, then design objects like this:



3) Save the diagram. This will generate JPA classes, a unit test and a helper class. The unit test will initialize the database with objects like this:

   private void initObjects()
   {
      EntityManager entityManager = persistenceHelper.getEntityManager();
      EntityTransaction tx = entityManager.getTransaction();
      tx.begin();
      user = new User();
      membership = new Membership();
      user.setName("Lars");
      membership.setX(100);
      entityManager.persist(user);
      entityManager.persist(membership);
      user.getMembership().add(membership);
      membership.getUser().add(user);
      tx.commit();
      entityManager.close();
   }

4) Implement the test method:

   public void test1() throws Exception
   {
      EntityManager em = persistenceHelper.getEntityManager();
      em.getTransaction().begin();
      User user1 = (User) em.createQuery("from User").getSingleResult();
      assertEquals(1, user1.getMembership().size());
      
      Membership membership1 = (Membership) em.createQuery("from Membership").getSingleResult();
      assertEquals(1, membership1.getUser().size());
      
      assertSame(membership1, user1.getMembership().iterator().next());
      assertSame(user1, membership1.getUser().iterator().next());
      
      user1.removeMembership(membership1);
      assertEquals(0, user1.getMembership().size());
      assertEquals(0, membership1.getUser().size());
      em.getTransaction().commit();
      
      em.getTransaction().begin();
      User user2 = (User) em.createQuery("from User").getSingleResult();
      assertEquals(0, user2.getMembership().size());
      Membership membership2 = (Membership) em.createQuery("from Membership").getSingleResult();
      assertEquals(0, membership2.getUser().size());
      em.getTransaction().commit();
   }

5) Run as JUnit test:



This demonstrates the answer to the question: You don't need to save any of the objects explicitly if you change them in a transaction.

My favorite Eclipse view

Have you discovered the Display view in the Eclipse debugger? It is my personal favorite. In it, you can execute any Java code in the context of a running debugger.

To open this view, select Window > Show View > Display.

When the debugger hits a break point, you can type some Java code in the Display view. The Java code can use the local variables in the currently selected frame in the Debug view.

To execute the code and display the returned value, push the button with a "J":

If you just want to execute some code that doesn't return a value, push the button with an arrow ">" and a "J":

The standard output will be printed to the Console view:

This is incredibly useful! You can for instance run some code in a debugger and run some complex debugging code, or prototype some code for a new method.

World War 2, Rationalism and Agile Development

One of the central values of the Agile Manifesto is Responding to change over following a plan. This doesn't mean that it's not agile to make a plan. You should have a plan, but you should realize the limitations of the plan.

Are there any parallels to this in history?

The Soviet Union had 5-year plans. Stalin was quite succesful in industrializing the rural country in a short span of time. But his methods did not work well in the battlefield. In the 2nd world war, the Soviet suffered severe losses in the beginning.

You can plan how to build and manage a factory, because the complexity is limited. But a war is far more complex. You cannot predict what the enemy is going to do. This doesn't mean that you should go to war without a plan. Far from it, you should plan as much as possible, but be prepared to improvize when something unexpected happens. As Dwight D. Eisenhower, a successful American general in World War 2, said: "Plans are nothing; planning is everything."

Building software is also complex. But some managers try to run a software development organization as a factory. They treat their employees like cog-wheels. They think that Employee #1093 can take over the responsibilities of Employee #835 with just some training. Or that 4 programmers can do in 3 months what one programmer can do in 1 year. It doesn't work like that. If software development was so predictable, then robots could do it. But they can't.

I don't like being treated like a cog-wheel. I am not Employee #1093, I am me. I am unique with my own weaknesses and strengths. If I am treated like a replaceable part, I may start to behave like one. Then, I will only do as I'm told, nothing more, nothing less. If nobody listens to my ideas, I will stop thinking creatively, and just do things the way we always did.

Since I don't want to become mediocre, I avoid environments like that. I seek challenging environments where my ideas are heard and I am given responsibility to work as I think is best.

I think that Churchill, the Prime Minister of Great Britain during WW2, was agile too. He was not afraid to change. He said: "To improve is to change; to be perfect is to change often.

He also said, "However beautiful the strategy, you should occasionally look at the results." That sounds like iterative development to me. You make a plan, and then review the results now and then to see how it works. (See more fantastic quotes of Churchill here.)

So how about Hitler? How agile was he? Not agile at all. He had a great vision and a plan that seemed to work very well in the beginning. But as we all know, he failed. I think this is similar to a waterfall project: He was 80% finished with conquering Western Europe, and 75% finished with Eastern Europe. If he had run the war more iteratively, he should have completed Western Europe first, and then started on Eastern Europe. Thank God, he didn't!

What did Hitler and Stalin have in common? Both nazis and communists were rationalists with a strong belief in their own enlightened minds. I dare say that Churchill and Eisenhower had a more humble attitude, accepting that they could not make perfect plans. I would also say that Churchill was empiric in that he wanted to review the results of his plan.

Rationalists think they can plan everything. Empirists believes in experimental evidence. I think that waterfall development is a result of rationalist thinking, but it doesn't work well, because our minds are limited. We cannot predict everything, but we are capable of planning to a large degree. Therefore, I believe iterative development where you make a plan but also review the results regularly is the most effective way.

PS: I am not a historian nor a philosopher, so please, enlighten me if I got something wrong.

Use examples to make your code easier to understand

Programmers are used to abstract thinking. To program is to generalize: A method is a general specification of what to do during execution. A class is a general specification of objects. A superclass is a generalization of several classes.

Altough our minds are capable of abstract thinking, concrete thinking is much easier, and concrete examples are the foundation for abstractions. For instance, when we were children, our parents didn't try to teach us about cars by explaining to us cars are and what they can do. Instead, they just pointed at a car that was driving by and said ”Look, a car!” When they had done that a number of times, we knew what a car was.

Another example is prejudice. We all have prejudices, because this is the way our minds work. If we have met a few people from Denmark in our lives, and those people were friendly, we ”know” that Danes are friendly. And this works even stronger for negative prejudices.

My point is that we learn by examples. Einstein said that examples is not another way to teach, it is the only way.

As programmers, we are highly skilled in abstract thinking and expressing ourselves in programming languages. But when the code becomes too complex, we try to comprehend it by making examples in our minds of what the code would do in different circumstances. We imagine or write down the values of different variables and what the behavior of different tests and loops will be with those values.

And when it becomes too hard to imagine what is actually going on by reading the abstract code, we use debuggers for help. A debugger is not abstract. It gives a concrete description of what is actually happening at runtime. It gives information on how methods execute in specific instances and which objects exist at a specific moment. This concrete information is so much easier to understand than the abstract code.

Therefore, it's a good idea to design concrete examples of objects and scenarios before implementing the abstract classes.

Unit tests are exactly such examples. They make it easier to understand what happens at runtime. And a well designed unit test is the best way to explain to somebody what the code does. If you ever had to take over somebody else's code, you know how hard it can be to understand it. Where do you start? Unit tests is a good place to start.

So, my suggestion is to think of unit tests as documentation. Write unit tests that are easy to read and that explain the intent of the code.

I believe in Java for future web applications

Will Java become the dominant platform for web applications? Actually, I think it has a good chance.

How will the future web applications be? I think we can expect the following:
* Highly interactive
* Complex, more work will be done on the client
* Collaborative, not only for games, but also for other applications
* Mobile
* Location aware

AJAX is fantastic compared to "web 1.0", and technology like GWT (Java) makes it relatively easy to develop complex user interfaces. But HTML and HTTP still sucks as an application platform, something they were never meant to be. I believe that if Microsoft hadn't managed to kill the Applet, Java would be the dominant platform for more complex web applications today.

I think Consumer JRE can be that platform. Java can be used to make complex, interactive applications that communicate with each other client-to-client.

This doesn't mean we have to implement in Java. When consumers have a JRE ready, there is nothing stopping us from developing applications in Groovy, JRuby and Scala and deliver them over the web.

I don't know much about Flash, but I am sure it is at least as good as Java for making flashy applications. But when it comes to complex collaborative and mobile applications, I think Java has the upper hand.

And Java has one more powerful force behind it: The community. The Java community has developed hundreds of libraries and frameworks. Too many, perhaps. But this community is capable of revolutionizing the way web applications are developed in ways we cannot predict now.

Passwords in phpBB 3

Here is a port of phpBB3's password handling to Java.

import java.io.UnsupportedEncodingException;
import java.security.GeneralSecurityException;
import java.security.MessageDigest;

/**
 * Port of phpBB3 password handling to Java. 
 * See phpBB3/includes/functions.php
 * 
 * @author lars
 */
public class PHPBB3Password {
  private static final int PHP_VERSION = 4;
  private String itoa64 = 
"./0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz";

  public String phpbb_hash(String password) {
    String random_state = unique_id();
    String random = "";
    int count = 6;

    if (random.length() < count) {
      random = "";

      for (int i = 0; i < count; i += 16) {
        random_state = md5(unique_id() + random_state);
        random += pack(md5(random_state));
      }
      random = random.substring(0, count);
    }

    String hash = _hash_crypt_private(
      password, _hash_gensalt_private(random, itoa64));
    if (hash.length() == 34)
      return hash;

    return md5(password);
  }

  private String unique_id() {
    return unique_id("c");
  }

  // global $config;
  // private boolean dss_seeded = false;

  private String unique_id(String extra) {
    // TODO Generate something random here.
    return "1234567890abcdef";
  }

  private String _hash_gensalt_private(String input, String itoa64) {
    return _hash_gensalt_private(input, itoa64, 6);
  }

  private String _hash_gensalt_private(
    String input, String itoa64, int iteration_count_log2) {
    if (iteration_count_log2 < 4 || iteration_count_log2 > 31) {
      iteration_count_log2 = 8;
    }

    String output = "$H$";
    output += itoa64.charAt(
      Math.min(iteration_count_log2 +
      ((PHP_VERSION >= 5) ? 5 : 3), 30));
    output += _hash_encode64(input, 6);

    return output;
  }

  /**
   * Encode hash
   */
  private String _hash_encode64(String input, int count) {
    String output = "";
    int i = 0;

    do {
      int value = input.charAt(i++);
      output += itoa64.charAt(value & 0x3f);
 
      if (i < count)
        value |= input.charAt(i) << 8;
 
      output += itoa64.charAt((value >> 6) & 0x3f);
 
      if (i++ >= count)
        break;
 
      if (i < count)
        value |= input.charAt(i) << 16;
 
      output += itoa64.charAt((value >> 12) & 0x3f);
 
      if (i++ >= count)
        break;
 
      output += itoa64.charAt((value >> 18) & 0x3f);
    } while (i < count);

    return output;
  }

  String _hash_crypt_private(String password, String setting) {
      String output = "*";

      // Check for correct hash
      if (!setting.substring(0, 3).equals("$H$"))
        return output;

      int count_log2 = itoa64.indexOf(setting.charAt(3));
      if (count_log2 < 7 || count_log2 > 30)
        return output;

      int count = 1 << count_log2;
      String salt = setting.substring(4, 12);
      if (salt.length() != 8)
        return output;

      String m1 = md5(salt + password);
      String hash = pack(m1);
      do {
        hash = pack(md5(hash + password));
      } while (--count > 0);

      output = setting.substring(0, 12);
      output += _hash_encode64(hash, 16);

      return output;
  }

  public boolean phpbb_check_hash(
    String password, String hash) {
      if (hash.length() == 34)
        return _hash_crypt_private(password, hash).equals(hash);
      else
        return md5(password).equals(hash);
      }

  public static String md5(String data) {
    try {
      byte[] bytes = data.getBytes("ISO-8859-1");
      MessageDigest md5er = MessageDigest.getInstance("MD5");
      byte[] hash = md5er.digest(bytes);
      return bytes2hex(hash);
    } catch (GeneralSecurityException e) {
      throw new RuntimeException(e);
    } catch (UnsupportedEncodingException e) {
      throw new RuntimeException(e);
    }
  }
  
  static int hexToInt(char ch) {
    if(ch >= '0' && ch <= '9')
      return ch - '0';
  
    ch = Character.toUpperCase(ch);
    if(ch >= 'A' && ch <= 'F')
      return ch - 'A' + 0xA;
  
    throw new IllegalArgumentException("Not a hex character: " + ch);
  }
 
  private static String bytes2hex(byte[] bytes) {
    StringBuffer r = new StringBuffer(32);
    for (int i = 0; i < bytes.length; i++) {
      String x = Integer.toHexString(bytes[i] & 0xff);
      if (x.length() < 2)
        r.append("0");
      r.append(x);
    }
    return r.toString();
  }

  static String pack(String hex) {
    StringBuffer buf = new StringBuffer();
    for(int i = 0; i < hex.length(); i += 2) {
      char c1 = hex.charAt(i);
      char c2 = hex.charAt(i+1);
      char packed = (char) (hexToInt(c1) * 16 + hexToInt(c2));
      buf.append(packed);
    }
    return buf.toString();
  }
}