Over the last few days I have been going through java build hell. The project in question is a neat micro finance solution called MIFOS. It’s current build is a mix of JSP,SQL Hibernate, Struts glued together with maven. Every thing checks out and is typical. But what is not typical is the testing framework. I think the intent behind extensive tests was well, but the outcome for the devs at least now is a horror story. Every time you do a MIFOS build it goes through a series of extensive integration and unit tests. And I don’t know any apparent way to avoid those tests and it takes awful lot of time (so far without any success on average of 90 mins, I just had a successful build last night and it took whooping 243 minutes, 10 times more than it should in normal cases)
The thing which I was doing wrong from the beginning was to use 64 bit JDK as I had not read the later sections about problems with 64 bit in the install guide. After few failed attempts and messing around with the GC Overhead workaround as well as adding more RAM to MAVEN_OPTS I got to 32 bit JDK and at least I was able to make it till the next step (ie selenium and acceptance tests). 
As of now the build gets stuck here and I am not able to make a significant progress.I can finally get a successful build, but its too slow to do any thing. I will be looking into speeding it up.
I think the whole approach to unit testing needs to be revised in MIFOS. There are probably too many tests in the entire system and they could be modularized. It wont be easy just separating out business logic in a production system, but clear steps need to be taken so as to reduce the build time for Mifos. This is probably the biggest hurdle for MIFOS community, to attract good developers you need a workable codebase and 10+ mins of buildtime is not a workable codebase in my opnion.
I will be looking into making the build run faster on my own system first and then perhaps see what I can do to make the whole testing thing better. Here is the build summary of my latest build
[INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] ------------------------------------------------------------------------ [INFO] Mifos - Parent ........................................ SUCCESS [2.626s] [INFO] Mifos - Common ........................................ SUCCESS [27.094s] [INFO] Mifos - Test Framework ................................ SUCCESS [14.122s] [INFO] Mifos - Service Interfaces ............................ SUCCESS [7.460s] [INFO] Mifos - User Interface ................................ SUCCESS [13.232s] [INFO] Mifos - Application Programming Interface ............. SUCCESS [4.165s] [INFO] Mifos - Service Provider Interface .................... SUCCESS [1.453s] [INFO] Mifos - Application ................................... SUCCESS [1:36:39.031s] [INFO] Mifos - Acceptance Tests .............................. SUCCESS [2:25:11.122s] [INFO] ------------------------------------------------------------------------ [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESSFUL [INFO] ------------------------------------------------------------------------ [INFO] Total time: 243 minutes 1 second [INFO] Finished at: Tue Apr 20 07:36:19 CEST 2010 [INFO] Final Memory: 87M/380M [INFO] ------------------------------------------------------------------------
Did you try barrier=0 in the options for the filesystem where you’re running the build?
@Adam
Tests were run on my laptop.
I will try barrier option now (i tried to get mysql working on ext3 based drive earlier, barrier option is dangerous for filesystem as some blogs would claim)
I finally got mifos build working under CentOS 32 bit
Actually the build time was halved to ~28 mins when I had all the deps already downloaded in the .m2 folder.
Right on, that sounds better! Thank you for following up with your findings.
I was just talking with some folks from ThoughtWorks that are currently visiting Seattle. They also expressed extreme dismay at the length of time required to run our tests.
I feel like we need to shoot for 90% of our tests being unit tests, 9% “integration” tests, and 1% “acceptance” (functional, web-based) tests. This requires sweeping changes to the code, but we need to get there.
In addition to improving tests, we need to do /less/ in Mifos. We can provide functionality via plugins and extension points, but the “core” of Mifos needs to be simple, robust, and maintainable. We’ve been working on a “technology plan” that covers a large part of how we’ll reach that goal. Drizzle (MySQL fork) is a good example of this kind of refocusing and restructuring of a large codebase.
We need your help improving our tests!
Yeah any time I find that a build time for a project exceeds 10 mins, I start finding how it can be split.I am interested to know that how much is the max amount of time that ThoughtWorks people would allow for build (for a project such a mifos)
Also integration tests which hit the db should be minimized as first priority. As for GSoC we (me and my team partner) will try our best to include tests which are absolutely necessary and add value to the system.
If you haven’t seen it already, check out my post to the mifos-developer mailing list with “faster Mifos integration tests” in the subject. I bet you could get your integration tests down to 8 minutes, too.
Doesn’t solve the larger test maintenance issues, but it doesn’t hurt to have faster builds until we get that stuff fixed, too.
@Adam
thanks a lot, I have been busy with my thesis work this week. I will chckout MIFOS over the weekend and see how much time it takes now. Ofcourse with under 10 minute build time people will (eventually) start complaining as ask it to be under 5 min