skizz.biz

Last week my ISP deleted my weblog.

I had asked them to downgrade my account from an ADSL account to something cheaper while I am in India. So they did, and in the process deleted the cgi space and MySql database that was hosting the weblog. Partly it is my fault, since (a) I didn’t keep my own backup and (b) I didn’t spell out for them in big block letters : DO NOT REMOVE THE 2 1/2 YEARS WORTH OF CONTENT. You’d think they might notice that it existed before going ahead and deleting it without a backup though. After all, even windows has an “Are you sure”.

So there I was with no backup of quite a lot of personal and professional thoughts and memories.

Fortunately Google has a really cool feature that allows you to see the latest cached version of any site in their index. As you can see if you do this query they archive quite a lot of stuff - my whole site included!

So a few days and lines of ruby later, I now have a newly retrieved webblog, hosted courtesy of my friend Damian Murphy.

Needless to say, I am also writing a script to regularly back it up…

The following is a
description of a system I have recently worked on. It describes the
thought processes we followed as we moved a large part of the code into
a declarative workflow engine. Its here because I think it shows some
of the thoughts that do into the continual design/refactoring of a
system like this. I warn you though - it is quite long :-)

The system was driven by a document workflow. The commands that a
user could perform would depend on the kind of document they were
viewing and also the state of that document.

We had recently moved to a declarative definition of workflow that
defined the Commands that could be performed on a document. When the
user viewed a document the client would call GetCcommandsForDocument
that would return a set of Command objects. Each Command returned would
generate a button on the view. When the button was clicked it would
instantiate that Command and send it back to the server via the
DoCommand method on an ICommandProcessor. On the asp client this would
delegate the call via remoting to the server.

For security reasons we separated Command and CommandProcessor, so
for example an AcknowledgeCommand would cause a CommandProcessorFactory
on the backend to reflectively lookup an AcknowledgeCommandProcessor,
inject its dependencies using pico-style DI (of
course :-) and call DoCommand on the processor. This meant that the
assembly that contained the Commands contained no domain logic code -
that was all in the CommandProcessors assembly.

Nice, but not nice enough.

Our users started asking for particular fields to be populated for
each action. For example, when the Acknowledging an invoice, the user
must correct any ambiguous dates (the invoices came from a document
recognition system and could come from many locales and languages).

At first we started writing and specialising custom Commands for
each action - previously they had been completely generic. As we went
along this path we found that we were dupicating these requirements
across Commands, since different Commands might have the same
requirement (eg. you must enter a comment).

So we abstracted a Requirement object, which encapsulated a single
such requirement. Then when the client called GetCommandsForDocument,
instead of getting a class which it would use to instantiate the
Command, it would get a set of Command instances populated with
Requirements. These were almost like Prototypes but we used them more like Flyweights. The view had a RequirementView corresponding to each Requirement, and we would use the Requirement as a model for the View.

Now when the user clicked on a command button, we would display a
new page (it would be a dialog in a gui) which would show the
RequirementViews backed by their models, Which were in turn populated
from the invoice data (which was in a DataSet. (There’s a lot of
different models here and it can get a bit confusing).

The user would enter data in the fields (RequirementViews) and click on the OK button, which would:
* Ask the RequirementViews to update their models (the Requirements in our flyweight Command)
* Ask the Command to validate itself
* If the Command is not valid, refresh the screen and indicate the fields (RequirementViews) in error
* Fire the command back to the server

* If the Command failed, refresh the screen and show the error (the
server could perform extra validation and would throw an Exception.We
could have returned an annotated Command object in the same way as the
client side validation above and had Requirement level server
validation but we never really needed it)
* go back to the previous screen and refresh it

Now we could declaratively build fine grained gui components from the back end. Our gui views were defined in an XML file that looked something like this:

<workflow type=”foo”>
<state name=”Unassigned”>
<action name=”Acknowledge” permission=”Acknowledge”
requirements=”ValidStartDate,Comment” destinationState=”InProgress”
destinationUser=”CurrentUser”/>

</state>
</workflow>

Here destinationUser is a Strategy - we had things like CurrentGroup, CurrentUser, TeamLeader, AccountsGroup etc.

We could have defined specialised Requirements like :
<validDateRequirement field=”startDate”/>
<validDateRequirement field=”endDate”/>

<commentRequirement field=”comments” required=”true”/>
but we didn’t feel the need so we used the simple attribute based approach above

Requirements became a very powerful abstraction in our system.

Having a declarative workflow also meant that we were able to
generate nice dot graphs for our BAs automatically each build, and
since we reloaded our workflow at each request (inefficient? who cares
- compared to remoting it didn’t slow the system down) we were able to
sit with the BAs, change the workflow xml file, reload the page and see
the changes immediately. It was great to see their faces light up when
two day stories became 5 minute changes to the XML
file - “Sit with me and we’ll do it now. Oh you want the fields in a
different order - hang on, how’s that? Cool? Ok I’ll commit it”.

One question this raises of course is that of how do you (or even do
you need to) test declarative systems like this. Do you need to write a
set of tests for every possible state transition? This seems a lot like
overkill - all you’re really doing is testing the contents of your XML file. But changing a part of the XML file can break the system from the users point of view. But fixing it again is really simple, so do you just give the BA’s
a tool to edit the declarative stuff and let them rework the system
themselves? If they then break the system from the users point of view
then that’s not a bug, its a misconfiguration - but the user still
can’t do their work.

Perrhaps we whould have a set of tests that act like invariants -
“In this state the date must be valid” and build them into the system?
That way the system will still function even if the details of the
Requirements are not necessarily as the user desires, or particular
actions fail due to invariant exceptions. Actually I quite like this
idea because it seems to cut across the declarative part of the system
at right angles. It defines invariants that guarantee thet the system never fails even if particular actions and commands throw errors - its also quite fail-fast.

So in the above example the system would never throw an “InvalidDateException - ‘23 Janvier S004’
is not a valid date” later in the workflow when the date was used for
some piece of domain logic, rather it would fail at the Acknowledge
step with some kind of “InvariantException - The date must be valid in
the state InProgress - This is probably due to a mis-configuration. A
message has been sent to the system support staff.”.

At 8:30 last Friday night I was yet again in the back of an auto-rickshaw off to the Majestic bus terminal, this time to catch a train to Dindigul for a trek to Kodai. We had sleepers for the journey and so the trip was an easier one than last time. I had a top bunk so I could hang my feet out over the end without blocking the passageway, and actually got some sleep!

Naresh woke us at about 5:30 in the morning as we approached Dindigul and we left and found a nice place for breakfast at the bus station. Great dosas and idlis here. We caught a bus for the 90 minute trip to Palani, and then all piled into a van for the last 7km to the edge of the forest.

We followed a path along the top of a wall for about half an hour, which lulled us into a false sense that the trek might be relatively non-strenuous. The day was warming up but there was plenty of shade and it was quite a pleasant walk. As we got deeper into the trees we started to see the unmistakeable sign of elephants hhaving been in the area recently. Fortunately none of us stepped in it.

Eventually the path ended in a small dam, and we started to follow the stream bed up the hill into the reserve proper. The stream bed was filled with boulders ranging in size from 1m to 5m in diameter, and there was no real path on the sides of the hill so we had to climb over all of them. Very soon it felt more like rock-climbing than hiking, and this continued for the rest of the day.

At about 1pm we broke for lunch in a tiny rocky clearing, and made several trips up to the small running stream to fill vessels and refill our water bottles. Again I was happy to have brought my water filter - the water was flowing out of all but stagnant pools made shallow by the heat and lack of rain.

Lunch was papads and Sambar and it was good alhough it took about an hour and a a half with all the gathering wood, cooking and then cleaning and refilling (yet again) of water bottles. So we set out again and re-acquainted ourselves with boulders and their infinite variety of shapes, sizes, handholds and opportunities to seriously injure you if you don’t pay attention.

We were making for a waterfall at the head of a very long valley, and it soon became clear that we were not going to be able to make it all the way. So when we came to a rocky flat area with two waterholes at about 5pm we decided to camp there for the night. We were also quick to go for a swim - the bouldering had been exhausting and the water was amazingly refreshing.

I managed to gain the klutz of the day award by forgetting to take off my glasses and then doing a bomb and promptly losing them to the bottom of the waterhole. Fortunately it was just over 2m deep and I managed to find them with my feet and get them back. It might have been a rather interesting time trying to climb over boulders while blind!

As the sun went down we made chai on the fire, lay down on the rock and watched the stars. And the forest fires. On the peaks on one side of the valley a forest fire was raging. For some time it seemed to be coming towards us down the hill, and then we lost sight of it behind a ridge, the only clue that it had not died out the orange glow of the smoke silhouetting the trees on the ridge.

Owen, who was watching it with me, said that “We’ll know it is coming towards us if that tree on the ridge catches”. Just as he said it a flame reached out from the smoke and lit it like a torch. Within moments the whole tree was a light and the rest on the ridge followed.

Fortunately the wind turned favourable for us and the fire was contained to the same section of the hill for the rest of the night. So we went back to chatting about politics, as some of the others played Anthakshari, a game where each person must sing a line from a song (normally from a Bollywood movie of course). The next person must then sing a line from a different song that starts with the same word or syllable that the last one ended with. Someday I will try it in english…

Finally we turned in and put out the fire - Owen went for a late night swim to rescue Fiona’s sleeping bag that had rolled down into one of the waterholes. I think she will be paying that one for a while. By that time it was *cold*. I slept reasonably well, despite literally sleepng on sloping rock. I did occaisionally wake to the sound of animals. Some of the others heard many moving around in the night, and swimming in the water, but mostly I heard the sound of bats flying overhead, and the occaisional falling log in the bush.

The next day we got up with the sun and Naresh made a breakfast of smashed rice. We washed and packed and compared notes on the animal noises of the night, and then some of us spent half an hour pumping water into bottles while the others started up towards the waterfall.

Start as you mean to continue is always the idea - and the day started for us with crawling through a tiny gap between two boulders onto a ledge covered in red ants. The rest of the morning was spent, as the day before, clambering over massive volcanic boulders towards the ever elusive waterfall. Towards lunchtime we got to an area with ancient spreading lava flows, complete with rounded fumaroles ready to trap the unwary trekker.

Ash drifted down from the many spot fires still smouldering on the steep walls of the valley, and we frequently came upon trees turned to ash from spot fires that fortunately seemed very localised. Finally we came to a clearing with a sandy floor and some other western trekkers who were having their lunch.

Their local guides were shocked that we had slept near the waterholes - they told us that that area was where all the local wildlife came to drink and was quite dangerous at night. In fact, the Elephant droppings that we hadd seen there were from a herd of 20 that had spent the night before us at the waterholes. The thought of what might have happened if they had come back was a sobering one, and the noises of the night too on quite a different significance. (For the record, the animals of the area include: Elephants, Nilgiri langur, sloth bear, spotted deer, giant squirrel, leopard, Indian bison and a variety of birds, although I can only claim to have seen the birds)

The guides pointed us to the path up the hill for the next part of the trek. This was refreshing in that it was a real path, clear and well trodden, and not involving any boulders. However it was also at times nearly vertically up the 500m wall of the valley. Since we were running behind schedule we needed to get foing as soon as possible, and so we skipped lunch and, again, some of us went to fill water bottles while the others went on up the hill.

Unfortunately we had also skipped dinner the night before (it had got dark and everyone was too tired to cook), so we tired quickly up the hill, and it seemed like we had to take a rest every 10 steps. However the views were spectacular, all the more so as we made our way up the side of the valley. As we rose we again crossed over areas which had been burned out - some of them we still hot from the fires. Across the valley another tree was burning and the smoke twisting into the still air.

We stopped briefly for a rest and some food about 20 minutes from the top of the valley, in a small stone overhang that provided some shade. Here we finished the scroggin I had brought, which made some small dent in our rapidly growing hunger. Finally we headed up the last steep pitch through thick trees to the top of the valley wall, and were greeted with a spectacular view back down the valley to the flood plain below.

Then, finally, the bliss of a downhill pitch. All the pain was forgotten and I almost ran down the hill path. This led past several small farms until I started to hear the sound of folk music from a tinny radio filtering between the trees. I came out past a river where the women ov the village were filling pots with water and doing their washing, and finally caught up with the others in a small clearing.

There was an exhausted discussion going on about what to do next - whether to eat and then catch a jeep back to a road where we could catch a bus, or whether to leave immediately and eat in Kodai Road. fiona and I skipped out and walked up the road towards the main part of the village, and sat in the dust with the kids watching the people get on with their lives.

Finally the decision was made by necessity - it was now too late to do anything but get to Kodai Road as soon as possible, so all 11 of us piled into the back of the only remaining jeep and drove up to the main road. There we were told that, being a Sunday, all the buses would be packed. Just to prove it a bus pulled up that was worse than the Northern line at rush hour.

It was clear that we would not get to Kodai Road like that, so we negotiated with a couple of Jeep drivers and headed off. We had 3 hours to cath the tran and the tandard time from that spot to Kodai Road, 110km away in the valley, was about 3 hours. So we were going to be pushing it.

Unfortunately someone told the drivers we were in a hurry. I was in thhe second jeep, and we only had one close call, overtaking a bus as two trucks came towards us and ducking in just at the last moment. The others later suggested that they had only one incident that was *not* a close call.

Belting through unlit towns with pedestrians everyewhere at 75kmph while blowing your horn to warn peple that you are not going to stop may be commonplace here, but its not much fun when you’re in the front seat and not wearing a seatbelt. If I wasn’t so tired I might have been scared.

As it was we made it to the train station by 7:30, leaving us an hour to have some dinner and calm our shattered nerves. Again, the food was excellent - great dosas and idlis and a truly superb tomato chutney. We all ate at a pace that it frightens me to remember - a long days trekking with only a small breakfast really took its toll.

Then it was off to the train and some well-earned rest. I don’t think anyone had any trouble sleeping on the train that night, and when we arrived at 7am Monday morning it was time for a quick shower and breakfast in the Diamond Cafe before going to work.

Phew. This was a much more tiring trek than the one a couple of weekends ago — but more rewarding too, with spectacular views and challenging terrain, and a great bunch of people who it was a pleasure to meet and spend a weekend with.

Yesterday (Sunday) Owen, Fiona and I were invited over to Naresh’s place to celebrate Holi.

You can read more in my life blog here

Yesterday (Sunday) Oween, Fiona and I were invited over to Naresh’s place to celebrate Holi.

Holi is a really cool festival associated with the Spring, and it is celebrated with an “exchange of colours”. What this means is that you buy packets of powdered dyes in specatacular colours, and throw them at each other, or to improve the effect, use water pistols…

As a result I am now stained a vibrant pink colour. I am assured that this will not last too long — a month at most…

It was also one of Naresh’s nephew’s first Holi festivals — so there was a special ceremony involving many women friends of the family who sat in a room singing songs, before taking him to the nearby temple for a special ceremony.

And of course no Indian street is complete without at least one contented looking cow…

On Friday night we met at 10pm at the bus terminal in Bangalore, ready to catch the K.S.R.T.C Rajahamsa Bus to Virajpet, a 6 hour trip. We were catching a Deluxe bus — so we had mod cons like reclining seats which were actually quite comfortable. No reading lights though so it was a case of trying to sleep through the blaring of horns.

We were pretty much on time thanks to the insane driving on the wrong side of the road that is normal here, arriving at about 5:30am. We had to wait until 6:45 for a local bus towards Tadyianda Mol, so we sat on the side of the road watching while the town slowly awoke. At 6am the mullah’s starting calling the morning prayer and people slowly started coming in to open their stalls near the bus station, while the occasional truck blasted through town joining its smoke to the fog.

At 6:15 we were the first customers at a small dosa restaurant — the food was quite good even if we had to rush it a bit to get to the bus. This bus was a more typical local bus which rattled and thumped over the single lane roads, blasting its horn at any corner to warn any oncoming traffic that it was about to be knocked off the road.

An hour later we were dropped on the side of the road about 3kms from the Palace and started our walk. It was quite overcast although still warm so the walking was quite pleasant. We got to the palace after another half an hour and had a quick look around and a rest, although the doors were locked so we couldn’t get inside.

The next 3 hours we walked up the hill, until we reached a place where Nuresh assured us there was a spring. Some of us sat with the bags while the others went to fill up their water bottles. Owen and I stuck with our safe firangi mineral water.

We kept on up the mountain to 1680m — Nuresh confirmed the altitude with his watch. I arrived later at the top than the others because I was straggling with Hema, and when we got there it was to see Owen holding court with some other trekkers who were also developers, telling them about CruiseControl and how we do distributed agile development.

We (well mostly Nuresh) cooked a lunch of sambhar and papads cooked on the coals before drinking the last of our water and falling asleep for 3 hours.

I think this was the point where I got sunburned, as both Owen and I had forgotten the sunburn cream. D’oh.

After about 3 hours of sleep we all started to stir. It was getting towards 4pm and as we planned to sleep on the peak, some of us needed to get back down to the spring to fetch more water. I went to get changed into my shorts as it was getting hot, and heard the tantalising sound of running water. Maybe we didn’t have a two hour round trip before us after all!

I told the others and we started trying to find it by bush bashing, but the jungle was far too tangled right there so I went around the outside trying to locate the source. It seemed to be coming from quite close but as I walked down what was almost a cliff at the side of the peak I couldn’t hear it any more strongly. I did try and get into the jungle again but there were some nasty thorny branches which dissuaded me (and I have the scratches to remind me).

At the bottom of this valley it was not possible to hear the water, so Owen, who had by now joined the search, went across the next ridge to see if he could see it there. We ended up searching for about three quarters of an hour to no avail (except to availing of a large amount of rather nice scenery).

Unfortunately by the time we finished we were not only exhausted again but it was far too late to get back down and back before dark, so we had to pack up and head back down to camp by the spring. We got there just as dusk was falling, and I got a chance to try out my microfilter, which luckily I had bought in Australia over christmas. It must have worked, because no-one got sick. Actually that’s a logical fallacy but I don’t think it hurt to filter the water. And I wasn’t going to volunteer as the control.

Anyway we set up camp, collected some firewood and cooked some dinner just as the sun went down. We chatted for ages about arranged marriages and religion and all those things that you’re never supposed to talk about at dinner parties. and then fell asleep telling ghost stories as the half moon slowly set.

Sunday we woke fairly early, refilled the water bottles (I am so glad we didnt camp far away from the spring) and after a breakfast including a really good sweet tea headed back down the hill. It was much hotter walking than Saturday as there was no cloud cover, but given that it was downhill it was not to bad. And we were lucky in that we only had to wait about 10 minutes at the bottom for the bus back to Vijrapet.

Even luckier was that when we got to Vijrapet, and walked back to the main bus station, we arrived literally 5 minutes before a deluxe bus left for Bangalore. I spent most of the time talking with Hema - this time about hindu religious practices. Really inteesting stuff, and amazing how many similarities there are between humanity’s different beliefs.

It made the trip fly (well no 6 hour trip really flies, but it definitely helped). we got back into Bangalore at about 9pm, and all of us headed back to a well deserved nights sleep.

All in all it was a great introduction to trekking in India.

I have been involved in a few discussions recently about the difference between refactoring and rewriting. And it got me thinking about the ways we characterise what we do.

On most of the projects I have been involved in we have talked at standups about the refactorings we do, in particular those that affect other developers. Often these are in response to new requirements that mean that a part of the system needs to be more flexible.

This is also useful information for the customer or BA, since it means that stories and tasks that concern that piece of the system may suddenly become lower cost, and this may change the sort of stories they think about for the system.

So I think it is entirely appropriate and necessary to talk about these refactorings in a stand-up.

But what is a refactoring? My last project was an invoice workflow system. The first release had a single invoice type and the workflow was defined in code and spread throughout the system. In the second phase (when I became involved in the system) we added 6 new invoice types to the system, each with their own slightly different workflow.

In one iteration we had a collection of stories that were estimated at about 50% of the entire iteration. We had wanted for a while to change (I am specifically avoiding the term refactor here - bear with me) the system so that the workflows were driven form some kind of definition file, and so we said at the planning game that we would put all these cards in one pile and one pair would work on this change.

Note that we didn’t write a separate card for the story. We chose to take the time from real stories. I have always found this to be better than separate cards since it means that you have a stopping point — when the story is complete — and have less tendency to keep changing the system beyond what is required.

We took the same time as 2 of the cards to implement an XML file to drive the workflow. and the rest of the stack of cards then became (literally) a 10 minute job of editing the new XML file. We also spent a short amount of time writing a simple DOT file to allow us to generate a workflow diagram. This became our primary tool for talking with our BAs and allowed us to discuss the workflow at a high level.

So back to the point: is this refactoring?

Many would say no, in the sense that it is a change to the architecture of the system. But all refactorings change the architecture at some level. Are two small refactorings a refactoring? Three? Four? Fifty? If the card takes one day and the build never breaks is this a refactoring? It can’t be the number of files changed, because a rename for an enumeration constant could change many files in different packages.

A problem in some teams with the term refactoring is a belief from the customer that refactoring is worthless. Since by definition you are changing code without changing behaviour, aren’t you wasting my time and money? Of course this is naive since we refactor to make change easier, so in the longer term it saves the customer time and money. But it takes an educated customer to realise this.

Many customers might reject the change — which we characterised as a refactoring — that I have talked about above. We were able to do it, and able to be honest about it, because we had gained our customer’s trust by openly discussing, delivering on and proving the benefit of smaller refactorings.

So whether you characterise these sorts of changes as refactoring or not is a grey area, and depends mostly on trust. If your customer is comfortable with it, call it a large refactoring. If not, call it a change or rewrite and you will need to spend more time justifying it to the customer.