Søvnløse tanker

Det er sgu ikke nemt altid. Man vil gerne alt, men når kun så lidt. Man håber på det bedste, men frygter det værste. <lorteindledning>

I dag var jeg til gymnastikopvisning. Min datter Freja var på eliteholdet. Jeg blev helt orange af stolthed (ja, jeg kender ikke farven, men det er i hvert fald ikke grøn som i grøn af misundelse). Hun lavede et meget flot split. 

Imorgen skal vi holde børnefødselsdag. Det er nok derfor jeg ikke kan sove. Øv. 

Jeg kom til at sige "Giv den gas!" til en meget travl (storløbende) far i regntøj, som jeg både mødte på vej ind og ud af centeret i dag. 

Årets julegymnastikopvisning er ret underholdende på mange måder. Det er jo hele hr og fru Tranbjerg, der er samlet på godt og givtigt ("godt og ondt" udtrykket stemmer ikke overens med den forfærdeligt vigtige management trend for tiden - vist kaldet: "positiv psykologi" ... Jeg ved ikke om det er samme trend der har fundet på at man skal spørge stressede medarbejdere om følgende: "Har du nok at lave?"..."klap i grøntsag!" Er det eneste svar jeg lige kan komme på! ... teksten: "Nu er den her parentes vist ved at tage overhånd!"... .)

Nå så da dada. 

Der er alt for mange udtryk, som er svære at tegne. Jeg nævner i flæng:

*Ben i næsen (Ser to sko hænge ud af næsebor)

*Kamme over (Ser en kam...?)

*Fik ben at gå på (Ser et juletræ med ben at gå på)

*Hvile på Laurbærene (bladene?) (Ser Master Wu hvile på en fin sammenfejet svævende bunke laurblade)

*En skygge af sig selv (rekursivitet?)

*lige til øllet (Øl-krus med afpustet skum?)

Ode to Dave Thomas

Most people will not find this post interesting. In fact, I think about one person, in the whole wide world, will do so. And possibly he won't find the time to actually read it.

I attended speakers dinner at this years GOTO; conference in Aarhus. I did not speak at the conference nor was I part of the crew or an official track host. So you could argue that I shouldn't have. 

Anyway I did (thanks to Liv Beswick Skov). After a nice meal, the rest of the party was held in a kitchen / cantina. It was a very relaxed atmosphere and everyone seemed to talk to anyone. 

At one point I observed that Dave Thomas started talking to a bunch of crew members. I liked the idea, that this Guru kind-of-guy talked to these young people. The party went on and I found a 23 year old (Rum) too good to mix with cola.

Dave is right in the middle of the picture. Crew folks typically in green.

About half an hour (seemed like hours) later, I realised, that it was as if 5 people had been frozen in the same position. Dave Thomas and the same bunch of young computer science students, that is. This stamina from an, how should I put it, grand old man proves to me, that he is more than just big words on a stage. He wants to influence young people and maybe even: he wants to make the world a better place.

Dave Thomas in the same position. At least one crew person repeated compared to picture above :-)

The Smallest Distributed System

I attended the "The Smallest Distributed System" talk by Mathias Meyer at this years GOTO Aarhus, and I quite liked it. The speaker got his message through to the audience and he touched upon a lot of the important points to remember when doing a distributed system. The points were nicely exemplified, by presenting his experiences with the Travis-CI system and the problem this system faced growing from 700 builds per day in 2010 up to currently 70000 builds per day. The main lessons that he advocated for were the following:

Visibility

You need to be able to monitor your system and at real-time be able to tell what is broken. When you get this visibility, it will lead to the need of responsibility because you need to react to the problems you see in production. The visibility obtained in the Travis-CI system led to a bunch of restructuring and refactoring of the system. Time-outs against external APIs (e.g. github api) were changed from 10 minutes to 20 seconds, and a retry mechanisme was introduced enabling the system to fail fast and also to ignore when non-important requests failed.

Uncertainty ... Modularity

You need to deal with the uncertainty associated with running a distributed system in production. There will always be something failing or performing badly somewhere in a complex distributed system. To be able to deal with this, the codebase need to be well-structured with simple and small dependencies between the different modules in the system. In the Travis-CI codebase, they had (and probably still have :-) a big ball of mud, where everything heavily depend on the travis-core module. So in short to deal with complexity and uncertainty you need modularity.

Simplicity

One of the problems in the Travis-CI architecture was the logging module and the need for ordering logs to be able to display and persist them in the correct order. The first implementation had to synchronise log events to ensure the ordering. Changing the solution slightly by having each log entry know its order, within the full list of events in the a build job, simplified this radically. Having the log entries know their order, made it possible to scale-out the logging functionality. So simplicity made it easier to scale. Simplicity also made it easier to reason about what for instance went wrong when having a breakdown i production.

Mathias concluded by mentioning that in the Travis-CI system, they still have a problem with scalability because of the relational Postgres database being a bottle-neck.

Q&A

The most interesting part of the session actually was the Q&A part. Here is my take on the questions and answers:

Q) What book can you recommend on this topic?

A) None that I know of. I read papers instead. You can find reading lists on various blog posts. Leslie Lamport (http://research.microsoft.com/en-us/um/people/lamport/pubs/pubs.html) has a written numerous papers on the topic which can be found on the Microsoft Research web page. Steve Vinoski (the Track Host) recommended the work published by the Distributed Systems Research Group at MIT and also Michael Nygaards Release IT book was mentioned.

Q) Are you using circuit breakers?

A) No but we are considering it and will probably implement it in the future. Matthias explained the concept, you can read about in Michael Nygaards work. In short it is a pattern that when implemented open and closes the connection to a service based upon a heuristic about how many erroneous responses the service has returned.

Q) Have do you share a temporal clock between the modules in the distributed system?

A) We don't. We have isolated the responsibility of incrementing counters for ordering log events within one module.

Q) (my question so the app actually works :-): Wouldn't it be obvious to use a NoSql database [like for instance Riak] (added by trackhost Steve Vinoski) for persisting the log-events?

A) I'm actually quite fond of Postgres, but yes - it could be a good idea to add a NoSQL database like for 
 instance Riak. Because of the isolation of different services in the current architecture it would be quite easy for us to exchange the current database with a NoSQL variant. But introducing a distributed NoSQL database also introduce an extra distributed system and thereby it also extra complexity a sources for failure.

Q) What is the Gatekeeper doing? (see slides for more details on the architecture)

A) The Gatekeeper transforms a commit into an executable build job

Q) How do you collect metrics?

A) We use Librata metrics with a custom collector. We need to improve it. We have assigned a unique UUID to each build job request and thereby making it possible to track its log entries through the system.

Q) Does all of Travis-CI only run on AWS?

A) No it actually also runs on EC2 and Heroku and we also have dedicated hardware for the system. L

Constrained Innovation

Abstract

My claim is that when you constrain people it fosters creativity. Let us dive into some exciting examples.

Danish Dogme

Lars von Trier made a complete fool of himself by throwing the dogme manifesto at participants of the Cannes festival in 1995. The idea was to constrain moviemaking with a set rules (Dogme 95). The hope was to catch a glimpse of reality. This set of rules started the Dogme film trend. Together with his three "disciples" the dogme-brothers made the movies: Idioterne, Mifunes sidste sang, Festen and The King Is Alive. This trend made movie-makers around the world rethink movie-making and at a point, it was almost considered inappropriate to not make handheld movies. I am a big von Trier fan so I could babble on about his greatness for a while but I will spare you :-)

Kashmir's constraints experiences

I'm also a big Kashmir and Kasper Eistrup fan. Kashmir's latest album E.A.R was created with a set of rules around the magic number 12. The rules were:

A. Record the album within 12 months
B. 12 tracks
C. 12 Instruments (max)
D. Release-date: 2012-12-12

At first you would assume that these constraints would make the creative process of writing music stop. It actually had the exact opposite effect! The rules resulted in the band having a lot of extra tracks (rule B violated). Reviewers loved the album which was released 2013-18-03 (rule D violated). Actually, I reckon, that the band only fulfilled two of the rules at the end, namely rule B and C. A danish blogger describes this process beautifully - Ørevoks

Gaming

Another example of how constraining people makes them more creative, is team building games like mashmellow-towerbuilding (The mashmellow challenge). My experience is, that these constraints make people extra creative and, if lucky, there is chance that this bubbling creativity lead to ideas. Ideas which in the end, if even more lucky, lead to innovation and when the probability approaches the improbable it might also lead to business value.

One of the talks, I still remember, from the GOTO Copenhagen 2012 conference was the Playmaking - Transforming Work Through Play talk by Portia Tung. The talk was designed with a bunch of games and she concluded that gaming is necessary for human beings. I could not agree more. When my life becomes dull, it is typically because the "play vs. no-play" ratio of daily activities is to low.

Boiling ideas

In conclusion

All the above exemplifies the general thesis, that when you constrain people it fosters creativity. The illustration above is my (best effort :-) attempt to illustrate what happens, when you place two professors in a large tea pot and start boiling the water. Hopefully they come up with an idea that prevents slow and painful death.

At this year's GOTO conference in Aarhus, I am looking forward to The beauty of Constraints, Faruk Altes talk. My hope is that Faruk will enlighten me further on this subject or even better he will surprise by talking about something (completely) different.

Agil...hvad er det lige det er chef?

Indledning

Lad os nu være lidt realistiske og indse, at der ikke findes nogen opskrift på god softwareudvikling. Ethvert forsøg på at beskrive hvordan den rigtige måde at udvikle software på vil altid kun være god i under bestemte forhold. Hvis man forsøger at beskrive hvordan processen skal håndtere alle mulige projektsetups, så går processen hen og bliver for tung. Derudover vil dem der skal fortolke processen have problemer med at finde ud af hvordan de skal gøre i en given situation. Her fejlede CMM big time og SCRUM er lige ved at fejle lige så fælt

I stedet skulle vi måske forsøge at fokusere på nogle ting der er gode at gøre.

Tidlig brugerinvolvering

Det ved jeg ikke noget om, men mange siger at det er en god idé og jeg arbejder lige nu på et projekt hvor jeg oplever dets fordele og ulemper. Ja, det er jo egentlig lidt en løgn men jeg ved at jeg taler på kundens vegne når han også opfatter sig som bruger af produktet.

Parprogrammering

Det er efterhånden et accepteret faktum at parprogrammering giver bedre kodekvalitet. Derfor gør parprogrammering det næsten unødvendigt at gennemføre efterfølgende peer-reviews af koden.

Cross-functional teams

Jo flere forskellige interessenter der er tilknyttet samme team eller endnu bedre sidder i samme rum jo større sikkerhed er der for at de oplever en success. Successen kunne f.eks. være at projektlederen blev glad, product owneren blev glad, salgsafdelingen blev stolt over det deres firma producerede og sidst men vist nok desværre ikke mindst kunden blev glad. Glemte jeg brugeren?

Pragmatisk Test-drevet udvikling

Jeg har det med testdrevet udvikling som jeg har det med sikkerhedsseler og cykelhjelme. Det eneste problem med TDD i forhold hertil er at TDD også kræver at man gør noget og bruger nogle værdifulde tastetryk. Lad mig sige det på en anden måde når jeg er ude på dybt vand og ikke føler, at det også gælder om at være en rigtig mand, så glæder jeg over den lille smule disciplin jeg trods alt til tider har. Det gør jeg fordi TDD sikrer mig en forudsigelig vej igennem dagen som kodeko / programmør / Senior Systems Engineer / Lead Developer / menneske. (så fik I også mit nogenlunde ærlige syn på titler)

Retrospektiv processforbedring

Retrospektives er gyldne muligheder for at et team eller en organisation kan få luftet ud og forbedre sig. Jeg har selv faciliteret en del retrospektiver og jeg synes tit de giver en del. De minder lidt om CMM's milestone reviews bortset fra at man typisk havde halvårlige milestones når man var lidt fremsynet :-)

Change by evolution not revolution

Problemet med Scrum tilgangen til agil udvikling er at den er revolutionær. Enhver ændring af en organisation er nødt til at ske evolutionært fremfor revolutionært. I Scrum bruger man meget ordet *skal* og beskriver nogle roller som har meget specifikke ansvarsområder og som har et meget fast defineret interface imellem hinanden. Det minder mig om en af de jobtitler jeg har grinet længe over (og som jeg senere faktisk også stødte på i SCBCD certificeringen) "Deployment Manager". Man kunne også kalde ham mr. Hudson eller mr. Jenkins for at være lidt med på beatet og samtidig en smule Oracle ditchene.

Bedste tips 2011 (top 3)

De tre bedste tips jeg har fået i år 2011:

1. Man kan afmelde telefonreklamer! På borger.dk kan man udfylde en formular og så stopper de fleste faktisk med at ringe
2. Strammer dine Nike Lunar Eclipse også lige omkring der hvor man binder sløjfen? Så stå så knæet hænger udover din fod når du binder skoene. Det fordeler trykket bedre. (Tak til @mwldk - Martin Westergaard Lassen)
3. I en presset situation så brug aldrig argumentet: "Jamen, jeg stemte på hende med de bedste bryster"

Æblemand