Lately, I’ve been working on a geographically distributed clustered web system and got several ideas I would like to share. All of them are written by blood, sweat and pieces of lost dollars.

Don’t rely on stability
“Yes, this is obvious” - you may say. But I was knowing this from start the project also. However during deployment and preproduction testing the life has shown many new tricks to me. Building network system all conscious developers will introduce some retries/recovery routines. However, absence of components stability has many implicit consequences.

(more…)