Latest Entries »

It so happens that a colleague was chasing me for about the 3rd time on a piece of work I am struggling to complete. The problem is not that I can’t get my head around it, or that I lack the skills, or even that I’m waiting on someone else to complete their part.

It’s interruptions. Pure and simple.

So with this in mind, I put together the following reply:

Dear Colleauge,

Whether I can complete the module in time for next week depends on how many of the following interruptions I get this week while I am duty 3rd line engineer:

Major Incidents:rescue2 Mentoring 2nd line:panic
Oh, and last week there was a delay because I was on holiday:
So sorry it’s late, but we need to do this right over doing it fast, we don’t want this:
We need precision:


Yep! Wrong, wrong wrong!

They really are

So, by the Monday morning following the last insane week, I was pretty
shattered, my sleep pattern having been blasted to hell and I always tend
to be a little on edge when on call, never knowing when the phone might
ring again. It’s just not conducive to sound sleep. So, I crawled groggily
out of bed around 7am, grabbing the first set of clothes I could find and
headed out on my bicycle in the dark for the train station.

Unbeknownst to me, the trousers from that set of clothes were a pair
which, two weeks ago, I had discussed with my wife regarding a rip in the
seam; she had examined them and pronounced them beyond repair. They had
been lying on the top of the chest of drawers awaiting disposal. So, I was
now wearing the clothing equivalent of a ticking bomb. Wallace and Gromit
– Wrong trousers, eat your heart out!

Suffice it to say, that at some point during that day, I sat down heavily
in my chair, snagging the pocket of the already terminally weakened
trousers on the chair arm. This caused the rear seam to rip – a lot, and
as I stood up quickly to try to stop the tearing, I worsened the situation
terminally, ripping the seam all the way from my arse right down the leg
beyond the knee. Realising my plight, I ran for the toilets, pausing only
to grab a stapler, hoping I could effect some kind of temporary repair
sufficient to last long enough to get me to a shop.

I sat in the toilet, trousers off, waiting for people either side of me to
go out, fearing the noise of the staple gun would give them the impression
of some weird perversion – so I sit listening to the various noises,
waiting for a flush or dryer to mask my stapler clicks. I seemed to have
hit peak hour, and there is a bloody queue of people  using the cubicles
either side of me, but eventually after about 20 minutes and 200 staples I
think I’m in with a chance, so I put the wretched trousers back on and
leave the cubicle.

Unfortunately the material was so frayed that the staples gave up without
a fight, and as I emerge from a meeting, a colleague leans over saying
“you do realise, your trousers are split all the way from your arse to the
floor…” as I leave a trail of small metal objects in my wake.

After reaching my chair and taking cover there for a while, I’m starting
to think in terms of how I can roll the chair all the way home, and how I
can get onto a train with it. Eventually I run for the loo again, this
time collecting a roll of sticky tape as well as the stapler to use as a
sort of reinforcement. Wish I had some duck tape. Anyway I reach the loo,
same ritual as before except worse this time, because I now had the added
pervy sound of tape peeling off the roll, but eventually I emerge, and I
am seriously dreading what anyone in the adjacent cubicles is thinking
(actually I’ve given up now, so it’s pretty much click – screeeeeeaapsh –
well, that’s the best approximation I can give in words to sellotape
peeling off a roll) and I exit with what feels like a large staple stuck
in my bum.

I notice the female COO (who lives in a glass office right behind my desk)
is now giving me very strange looks – probably because she’s been mooned
at all morning and has seen far more of me than she feels is necessary -
she’s not been in her office since Monday now!

Finally it’s time to go home, and it seems the temporary repair has held
up. But then, disaster – my jacket sleeve has wound itself around the
wheel of my chair, and the only way to untangle it is to bend over and
pull it out. If i bend over there’s no telling what might happen.

Why is fate conspiring against me? Was I Genghis Khan or Atilla the Hun in
a previous life?

Finally I resolve the problem by picking the chair up, placing it on my
desk, and wrenching the damn sleeve out of the wheel. By now though,
everyone in the near vicinity, including the COO behind is looking at me
as if I am a complete lunatic wondering why on earth I am putting a chair
on my desk, yanking at a jacket while I am looking as if I’m about to
burst a major blood vessel and mincing around trying to keep my arse
tucked in.

Finally, I move stiffly, but as quickly as I can out the door heading for
the station.

The repair ½ lasts for the change of train at Surbiton, where the repairs
start giving way again, but I am able to conceal it by sitting until the
last second until the doors of the incoming train are about to close. By
the time I reach my stop, the flaps are wide open again, but I’m on home
ground now, and run for my bicycle. Amid quite a lot of hoots from
following cars, I pedal home frantically. Wife is just hysterical with
mirth, and I have to admit, even I at this stage see the humour in it,
though Daughter is actually chastising her mother for being so mean to me -
“poor daddy”.

On changing into a less damaged pair of trousers I find several strands of
sticky tape stuck to my underpants, and a number of staples to boot. They
doggedly remained attached to there, where they hadn’t been stuck, but
refused to  bind to the wrong trousers. Typical.

NOTE: As of 14 Jul 2011, some mirrors are still updating and don’t yet have all the RPMs and/or metadata uploaded. Your mileage may vary.

To get Vbox Guest Additions installed and running on CentOS 6, get a terminal up and enter the following:

  yum -y update

  yum -y install gcc kernel-headers-`uname -r` kernel-devel-`uname -r`

Now, from the VirtualBox top window menu, select Devices->Install Guest Additions

If the autorun window comes up, let it run the script. If not, go back to the terminal and do the following:

  cd /media/VBOXADDITIONS_4.0.10_72479/     (the version may be different – this is as of 14 July 2011)


or if you are not logged in as root,


Now check the vboxadd service is running with:

service vboxadd status


sudo service vboxadd status

If it’s running, now log out and back in of the CentOS virtual machine. That should do it.

Synopsis: If you have a Linux instance, on booting it will run fsck on disk partitions configured with default settings in /etc/fstab. If the check fails, the boot will halt before bringing up networking, and offer you the choice of continuing, ignoring the error by pressing ctrl-D, or entering the root password to attempt a manual fix. This is not helpful as AWS (and Eucalyptus) provide no interactive console service, so you can see the prompt, but not respond to it. However, all may not be lost.

Here I shall present a method for recovering an instance which is failing to boot because of failing disk integrity checks on non-root partitions when using an AWS instance store AMI.

The first thing to note however, is that if the fsck fails on the root device, you are royally screwed – as far as I can make out there is no way to recover from that. I’m looking into it though.

The second thing to note is how to prevent it happening in the first place. All you need to do is prevent fsck running at boot time, this can be accomplished by editing /etc/fstab and modifying field 6 of any mounted file system entries to 0 (zero) like so:

/dev/sda1  /         ext3    defaults        0 0
/dev/sdf   /var/www  ext3    defaults        0 0

If those fields are non-zero fsck is liable to run after a certain time duration or number of reboots. Setting it to zero means that fsck will consider the drive as not requiring checking.

So, I had a customer who had this problem while resizing a volume, the aim being to increase the size of an EBS partition. They had added additional volumes, mounted them manually, copied all the data across from the existing partition then swapped the mount points by editing /etc/fstab, then rebooted. Problem was, the instance then failed to boot as fsck failed on the old partition, which was running a RAID0 partition and somehow got damaged during the reboot, or before. Whichever, the problem only became apparent on boot. It could have been prevented by unmounting the partition and running fsck on it before the reboot, so there ‘s a lesson in itself.

Ironically, the old filesystem was not even required any more, since all the data had been successfully copied across to the new one, and it was now this redundant partition which was preventing the boot completing. Detatching the volumes is no good, since fsck will still obey /etc/fstab and complain that there is no superblock.

But, since this data plays no part in actually bringing the instance up to the state where an administrator can ssh in, we can spoof it.

My solution is as follows:

Create a new instance and 2 new small EBS partitions (you can use the AWS console or command line tools, whichever you prefer for this) of 1Gb each.

  1. Attach both volumes to a new instance.
  2. ssh in to the new instance
  3. create a new RAID0 array
  4. create a filesystem on the RAID0 array
  5. fsck the new filesystem
  6. check /etc/fstab settings
  7. mount the new filesystem to confirm error free
  8. unmount
  9. detach NEW volumes from NEW instance
  10. detach OLD volumes from OLD instance
  11. attach NEW volumes to OLD instance
  12. reboot OLD instance

Now it should boot, as fsck will fly through and find no errors.

Good luck.

Well, I’d spent a long time getting this beast up and running, and like an unbroken stallion – it wasn’t coming quietly. I had a mountain to climb, and I named it Bustboot Mountain.

Score to date, Gremlin: 2, me: 0

The beast in question was a supermicro blade server based monster, with 24 cpus, 28gb RAM and around 40Tb of RAID5 storage, controlled by 2 fast LSI Megraid cards. There will be more of these, so the task was to get the installation automated by Cobbler such that a minimal CentOS 5.6 was installed and reachable from the install server.

Of course, the limitation on the kickstart install did not help, an unhelpful message about the GPT partition not being usable as there were storage volumes over 2TB available – why does this matter when 2 system volumes had been created of 1TB each off the local SATA plane?

I finally breached the enemy’s defences by hacking the python anaconda partition manager script, and rebuilding the stage2 squashfs image – then re-inserting it into the package directory on the cobbler server.

Gremlin: 2, me: 1

However, it now installed, but then refused to boot, ending up in a cycle. Gremlin was far from finished yet.

The addition of a super grub image to the PXE menu helped cure the next ills simply by writing the MBR correctly. It now presented a boot menu, but failed to boot from the default option on hd(6,0)

Gremlin: 2, me: 2

So then a quick mod to the installed boot hd(6,0) to hd(0,0) finally made it wake up and reboot unattended. Victory was mine, this day. I just needed to mod the kickstart file for write the MBR to both individual RAID1

Gremlin: 2, me: 3

That done, I went to shut the kvm panel, only for the beast to have it’s final word and snap the screen shut – on my fingers. I know now that gremlin is still there, still waiting for his chance for vengeance.

Gremlin: 3, me: 3

Here are a list of danger signs to watch out for in adverts. I list the key phrases and their real meaning…

Strong team-player: We are looking for someone who is so desperate for a job that they’ll endure crap from lots of over-aggressive, lazy, annoying people whom you’ll have to continually chase to get off their arses and do the simplest things.

Ambitious and Energetic: Senior Management want another “Yes Man” so you must be a total brown-nosed bastard whose sole mission is sucking up.

Customer-focused: you will take all manner of abuse from customers without any backing from your own management (who will certainly throw you to the wolves if you challenge the customer on anything at all) and still be glad you have this shitty job.

Self-starter, must be able to work on your own initiative: – We have absolutely no inclination, time or resources to brief you, so we expect you to figure everything out for yourself, like yesterday.

Attentive to details: We have anally strict policies and processes and won’t hesitate to target you as a scapegoat as soon as a mistake is made.

Flexible and Innovative: – We are totally disorganized and indecisive, so you’ll need to be able to clean up our messes and do stuff that isn’t in your job description and/or you have no experience or training for at our whim.

Results Orientated: – you’ll be inheriting projects which have failed because of a group of influential people who won’t accept change.

Able to work under pressure: – Our managers are hopeless at prioritising, so they class absolutely everything as mission critical and will micro-manage your activities to a maddening level.

Goal-Oriented: – We have lots of utter messes we have no idea how to clear up so we need someone to figure out how to do it with zero guidance, in the face of considerable resistance and without any other resources.

Some observations made from a long career in technology.

  1. Invite as many people as possible (notwithstanding point 2), duplicating representatives of specialist teams and ideally external suppliers wherever possible. We all need time to waste, and you wouldn’t want to hurt anyones feelings by not including them. Also make sure you tell plenty of lame jokes, it helps to finesse the time wasting.
  2. Avoid inviting people who can actually make decisions and/or provide critical input on relevant topics the first time around. This way you have an excuse to hold more meetings to fill those gaps in your schedule.
  3. Send out meeting invites at ridiculously short notice, especially to people who must travel to attend. Meeting invites sent in evenings and very early morning after/before normal working hours are an excellent example. This also shows how hard you are working and wastes even more time because people don’t have time to react. It has the added benefit that it can also cause them to reorganise their entire day to comply with your requests.
  4. Don’t start the meeting until everyone has arrived/dialled in. Starting a meeting on time would be rude to those who are late. Plus starting a meeting late helps everyone feel more comfortable about being late next time, which almost guarantees that you will be able to start your meetings late every time.
  5. Check busy people’s diaries and try to arrange meetings accordingly to ensure that all their available slots are taken up with no gaps between meetings to carry out their actions. This ensures they are using all of their time effectively.
  6. Don’t circulate an agenda. Agendas create focus – which can stifle tangential and random topic discussions. However, DO send out emails after the meeting purporting to summarise things. These should contain as much verbose detail as possible, and be as large and comprehensive as you can make it. Ideally, include lots of attachments, which may be other email messages, documents, video clips etc. Ideally many of these will be irrelevant to the context of the meeting, the purpose being to ensure the no-one actually reads beyond the first two sentences without losing the will to live. Failing to do this creates a risk of actual progress being made and we don’t want to introduce additional risks to the projects.
  7. The PM should do most if not all of the talking. Since he knows best, it makes the most sense that he or she talk the most.
  8. Punctuate sentences liberally with common verbal tics such as Err, Umm, “You Know”, “Sort of” etc. and play with various combinations of these to get the signal to noise ratio as low as possible for everything you say. This also serves to maintain everyone’s attention span at minimum levels.
  9. Give people freedom to “multitask” during meetings. Let everyone know at the start of the meeting that if they need to do other things such as texting, reading/sending email or answering calls to go right ahead. It will make the meeting more productive for everyone.
  10. Never end a meeting on time. Doing so means you failed to apply the items above correctly!

Get every new post delivered to your Inbox.

Join 34 other followers