|
Batch Job Bug and Nuclear Submarine There were four of us in the room, although only Tony and I were really working. Leroy, his face lacking any expression, was from time to time yawning, which made me even more uncomfortable than I would have been otherwise. I had worked with Leroy as night shift operator on several urgent bug fixes and knew him as a reasonable man, relatively competent technically, ready to repeat the same things as many times as necessary – but his relaxed observation of events made me nervous. The situation was difficult, to say the least. If we could not fix the bug crashing the batch, two hundred thousand or so disbursements would not be issued. Not a big deal on the scale of our galaxy but for our corporation – a cosmological catastrophe. The fourth of us, Stan, I didn’t know that well. I felt reluctant to call him but Tony insisted. Yes, this was what we had to do: elevate the response level by calling a night shift manager on-call. “I will be there in forty minutes”. OK, he is here now. But what can he do to help Tony and me? Nothing. Tony is the main expert in disbursement software; preclaims and claims is my area. The bug is somewhere between the two. Any third programmer, even the brightest and smartest, Chuck let say, would only distract and slow us. At 4:15, there is no way to panic. But there is no way not to panic either. Disbursements must go out by 6:30. Period. Stan was sitting still; he hadn’t pronounced a word after initial greetings. I was pleasantly surprised: usually night shift managers, on those rare occasions when they arrived at all, made suggestion after suggestion trying to show their extraordinary effort. Probably they were writing imaginary essays that would be read to our VP aloud afterwards. Stan did not look like your normal manager in an IT company at all. He was short, very wide in shoulders, with powerful biceps that one’s eye could immediately detect even when Stan wore a jacket. His neck was short and the head had been placed on it at some unusual angle, which made Stan reminiscent of both a matador and an ox. If I were asked what Stan is judging from his appearance only, I would alternate between a truck driver, a policeman, and a mobster. Even despite clearly non-aggressive, tranquil expression on his large simple-minded man’s face. It was more or less clear that the bug was related to the recent code modification by Malati. The modification was simple, compact, reasonable, well commented. It looked very much like Malati herself: small woman, her facial skin so delicate that it often blushed despite of Malati’s Indian swarthiness. It was difficult to assume that there could be an error in those nicely written lines, which passed one of the best QA testers on their way to the operational system’s patch. But since nothing else had changed that could be even distantly related to disbursements, Malati’s changes had to be suspected. Tony and I had already started the third round of reasoning; just out of desperation we were repeating the same discussions. Well, why did not it fail last Wednesday night – there were no code changes between then and now? What it has to do with claims at all? Why not everybody with a claim against their loan cause the failure but about each third? We asked each other the same questions and did not really hope to get new answers… We had already tried brute force several times by just removing each time the borrower whose loans had caused the crash and restarting the batch – only to watch the next crush after about the same time. Brute force did not work. “Tony, - said I, - Let’s examine once more these five records: do they have anything in common?” Tony shrugged his shoulders: “OK. But…” “You had already been here, guys, – Leroy couldn’t withstand the tension in the air. – You had already found that they have nothing in common”. Stan’s bull neck became a little darker. “Hey, Leroy – he said – you cannot help them. Be quiet please”. And without a pause he asked us: “Cold soda, hot tea – what would be better for each of you”. “Nothing. Thanks.” – I answered immediately, without slightest thinking. I did not want any distraction. Any! Tony was not such a hurrying squirrel. He took a little time, ten seconds maybe, and then answered: “Soda for me. No caffeine, please. Lemonade would be the best. Thanks, Stan”. He hadn’t even finished yet when I sensed unbearable thirst. I almost felt that icy lemonade in my mouth. I licked my lips. Stan looked at me with a little, almost invisible smile: “Maybe, the same – for you?” I just couldn’t help smiling back: “Actually, yes”. Stan quickly strode to the door but stopped for a moment passing by Tony and clapped Tony on the back. Then he looked at me, and at Leroy. “Relax, guys, - said Stan, - relax and you will resolve it. I have a gut feeling that you are close. I believe my gut feelings, we, sea-folk, develop some over time”. And out he stepped. “Was he a seaman?” – asked I. “Oh, yes, - Leroy answered, - Stan was in the navy. On a nuclear submarine.” “Look, look here, - Tony whispered suddenly. – All these guys have four disbursements, and for all of them tonight’s disbursement is the last one. Maybe, this is a clue.” We looked once more. It took only couple of minutes to find a guy from Missouri with four disbursements and the last one being disbursed, which did not crash the batch. “Well, we are back to square one” – uttered Tony. But now it was my turn to make a discovery. “Wait a minute! – from anxiety I started to gesticulate and almost hit Tony’s face, - let me check on what kind of claims these guys have against them!” “Does it matter? – Tony looked skeptically. – Malati modified just maturity control…” But I already knew, I have already understood what was happening; I had only to check and verify. “Wait, wait! Yes! No! Well, yes! Permanent disability, death, permanent disability again… Yes, of course! And that guy?.. Fine, just delinquency. Great! Where is Malati’s code? – my speech became incomprehensible because thoughts pushed each other aside and could not create any sequence. – Yes, that’s it! Malati did not filter claims by their kinds. We will fix it in a moment, Tony! You are a genius!” But even being called genius did not completely convince Tony. “Well, what about the last Wednesday batch?” – asked he. “Last Wednesday? Last Wednesday… Don’t know.” I was confused. “Anyway, - I realized that Stan was back only now, hearing his voice, - Anyway, let’s make this fix. Time is running out and this is our best bet now.” Stan was standing with three large glasses of lemonade: “Sorry, Leroy, I forgot to ask you. Hope lemonade is OK”. “Sure, - said Leroy and added calmly. – By the way, I have an idea about last Wednesday. We had only about a thousand of disbursements then, probably even fewer. Now there are two hundred thousand. It is possible such borrowers just weren’t in the batch last Wednesday”. In about forty minutes we restarted the batch and some twenty minutes later it was clear that the problem has gone. “Well, Stan, - said I, - probably Tony and I can leave now. I am tired”. Actually, I was not sincere. I had all the reasons to feel tired but I did not. I was too excited to be tired. I said so to prompt Stan for some expression of his appreciation. He did not act so far in usual managerial manner, he did not say in many nice words how good a job we did with Tony. “OK, you may leave, of course. But just in case it would be better if you stay for a little longer. And pizzas will arrive in a few minutes, by the way. I ordered two large, just enough for four hungry men.” * * * The pizzas actually did not arrive so soon, we had to wait for a good half-hour. This was even better, however, because the disbursement batch had been completed just a minute or two before they arrived, and each of us breathed out the remains of the tension. “Stan, how did you know that they were already close?” – asked Leroy helping himself to the first slice of pizza. – “I love this, with pepperoni, the most”. “Don’t know, just felt this way. Well, everything ends well much more often than badly. I often am wondering why it is so.” “What do you mean?” – asked I. Stan interested me more and more. “He is just a natural-born optimist, like me. Correct, Stan?’ – we all were happy but at the moment Leroy seemed be the happiest. “Listen, guys, this has nothing to do with optimism or the lack of it. This is what I see around me. Most folks do not die in traffic accidents but almost everyone can list several cases when he could die and did not. And many think they were saved miraculously. This may be true for some cases but miracles cannot happen to many. Now, let’s look at our software. We all know that it has a lot of bugs, and some of them could derail all the process. And several times a month we have these night alarms. Mostly they are not as big as this one, - Stan waived his hand in the direction of operational monitors, - but, nevertheless… And during my three years in the company, we had just one, no, wait, just two major problems with customers due to the batches. And I can tell you several stories, if you wish, about much more serious real life situations when things went well against the odds.” “Sure” – nodded Tony. “It was in Israel, in Haifa. Our submarine arrived there for a planned stay with some repair works that had been scheduled in advance. We had to spend a lot of time there. On the second day after our arrival the captain allowed shore-leave for a big group of us. Do you know what sailors do on shore-leave?” “Whores?” - asked Leroy, his face shining. “Yes, the whores too. But this is secondary. The main task is to make yourself as drunk as possible. I don’t even understand now why it was always the absolute goal. Regardless of port, regardless of country. Even folks that had a friend, or a relative, or even a girl in town – they had to drink as much as they could first. Typically, shore-leaves are 24 hours long, from early morning ‘till the next early morning. Usually, from five to next morning’s five. So, there are good cheap places around each port to drink. And women are there also, of course. Israel is a good country, - Stan casted a quick glance at me, - the wine is good, the women are good too—“ “Beautiful? Hot? What do you mean by ‘good’?” – Leroy looked excited. “I’ll tell you later, let me finish the story. So, we drank first thing in the morning and then walked around the city a little, drunk but not completely drunk yet. It was a nice day. Haifa is a nice city, - Stan cast another glance at me, - then we drank again, then we went to some dancing places, then we had prostitutes, then I slept, an hour maybe, what a waste of time – to sleep on shore-leave. Why? – when we are back to the boat nobody will bother us before the next day, they understand quite well how drunk and tired we come back. So, then we drank again, and help each other to walk to the boat, and guys on the submarine helped each of us to his bunk. So, we came at five, and before six the captain’s mate wakes me up: “Get up, we must leave Haifa immediately! Do an emergency start of the reactor.” What? Emergency start? I thought first it was a nightmare, tried to sleep again. But he started to shake me, very physically: “Get up, Stan! There is an unusually powerful cyclone formed in Western Mediterranean, it is moving here, the harbor is too shallow for us and the boat must leave immediately! Get up, son of a bitch!” But I just couldn’t get up. I was too drunk and all my limbs were out of my control. I did not remember how I got to the boat. And there are two completely different procedures of starting the boat’s nuclear reactor. The normal one takes a little more than 24 hours; I performed it several times. The emergency start I did just once before while in training. It takes less than an hour and is not good for the reactor. It is also more dangerous, of course. So I said: “I cannot do it now, Sir. I absolutely cannot. Give me one more hour of sleep please.” – “Hour of sleep? Are you crazy?” While we were speaking, the captain himself entered the crew quarters. He addressed me in a soft, kind voice: “We understand everything. But there is no other option. You must do the emergency start now”. And then, without a two second pause, he ordered: “Sit up!” I sat up on my mattress but immediately fall down. “You two, help him!” – ordered the captain. “Captain, Sir, - said I. – I cannot start the reactor. I don’t remember how to. I may blow the boat up”. “Shut up! – shouted the mate. – Shut up and get up!” The captain raised his hand to silence the mate. “You will recall it as you start doing it, - said he to me. – We will help you. And you have your manual.” I realized that he was right even before he finished; I started to recall my knowledge about the emergency start procedure. “Maybe somebody else can push the buttons, Sir. I will tell which ones to push. My hands don’t obey me, Sir”. The captain frowned: “No, Stan. You know that you are the only one aboard who is permitted to start the reactor. But we will help you”. “Wait a moment! – interrupted Tony. – Are you serious? How could it be that such an important operation was not doubled? And if you had died?” “I don’t know. But this is how it was. I was the only one with the special training.” “Is it still the situation?” “No idea. After I left the navy I was not really interested. So, what did I stop at?.. Well, I dressed somehow, forced myself to sup stale coffee… Two guys made me walking to the reactor’s console, they almost carried me actually; there somebody put me on a high stool – I didn’t even know there was such a stool on the boat. I sat on it instead of standing by the control panel and these guys help me with moving my hand – one with the left hand, another with the right. I recalled what to do after what - there were no problems with it. To cope with my hands was much more difficult. I swore violently at the guys who were helping me, called names. “Not here, there, not there, here, not the left hand, no, this is not the hand! You are such and such sun of a bitch!” How did they tolerate me, thought I afterwards. All right, all right it all went; they just made fun of all this later, mimicking and mocking me. So I started it, that damned reactor.” “And then? – asked Leroy. - Could you get to sleep then?” “You bet!” – Stan smiled and got a big sip of his diet coke. * * * When Lynda and Joshua, morning shift operators, came, we were almost done. Only two slices of pizza left – just one for each of them. “Thanks, - said Joshua. – How was it? Problems with disbursements?” “Yea, small ones,” - smiled back Leroy. “It was great!” – said I enthusiastically. And I meant it. Firstly, I was happy that I was able to fix the batch problem (being excited, I forgot about Tony’s role), and secondly, I learned that the Yom Kippur War was not the ultimate danger to the Israel’s existence. The drunken short man, good manager Stan, was. On my way back home I thought about Stan’s statement that things end well more often than badly. At one point I dived into my thoughts so deep that crossed an intersection on red light. Fortunately, there was neither traffic, nor a policeman. Probably, Stan was right. 2013
|