In 1997 I was working for Science Applications International, Corporation (SAIC).  I was told to meet with the IT Director of British Petroleum (BP) for European Operations in Germany and discuss doing a Y2K mitigation contract.  It was an unusual assignment for me.  I should have realized there was something odd.  I had never done any contract work outside of the U.S.  I had not done any Y2K mitigation contracts.  I was, of course, aware of the news media material promoting a state of fear about the pending disaster because all of the computers would fail as the date changed to 01/01/00.  And, I understood the specific technical nature of the problem.  I had written a number of applications where I did not code for century changing from 1999 to 2000.  It was a standard habit to drop the 2 digit century because it was always 19.  It was not until the 80s that cost of computer data storage had dropped to where the cost savings of dropping the 19 was no longer material.  I had no idea if any of the applications I had written were still running.  It didn't seem likely but it was possible. 

I was aware of the IT technical press and that there were a lot of companies doing contracts to find the date routines and fix them.  I knew that several companies were promoting the position that mitigation would cost a little over $1 per line of code.  There were thousands of companies with over 100,000,000 lines of code in their inventory.  When it was confirmed that I was to go to Germany, I got in touch with the SAIC Y2K remediation center and asked for our position and what corporate support was available for Y2K contracts.  I then read through many articles in the trade journals.  I thought I was well prepared for the meeting.  I was wrong.

I arrived at the German BP office at the appointed time.  I was taken to meet Horst, the IT Manager for Germany.  We spoke causally for a few moments and then went to a conference room, a very large and formal executive style conference room.  It was set up for at least 40 people to sit around a single massive and very impressive conference table.  The chairs were of the massive type and so padded you thought you might just get lost in them.  The 2 of us sat down.  Robert, the IT Director for European Operations entered within a couple of minutes.  He sat down opposite me and said, "OK, what's my problem? What are you going to do for me?" The questions were not asked politely.  He was not in a good mood.  Something was going on and I had no idea what it was. 

Later I learned that a few weeks previously the official corporate Y2K Remediation Team had given Robert a presentation with a price tag of just over $1 per line of code.  They had a known inventory of at least 100,000,000 lines of code.  The proposal was for $120,000,000 plus additional fees for every line over 100,000,000 lines.  When it became clear that the corporate team had no idea about the condition of the code or the types of systems in use at BP, the IT Director had become furious and quite literally threw the SAIC team out of the office.  To this day I do not know who decided or how it was decided that I should go meet with Robert but I think it would have been nice if I had known this little piece of background information.

In a situation like this, facing a very senior executive with a forceful and upfront personality and who is accustomed to taking charge and clearly expected results, there are really no alternatives to be considered.  There is only one option; the truth.  My response was, "I don't know.  Here is how I propose to find the answers to your questions." I then took about 5 minutes to outline a very simple and straight forward exploratory project.  He said, "Can you do it for $100,000?" I said, "Yes." “When will you have the results?" "Two months." He turned to Horst and said, "Give him whatever he needs." Robert then left the room.  The next day he gave the SAIC sales representative a hand written purchase order on which he had written, "Y2K pilot, $100,000." There were no further formal details defined for the work.

This was Monday.  Horst and I worked out further details and I spent the week discussing the systems with his staff.  I left Friday with source code from 4 different systems, provided in 4 different computer media, and written in 4 different computer languages.  When I was back in my office on Monday, I called the corporate Y2K office told them what had transpired and asked for their assistance.  They said, "What you proposed can't be done.  You are on your own."

I discussed the situation with my boss and we laid out a plan.

1. I would use our personal networks to locate places that could read the media I got from Germany and put it in a format that we could read on a PC. 
2. Two of the applications were in COBOL.  One was a standard version and the other a not too often used variant.  We would subcontract with a firm in India that was specializing in Y2K mitigation for these.  They used an industry standard tool to scan the code for instances of date routines that were likely to require changes. 
3. One of the applications was a refinery control system written in FORTRAN.  We would subcontract with the local University and use graduate students who knew FORTRAN.  They would write a scanner to examine the code. 
4. The last system was in a language I had never heard of before.  We could locate no one who had heard of it.  I fortunately had the foresight to bring a coding manual back with me.  We decided that I would do this one.  I would write a general purpose scanner that we could test on the COBOL and compare it with the existing scanner for COBOL and we could also cross check against the FORTRAN.  The scanner concept was essential because of the vast number of lines in the full BP inventory.  The process needed to be supported with as much automation as possible. 

We proceeded as planned.  When we had the initial results of the three teams, we compared my scanner against the FORTRAN Team, the results were the same.  When we compared my scanner against the industry standard COBOL scanner there was a significant difference.  Well, the purpose of the test was to confirm my scanner was working properly.  I must have over looked something about COBOL.  After a careful analysis, I decided I had not.  The industry standard scanner was significantly over reporting the number of lines with possible date problems.  Interesting!

I went back to Germany as planned on time with the results.  Part 1 was to have the German technical staff confirm that we had correctly identified date problems and had made the appropriate change.  We had.  Part 2 was to present our results to Robert, which were

1. how I extrapolated to repeat the same process for all of European Operations (14 Countries)
2. the total cost, and
3. the risk of not doing the mitigation. 

My total cost estimate was $19,000,000.  In the programs we examined I found no significant risk if the programs were not changed.  They would fail, but each failure would result in only minor problems in reporting.  There is no doubt that the problems could easily be handled individually by the existing trouble discovery, reporting, and maintenance system.  The difficulty was, of course, the possibility of a spike in problems as the century changed.  Incidentally, most of the spike would be at month end, not January 1.  There is reason to consider separating the inventory into those that would need rapid fixing from those that could wait a month.  And perhaps mitigate only this part of the inventory, just in case the maintenance team got over loaded.  I did not have enough information to decide which systems would be in each category or how much load the maintenance team could handle.

The strongest reason I could offer for doing the work is what I called legal due diligence.  I found nothing in the pilot sample that indicated any concern about legal liability.  Because of the excessive press about the pending Y2K disaster the attorneys were getting in the act.  Should something happen, it would be a difficult situation if BP had not done enough to satisfy a legal due diligence.

Robert said, "He had the German team duplicate my pilot and develop an independent estimate.  They were a million and half lower than you.  They should be.  They know the systems better.  I need them to do other things.  You have the contract.  We will prioritize the inventory and stop where it makes sense.”

During the project we found several things that needed to be changed.  None of them were serious that would have created a major problem. 

During the project I had a very interesting conversation with Ton in the Netherlands.  Ton was the Information Technology manager for a BP oil refinery.  He had previously been the refinery operations manager.  He was in a unique position to comment on the potential of a problem in either the IT systems or the refinery computer controlled systems.  One of the big scare publicity items in the media was the disasters resulting from computer failures in control systems, such as in aircraft and oil refineries.  As a part of the due diligence I was to make a trip and inspect the automation system at all of the refineries.  Robert knew, as I did, this was simply for the attorneys.  I asked Ton, "What do you think the potential is for a computer system failure here at the refinery as a result of the century change problem?" Ton, "I expect failures." I then asked, "What do you plan to do about it?" Ton, "Nothing.” If every single piece of automation were to fail we would operate the plant manually.  After 12 hours, we would be forced to shut done, but only because we could not maintain the quality of product longer than that." It should have been obvious from the beginning.  High risk operations always have back up to the computer systems.  Computer systems fail for more reasons than a century change.  If the century change could cause a disaster at a refinery, BP had a much bigger problem about refinery safety. 

Ton went on to explain that he was not going to look for a problem before hand.  Some years previous, when he was the operations manager, he had been told to start a preventative maintenance operation.  He was to open up and inspect every control valve, control point, and control system on a regular basis.  He opposed the plan because he felt it would introduce more failure than it prevented.  He was given no choice.  So he kept records.  After 2 years he proved he was right.  Actually, I had already experienced this point in doing the avionics project for the Air Force.  Some types of preventative maintenance increase failures.

Shortly after the BP project I had an opportunity to discuss the Y2K issue with the person who was responsible for Y2K remediation at Boeing Aircraft.  He, not too surprising, gave no direct answers to any question about what they were doing or what they were finding, but the situation was obvious as we discussed this issue at a conceptual level.  They had very significantly scaled down to a relatively small project for Y2K.  They had found no cause for concern for flight safety.

Because of my success at BP and because of the all purpose scanner I had built I got a contract to assist with a Y2K project at a very large business in the health care industry.  This time I'll leave out the names of the companies.  They had an estimate of about 150,000,000 lines of code.  The marketing folks were in a frenzy as they looked at the prospect of selling a nearly $200,000,000 contract.  But the technical team doing a discovery project was stuck.  They were not getting the information they needed from the various divisions that own portions of the code inventory.  They also had several computer languages for which there was no industry standard scanner.  When I got to the project I scanned all of the programs that they had gotten copies of so far.  There were very few date problems.  Once again my scanner showed far fewer problems than the industry standard scanners.  Direct personal examination of many of the programs quickly revealed the reason.  The programs were already set to handle the century change.  Of course, most of the applications had been written within the last 10 years.  Any competent programming staff would by the late 80s be coding for the century change.  Only programs written before the early 80s ignored the century issue.  We had a list of all of the programming project managers and a list of the systems for which they were responsible.  The team had not been successful at getting code or even an inventory list from these people.  There were about 500 names on the list.  I suggested that the 5 of us take 100 names from the list and start calling.  We would quit after we had answers from 12 people for the following questions.  Do you have any programs for which you have known Y2K problems? How may of these will be replaced before the end of 1999? Do you have programs that you have already corrected for the date problem? Do you have any programs that you have scheduled for corrections?

Over 70% of the program inventory that required changes was already changed.  For the last 10 years whenever a change was requested in a program that was likely to be around in 2000 the programmers corrected the date problem.  The corrections were actually very easy to make.  The real effort was in the testing protocols to make certain the programs still worked.  But since these programs had to go through testing anyway there was no significant additional cost to take care of the date issue during the routine changes that were already being made.  Of the remaining 30% of the inventory 25% was scheduled to be replaced before 2000.  That left 5% that still needed changes.  3% were already scheduled for changes.  Thus the actual scope of the problem was at most 2% of the inventory.  I concluded that if left to themselves the staff would take care of this.  We documented our findings and sent them to marketing management.  The customer never saw the report.  At this point my work was done and I left the project for other work.  I do not know if there was or was not a large Y2K contract that eventually got signed.

I do know from first hand knowledge that at least one bank, one insurance company, and one large telecommunications company signed up for Y2K remediation projects each worth between $150,000,000 and $250,000,000.  I also saw promotional material from several computer services companies reporting over a billion dollars in Y2K remediation projects.  I noticed that all the home improvement business had large displays of power generators because of the pending Y2K disaster.  They could not get enough stock to meet the demand.  The grocery stores had a run on products; they could not keep up with demand just before New Year’s Eve. 

And what really did happen? Why nothing! Except some consultants got rich and some computer services companies made billions.  And NO it was not because of a job well done.  If that had been the case there would have been at least a few places that failed to do the job.

Why? How did it happen? It was simply a faulty belief.  A faulty belief promoted by the media and corporate marketers.  It was all a very bad joke.  I hope it was not at your expense.

ç  Prior Page of Text     Next Page of Text è
(C) 2005-2014 Wayne M. Angel.  All rights reserved.