Tuesday, May 09, 2006
The TA-15 CREM Incident and Aftermath
On July 6, 2004, two CREM items appeared to be missing in DX-3. A wall-to wall inventory of all DX-3 CREM had revealed a discrepancy in the amount stored in a safe. The discrepancy was reported to the DX Organizational Computer Security Representative (OCSR), DX-3 and Division Office management, as well as to the Security Division, in strict accordance with Laboratory requirements. When the discrepancy could not be reconciled within 24 hours, NNSA was also notified. A local Security Investigation Team (SIT) was convened, and over the course of the next few weeks, a chaotic series of thorough and intense physical searches was conducted, both inside and outside the building where the safe resided. At about the same time, the SIT sequestered the nearly dozen people who were authorized access to the safe, and interrogated these individuals repeatedly and at great length.
Though not very likely, the possibility that the missing items could have been mistakenly put in a different repository was explored, along with thorough searches of the building where the CREM was stored and the surrounding area. Finally, a few of us who were on administrative leave, because we were authorized access to the safe in question, began to look at the CREM tracking system, past inventories, and all classified portable computers that had been used with the CREM over the past year. A pattern began to emerge that suggested there were fewer actual CREM items than the inventory listed. We analyzed all the records from the electronic tracking system all the way back to a classified international conference we participated in during the fall of 2003, where the CREM was initially generated. During this conference the CREM custodian had mistakenly entered a set of bar-code numbers into an electronic accountability system, even though the amount that had actually been assigned to specific media was less. A series of previous audits that failed to reconcile this mistake were the root cause of the difficulty in reaching our conclusion.*
When this plausible explanation was advanced by our OCSR to the SIT, it was not only rejected categorically, but also, in very short order the pertinent records were confiscated, and the OSCR was forbidden to speak to anyone about this line of thought. After nearly two long weeks of tense meetings devoted to the CREM issue, this threatening confrontation was the final insult that provoked The OCSR’s decision to resign from the Lab. Alarmingly enough, this development strongly suggested that the SIT had already reached a conclusion about faultfinding and was unwilling to consider the strong probability that the allegedly missing CREM never existed. However, this information leaked out and may have influenced the Lab to hurriedly assemble an independent team to test the hypothesis. Meanwhile, the Director had already forced a work suspension across the board, because a laser safety incident, with injury, complicated the Lab’s operational picture. Moreover, the FBI had begun to investigate the missing CREM. But before long, the Lab’s story began to change, and in an August 11 news article, Senator Domenici revealed the possibility that the CREM issue might actually be a case of a “false positive,” or that of a system indicating that something was missing, when it actually wasn’t.
In August, though the account custodian attributed the CREM issue to error and was later terminated from LANL, a series of Case Review Boards were convened to make recommendations for punitive action against others with authority to access the CREM repository. In my case, curiously enough, the Board recommended 3 week’s suspension without pay, essentially for taking excessive training and being on vacation when the CREM issue developed. Nevertheless, the Acting DX Division Leader proceeded with my firing on September 23, 2004. The Board’s review was little more than subterfuge, for the Director had carried out a threat made against me in early July for not rushing home from vacation as the CREM crisis began.
* Los Alamos National Laboratory, the Federal Bureau of Investigation, and another independent panel have supported this conclusion.
The TA-15 CREM Incident and Aftermath
In a late December 2004 brown bag lunch held by Los Alamos National Laboratory (LANL) Director Nanos at DX-3, the Director spent more than an hour regaling attendees with stories of his military career. Posing with his black cowboy hat, he explained that he had proudly shot the cowboys who were responsible for the safety and security problems that prompted the July shutdown of the Lab. He said that he would make certain “those SOBs” would never work at Los Alamos as long as he was Director. He continued by bragging that his “government-funded attorneys were cheaper than the cowboys’ private attorneys” and he could “outlast them indefinitely”.
Until now, John Horne and I have been silent regarding the “cowboy/butthead/C-student” accusations voiced by the Director in his first all-hands meeting in July 2004. However, after the DX-3 classified removable electronic media (CREM) incident, lengthy investigations, Case Review Board recommendations, disciplinary actions at odds with those recommendations, and an official investigation by the Federal Bureau of Investigation (FBI), it is our hope that a narration of events leading up to and following the CREM “incident” is now timely. The Director has failed to maintain his composure and his objectivity during the entire course of the CREM investigation, unduly influencing the latter through actions and rhetoric that reflect poorly on him, the University of California, the Laboratory, the National Nuclear Security Administration (NNSA), the Department of Energy (DOE), and the Nation. His conduct, in a word, has been unbecoming and deserves condemnation. His consuming interest was in finding scapegoats, and his quest was thus far successful. Pity that he was indifferent to the truth.
We now know the official results of the FBI investigation have become public and their conclusion matched our own, that no CREM were missing at any time. These findings suggest we have every reason to expect exoneration, yet we continue to be vilified by the Director at every turn, and perhaps have only faint hope that promising careers can be rescued from the scorn and suspicion of unenlightened managers, who must demonstrate fealty first and integrity second.
Procedures to account for CREM derive from corrective actions taken following the well-known Wen Ho Lee security incident of 1999, and are called Requirements for Accountable CREM. These requirements describe a lengthy assortment of nonvolatile digital storage media (removable hard disk drives, laptops with removable hard drives, floppy diskettes, CDs, DVDs, Zip and Jaz disks, etc.) that can qualify as CREM, and dictate how these items must be catalogued, protected, and tracked. Traditionally, technical presentations containing text and image view-graphs are created with computer software, stored on electronic media such as a hard drive or disk, and displayed through the use of a computer-linked projector, eliminating the need for transparencies. If the display material is classified, then the computers and media used must be authorized for such use and are accountable as CREM.
At the time of the latest CREM incident in July 2004, so long as the general stipulations in Notice #0136 were met, line organizations at Los Alamos were free to develop and implement their own procedures for handling and tracking CREM. While barcoded serial numbers were required for marking CREM, a shortage of barcode scanning equipment forced many organizations (including DX) to visually read the numbers of each barcode and enter those numbers into accountability, increasing the likelihood of entry errors. Software needed for tracking CREM items could also be developed locally, and so the only common feature of LANL’s CREM process was the haphazard approach taken by the line organizations in attempting to satisfy the general requirements. The CREM custodian, charged with oversight of the CREM process in each organization, generally took on this job assignment with the understanding that it was only a part-time job and an added duty that wouldn’t interfere with the person’s primary job function. The approach to training can be similarly casual and treated as a chore that could be postponed until the training time was convenient for the assignee. This assignee generally was not proficient in computer skills, but was persuaded that the simple barcode accounting task was easy to master and was not a burdensome or particularly risky task. Verification of the CREM inventory was not required and seldom occurred when custodians changed, increasing the likelihood of proliferating errors without prompt detection. Those personnel likely to generate CREM were sometimes given blocks of barcode numbers to apply to generated CREM over some period of time, and the numbers were only to be entered into the accountability system when the CREM custodian was notified that one or more of the barcodes were actually affixed to created CREM. Since the CREM incident, requirements for creating and accounting for CREM, as well as training requirements for custodians, have become more rigorous. In fact a recent job ad for a CREM custodian required “flawless performance” as one of the job criteria, prompting several letters to the editor of the Los Alamos Monitor and causing much astonishment among Lab staffers. 
During the fall of 2003 I served as chair for a classified international conference at the Metropolis Super Computing Center. To support classified presentations at this conference, John Horne received from his CREM custodian eight barcodes to use as necessary to mark any CREM that might be generated. Unbeknownst to us when the eight barcodes were issued to John, the issuing CREM custodian, who had held the job for only one month, immediately entered all eight barcodes into the accountability system. In the course of preparing presentations for the conference, John actually applied only six barcodes to CREM, which were delivered to the custodian by placing the properly marked CREM into a safe along with a conveyed message to enter the disks into accountability. At that point John’s obligation under existing CREM procedures was fulfilled, there being no requirement for him to oversee or witness the custodian’s subsequent activity. Thus, because of an accounting error at the conclusion of the conference, eight disks were in the CREM accountability system, while only six were actually generated (thus the “missing” two disks). Nearly nine months would elapse before we would stumble onto this accounting error.
DX-3 CREM Inventory Crisis
On July 6, barely a month after an institution-wide CREM inventory had revealed an accounting error in the DX-7 inventory (totally unrelated to this current crisis), the DX-3 inventory of CREM items assigned to John indicated that two items could not immediately be accounted for. A frantic search for the allegedly missing items ensued, but when they couldn’t be found in the first 24 hours, notification beyond the Division was required.
At the time of the DX-3 CREM incident, I was enjoying a long-delayed family vacation in Washington, D.C., where I answered a page from my Deputy Group Leader on July 6 (my 20th Wedding anniversary). Upon phoning in, I learned for the first time about the inventory discrepancy. I was asked to return to Los Alamos immediately to discuss details that could not be revealed over the phone. Reluctant to terminate my vacation at that point (for the CREM items in question were not mine), I suggested instead that because I had my badge with me, I could travel to the Forrestal Building (DOE HQ) and arrange to discuss the manager’s concerns over a classified (STU) phone. The manager agreed to this offer, and in brief conversations over the course of the next two days, we discussed the problem, and the manager’s concerns were alleviated; he agreed that there was no reason to rush back, and gave me permission to continue my vacation. After all, the CREM items were not in my name, and it looked like I could not be of much help even if I were at the Lab. I would later learn that this delay in my return, even though approved by my manager, had enraged the Director, who demanded that I be fired. It would take a few months for him to make good on his threat, but he clearly made up his mind that day, and all his actions since then point to that very plan.
Near the end of my vacation, my deputy group leader paged me again. When I phoned him this time, he told me that I was really needed back at Los Alamos and must return right away. I thanked him for giving me the few extra days and agreed to fly back the very next morning. I left my family to fend for themselves and returned to the Lab. Upon arrival at DX-3, my badge was confiscated and I was summoned to what would be the first of numerous meetings with the LANL Security Investigation Team (SIT) that had been convened for the case.
Once the CREM inventory revealed that two items could not be accounted for, the searches unfolded in a series of increasingly anxious efforts in ever widening circles and in all directions—up, down, and all around, canyons and ceiling tiles included. News of the suspicious incident had spread swiftly to UC, NNSA, DOE, Congress, and the media. By the time the SIT appeared on the scene (July 8th), the focus was on the roughly dozen people authorized to access the safe where the allegedly missing CREM should have been stored. Badges of the dozen people were confiscated, and everyone was treated with dark suspicion, though none more than John and myself, even though my role seemed to be little more than incidental. The presumption now was that misconduct (if not outright espionage) might well be in play, and the trail was growing colder with the passing of time.
As pressure increased, the SIT investigation stalled, with no discernible progress being made in the face of a daily requirement for a SIT status report to the Director. Our situation was complicated by the mid-July appearance of Federal investigators, who immediately targeted us for scrutiny. Frustrated by the lack of progress, though, the Director lashed out at DX managers, and in one meeting the DX Division management blamed the current crisis on the DX-3 “cowboys” who simply wouldn’t follow the rules. By declaring helplessness in the face of these few uncontrollable workers, management had attempted to deflect criticism from themselves directly to John and myself. The “cowboy” term resonated with the Director, who then used it on the frequent occasions in July and August when the CREM issue was a subject for discussion. The Director also seized on his discovery of a bumper sticker (“LANL: Striving for a Workfree Safety Zone”) as yet more evidence of a deplorable cultural mindset at the Lab. He may have seen the sticker somewhere other than DX, but the way he used it in his first All-Hands Meeting did not make that point clear. Rather than to be stimulated into probing to find out what procedural frustration might provoke someone into displaying that damning slogan, he chose an illogical extrapolation that gave him license to declare it was indicative of a general contempt for rules by LANL workers. In mid-July a laser safety incident with injury occurred and was used along with the CREM incident to justify forcing a suspension of work at the Lab by the Director. The “cowboy” label continued to be used, but now its derogatory meaning had expanded to apply to anyone who demonstrated even a hint of skepticism about the decision to suspend all work at LANL. Indeed, line managers were directed to identify “cowboys” in their organization so that these personnel could be isolated and eventually removed from LANL’s ranks. The Director was widely quoted in the press, with two notable newspaper articles appearing in the July 13 and July 20. By this time, personnel in DX were being blamed for putting the UC contract at risk, contributing to a cultural attitude that needed to be stamped out, and displaying academic arrogance that dated all the way back to the Manhattan Project.
In this atmosphere and with this kind of treatment to contend with, we had enormous difficulty concentrating on the CREM problem that was our daily and nightly cross to bear. Because the issue seemed to track all the way back to the fall 2003 conference, it was hard enough to try to piece together events that long in the past. Meanwhile, the CREM investigation was going nowhere, and so a few of us who had come under accusation, along with other colleagues, decided to go back over the records from the CREM media tracker and the computers with extreme care, to see if we could find any clues about the allegedly missing two items. Many days were spent at this task, but finally a pattern began to develop with the periodic activity and inventory records that showed the two missing items being inventoried, but then being removed from the database three times. I found this unusual activity strongly suggestive of a case where two items might have been mistakenly entered into accountability and then eliminated when a subsequent inventory failed to uncover them. As this sequence repeated with each of the inventories (supposedly verified three times since the international conference), it made no sense. Interrogating the registry on the computer confirmed the only media, which had touched the computer, had been barcoded and accounted for and that the unused barcodes never existed. This suspicion was shared with the DX Division OCSR (Organization Computer Security Representative), who then in turn revealed it to the SIT. The OCSR’s information was not only rejected, but she was also then warned not to reveal this discovery to anyone else. Nevertheless, the information was apparently credible enough that the SIT came in after hours and confiscated the OCSR’s records. Alarmed by the combination of the threat and records confiscation, which was just too much for her after nearly two weeks of pointless searches and tense meetings on the CREM issue, she abruptly resigned from the Lab after more than twenty years of dedicated service.
It was this cover-up that persuaded John to hire counsel for his own protection, as the SIT’s astonishing and probably illegal action strongly suggested that John could easily be blamed for the loss of nonexistent CREM. He was convinced that he had retained the two barcode numbers and simply kept them, later shredding them with other items he no longer needed to keep during a midyear cleanout, but he couldn’t prove this claim. Now it looked like the most likely explanation was not only going to be rejected, but also any reference to it was going to be suppressed. The SIT had made it known that the most important goal of the investigation was to protect the credibility of both the Laboratory and the Director, and here was appalling evidence that the SIT would even conceal exculpatory information to achieve its aims. Nor could he necessarily rely on his own management, for the DX-3 Deputy Group Leader had recently declared at a Group Meeting  that “HR and S Divisions exist solely to protect the Director.” Despite the attempt to suppress the information we had given the OCSR and SIT, it leaked out anyway, and the Lab was then apparently forced to assemble an impartial team consisting of DX employees to test this theory. Only then did the Director suddenly change his official story and admit the possibility that the missing CREM might never have existed at all. This change would eventually lead to Senator Domenici’s comment on August 11 about the possibility of “a false positive” in the CREM case. Though my concern had grown sharply, I still felt that we had successfully shown—with the actual records—what probably happened; besides, I was still only peripherally involved.
The Director’s position continued to harden. A July 15 safety incident with injury tipped the scales, and he declared a work shutdown shortly after that date, again blaming “cowboys” who wouldn’t follow safety or security procedures as undermining his confidence in the entire institution. In one GLIM (Group Leader’s Information Meeting) he exclaimed, “ If innocent people get caught up in this, tough, I can’t worry about the buddy system here.” He went on to say that he was going to take them out, and if he had to restart the Lab with only 10 employees, then that’s what he was going to do. Anyone who disagreed with him was threatened with dismissal. FBI investigators had been called in to reinforce the investigation into the missing CREM. Now, however, the leakage of the plausible explanation of the two CREM items clearly forced the Director’s hand, leading to his previously mentioned admission that the CREM items might not have gone missing. Further speculation by the Lab on the CREM issue ceased, and we can only sadly conclude that this shift in emphasis was influenced by the need for the Director to justify his earlier rush to judgment in his hysterical claims about worker misconduct. Firings of scapegoats were then essential, or a loss of public trust would be the inevitable result. For the CREM issue, the ones chosen for firing were those who were not yet represented by counsel (including myself, having also been targeted back in July for not returning from vacation on my 20th wedding anniversary!) Again, the sole motivation appeared to be the protection of the Director and Laboratory management.
Case Review Boards
In August, though the Laboratory was silent about progress in the CREM case, and while we were still on investigatory leave, Case Review Boards were assembled to consider charges against us, although at the time the specific accusations were unknown. I was grilled by two specialists from Human Resources for hours on Aug.22. The central points had nothing to do with CREM, but instead concerned my interactions with colleagues, weekend and afterhours time spent at work, and my training records. On September 16th, to my complete astonishment, I received an adverse action letter that invited me to supply written arguments to show why I should not be fired for cause (supposedly the Board’s conclusion). The principal charges against me were clearly trumped up and included ridiculous allegations of misconduct, such as taking too much training. I defended myself with pertinent facts at some length, but was then terminated on September 23 by the Acting DX Division Leader, who said my arguments were not convincing. Weeks later, when I learned that I could use the Freedom of Information Act (FOIA) to request a copy of the Case Review Board findings, and finally received a redacted version of the same, I learned to my utter bewilderment that the Board’s actual recommendation was three weeks suspension without pay, contradicting the claim by the Acting DX Division Leader that the board had recommended termination. In the summary, much was also made of my delay in returning from vacation, though not included was the fact that I had permission from the DX- 3 Deputy Group Leader to remain in Washington longer. The Administrative Mandate (AM) 112 requires the examination of one’s career and a bottom up investigatory approach. These requirements were not observed, and of the 22 people questioned about me, astonishingly (in violation of AM 112.09), my own Group Leader was excluded from such discussion because of an unclear time constraint. I suspect the information she would have given would have contradicted a significant portion of the review board report and was intentionally omitted.
In a brownbag luncheon meeting held at DX-3 since my termination, the Director said that he fired me because I had been bullying people since my days at the Nevada Test Site (NTS). Because I don’t bully people and I’ve never worked at NTS, it is obvious that this information was manufactured to justify my termination. Clearly the Director believes that the power of his position affords him the freedom from accountability for his shameless mischaracterizations, untruths, and exaggerations. In my case, he has demonstrated that the end justifies the means, however sordid, and he apparently has the full support of UC and NNSA for this behavior. One further point deserves mention. When I filed for unemployment compensation in late October to help me at least feed my family while looking for employment, the Lab protested in writing that I had violated policy and deserved no consideration. I successfully refuted this claim at a formal hearing on December 17, as the official record shows. This time I was not surprised by the Lab’s actions, for I had become accustomed to its propensity for defamatory and slanderous actions.
Meanwhile, John was unaware that a Board had also been assembled to consider his case, until he also received an adverse action letter on November 23 soliciting his arguments to show why the Board’s recommendation of two weeks’ suspension without pay should not be followed for his failure to see to it that CREM under his name was not properly entered into accountability. This letter from the Acting DX Division Leader, refreshingly enough, admitted that the two CREM items were not missing after all. We are disappointed that the Director did not officially share this conclusion with the public, although he allowed a manager to put the statement in writing. John had already passed a lie detector test on November 10 before he had been reinstated, and his detailed response to the letter was confidently written with the expectation that the Board’s recommendation would have to be rescinded. Instead, he was told on December 16 that the suspension would be carried out, and that the period chosen was from December 6-17. Having no other recourse at the time, he agreed, adding that he certainly could not bear the thought of missing a Nanos brownbag lunch scheduled for December 21. Curiously, later that same day, he was told that suspensions couldn’t be postdated, and his actual time off would have to be taken from December 20 to December 31. It seems likely that management did not want any employees with detailed knowledge of the events at that meeting. A subsequent meeting held the next week required questions to be prepared beforehand and in writing. When John obtained his redacted copy of the Case Review Board’s conclusion, the redaction was so extensive that he could read little more than the date of the summary document. He is seeking a more sparingly redacted version, although he is saving the original copy in case journalistic interest in his predicament might be expressed later—perhaps before there is a decision on the UC contract competition.
In looking back, John and I must wonder whether our joint 42 years of spotless service at Los Alamos merely represents wasted sacrifice. Why do our personnel files not count in our defense instead of suddenly becoming weightless on the scales of justice? What was the point of sacrificing countless nights and weekends in the creation of the elaborate hydrotest procedural checklists, detailed schedules, and redundant signature protocols that were once so highly prized by the Director, UC, and NNSA? Why are our unique developments there still singled out as the gold standard for complex, multi-diagnostic, hydrodynamics experiments at DARHT, PHERMEX and other firing sites, exemplifying our dedication to safety and security, but now only in our absence? What is the gold standard for management conduct at Los Alamos, or is bronze even too much to expect? When the human cost is so severe, why is justice so elusive? If we had not been able to figure out the significance of the swing of numbers in the CREM tracking records, the terrible conclusion we would be battling right now is the assumption of two missing and highly classified disks, and I suspect that many other workers would have been wrongfully terminated.
(Communicated to Doug Roberts on March 2, 2005)
Not wome whiney compaint about diminished benefits under the new contract.
Not some stupid, cone-headed query about "Why can't I be allowed to continue to do whatever I want to and have the government keep paying for it?"
A post about moral values.
Or lack thereof.
Todd's wife and kids can be proud that he stood up against Authority run amok. It is sad that he had to pay such a high price, but I feel honored to have known him.
Los Alamos is populated by a bunch of cowards. Only a few people had the guts to stand up to Nanos when he was here, and Todd was one of them. What did the good citizens of Los Alamos do when Todd got into trouble for standing up to the bully?
They ran back to their comfortable little Los Alamos homes and locked the doors and shuttered the windows. Not a peep of protest was to be heard from the general populace.
should be etched into metal plates and placed at the front doors
of the new NSSB building for all future LANL employees to read.
Instead, the new NSSB building will soon be christened with great
fanfare by LANL managers who will act as if the earth shattering
events that took place during the tumultuous Summer of 2004 never
even happened. How sad.
Doug, when you pull the plug on this blog on July 1st, leaving
this post at the very top of the blog would be a most fitting
tribute to Todd. May we never forget.
God bless the few who have sacraficed so much for so many that do so little to help even just themselves.