Commercial Off the Shelf (COTS) - A Double Edged Sword

Yankee Sailor, an active duty "surface" type put up this post today on the topic of using Commercial Off-the-Shelf "stuff." Not only is it an interesting topic, for the military, as well as industry, I bumped across an article in Avionics Magazine on the very topic a few days ago. I saw some of the first big COTS projects for the Surface Navy, back in the mid-80s. The Joint Operational Tactical System (JOTS), which was fielded under the oversight of Adm Jerry O. Tuttle, gave us incredibly powerful computers that did an incredible job as providing mobile staffs and older ships with almost as much horsepower as a full blown, military developed Combat Direction System, and JOTS had 16 colors! That's not said lightly, for color tagging made for quick mental integration of the mountains of tactical info in hi tempo ops. I saw the HP9020s come in the "rapid prototyping" model, and our staff, in particular OSCS Jim Koch, had access to the JOTS code (which was running under UNIX and written in Rocky Mountain BASIC), and regularly sat with the programmers, one I recall was Dr Lee Whit, and Jim and I actually found and fixed a few bugs, which ended up in the fleet. From a configuration management standpoint, it was a software developer's nightmare, as the programmers went from ship to ship, and eagerly took "inputs" from the users, and in many cases, we (yep, I was party to it) had what we asked for last night the next morning. JOTS was first developed for use aborad ships, but became to centerpiece of the Joint Maritime Command Information System (JMCIS), being used fixed bases and with mobile units ashore In the later part of my tour on CDS 32, we also saw the same systems, using the Prototype Ocean Surveillance Tracking (POST), which morphed into the Ocean Surveillance Information System (OSIS) program, put out in the CICs of ships to help pre-process communication traffic for the Tomahawk cruise missile. Those POST instalations flew aboard the ships, after our staff reported out our results of some Operational Test Launches (OTL), to address some manpower loading issues for the THawk DataBase managers. The system ran on the HP9020 family of computers. More COTS. A then captain (later Admiral) John Gauss was the driving force behind all of that. He's also the only Captain I even saw look across the table in a meeting with Admiral Mike Boorda, and say "If the CTs steal one more of those systems, I'm pulling them off all the ships!" Adm Boorda turned to his senior cryppie and told him to fix the problem, NOW! I proceeded from CDS32 to Naval War College. One trimester was "National Decision Making." Those were fancy words for my indoctrination tot he McNamera methodology of system procurement. It was fascinating, and the most significant lesson I walked away from that trimester was: Once you buy into a level of technology, you are fundamentally stuck with it, so choose very carefully." In other words, you buy something like a fleet of radial engined cargo planes, then they get old and cranky, but if you decide to go jet, you better have a huge pile of cash, not only for new planes and engines, but to retrain your entire maintenance force, and buy all new spare parts, not to mention changing from AVGAS to JP series fuels. It goes further than that in this day and age, where we in the US don't make much of anything. Even "Made in the USA" means assembled from parts from all ove the far east, in the electronics field. Years later, I was the Systems Engineer at Fleet Combat Direction Systems Support Center, Dam Neck, VA (oh, hurt me with shore duty on the beach!). One major project I had oversight on, and was part of the training committee for the Program Manager, Capt Herb Kahler, was the Battle Force Tactical Trainer (BFTT). I also had the programs for the FFG-7, DD-963, DDG-993, non-AEGIS CG/CGN, and JTIDS projects in my shops. Those were all military developments, hardware and software included. Lots of contrast. BFTT could not have blossomed as it did w/o COTS. Capt Kahler had a novel idea: Figure out what the BFTT project would do, them instead of building it from the basement up, see what other "puzzle" pieces already exist, and with minor modification, can be integrated for the final functionality. He found the projects. He began with the Tac3 series of computer parts, and Versa Module Europa (VME) card cages. Brilliant idea. UNIX was the OS. He found a project named Tactical Decision Making Under Stress (TADMUS) then out of a Navy training command office in Orlando, headed by Dr Jan Cannon-Bowers. That program had been developed as a fall out of the problems noted with USS VINCENNES CG-49). It is difficult to practice for the real world, when you're not getting the dimension of stress. TADMUS addressed that. I learned more about Industrial Psychology than I wanted to, but that was part of the program. He also found a program written by FLTAC, Corona, which used 3D modeling to reconstruct what happened on the open ocean range testing. That was a good tool to rerun the problem for the crew critiques. On and on the list went. Herb saved us many dollars, and he "broke the mold" by not holding his entire development close to his chest. He found already spent dollars, and "recycled" them by using the functional hardware and software in the inventory. It risked him not getting all the credit, but it got us a remarkable program, which is still alive and well, as a result. The description above shows it was a mix of COTS and military work, but the COTS was the hardware component. Side notes (and bragging rights) on BFTT: The Navy was the first service to successfully conduct an system demonstration of the intergation of live, constructive and vitrual "entities" in a traing scenario. BFTT was alwys planned to be able to be intergrated into a larger Joint training and simulation system. I also posted some interesting developments on BFTT for the gaming industry in a post last February. Side note to the side note: It looks like Dr Cannon-Bowers is entrenched in the gaming aspects of Industrial Psychology. That may have been a result of decreases in funding for R&D, or more lucrative funding from the commercial developers. When I knew here, she was hot on the trail of figuring out the interactions of people in teamed environments. A noble, if not incredible difficult job, since we seem to not have personal psychology down yet, but tha't my layman's opinion. As the Systems Engineer, I also had a separate shop, where a small group of man had written the entire program for the Advanced Combat Direction System (ACDS) Block 0. It was written to run in the ruggedized MILSPEC AN/UYK-43 computers, which were the upgrade of the long term workhorse, the UYK-7 Tactical Computer. The OS for ACDS was specifically designed for combat operations. Something as simple as interrupts to the OS were managed, based on tactical prioritization. If a printer went off line, then a missile alert followed the first interrupt, then missile detect alert moved to the top of the queue. Try that with Windows based operating systems.... From a practical standpoint of equipment maintenance in any environment, just because you and I can buy the newest dual layer DVD burner, and come home and install it, it's a more significant logistical problem when your assets number in the hundreds of thousands, and are spread around the globe, some in operations and some in the repair overhaul cycle. And don't forget the ready spares warehoused all over the world. Just think of the plan to haul out all the ones you aren't going to use (which have been paid for by tax dollars), and send them to the surplus disposal system. How about those stories where someone buys a few pallets of "non longer needed" material from DMRO (I think that's the disposal office, only to have someone find out they really needed more, so they sell them back and make a killing (equals more $$$ spent). I was trained as a software safety engineer as well. In the process of ensuring weapons aren't fired while doing training, and important stuff like that, you need to examine the software code at some point in the analysis process, where you have developed safety issues. When it's a military paid for application and OS, you can do that. When Bill Gates lives under your missile fire control and engineering plant controls, he's not going to turn the source code over to you. Will a printer off line alert keep a busy operator from seeing a missile inbound alert? You'll only need about 20 seconds to find out, in the worst case scenario, and it most likely won't be pretty. Up until earlier this year, I was in the electronics recycling business. I made a pretty decent living selling "old" parts. I found there are two "spikes" in the pricing of an item or technology. High when it's new, then as it's widely used, the price goes down. The next technology comes along, and while new companies can buy the new stuff, and the old stuff is cleared from the warehouses, the old company can't replace every zip drive they have in their multinational corporation. They had to come to the electronics recyclers. In this period of the lifespan of an item, it rises again. Then, as you horde these parts as a recycler, all of a sudeen you realize everyone in the world has upgraded their systems and the stuff you have clogging your shelfs, while you dream nightly of that phone call that will allow you to retire rich and famous has been superceeded. I think it was in 2001, I had a large company call me to find a controller for an IBM mainframe system. Pentium IIIs were pretty well established and PIVs almost out. I ended up finding their controller, at another recycler, and paid $2400 for it. I sold it for $3800 and they thought they were getting a "deal" and they were, for they didn't have to replace the multimillion dollar mainframe installation. Essentially, I sold them 80386 technology. That controller wasn't much more than an IBM PC, with a few specific bells and whistles for the mainframe proprietary interfaces. The Space Shuttle computers use 80486 chips, but not just any chip. Since the coding language is Assembelr, it is important that the chip architecture is exactly as the program is desiend to work in. NASA pay hundreds of dollars for these chips, and they are out there on eBay looking and buying, as well as working with select recyclers, who know what to look for. They can't afford to rpelace the computers. In such a mission critical environment, because to development and testing costs are not in the budget. As a side note, the POLARIS Missile project was where software and system safety were initially developed and it has been NASA who has been one place responsible for continued improvemes in this discipline, hence their understanding of their current position in regard to being "fast and loose" with system procurement decisions. I like the COTS idea. I professionally benefited greatly from it. I served on both sides of the equation, as a user and as a logistical support manager for end users. It's not that it is inherently a bad idea, it's just a decision one does not approach lightly, and one where you must project into the future as best you can to weigh the present day cost savings, versus the long term consequences of support costs in the lifetime of the product. We have seen the industry crank out new processors, storage technologies, video graphics cards, etc, etc, etc like there's no tomorrow. Just recall how quickly something becomes "yesterday's news." In my PC, nothing is so critical in the type technology that I can't replace it with a similar part (right now), but if I had to use a program that would only send output to an HP1100 laser printer, then I'd have to go scavenging for one, for they are long out of production. I have a smei-issue in this vein, with my own equipment. I do have an Orb Drive I bought in 2000. It stored 2G of data on a removable cartridge, and cost less that double that for a 100M Zip drive. While I found others had reliability issues with their Orbs, I never did. I bought the external parallel port version, so as I did consulting, I could go to anyone's computer and back up, etc. It was a great idea until WinME came out, and the company that produced the Orb quit writing drivers for the parallel post interface. This is my COTS nightmare. I have to load up a system with Win98 so I can pull my old archives forward. I would like to have the data, but if it got lost, I'd not be in a life threatening condition. The Orb is around, but you now need the USB version to have it supported. In this case it's not the hardware compatability issue, but one of support at the OS level that makes it difficult. "Total cost of ownership" is the filter you must evaluate COTS purchases with.