One of the more interesting comments that kept coming up in both plenary and side-bar discussions last week at AFCEA/USNI West 2011 was a concern that the USN simply does not do enough experimentation. The reasons why are many, but here are two that came up the most; one materiel and one psychological.

1. Our Fleet is too small. Our OPTEMPO and internal churn gives us too few platforms to experiment with. While we have a few examples out there, we don’t have the “white space” on the scheduler’s XLS to put into putting ideas into the Fleet and “see what they can do with it.”

I understand that perspective and think that does have a lot to do with it, but I don’t think that explains the whole problem.

2. We Demonstrate, We Don’t Experiment. There is a distinct difference between a demonstration and an experiment. Demonstrations have as a goal “success.” There is very little risk involved. Success and a green up arrow on the PPT is the expected outcome. The downside is that something that is known and mature also tends to be by its nature to have little risk and little new, novel, or or hmmmmmmable.

An experiment though has failure as an option – and in many cases a failure is just as good as a success, as with failures you learn and can refine. With new concepts experimented on proven platforms, you isolate technology risk such that if the experiment shows promise, the path to demonstration then deployment is short. Also, a splendid idea once experimented properly could turn out to such an unworkable concept that it is halted before more effort is wasted trying to operationalize it. You also avoid embracing the happy-talk and rosy-scenario so much that you put too much experimentation in new platforms – spiking technology risk – and as a result making ships that do too little for too much money. Huge aggregated developmental costs – little operational use.

Experiment – Demonstrate – Deploy.” Much better than “Deploy – Cancel – Replace – Fix – Feed Money – Spin – Deal With It

Using what we learned in PSYCH101, I think our apparent bias against experimentation is easy to trace to it’s source. We are soaked in a culture that encourages happy-talk and self-esteem based intellectual fluffery. Look how we write out FITREPS. Look how the “Every Child Gets A Trophy” mentality has turned the NAM into simply an item on the PCS out-processing checklist.

We are very bad at both giving or taking anything less than perfect. Every Sailor is above average, every system must be transformational. Every program, concept, or process must be sold as, “Never before has .…. ”

What is a possible secondary effect of this lack of experimentation? Simple; as stated earlier – compounded technology risk in our programs.

New and immature systems almost always have growing pains. These pains cause timelines to shift to the right and for costs to increase. If you pack too much in one platform, the natural challenges with new systems compound. They compound. One delay costs another which then drives up costs downstream – rinse, repeat.

How do we re-invigorate experimentation? Well, we aren’t going to have a larger fleet, so we have to address the intellectual bias against experimentation. That will take leadership that supports creating a climate that can see failure as part of learning how to win. Culture. Money too – but money follows priorities. Leadership sets priorities.

We failed, but this is what we learned” should be rewarded and not seen as a waste. Not having everything on a stop-light chart would help too.

Do we experiment enough or not? Do the numbers back up this recurring theme – or is this just a feeling people have? In a climate of shrinking budgets – how much do we protect experimentation?

Posted by CDRSalamander in Innovation, Navy

You can leave a response, or trackback from your own site.

  • You know, this is one thing that really struck me when galrahn and I were on LCS-1 last week. I don’t want to get into that debate here because the question of experimentation is an important one, but talking to the Blue CO and XO of USS Freedom, two things became clear.

    The first was that they were entering uncharted territory — and not in some abstract discussion of requirements but at sea aboard a commissioned warship. Like the program or not, the next few years with LCS-1 and -2 may offer some of the most significant experimentation the surface navy has done in a generation in everything from manning and crew size (though it also occurs to me that LCS program has considerable chronological overlap with the era of optimal manning) to warships with fundamentally different characteristics and capabilities than the current surface fleet.

    (Whether we should be committed to a 24-ship buy at this point is probably a question for another time, but there is also certainly a danger in a model of experimenting too late in a program that has already been deemed essential.)

    The second thing that struck me was the pride and ethos of the CO and XO. They believed in LCS, and they did not hesitate to point across the pier to a Tico cruiser and make comparisons about what each was doing, what each was capable of and how many crew were available and required for a task. It was by no means derisive, but their pride in their way of doing things was clear. And that is important — a cadre of officers and sailors who not only see room for and value in doing things differently, but who have both an avenue and a career path to advocate for and experiment with it.

  • In my mind, experimentation does not require the movement of ships and aircraft, at least in the early stages. The war games conducted at the Naval War College in the 20s and 30s were a form of experimentation, and provided a wealth of information on what would be the likely course of war in the 40s. Table top exercises and simulations can also be used to experiment and understand the relationships between concepts and weapon system capabilities. Demonstration can be effective after those broader looks have thinned the herd of possibilities. The DOD Command and Control Research Program has published several books on this topic, including a Code of Best Practice for individual experiments and a discussion of Campaigns of Experimentation. There’s even a paper by Captain Rubel on Using Wargames for Command and Control Experimentation.
    The challenge of experimentation comes in managing the expectations of senior folks when they sponsor these types of experiments, and preparing them to accept the value of those cases that demonstrate failure. They must understand, going in, that if the experiment does not find a failure point then it was not worth the effort.
    A secondary and intertwined challenge is in managing the stakeholders who procure the weapon systems. None of them has an interest in demonstrating that their system lacks some capability, and many will actively prevent the use of their data if such an outcome is likely. Very senior top cover is needed early in the planning for an experiment to prevent this from being a problem.
    The third challenge is in keeping an experiment focused. Too often, I’ve seen so-called experiments that try to cram too much into their various cases. Recently, I was involved in one that aggregated many different system elements into three factors with two levels — Sensors, Networks, and Weapons in an AS-IS and TO-BE state. Certain capabilities were forbidden from exploration (for various reasons). When we looked at the overall outcome we found a statistically significant but operationally meaningless difference.
    How do we reinvigorate experiments? I’d suggest that successful execution of relatively inexpensive experiments, under very senior sponsorship, is the only way to make it happen. Bottoms-up could generate some interest in the techniques of experimentation, but then staff officers at higher levels will train their commanders to respond with the typical “so what” and “not invented here” that kills any good idea.

  • These observations were about experimentation in general, and not related to LCS.

    As for experimentation on LCS, the goal of experimentation is to try novel concepts on proven platforms. That isolates the experiment so that you can have an effective evaluation of the experiment. Running experiments on unproven platforms can easily lead to confusion as to where the positive or negative results are coming from – the experiment or the platform.

  • B. Walthrop

    I’m not trying to be exceptionally obstructionist, but I’d like to hear what CDR S. defines as a proven platform. I’m also not sure that the USN is particularly deficient in experimentation. FSF-1 and Sea Shadow are examples of ship scale experimentation. The LVI LO/LO crane at ONR is an example of system level experimentation.

    Also, if you confine the design of your experiments to “proven platforms,” are you unecessarily constraining the possibility of conducting high risk, but high pay-off experiments to only moderate or low risk but moderate or low pay off solutions? I don’t entirely disagree with your thesis, but I suspect it is a much more complicated issue than can be covered in a single blog post.


  • Mike M.

    Sal, you’ve got some good points. But I’ll add a couple more…

    1. Few advocates for experimentation in the acquisition community. The acquisition community is biased in favor of normal acquisitions, especially the big ACAT-I programs. Experiments are often regarded as sideshows diverting money and manpower from the big programs.

  • Mike M.

    2. Hand-to-mouth resources. OPTEMPO is part of this, but money is another. Every penny is being spent either on current operations, or on orthodox acquisition programs to replace existing systems. Any experiments get the table scraps.

  • Mike M.

    3. Success gets deployed. If your experiment works, it stops…and gets sent off to the war. Ask any UAV program.

  • BW,

    To clarify, I am not referring to experimenting on proven platforms, not with proven planforms. For instance – a DDG-51 Class is a proven platform. LCS is not.

    If you want to run an experiment, make sure it is “riding” on a control system that is fairly well known. If your experiment is “riding” on an experiment itself – such as LCS – then your data on the performance of that experiment will be difficult to interpret.

    For example; if you have an experiment – run it on a DDG-51 or FFG-7 ship, not LCS.

    As for a complicated question – you be it is. The defense industry pros at West2011 in San Diego had trouble putting their finger on it. That is why it is here! Free-form pondering – what a blog is best at.

  • Sal,

    Great point about how to go about this — on something with known parameters, not something where even the most basic parameters are still being understood. This method for experimentation you suggest; there’s something almost scientific about it. I feel like I’ve heard of something like it somewhere.

    Anyway, there’s a different example that also might be helpful. David Johnson wrote a fabulous account of the interwar period entitled “Fast Tanks and Heavy Bombers that I highly recommend. In the case of the former (to your and Mike M.’s points), tanks were divided between the infantry and the cavalry, with nascent armor officers without either money or institutional support. And during a period of immense fiscal austerity, experimentation with the tank both technically and doctrinally was severely constrained.

    The other, almost counter-example, is the way in which the separated Army Air Corps made immense strides during the interwar period but also drank its own kool-aid in terms of believing a properly designed bomber and a properly configured bomber formation was essentially a highly-accurate and unassailable force of modern war.

    It seems like we are often more in danger of the former scenario where we aren’t really changing how we do things, but we also face the latter, where we’re so committed to something we aren’t really checking basic assumptions.

  • Mike M.

    I know it’s late, but I’ll toss in one other observation…

    Experimentation is a habit the Navy has gotten out of.

    Take a look at Naval Aviation. In the late ’20s and early ’30s, the Navy was experimenting. This was followed by a decade of preparation for war and warfighting. In the mid 1950s, another experimentation period. Which produced the angled flight deck, routine night operations, and modern CV operations. There was an abortive experimentation period in the late 1980s and early 1990s, but it was strangled by budget cuts. A second experimentation attempt in the mid-00s – strangled by GWOT demands.

    It’s been so long since the Navy did experiments on a regular basis that the institutional memory of how to do it properly has been lost.

  • YN2(SW) H. Lucien Gauthier III

    I remember us expiramenting once aboard SAN. I won’t go into the details of it. But, when my CO gave his comments regarding it, it was fasinating, and one of the few OPS/INTEL briefings I can actually recall.

    I was lucky enough to obtain copies of Proceedings from the 1800s (Thanks again Mary!!!). One copy from 1895 has a JO writing about the bore pressures of various gun types with various types of ammunition. The article gets into the weeds by showing the equations involved in calculating the pressures. Today, when I look at Proceedings they’re all written as articles to be readily understood by the average officer/naval enthusiast.

    When I was in grade school, the school systems switched to test based advancement, where I had to pass the State mandated test to advance to the next grade. The effect this had on the teachers was that they instructed based upon the requirements of the test, and wouldn’t deviate because they time alloted to them in teaching the material was just enough to get through all the material. For those students ‘unlucky’ enough to be in class with me, they would end up days behind because of the random and indepth questions I would ask beyond the material presented (that is if I were not sleeping 😉 ).

    The same thing is happening in the Navy. From my perspective, I have no expierence in platforms that have not been fully introduced into the Fleet. I only know what it is to exercise a crew in shipboard evolutions. In these evolutions all I’ve seen is us practising the test, and I mean that, THE test. The same drill we’d run for weeks/months was what ATG would come and grade us on, with some unsubstantial change. Again and again, I’d ask my fellow POs what would happen if we had to do DC for three days solid like the COLE did. Or, how we would respond if an entire repair locker was where alfa hit. But, we couldn’t get beyond the cycle that ATG and above had us locked into, we couldn’t go beyond the standard demanded by the Fleet. We couldn’t expirament. Those in the duty section didn’t have the energy or inclination, shipwide GQs were done to ensure we met standards. The DCmen I’d talk to could go all theoretical, especially DCC Griffiths (A man’s man, and a Chief’s Chief… Loved that guy, which is why I am giving him a shout out here). I am sure the case is the same with the OSs up in combat, I would imagine that during their long mid watches they’d just start spouting off ‘what ifs’ out of sheer bordom… The officers do the same in the Wardroom? The messdecks wouldn’t.

  • Michael

    To what extent could the ships of the ghost fleet be reactivated for the purposes of experimentation? Assuming that personnel and funds could be found, using such ships would seem to be a good way of minimizing disruption to operations (since they’re no longer on active duty) and false data from unproven designs (a ship mothballed because of age is about as proven as they get).