TAPE 1, SIDE 1
MR. MARTIN COLLINS: Last time we talked about--besides your TRW experience--your initial impressions and activities at NASA. One of the things you were most concerned with was developing a management structure that would allow you to direct the Apollo program. And one of the novelties of this approach was a dual reporting structure for the center directors. They reported to you as well as to the associate administrator, that is the head of the institutional chain of command.
DR. GEORGE MUELLER: That isn't quite right, Martin.
COLLINS: You can clarify that then.
MUELLER: Yes. One of the problems had been that dual reporting channel that was Brainerd Holmes' difficulty, because the center directors used that as a mechanism for doing whatever it is they wanted to do. And so when I came in, one of the requirements was that the center directors would report directly to me, those who were involved in my projects, and that was carried over to the other center directors. So that at that point in time, we set up the structure where center directors reported to associate administrators for particular areas of responsibility. Like in the case of manned space flight, we had Kennedy and Johnson [Manned Space Center], what are now Kennedy and Johnson, and Marshall reporting to me directly. This dual role I set up was one of program offices within the associate administrator structure in Washington, who had counterpart program offices in the centers. Those counterpart program offices had a dual reporting responsibility, to the center director and to the program director in Washington.
And that was an innovation that the center directors found it somewhat difficult to understand for some period of time. They said, "Well, what's our role if we have direct control from Washington's program office over our program office? What are we supposed to do?" And the answer was, "Make it work." But that took a fair amount of doing because they were not used to and didn't like having to have anyone have direct access to anyone within their department. At headquarters we didn't have that kind of a problem since Bob Seamans was used to working in a matrix management style, and Mr. Webb was so concerned about the external world that he didn't spend a great deal of time on the internal world, except when there was something in the newspapers that caused a flap.
COLLINS: So Bob Seamans didn't see any problem with this approach to handling the program element.
MUELLER: At one time, of course, he had set it up as a matrix organization with himself having the center directors reporting in and the program offices at headquarters. What he didn't have was any reasonable mechanism for the program offices at headquarters to control what was happening in the centers. Yes, small projects, then it's better to decentralize them, but if you have a project that goes across several centers, you've got to have a central control point or else you run into all sorts of problems--the lack of coordination, lack of real understanding of what the other centers are doing.
COLLINS: A control point that reaches down to the working level of the program.
COLLINS: Not just through the center director.
MUELLER: Not step-by-step bureaucracy. At the same time that we put in the program offices, I did another thing which was quite unusual at that time. I set up a structure under the program director of five elements of reporting, each of which had counterparts in the centers and each of which had direct communications with their counterparts. Now that doesn't mean they had direct authority but direct communications so, for example, there was one on program control and one on systems engineering. But because of my experience in the ballistic missile program, unless somebody became responsible for testing and that testing went all the way down and up through the organization, you really never knew how the test program was going. And so I had a separate line of communications and really control on testing, one on reliability and quality and one on operations, so you had all of the major elements of the program separated out so that there was somebody reponsible for each one, all the way down to the contractor.
COLLINS: When you say somebody, are you talking about a group of people or an individual? How was that responsibility assigned?
MUELLER: It's to a person who in turn has an organization supporting him at each place, and the organization supporting him may be only one or two people or it could be a whole group, but in any event he was responsible for the testing on Apollo. He had to make sure that the proper sets of tests were defined, that the proper control over those tests, making sure they were observed, making sure they took place, making sure that results were understood before he cleared that test as being complete. So there was a whole infrastructure that provided a clear set of interfaces between the engineering and the delivery to the operational elements.
COLLINS: And the counterpart offices at headquarters in the program office were to essentially coordinate various center activities, is that correct?
MUELLER: Coordinate and direct. They were responsible for setting up the guidelines for testing. I'm using testing as an example. So that the right set of tests were developed because you had to develop the tests. It's important to do the testing that's needed and not testing that's unnecessary, and they worked in that area. They worked both with engineering and with reliability and quality so as to be sure that they understood what the engineering was about and how to test it. Also the reliability people wanted to know how much testing they had to do and what the results had to be in order to clear them for flight. And that then from headquarters filtered down so that there was then a structure of testing defined, and the actual tests were defined at the field centers. Really they were defined at the contractors, and the field centers made sure that that set of tests were properly documented and properly carried out. So there was a chain of increasing detail all the way down to the actual guy doing the testing, but there was also a chain of communications and a chain of understanding working both ways through that organizational structure. That gave us some assurance that they were being carried out, and somebody was paying attention to the results.
COLLINS: Right. But again using the testing as an example, did the testing office in the center report to the testing office at headquarters?
COLLINS: It was a line relationship?
MUELLER: With respect to the testing. B ut, of course, he also reported to the program project director at the center, and he, the test guy at headquarters, reported to the program director at headquarters. He couldn't independently say, "Gee, I want to double that testing." He had to go back and say, "We need to do more testing, and it's going to cost this much money and going to take this much time. Do you agree?"
COLLINS: So there were actually at least a couple of levels of dual reporting.
MUELLER: Yes, indeed. That's one of the small problems that took a while to get established. And one of the things that tends-- it's a matter of management style and tends to change from manager to manager. Later on when John Yardley took over, much of the form stayed in place, but the function disappeared. So you had a bureaucracy doing things now, rather than people who really understood or who were really responsible for what was happening.
COLLINS: I'm not sure what the distinction is there. You've still got people manning the bureaucracy. I mean your organization is in a sense a bureaucracy, as a large institution. What's the difference in the two situations between an individual having responsibility and simply a bureaucracy working through a problem?
MUELLER: It's one of those subtle distinctions which make all the difference in the success or failure of a project. But as time passes, the program director decides he's going to work more directly with the contractor, for example, and bypasses this chain of command. Once you start bypassing it, its effectiveness dwindles rather rapidly. So if John Yardley, for example, spends time with the contractor saying, "Now these are the tests we want to run," the first thing you know, the test director up here at headquarters is nothing more than a bookkeeper and has given up his feeling of responsibility for what those tests should be. And that kind of penetrates through the entire organization. I'm not picking out John Yardley particularly, but it's just an example.
COLLINS: From your point of view and your key peoples' point of view, there's a certain vigilance necessary to maintain the spirit of the reporting structure.
MUELLER: You've got to have people feeling responsible and holding them responsible for the results in each of these major areas.
COLLINS: And, in part, it seems what you're striving to do was to get people at comparable technical levels feeling responsible for particular areas.
MUELLER: I'm not quite sure how to phrase that, but it is to build in a feeling that they had to know what was going on and be responsible for being sure that it was right. That it was going to do the things that needed to be done in order to assure the safety first, but secondly, to make sure it was going to work when you got it up there.
COLLINS: Yes. This may be a difficult question to answer but, depending on the approach, say, your approach of individual responsibility versus--for shorthand, call it a bureaucratic approach--what effect does that have on the contractors in terms of how they respond to the organization, to NASA?
MUELLER: Well, if you don't have people who are in detail responsible for these various elements, the contractor tends to do what he thinks is the best for the total program. I don't know of any contractor who deliberately does something that would be inimical to the success of the program, but without some emphasis on all of these elements and an equal emphasis on them, because each are equally important, you find that one gets emphasis and the others don't. That's where the real danger comes in from relieving the individuals of the responsibility for their particular segment working. And the contractor will do what he thinks is best, which may be not necessarily best for the program as a whole, because he doesn't have the picture of the program as a whole. That was one reason why I insisted on having these meetings of all of the key contractor personnel on a regular basis so they had a chance to see what other people were doing and what the problems were for the program as a whole.
COLLINS: This is what was known as the Apollo Executive Group?
MUELLER: That was the upper level of that, but underneath that, the various program directors also got together, so that was a multilevel approach as well. The project directors from the centers and their counterparts in the contractors pre-met before the Apollo Executives Committee met, so that everybody knew what everybody else was going on, and they could brief their bosses. That was why we wanted the Apollo Execs Group, so that the people down the line would have an opportunity to recognize that what they were doing was going to get visibility by their boss's boss's boss. You know, if you can really make sure that people understand what it is they need to do, they can do it. But if they don't know what it is they need to do, they'll do something.
COLLINS: Were these meetings scheduled on a regular basis or as needed? How did that work?
MUELLER: They were scheduled on a regular basis. We tended to meet sometimes twice a year, sometimes four times a year. During the crucial times it was four times a year.
COLLINS: To what degree did you need to be concerned about, not necessarily the corporation's technical expertise, but its management capability in handling the contract work? To what extent did you get involved in management problems within the corporation? That was essentially one of the things that TRW had to be concerned about when it was working in the ballistic missile program. To what degree did you see that again when you were working on Apollo?
MUELLER: To some extent we were probably as aggressive in managing, in structuring the management of the contractor in Apollo, maybe more aggressive, than we were in the ballistic missile program. But the Apollo Executives Group was an excellent forum for that because that gave me at least the ability to talk to these executives on a first name basis and with a real understanding of what we were trying to accomplish and, I believe, built in them some confidence in our ability to know what the problems were that we were having and that, in turn, they were having. We had to restructure Rockwell with North American twice, three times.
COLLINS: Including before the fire?
MUELLER: Oh, before the fire, yes. And after the fire, too. But that was only one. We did some fair amount of counseling with other contractors whenever we ran into a problem. You know, this idea of central program control was not one that was widely used at that time. In fact, it was almost unheard-of. And so it took a while to get people who could understand and work in that framework in the key positions through the structure. But whenever it was possible, we did that. If we had to make changes, we tried to do it in a way that the contractor himself felt like he had made the decision. The real key to having something like this work very well is to get people to do what they ought to do naturally. Let them invent the solution.
COLLINS: A couple of questions. The first one, were there no comparable projects in DOD [Department of Defense] that were managed in a way similar to Apollo, under their large development projects? Is that what you're saying, that the contractors had not had the experience of working with an Apollo-type organization?
MUELLER: In large measure that's correct. There were comparable things in the DOD, but remember there was a revolution in management of programs at that time. Benny [Shriever] and Si Ramo and Dean Wooldridge and a small group at Ramo-Wooldridge began this process, and I got myself involved in it, of course, and that revolutionalized how the Air Force and eventually the rest of DOD went about organizing and carrying out programs. The submarine program was a direct offshoot in management style of Benny's organization. So this was a recognition because those programs were large compared to any programs that we had undertaken as a nation before, and so we had to develop new management techniques to make them work.
COLLINS: A large part of the literature relating to the NASA effort talks about the NASA/industry partnership. Clearly this notion of conveying the need for organizational changes in the corporation at some particular point of development, this seems to be a test of this notion of a partnership. I wonder if you might just look at North American as an instance of how it actually worked and say what was the first case, if you can recall, where some restructuring of corporate resources took place?
MUELLER: Let's see. You can start out with the fact that the S-2 stage, when we first found that the engineering, the technology that was being adopted for building the S-2 stage was not working. We had some very competent people at Marshall in mechanical structures, and so we put together a task group that went out to review the work that was going on at that time at Downey. They had just been building the Huntington Beach facility. The task group reviewed their whole process, decided that the person that they had in charge of it wasn't going to get there from here. And so they met with Stormy, who was running it. Stormy [Harrison Storms] at that time, at least, had a well-deserved reputation for storming, and so there was a fair discussion of that. Eventually it got to the point where I had to go out and spend some time chatting with Stormy about the fact that, look, we just can't get there from here. You've got to get some real talent into this thing. And so he brought in their chief engineer, Ralph Ruud, who came down and began to oversee that development process. Tremendously capable person. And in three or four months it began to get better. And so it did.
COLLINS: In this particular case, what was one importing into the situation by bringing in the new individual? Was it just more experience with managing a program of this type, more technical insight? What was brought?
MUELLER: Someone who could bring in the talent necessary and the dedication to getting it done. That combination of technical, management, and inspirational leadership, which isn't widely distributed among people. You've got to get the right person with the right background in place so that you can get it done properly.
COLLINS: Right. What specifically, if you can recall, was the technical problem that North American/Rockwell was running into with the S-2?
MUELLER: One was the welding. These were huge cylinders, and getting a uniform weld--they were having hundreds of imperfections in the welds. They had to develop a whole new technique for welding. You just didn't do it by hand. You had to get an automated system for doing the welding, and that required a major investment, as well as some understanding of how to go about doing that. That was a whole new technology that was developed and eventually became used widely. But you had to get the right person in because the guy that was there was used to hand welding, and he thought, gee, all I have to do is improve the quality of my welds by learning how to do it better and training people better.
COLLINS: Was the hand weld the typical method in industry at that time?
MUELLER: Yes. Although automated welding had been developed, not for anything as complex as this S-2 stage interface was. The other one was the business of insulating it. Originally there'd been thought of using insulating tiles slaved on the inside, but that was taking a very large amount of time, so there needed to be some innovation as to how you go about insulating such a structure. They developed a foam that could be foamed in place and then machined down. You know, you had two choices. One is insulate on the outside where you could use tiles, but that was deemed to be less effective than insulating on the inside, where you didn't then have all of the conducting structure exposed to the liquid hydrogen, liquid oxygen.
COLLINS: So through the people in the centers who had responsibility, in this case it would be Marshall, for keeping abreast of the oversight in North American, these problems were discovered.
COLLINS: Did they then pass this up? How did the resolution of these things work out? Did the Marshall people first try to work directly with North American, and then after not succeeding in getting it fully resolved, bring it up to you? How did that mechanism work for resolving these kinds of issues?
MUELLER: We were on a tight time schedule, so there wasn't a lot of time for reiteration--I mean for solving the problem several times. Sure. First of all, the contractor tried to solve the problem. Then, when it wasn't getting solved, Marshall sent out their task group. But when it wasn't getting solved the first time, this chain of communications highlighted it back in our monthly management meeting, so a problem like that would never be more than a month old before it got--and usually when a real problem occurred, you heard about it that same day. In this particular case, it took several months for it to become obvious it was going to be a real problem, so then you went through the various stages. Eventually our engineers at headquarters were always involved in the various staff meetings and the meetings with the contractors, so that we had a direct involvement all the way down.
Now that was one of the requirements that we set up in this rather elaborate structure for program direction. They weren't directing the activity but they were observing it so that they had firsthand knowledge of what was going on, and they were very competent people. Our systems engineers at headquarters were the Bellcom folks, and that worked out very well. Not that they provided direct direction to anybody, but they were able to influence, cause people to understand what the problems were, in a way that was most effective. And we tried to do the same thing at every level. The systems engineers at the centers were not the engineering specialists that would go out and solve a problem. That came from the engineering organization which was the center's strength. But they were the ones that really understood the whole implication, and they were systems engineers versus detail.
COLLINS: So in the case of the S-2, it was convincing North American that there was a problem and the problem needed to be handled in a certain way, in this case by bringing in the right person to provide the management and leadership.
COLLINS: Were there any other examples?
MUELLER: Sure, there was another one with North American. You know, North American had a major part of the program. J-2 engine and the F-1 engine were their engine things. We had some instability problems. They had what they called pops. Those pops were fairly substantial explosions that blow up part of the engine, and there weren't very many of us that were enthusiastic about having it pop when we were flying. So we paid a great deal of attention to getting the testing done properly and finding out what the mechanisms were for the instabilities. In the course of that, again, we found that the people who were originally involved in the design were less of the analytical variety. They really did it more as an art form. It still is an art form. But they didn't use all of the analytical tools that were available at the time. So we influenced them to increase the amount of analysis they were doing substantially and also to bring in some people on the engine program who could, in fact, understand in a theoretical sense what was going on and begin to get a better relationship between the engineers who were building it and the people who were trying to understand how it was working really.
COLLINS: Did Marshall also oversee the engine?
MUELLER: Marshall oversaw the progress on that.
COLLINS: I guess it's a question of, how did these things happen in the corporate setting? Was it that they thought they had good people, but they weren't quite as good as they thought they were? Or was it that they had a lot of other activities going on, and perhaps some of their better people were doing other things, and some of their second tier people were assigned to these efforts?
MUELLER: All of these things happened. In the case of the engines, it was more the people who had grown up with the Navajo were doing the design and the development. They hadn't brought in the young folks who had a theoretical background to the same extent as they should have, because they knew how to do the engine. So you really had a cultural problem there. In that case we were the only engine people at the time, so it wasn't a conflict for other purposes, but there was a lack of understanding of what one could do in this time frame. So what we did was get some of the research people over working with the engineers and get a better set of things going. Looking back, there were a myriad of those problems. There wasn't any lack of challenge. One of the things that became most important is to be sure that the people doing the actual work understood how important their work was in making the whole thing come out right.
COLLINS: You mean in the corporate setting.
MUELLER: In the corporate setting. And for that matter in NASA as well, because you've got to start from all the way down and make sure everybody understands how important it is that they do their job right. You know, that is a problem in a place like North American where they're unionized. You've got to get the unions involved in this, and they are not by themselves anxious to let the management tell them how or what to do. In any corporate setting you have a whole set of constraints that--and you now have them in government, too--which are overriding to some extent the ability to directly control and cause things to happen. And so it was essential to build in a rapport with the unions in this process as well. In fact, that later on turned out to be a very important ingredient because particularly in constructing these new facilities, we had to have nonunion people working with union people because some contractors were unionized, some weren't. We had to bring them all together to get this thing put together and built down at the Cape. We had a very senior person who spent his full time eventually getting the unions all working with us to get this thing done.
COLLINS: What were some of the typical constraints in working with the unions? Was it work rules? What were the basic elements that sometimes came into conflict between what NASA and the corporation needed to do and the union's way of doing things?
MUELLER: There's work rules. There's also the ability to direct activities. In many instances you don't want to wait for a welder to do his thing, and then a wire man to come in, an electrician to come in and do his thing, and then the welder come back because that takes hours. Whereas you can do the whole job if you get them both working at the same time and do it in minutes. So work rules were an important ingredient. Directing, the ability to direct people. Because normally in one of these shops, you direct, you talk to the shop foreman, who then talks to the workers, and you get that chain going. So the engineer can't go and say, "Look, that's not right." He's got to go to the floor manager. Eventually what we had was engineers brought down on the floor, along with the shop foremen, along with the union foremen, and got them all working together on a problem instead of trying to do it serially or writing notes to each other.
COLLINS: What other constraints on the corporation were there? You mentioned unions. What other constraints did you have to work within, in terms of being able to work effectively?
MUELLER: In any corporation there's a whole set of procedures that they've developed, which may or may not make any sense in your particular project, but which are almost impossible to change. Their bureaucracy is such. You had to either work around those procedures or get them to change. In many instances that meant you had to change people, since they had invented the procedures, and they couldn't conceive of how doing it some other way would be better. When you think of the number of subsystems within Apollo, for example, and recognize that each one of those had its own set of people working on it and its own set of problems, it was essential to provide an in-depth understanding of all of those. And so you had real decentralization of responsibilities to accomplish, but the ability, when there was a problem, to surface it rapidly and get enough talent brought to bear to solve it. So we didn't arbitrarily change procedures, but on the other hand they represented a constraint, and many times we had to change them.
Introducing PERT, for example, is a good example of trying to change a set of procedures because every contractor has his own way of keeping tabs on what's going on. And overlaying a common system is a problem because "they weren't invented here." Obviously that can't be the best way of doing it. And I'm not sure we ever really succeeded in getting PERT as being a common system that everybody actually used. Everyone gave lip service to it and tied it in to their existing systems and that was fine because then you had a common reporting system. But if they didn't make the tie well enough, then PERT went off on its own, and you discovered somewhat later that there wasn't any relationship. And then we had a new manager.
COLLINS: I guess then also PERT had its variations as well. There was no one "the" PERT system?
MUELLER: Sure. Well, as a matter of fact, there was. We had a standard PERT system that would go all the way up to headquarters.
TAPE 1, SIDE 2
COLLINS: When you were confronted with a problem, it sounds like the North American ones were solvable. But what were your options? Did you ever have to, say, consider or want to consider taking a contract away from one corporation and trying to award it to someone else that you thought might be more competent?
MUELLER: We did that a couple of times. There were a couple of contractors who went bankrupt in the process, and we were forced to move in and essentially take them over. Particularly in the development of the crawlers, we had a couple of construction contractors that ran into problems. And because we needed them, we didn't just default them. We went to work and essentially took over the management of their contractor.
COLLINS: These must have been the subcontractors.
COLLINS: I'm not clear. Does that mean actually moving in NASA people?
MUELLER: No, we got our prime contractor to do it. Well, in one case I guess we actually had a NASA guy go down and run it, just to get it done. But that was down at the Cape. Wherever we could, we had two contractors. We tried to get at least two contractors building wherever there were a number of systems to build. You couldn't do that on the J-2 engine, but you could do that with some of the parts of the J-2 engine. Occasionally one guy failed to produce, and so we had to change contractors. We didn't change any of our major contractors. It was always at a subcontractor or vendor level that we shifted.
COLLINS: Was that a sense that either their job was adequate enough that it could be improved to the appropriate level, or that there just wasn't a sense you really could go anyplace else? You had to make this particular situation work?
MUELLER: I think it was clear that they had the capability of doing it, but changing a major contractor is something that one doesn't do very readily. I mean you had tooling and everything else in place, so you're stuck once you make that kind of decision. The only thing you can do is to change some personnel and bring enough pressure to bear to cause something to happen. But you aren't going to change contractors. There's a hue and cry in Congress to change out Thiokol. Well, you start to do that. You're going to start over from scratch and build a new system. You're not going to change the contractor and expect to build a Thiokol rocket.
COLLINS: Does this tie in with your interest as a manager in having parallel development throughout?
MUELLER: Absolutely. That's the best way I know of to achieve a program in a reasonably cost-effective fashion. It is amazing what that kind of competition does accomplish in terms of saving time, costs. And time is money, and improving the quality and the results.
COLLINS: Improving in a sense. If you have a parallel development, the two corporations who are involved are aware of it. And that's the motivating factor for them to produce something of quality, on time and in budget?
MUELLER: Oh, yes. And they have different approaches, and at the end you have an opportunity to choose the best of the two. Certainly that was illustrated in the ballistic missile program very well indeed. There Benny had five different missile programs going in parallel. Eventually all of them produced usable vehicles, but the one that eventually won out was the last one started, which was Minuteman. In Apollo we didn't have that luxury.
COLLINS: That takes us back, I guess, to where we started. And that is, when you walked in and became associate administrator for manned space flight, many of the elements of the program were already set. Most of the major contracts had been let. The basic organization within NASA had been set up. So you had, I think, a difficult task. You didn't have a lot of latitude for redirecting or organizing the thing, conceiving it from its initiation. You had to work with what your set of circumstances were. I guess I want to get a sense of what it was like walking into that situation, as opposed to having been on the ballistic missile program there at the beginning, and walking into a situation in which many of the key decisions had already been made.
MUELLER: It wasn't exactly a new situation for me because while I was at Ramo-Wooldridge, we had begun a parallel effort with Marshall in building the first of the probes. Then when STL was spun off, we started to work on the Apollo program before it got to be called Apollo. We were one of the bidders with General Electric for the capsule for the Apollo program. And we had bid to Marshall to take over systems integration for the program because it seemed to me important that we have a systems integration contractor, and Marshall thought that would be a good idea, too. Wernher went back and sold it to Jim Webb, except that he didn't sell us; he sold the idea. And Jim said, "That's a great idea. We'll give it to GE." He was distributing contracts around. Unfortunately, GE is not particularly well-qualified to be a systems integration contractor. They didn't really have the background or the people to do it at that time. So I was fairly well current as to what was going on. Joe Shea, who was really the chief engineer on Apollo, used to work for me at STL, so I was fairly well aware of what he was doing.
I got there at a time when they were still thinking of building the Nova, but they had sort of settled in on the Saturn V and the Saturn I vehicles. The Nova, you know, was going to be an even bigger vehicle that was capable of direct flight to the moon and back. The argument about lunar orbit rendezvous versus earth orbit rendezvous was almost settled, but it was still up in the air, and I got there in time to defend the decision for lunar orbit rendezvous. That was, under the circumstances, the right decision, but you could just as well have done earth orbit rendezvous, but it made it somewhat more complex. You would have been better off energy-wise to do an earth orbit rendezvous and a lunar orbit rendezvous. That's the best energy-efficient system because it's got more stages. But the complexity then became considerably more. So we went with the lunar orbit rendezvous but not without having to commit yourself before the President's Scientific Advisory Board because Jerry Wiesner didn't believe in that.
And we, coming in when it was still amorphous, had begun to get focussed. Coming in at a time, also, when the original contracts were all overrun by substantial amounts because there hadn't been any clear definition of what it was that they were going to build before they started the contracting process. And there was a sort of an adversarial relationship building up between the contractors and the various contracting officers in the centers. Because on the one hand the contracting officer said, "This is what the contract says," and the contractor says, "Yes, but we didn't know what it was you wanted." And so you get into one of these things that create problems. Congress had gotten involved with the interplay between Jim Webb and Brainerd Holmes, and they'd also gotten wind--because they always hear about contractor problems--of the fact there were massive overruns coming up. Their answer to that was, "We'll cut back the funding." So I got there just in time to have them reduce the funding by, I don't know, 750 million dollars or something for that year. So it was an exciting time.
COLLINS: How did you begin to resolve this issue of better defining what was to be asked of the contractors?
MUELLER: We set up a couple of months of really intensive study of what could be done and what should be done and what were the probable schedules for things. It became apparent that if we did things the way that they were then planned to be done, we would never get to the moon by 1970. So we decided to do some bold new initiatives, and I introduced the idea of all-up testing. I eliminated the Nova. I severely restricted the work going on on the Saturn 1-B. I left it in as a backup, though. I constrained the Gemini program and gave them an ultimatum: either they got it done or they didn't. We set up a schedule that they said was impossible and generally put together a new program and defined it.
And then began to do a serious amount of work on defining, on developing development plans for each of the elements, getting people to commit to what really was required, to get specs written, all of the myriad of details that you need in order to assure that you've got all you need and that they are all going to work together. One of the first things we did was introduce the idea of interface specifications and make sure they got created so that when you built the S-4B down at Huntington Beach and the S-1 stage down at Michoud, that they would, in fact, mate and that they had the same set of wires going across. And that was not at all clear when I came in, that that really was being done effectively.
COLLINS: Was it a question of things like design specifications having not been done, or interface specifications not having been done, or that they were done but somehow not getting to the right places, the right people?
MUELLER: In many instances, they weren't done. They were talked about. You get a piece of paper like this one, and it says all the right words, but somebody has to go make sure it happens. And there wasn't really the awareness of this thing as a system that all had to work together. Everybody was working on their pieces. But it takes a considerable rigor in order to get all of these pieces to work together and to avoid having to rebuild them when you get them down at the Cape. And that kind of discipline was new to the program, or at least was fairly new to the program, and that's where we spent a good deal of time implementing it. The other thing, of course, is we had to renegotiate all the contracts because they were all building something different than we wanted, than we needed at that point in time. And so we went through and introduced the idea of incentive contracts and renegotiated all of the fixed-price contracts over to cost-plus incentive fees.
COLLINS: You mean cost-plus fixed fee. Is that what you mean?
MUELLER: No, I mean cost-plus incentive fee.
COLLINS: No, I mean the original contracts were cost-plus fixed fee?
MUELLER: Some were cost-plus fixed fee. Some were fixed price. You had a whole myriad of contracts. But we eventually changed, I think, all of them over to cost-plus incentive fees. And that was effective in almost every instance because it got management's attention.
COLLINS: But then obviously it was more incumbent upon you to provide precise specifications for what it was that you wanted.
MUELLER: And that was one of the great benefits of having the incentive fee structure. It forced us in NASA to define what it is we wanted quite clearly and to measure our progress against the contractor's progress. Yes. It's a two-way street. And most effective in a research and development program.
COLLINS: Surely some of the people at NASA when you came in must have been familiar with the general approach for project management developed through the ballistic missile effort.
MUELLER: Oh, yes. Remember, of course, Marshall had been building systems for some time, and they actually built the Jupiter series of vehicles. They had a fairly good project structure, but it was not one that was independent. Wernher was the project manager--Eberhardt Ries was the project manager really, and so they weren't really geared up for building three or four stages all at once. Or in this case they had five different contractors working away from their plant because most of the Jupiters were actually built at Marshall. So it was a major restructuring for them.
In the case of what is now Johnson Manned Spacecraft Center, that really hadn't been started. They had made the decision to put the center down at Houston, but they were still partly at Goddard and partly living in various buildings around Houston while they were still building the complex at Clear Lake. So it was an amorphous thing, and they didn't have much of any structure, just some really highly dedicated people who were out there working their tails off.
COLLINS: So if I can kind of put words in your mouth, your contribution was to come in and give this effort strong central direction and lay out those tools that were required to do the job. That is, especially these various kinds of baseline documents, design specifications, interface specifications, and that kind of thing that would allow one to effectively and quickly move ahead. Is that a fair summary?
MUELLER: Partial. Because I also had to do a fair part of the decision-making about the technical approaches that we were going to use. Interestingly enough, I got myself and our team at headquarters involved in many of the technical decisions. Joe Shea, of course, was the one who carried out the lunar orbit rendezvous arguments and developed that. George Low was involved in many of the spacecraft decisions. We worked as a fairly closely knit team in doing the technical review because the basis of our schedule was a very clear understanding of the technical characteristics of these various elements, and we played a significant role in the beginning in defining what technology we were going to use and what direction we were going to take.
COLLINS: So that the goal could be achieved within the time frame.
MUELLER: Yes. You can't just do the structure. You've got to have an understanding of what it is, the system is, if you're going to succeed in these things. That's one of the failings of our modern management approach, modern management religion, and that is anybody can manage anything. The fact of the matter is unless you understand what it is you're going to be building in some depth and detail, you aren't going to succeed. We managed to demonstrate that fairly conclusively in a number of instances. So I don't want to overemphasize the structure of what we set in place, although that is an essential and a necessity. It is also equally essential and necessary to really understand at every level in the organization the technology you're dealing with and make the right engineering decisions. You can't really do that with a structure. You've got to have people.
In fact, that's what worries me right now about the Space Shuttle, the things they're doing in terms of trying to get it relaunched. They've got themselves a marvelous set of paper. Everybody and his brother has to sign off on every element that's going in down at the Cape. But I'm worried that there isn't anybody going to look at the piece that's there and make sure it's working. You can sign all the paper you like, but that little black box has to be operating properly. Somebody has to really know that it's operating properly and be willing to say, "Yes, that's going to work." You can overlay enough paper so that that no longer becomes important as to whether it's working or not. It's just that you've got the right set of signatures.
COLLINS: I would like to go into this management structure here if we have time. Do we have a few more minutes to discuss this?
COLLINS: Okay. We've discussed kind of the basic manner in which things have operated, the kinds of changes that you wanted to institute, did institute, to bring the program on track as you saw it. I'm curious just how your sense of how to handle this large program fit in with the existent structure for talking about project management and that kind of thing. There was at the inception of the project, as laid out in this document here, which is entitled "General Management Instruction No. 4-1-1, March 8, 1963": a project begins with a project proposal accompanied by a program approval document. It went up the chain. And then that to be accompanied by a program development plan. How did that setup fit in with your sense of how projects ought to be handled and how accountability ought to be assigned?
MUELLER: Brainerd Holmes, of course, came from RCA, and RCA in turn had worked with the Air Force for some period of time, and so did Bob Seamans come from RCA. Both of them, then, had been exposed in the course of their latter careers to program management as practiced by the Air Force. This particular document is one that I didn't happen to see, but the NMI is one that we worked with because it fit in fairly well with my approach. The only difficulty was that the Program Development Plan, which is called for here, didn't exist. The PAD existed, but it did not really define the program. Again, here's a case of having the form but not the substance of the program. And like in any of these things, some parts of it were fairly well defined. Other parts were fairly well undefined. It wasn't until several years later, for example, that the LEM really got itself defined, the Lunar Excursion Module, because the decision to go to lunar orbital rendezvous had just been made, and that meant that the Apollo spacecraft was not defined either. And so although the concept was there, it was not even reduced to any drawings.
Just looking at this, I would have guessed that that was written in part by Brainerd Holmes, in part by Joe Shea, and in part by someone over in Bob Seamans' organization. Just a guess. Field center roles, as is described here, is essentially the role that the field centers had written in for themselves. The problem with this particular instruction was that it gave the field centers the impression that they were--and they were--responsible, and the responsibility of headquarters was to get the money they needed in order to carry out the programs that they were responsible for. That's the syndrome that continues to this day in NASA. That the function of headquarters is to provide funds and protection from Congress and other OMB and stuff, and that the centers are, as they really are, the centers for getting the work done. Now there's a subtle difference between that and having the headquarters describe what the work is that's going to get done. That's where much of the confusion comes time and time again. So the form of this is one that is clearly a very useful thing. I think it persists to this day. What happens is it depends on how you implement it and in particular what the substance is that one builds into this overall structure.
COLLINS: The most fully articulated document is the Program Development Plan. Did you play any role in suggesting to the people in the manned spaceflight area to what level of detail this thing needed to be filled out?
MUELLER: Oh, yes. Oh, yes. One of the first things we did was begin to develop a Program Development Plan because although they had something called a Program Development Plan, it didn't really tell you anything about the program. We had to make that a living document, and that's not something you write up here at headquarters. That's something you cause the centers to write, and within a framework that you've established at headquarters. That was one of the chief contributions of Bellcomm is to make sure that we did, in fact, get a coherent Program Development Plan and along with it the necessary program control and scheduling documents. So it was a combination of Bellcomm and our program control office that built this Program Development Plan structure and corresponding elements at the centers and corresponding elements at the contractors because you have to get the contractors involved, as well, if you want to have the necessary detail. So there was a whole set of task groups set up to--let's get this done.
COLLINS: And were the key working documents, design specifications, interface specifications, a subset of the Program Development Plan?
MUELLER: That's right. Here you have a hierarchy of documents to run down. The Program Development Plan is a fairly top-level one, but system specs are the controlling documents and the system schedules and the funding.
COLLINS: And Bellcomm provided a systems perspective to make sure.
MUELLER: And we had a very strong program control organization that provided the reality in terms of schedules and dollar funding and the relationship to the engineering. The Program Development Plan was chiefly the responsibility of the program control organization. The system specs were chiefly the responsibility of the systems engineering organization, but they cross-couple, obviously, so they worked very closely together as a task group to get that done. And the other elements, the testing and operations and so on, all play a part because a key part of the Program Development Plan is the test plan and the operations plan that goes with it. Of course, there's an overlay of reliability and quality that goes over that as well.
COLLINS: So, in part, these specific program area offices that you created not only exercised an oversight to the actual work but were a very useful part of creating the documents that guided the work.
MUELLER: They were the key people in making sure that those documents--that the concepts were reduced to something that was buildable. Testable and operationable.
COLLINS: Maybe that is a good stopping point for today, if you wish.
MUELLER: Yes. Sounds great.
MUELLER: Thank you, Martin.