We tend to marvel at the Darwinian perfection of organisms now, saying 'this must have been highly selected for, it's a tuned and sophisticated machine'. In fact, it's a mess - there's so much unnecessary complexity.
Bear with me. The preceding passage is from an article I found on the BBC regarding flaws in proteins that are believed to be linked to more complex biological structure emerging. While not directly related to Walt's post on bureaucracies, the passage quoted above just sort of struck me as appropriate.
Walt's basic argument is that America's foreign policy, being one that he generally describes as imperialistic, could not be sustained without a large and complex bureaucratic organism that can manage foreign policy activity. These bureaucracies, Walt argues, are more prone to secrecy because it makes the perpetuation of American foreign policy easier. Basically, secrecy allows bureaucratic actors to pursue their own policy preferences and use the tactics and strategies that they want to use. Without secrecy the public may find out about the reprehensible policies that bureaucrats are employing and put a stop to it.
Like the preceding passage, Walt's argument seems to assume that the functions of the bureaucracy are finely tuned and selected for because they better enable the bureaucracy to do its job. The first mistake here is treating the national security bureaucracy (as he refers to it) as a monolithic entity that controls foreign policy. Indeed, one of the core elements of bureaucratic politics theory is the existence of multiple bureaucratic agencies, all with their own particular mission/mandate, and that are often "competing" with one another to influence the direction of foreign policy. These theoretical models often focus on the individuals within these bureaucratic agencies, arguing that membership in a particular agency, previous experience, and one's rank within the organization all determine views and preferences. It may be more appropriate to say that these factors partially shape preferences, and partially determine the extent to which an individual can pursue their preferences. The "previous baggage" component suggests that preferences are partially determined by factors other than an individual affiliation with a particular agency. Nevertheless, bureaucratic politics theory assumes that these individuals will seek to enhance their own standing and, as a result, the standing of their organization within the broader organizational context (i.e. the Pentagon/DOD will try to maximize its influence within the broader foreign policymaking community).
Now I don't doubt that some of these bureaucratic actors enjoy having a certain level of secrecy associated with their work, as Walt suggests. This surely opens up opportunities to do things that would not otherwise be possible if these actors were subject to a higher level of public scrutiny. But I think one mistake is thinking that this is all completely deliberate. The redundancy of bureaucratic functions likely accounts for a substantial amount of the secrecy that exists in the American foreign policymaking "apparatus." Since there are multiple organizations all competing for influence within the system there is a strong incentive to not share information with those actors/organizations that are perceived to be your competitors. The FBI, CIA, NSA, DIA, and other intelligence gathering agencies are in many ways rivals when it comes to gathering intelligence. Indeed this was argued to be a major problem in the wake of the September 11 attacks. The solution? Create the Department of Homeland Security and the Director of National Intelligence to oversee much of the broader intelligence and security community.
The result? The CIA, who had arguably been top-dog in this field prior to the creation of the DNI is still far more entrenched in the national security bureaucracy than the newly created DNI. This is largely due to path dependence and the fact that the CIA had 60 years to establish its own contacts and relations with other agencies. Again we see the CIA in a position where there are strong incentives to jealously guard any agency-exclusive information in an effort to stave off the DNI's growing influence. The CIA is therefore in a position to undertake its own operations and only report successes once they have become, well, successful. Failures of course should never see the light of day.
Now this may sound a lot like what Walt is proposing. However, the major difference is that a great deal of secrecy is introduced into the system as a byproduct of other actions---not because a high level of secrecy is optimal for the pursuit of some foreign policy goal. This secrecy also exists in an intra-governmental fashion, and is not simply an issue of external transparency. Again, as we have seen over the past few years, and as many folks have argued, the refusal to share information between government agencies has resulted in a decidedly suboptimal system for ensuring our national security.
The current bureaucratic structure has not evolved, its characteristics have not been selected for, because it is the best suited system for pursuing national security. Certainly policymakers, while delegating substantial powers to the actors within these agencies, can face very negative consequences when bad policies or tactics are uncovered, so internal transparency is not necessarily desirable for political leaders either. The creation of the DNI and Homeland Security have definitely increased the size of the national security bureaucracy, and have probably increased the level of secrecy in the system, but I don't think this is the intent. These agencies were created in an effort to facilitate communication between other agencies in the system and to keep political leaders better informed. While I don't totally disagree with Walt's assessment, I think there are probably some more nuanced organizational factors that contribute a great deal to our understanding of why we see the structures and behavior that we see.