Ostensibly to support the information needs of the Undersecretary of Defense for Personnel and Readiness (and of other entities affiliated with the DoD), the DMDC maintains a highly aggressive schedule of personnel surveys, such that individuals may be contacted for multiple surveys each year. Information that derives from sample surveys could potentially be useful in formulating personnel policies and decisions. However, the statistical methods used in DMDC surveys are so shitty that really none of the numbers or figures provided in DMDC reports should be trusted at all. The DMDC's poor performance severely affects the DoD's ability to carry out its missions, and reflects gross mismanagement and an egregious waste of funds (millions of dollars per year). That is millions of fucking taxpayer dollars, per year, plus the time of hundreds of thousands of people contacted to participate in the surveys, to produce ridiculous piles of shit that are supposed to satisfy some insipid DoD bigshots' information needs.
The root cause of this hideous waste is toxic and abusive leadership within the DMDC. Toxic leadership is acknowledged to be severe problem throughout the DoD, but is outrageously so at the DMDC. The DMDC suffers from a lack of professionalism at all levels, but most acutely so within the managerial class. An employee who dares display a modicum of integrity risks being subjected to a “Perfomance Improvement Plan”, which entails many hours of opprobrious and excruciating supervisory “counseling”, and almost invariably culminates in the employee’s removal from the federal service. The DMDC will simply not be able to correct its statistical mistakes, and begin to generate useful products, until its virulent management troubles have been fully rectified. Unfortunately, the DMDC seems to be run by incorrigible, poopyheaded scoundrels who just don't give a damn.
Deficiencies in DMDC Statistical Methods
From September, 2006, until July, 2008, I held the position of Lead Survey Statistician with the DMDC. My principal duties included providing advice on the quality of databases, and on appropriate methods of statistical analysis for sample surveys. To that end, I identified a number of critical deficiencies with the DMDC's surveys whilst tasked to write statistical methodology reports to describe the sample design, sample selection, weighting, and estimation procedures for the DMDC's 2006 Survey of Active Duty Spouses (2006 ADSS) and for the DMDC's 2006 Survey of Reserve Component Spouses (2006 RCSS).Briefly:
- stratification of the sample without proper consideration of the survey objectives rendered impossible the attainment of reportable information for many desired population groups;
- when problems were observed in the sampling, no efforts were made to re-stratify to correct the problems--instead, the decision was made to accept the fact that some of the wanted information would be unattainable;
- extreme over-stratification caused many of the sampling strata to have very small numbers of respondents, both expected and actual;
- logistic-regression models (used first to adjust sample weights for unknown eligibility and subsequently for survey completion among eligible respondents) contained an extremely large number of explanatory variables (plus two-way crossings), which led to absurd weight adjustments-- most weights were adjusted very little (or not at all), while a few weights received enormous adjustments (sometimes more than 100-fold);
- no efforts were undertaken either to examine or to mitigate the effects of excessively variable weight adjustments, which can cause severely warped estimates;
- post-stratification cells (used in the final post-stratification adjustment to the weights) were inconsistent with the survey objectives;
- the post-stratification adjustment demonstrated that many survey estimates following the second logistic-regression-model adjustment were off by quite a lot (as much as 42 percent);
- sampling strata were collapsed together to form new “variance strata” for variance estimation, which effected a downward bias in the variance estimates, in order to make the survey results appear more precise and accurate than they actually were. The variance estimates that resulted from the creation of the “variance strata” were inappropriate because the variance estimates did not reflect the actual sampling design. The margins of error presented with the 2006 ADSS survey results grossly misrepresented the actual uncertainty associated with the estimates.
DMDC Management Response
Consequently, on April 5, 2007, my supervisor presented me with a “Letter of Warning”, wherein she stated that I was performing at an "Unacceptable" level in the Critical Job Element "Team Participation and Organizational Orientation." The “Letter of Warning” included the following accusations: "You have attempted, for public consumption, to characterize DMDC statistical methodology as wrong, even dishonest...Specifically, the draft methodology reports received February 8 and March 4 for the 2006 Survey of Reserve Component Spouses and the 2006 Survey of Active Duty Spouses, respectively, were essentially commentaries, critical reviews, evaluations, text book-proposals for redesign--anything but methodology reports that simply described procedures and results. Stratification, allocation, design effects, weighting adjustments, variance estimation were all explained in a manner leading to the conclusion that DMDC's methodologies were wrong. Statements such as these from the February 4 draft of the 2006 RCSS report are absolutely inappropriate: 'The fact that all of the design effects for all of the reporting domains were greater than one probably should have raised an alarm that something was amiss with the design.' 'Creating "variance strata" that are completely different from the sampling strata is not an entirely honest proposition.'"Another point that my supervisor made in her "Letter of Warning" was that the three statisticians (whose work I was presumably leading) were "highly knowledgeable about the mechanics of the sampling and weighting process...but they are not equipped to make independent statistical judgments." Well, why the bloody Hell not? They all had at least as much education as I did (one of them even possesses a God-damned PhD, for Christ's sake), and many more years of agency experience. Why weren't they equipped to make "independent statistical judgments?" Does she consider them to have been a bunch of incompetent, fucking retards? "You have proposed ideas and identified errors, but often left them without hands-on guidance for execution. Yon have also passed on to them work that they should not be expected to do independently." If such is the case, then why the fuck should the DMDC want to retain these knuckleheads in their positions?
The “Letter of Warning” established a 90 day period during which I was to be provided with an opportunity to improve my performance “to at least the minimally successful level in this critical element” for me to be retained in my position. Which, of course, was utter bullshit. Government human-resources specialists generally refer to a supervisor's purported expectations surrounding such a 90-day period as a "Performance Improvement Plan" (usually abbreviated to PIP, given their intense fondness for Three Letter Acronyms, or TLA's).
On June 25, 2007, shortly after taunting me about my disability, my supervisor informed me that she had decided to extend "the formal opportunity period to demonstrate at least the minimally acceptable level of performance" from July 25 to August 28. I asked my supervisor about the likely outcome, and she replied with a sneer that it was “going to be a war”, and that the extension to the opportunity period was “just postponing the inevitable.” My supervisor evidently extended the "formal opportunity period" for the sake of affording her the opportunity to compose a few damaging and insolent memoranda, that she could use to support her case for removing me from the federal service, and which she had left neglected during the initial 90-day period. During her extension, my supervisor presented me with a series of strenuously damning, denunciatory and nasty documents, each entitled "Memorandum of Counseling." The August 2, 2007 “Memorandum of Counseling” stated that "DMDC's statistical methodology has been informed by private and academic research organizations and driven by real-world trade-offs. You find fault with many aspects of the methodology; indeed, you raise valid issues, but not one of them is news to DMDC."
Finally (surprise, surprise), on September 21, 2007, my supervisor accorded me a “Proposal of Removal for Unacceptable Performance”, based upon “unacceptable performance in Critical Element 5 of your Performance Plan: Team Participation and Organizational Orientation.”[5] The attack began as follows: “In terms of Critical Element 5, your performance of the above duties and tasks has failed most notably with respect to Organizational Organization (sic erat scriptum); the sub-element Seeks to bring credit to DMDC has been a serious problem. The performance standards describe the unsuccessful level of this element as: Uninterested in DMDC management goals and team decisions. More precisely, the problem has been your failure to accept the body of DMDC’s methodological decisions and to incorporate the resulting practices and conventions into the performance of your duties. This was the problem in February with the 2006 RCSS and 2006 ADSS statistical methodology reports, in which you attacked DMDC's methodology as wrong, even dishonest…Costs of these failures in Organizational Orientation were high…the issues you raise are valid but nothing new; your contribution to re-examining them would be welcome.”
Note that the February time frame of the statistical methodology reports precedes the start of the 90-day period (April 5, 2007) during which I was ostensibly to be afforded the opportunity to “improve my performance to at least the minimally acceptable level.” Moreover, my supervisor’s choice of words, that I had “attacked DMDC’s methodology as wrong, even dishonest”, demonstrates that she believed me to have been reporting gross mismanagement. As sensitive as folks at the DMDC may be to perceived “attacks” on DMDC methodologies, the Whistleblower Retaliation Act prohibits (at least officially, if not in practice) "any federal employee who has the authority to take, direct others to take, recommend, or approve any personnel action from taking, or threatening to take, a personnel action with respect to any employee because of any disclosure of information by an employee which the employee reasonably believes evidences gross mismanagement."[6] Which isn't to say that anyone at the fucking Office of Special Counsel actually gives a shit.
The “Proposal of Removal for Unacceptable Performance” continued with a long series of libelous comments, vicious tripe, and inane accusations, and concluded with: “DMDC survey products support the personnel information needs of the Under Secretary of Defense for Personnel Readiness (sic) and other DoD customers. The information they provide is critical to formulating a wide range of personnel policies and decisions. Your poor performance has negatively impacted the organization’s ability to carry out this mission. Before proposing this action, we considered the feasibility of assigning you to another position at or below your present grade in lieu of removal from the Federal Service. However, there are no suitable vacancies anywhere in DMDC. Because of the nature of DMDC’s work, the likelihood of your success in any other position would be no greater than in your current position.” What a sweetie.
The basic templates for establishing the paperwork required to remove a career employee from the federal service, based upon alleged “unacceptable performance”, were provided by a human-resources specialist, and my supervisor told me that each document that she had prepared was reviewed and approved by a DoD lawyer. The “Proposal of Removal for Unacceptable Performance” was ultimately found to be ridiculous and unacceptable, and was consequently rescinded, as I successfully rebutted each of her points. It took me a long time (about two weeks), as her proposal went on and on for about 20 pages, and her supporting materials filled a good-sized cardboard box. Her commentary was deliberately hurtful and insulting. But, once I got started, it was as easy as goosing federal managers at a donut-eating contest--just very time-consuming, as I was compelled to compare each accusation against her supporting material. The bulk of her claims consisted of outright lies, dastardly distortions and egregious exaggerations.
For example, she wrote: "In terms of Critical Element 5 your perfomance has also failed with respect to Team Participation....the problem with the sub-element Cooperates with Others has been that in working with others, you have attempted to pass off your responsibilities to them. This was the problem all April 26 when you requested that the FCAT-M Statistician produce a new allocation reflecting response rates based on email coverage; the Statistician did not know how to compute those rates; in fact it was your responsibility to provide them to the Statistician." In fact, my supervisor's records clearly showed that I did compute those rates, that I did not ask the Statistician to compute those rates, and that I did provide those rates to the Statistician. Moreover, those rates were very simple to compute, and the fucking Statistician should have been able to compute those rates by himself, without help. He had a fucking master's degree, in Statistics. What, indeed, would be the point of keeping such fucking imbeciles on the payroll?
The fucking agency’s lawyer made certain to include verbiage in the Settlement Agreement that I would waive any rights that I might have had, including any entitlement to accommodation, under both the Rehabilitation Act and the Americans with Disabilities Act. The lawyer may have been unaware (and probably didn't care) that the Americans with Disabilities Act doesn't even apply to federal employees. Further, the agency’s lawyer insisted on including: “each party to pay its own attorney’s fees.” My supervisor never paid a dime for the many hours of legal advice and assistance that she received, over a period of several months, from the fucking agency’s lawyer. After deliberately placing me in a position where I needed to hire a lawyer, the very least that the fucking agency could have done would have been to offer to pay my legal fees.
Eventual DMDC Response to Statistical Issues
In what might charitably be described as a pathetic attempt to cover the DMDC's shitty little ass, three very-well-compensated members of the DMDC's payroll conspired together to write a response to my comments that had been published in Armed Forces and Society. Yes, I know, one of the three is actually paid through a God-damned government contractor that skims off a significant profit. But, as far as anyone ought to be concerned, they are all just suckling at the federal teat. Federal employees are barred from lobbying and engaging in other political activities. Private contractors aren't, and consequently carry considerable clout.They began with the patently false assertion that my observations had been based upon a “preliminary data set with incorrect weighting variables”, rather than a final “publicly-available” data set, and thus lacked “the necessary empirical support to be useful in improving the DMDC survey program.” Utterly reprehensible bullshit. First, the DMDC never makes any data sets "publicly available." The DMDC hardly even makes any survey results "publicly available", even on its fucking website. Second, I did use the right fucking data set. Moreover, their assertion that "it is clear that eligibility status adjustments should be as close to 1.00 as possible" is wrong, and demonstrates an utter lack of understanding of the basic subject matter. Most of the weight adjustments were close to 1.00, which would mean that response rates had to have been very close to 100% in most of the sampling strata. Response rates were nowhere near 100% in any of the fucking sampling strata. The DMDC's absurd weight-adjustment methods simply fucked up whatever credibility the survey might otherwise have had.
The three very-well-compensated members of DMDC's payroll proceeded to claim that “each iteration of the weighting process goes through vigorous quality control checks to assure accuracy of the weights.” Utter bullshit.
The three very-well-compensated members of DMDC's payroll went on to assert that the DMDC “expends a great deal of effort to ensure that its methods are in line with current research standards” and “is a firm believer and practitioner of continuous process improvement, including enhancements to its statistical methods.” As evidence, they stated that, in 2008, the DMDC began using the Chi-squared Automated Interaction Detector (CHAID) to “identify the best predictors for inclusion in the non-response logistic regression models.” CHAID is type of decision-tree method (based upon adjusted significance testing) that has been around for more than three decades[28], and is an exploratory technique that is an alternative to multiple linear regression and logistic regression, particularly when the data set is not well-suited to regression analysis.[29] In adjusting survey weights for non-response, CHAID (and other branching algorithms that have been developed more recently) is typically used to form weighting classes directly, thus avoiding the need for logistic-regression models.[30] Using CHAID to identify predictors for inclusion in non-response logistic regression models is highly irregular, and the three very-well-compensated members of DMDC's payroll really ought to reflect upon their approach.
The three very-well-compensated members of DMDC's payroll concluded that the “DMDC takes its mission to collect, analyze and report data to support the military community very seriously. While there may be steps that DMDC can take to advance this mission, the suggestions made in Losinger (2010) are not among them.” The problems that I had pointed out in the DMDC surveys comprised very basic mistakes with regard to issues that are well discussed in the statistics literature, and constitute relatively simple concepts that could be corrected by reviewing standard texts. Neither supreme nor extraordinary intelligence would be required. The clowns at the DMDC are just either too fucking proud or too fucking lazy to do it. The Secretary of Defense should either compel them to make the corrections, or simply shut down the program. The Undersecretary of Defense for Personnel and Readiness could just as well sacrifice a sheep and hire a haruspex to interpret the entrails for the agency's information needs. This would at least save the government millions of dollars that we don't have, and afford the Undersecretary's staff something to eat.
