Pseudoscience, data fabrication and malarkey:  NMFS/NOAA’s “analysis” of simulated steelhead consumption by cormorants


Shugart, Round 2 of Comments on DCCO FEIS CENWP-PM-E-15-01, 7 March 2015 (Misrepresentation of Monte Carlo...)

[Many comments with responses by ACE et al, were included in the 1115 page DCCO EIS available on the ACE website for comment period that expired August 4, 2014.  However only a few responses from the follow up comment period February 13, 2015 -March 16, 2015 were mentioned in the final record of decision (  Mine and many others (?) were simply ignored.  Mine was "Misrepresentation of Monte Carlo Simulations in Wildlife Management: An example using piscivores (Double-crested Cormorants)". (link above)   

The crux of my comments outlined the general misunderstanding or misrepresentation of Monte Carlo simulations that have been used to simulate consumption by DCCOs in the current EIS and CATEs in a previous EIS.  In the DCCO and CATE usage, estimates of number of birds, energy requirements, assimilation efficiency, proportions of prey in the diet, energy/prey item, and average mass of prey, etc, to calculate the number of prey items consumed.  The simulation was based on  Roby et al. (2003) (OSU) and titled Bird Research NW (BRNW, consulting firm) Bioenergetics Model for use on DCCOs.  Briefly, Monte Carlo simulations involve replicating or iterating input variables, 1,000 times in the case of the OSU/BRNW simulations, then taking the average (or median) result as a "best estimate".  Multiplying by 1,000 then dividing by 1,000 would produce a similar result, but there would be no variance associated with this questionable approach.  The Monte Carlo approach adds a step of introducing random variation in the replications.  But in the end the estimate will be the same if done once with a single calculation or with 1,000, 10,000 or a million iterations, except for noise generated by the random numbers generator and mistakes or manipulation of simulated data in calculations.  Running the simulation or input data through the simulation or model does not enhance or validate "best estimates".  The best, repeatable, and reliable estimate is one using a single calculation using the input variables used to seed the random numbers generators.  The problem with a single calculation is that it has no indication of statistical validity.

The purpose of Monte Carlo is to estimate variation usually expressed as a confidence interval of the result thereby providing some statistically validation. A relatively small 95% CI indicates the estimate is reasonable and that if the study was done 100 times, the estimate will fall within interval.  It does not mean that the estimate has a 95% probability of being correct.  Based on a glimpse of how the model worked, variation & resulting CIs were grossly underestimated due to mistakes, miscalculation and manipulation or other adjusting of random numbers generated (as stated in the FEIS).  A glaring error was the use of variance rather than standard error as a seed values for number generators for a prey proportion.  Rerunning the simulation with correct SEs and without manipulating the iterated values and CIs are 2-5x greater including a large negative consumption.  The negative consumption could be truncated after computation.  However as revealed in the FEIS, negative consumption was prevented in the simulation with Rube Goldberg-like machinations. If done without manipulations, the CIs are so large as to render the estimates meaningless for management purposes.

An additional point that came out in analysis of the Tillamook Bay workbooks and then revealed in the FEIS was that proportions of salmonids in the diet where based on simulated proportions using a 10+ step process to convert frequency of occurrences, which were pooled for all or part of the study period,.  Going from the frequency of occurrence data to salmonid proportions involved a 10+ step process, or simulation, with each step introducing error that was not considered in the simulation.  More details in the links above..

My interest is not how many salmon are consumed or how many DCCOs are shot, but was there any statistical validity behind the estimates that have been simulated for the past 20 years for CATEs and DCCOs?   As an example, consider extrapolations (more details in link).  The detection of a steelhead in a DCCO stomach may have been extrapolated to 1/4 to 1/3 million consumed.  In five years, yearly projections of 1-2 million steelhead consumed, the poster species for the EIS, could be based on as the detection of as few as 5-7 fish per year (see page 870 of the EIS).  This is over a 10+ week period of sampling for 20,000 bird plus chicks this simply isn't credible (Note that is is based on reverse extrapolations from summary results since the raw data are not available.).  No doubt that DCCOs eat many salmonids, but given lackadaisical way the simulations were done and lack of detail regarding methodology (the code used for calculations as been deemed proprietary) all that is known is that DCCOs eat fish and some of them are salmonids.  Raw data or estimates, SDs and sample sizes used in calculation and the actual code used for calculations need to be provided in a data archive or tabular form (e.g., see Appendix D in the FEIS).