In the case of the Pacific Surfliner, a partial explanation may be found in the studies which show that riders see no difference between Amtrak and Metrolink; it is possible then that a significant portion of what would be Surfliner riders are captured instead by Metrolink's Orange County and Ventura Lines. I find this doubtful, however, given the similar failings in Capitol Corridor and San Joaquin ridership.
Another reason for the major shortfall could be a failure to complete work which would expand capacity or decrease travel time which would be necessary for the validity of the ridership projections. Certainly not all of the projects which were planned for were completed within the past ten years and there has been no time reduction between San Diego and Los Angeles in contrast to the 30 minute time travel reduction expected by 2005. Additionally, the loss of the planned Los Angeles-Las Vegas train almost certainly cut Surfliner ridership since 19% of current PSL revenue is derived from connecting to long distance trains (page 5). Still, a significant fraction of the ridership gap is almost certainly due to overly optimistic projections, even taking into account failed improvements and an economic downturn.
It is a sad reality that what gains Amtrak has made in terms of ridership has come almost solely from degradations in service and speed by air and automobile. The forthcoming creation of a LOSSAN joint powers authority may allow for local funding sufficient to fund these improvements via a variety of means, including selling the PSL franchise to another operator. Given that Amtrak has become myopically focused on the Northeast Corridor, to the near total exclusion of investment plans in the rest of the nation's passenger rail network, this does not strike me as a poor solution. Indeed, that focus helps demonstrate why, though the overall purpose and tack is wrong, Republican arguments for the privatization of Amtrak and a degree of free market competition in passenger rail service have an element of truth and public good to them.