San Diego — On October 26, 1999, the county Board of Supervisors voted to approve a $644 million, seven-year contract to "outsource" the county's computer and telephone operations. The move was heralded by the supervisors as a bold step that would save taxpayers millions of dollars and provide the citizenry with new technological marvels, such as walk-up computer kiosks to provide automated ways of paying taxes and fees. The arrangement was opposed by the county employee union, Service Employees International Union, Local 2028, which argued that the arrangement would lower wages and benefits for workers and result in degraded services.

A year into the contract, things have not gone as smoothly as the supervisors planned, according to a series of internal documents, memos, and e-mails recently produced by the county in response to a request made by the service employees union under provisions of the state public records act. Phone systems have gone down, animal-control databases have crashed, real estate recording systems have malfunctioned, security has been violated, and law-enforcement agents have complained about not being able to log onto an online network called "CLETS," the state's main crime-fighting computer tool.

Not to worry, says the county and its computer contractor. The complaints manifested in the mountain of letters, memos, and e-mails unearthed by the union are just temporary fallout of the switch to private-sector management of the county's data-processing empire. Untangling the mess will take time, they say, but the promised data-processing nirvana will be reached. The service employees union argues that it was right all along and predicts even more trouble ahead.

In either case, the memoranda quoted below provide a glimpse into realms of local government. Excerpts from the letters, preceded by a short glossary of terms, are presented in chronological order.

PA: Pennant Alliance, the group of companies that won the county's data-processing contract; made up of Computer Sciences Corporation, Science Applications International, Pacific Bell, and Avaya.

CSC: Computer Sciences Corporation.

CLETS: California Law Enforcement Telecommunications System, "a high-speed store and forward-message switching system that has been operational since 1970. For almost 30 years, this system has provided law enforcement and criminal-justice agencies with a means to inquire and update files of the Criminal Justice Information System (CJIS) over the California Department of Justice (DOJ) secure law enforcement telecommunications backbone," according to a county description.

SUN: A computer system that "gives online users access to CLETS," according to a county document.

IBIS: Inmate Booking Information System

April 6, 2000

12:05 p.m.

From: Kathy Wilder, CLETS coordinator

Subject: Problems at the FBI office

This month when the SUN report # CX0502-01 was mailed out, the FBI's report went to the wrong location. (It went to George Bailey Jail) where it was opened by the employee at the jail, thinking it was their report. There is highly confidential information included on this printout. This morning the FBI called to complain, and needless to say were quite upset.

Last month I was made aware of at least one other report that was mailed to the wrong division within our department.

These reports need to go to the correct agency and division. Can this please be corrected ASAP?

June 8, 2000

3:34 p.m.

From: R.J. Warner

To: Kathy Wilder

As I am writing this, at 15:25 on Thursday, SUN is very slow at both Records and the Comm Center. Slow means one minute plus for replies from CLETS and many seconds to change screens. This slowdown has been reported by the DSD helpdesk to the PA helpdesk.

I did not see you today and I know you will want to follow up on these problems. I am not trying to blast the PA. This is their chance to show they are on top of things and they follow through on things.

June 9, 2000

4:58 p.m.

From: Neil Rossi, County Chief Deputy Treasurer

To: Tom Boardman, County Chief Technology Officer

Subject: Again needing your help

Tom,

Once again I find myself in the position of needing your help. I am beginning to feel that everything the Alliance touches goes to hell. Our current problem revolves around the issuance of warrants.

Under California Law, we must refund payments to taxpayers within 60 days or we have to pay them interest. As this results in a loss of funds to the County, we have always made every effort to meet this requirement.

The researching and refunding of taxes is not an easy process. We seem to have residents that make a habit of overpaying or double paying their taxes. In the past we have been able to complete this process on our end and request and receive a warrant in sufficient time to meet the 60-day requirement.

However, now that we have transitioned the data entry and warrant printing processes to the Alliance, what once took 1 week is now taking in excess of 3 weeks, and more typically about 4 weeks. Because of this we are in jeopardy of not meeting the 60-day requirement. In addition to causing poor customer service, the potential financial impact of this delay on the County could be substantial.

Therefore, I am asking for any assistance you can provide in speeding up this process. While I think it should be no slower than the past process for any warrant, at a minimum we need to speed the process for those warrants that cause, or may cause, a financial cost to the County.

Please let me know your thoughts on how we can improve this process. As we approach the closing of the fiscal year, we will be sending out a large amount of warrants, so we need this process resolved by early next week.

Please do not hesitate to call if you have any questions or need additional information.

Neil

July 6, 2000

12:43 p.m.

Joe:

This is documentation of our phone call on 7/6/00 at 1225 hours.

I have reported slow CLETS responses to Pennant Alliance on several occasions. Today while you, RJ, and myself were in a meeting discussing this very issue the problem happened again. The dispatcher on Vista radio ran a license plate to check and see if it was stolen. She did the first inquiry at 10:13:09 with no response. She did another inquiry at 10:13:31 with no response. She finally received the response from CLETS at 10:22:00 that showed the vehicle as a stolen vehicle.

This response time is unacceptable, and I'm sure you understand the seriousness of this issue. I know you wanted to bring people over to see this when it occurs, but we can no longer wait for that type of troubleshooting.

In addition to the response time, the dispatcher called P.A. Help Desk twice. She was told once that it was just "running slow" and another time that it "looked up." As we have discussed in the past, this is a chronic problem with the help desk. Previous discussions have not accomplished the desired outcome. Please report this to the help desk manager, Karen Hess.

When we spoke you were going to check the problem out, and I'm sure you will do the appropriate follow up. I'm looking forward to speaking with you at our next meeting.

Kathy Wilder,

CLETS Coordinator

July 31, 2000

10:18 a.m.

From: Ron Lane, [County Public Safety Information Technology Manager]

To: Gary Clarke, [County Technology Manager]

Subject: Sheriff Outage

I had a long conversation with John Pingel this weekend when he called me to advise of the Sheriff IBIS outage. This appears to be an issue similar to the one you were concerned about last week -- lack of on-going maintenance of servers and systems. For a mission critical system such as IBIS, a system that is used in the booking and releasing of prisoners, to run out of disk space and go down for five hours is a self-inflicted outage that is completely avoidable.

I have asked for the RCA, which I understand indicates that the outage was avoidable. Anyway, this may be another example to add to your list in support of lack of ongoing maintenance.

CC: Thomas Boardman

August 9, 2000

REPORT FROM THE DEPARTMENT OF ANIMAL CONTROL

COMPUTER PROBLEMS

Monday 8-7-2000 until Tuesday 8-8-2000 at 10:00 a.m. we had no database or e-mail service at the North Shelter and the South Shelter.

Business could be conducted on a limited basis by requesting the staff at the Central shelter to make inquiries in the database and FAX the information to the outlying shelters.

Monday 8-7-2000 until Tuesday 8-8-2000 at 10:00 a.m. staff at the Central Shelter could not send or receive e-mail messages outside the building and had no access to the mainframe or Internet.

Wednesday 8-9-2000 from 9:00 a.m. until 5:00 p.m. there was no access to the Chameleon database system at all three shelters.

IMPACT OF THE

CHAMELEON DATABASE OUTAGE

ANIMALS

No euthanasias could be done for three days at South and North, which also adds to overcrowding in the kennels and catteries.

* Overcrowded kennels and catteries increases disease among the animals.

* Overcrowding lowers adoptions because the customers will become disgusted and leave.

* Spay and neuter surgeries can't be done on the animals, which results in delayed or aborted adoptions.

* Medical treatments are delayed, which results in higher disease, lower adoptions, and more euthanasias.

Overcrowded kennels and catteries make it unsafe for the kennel staff, as they have to deal with fighting animals.

* Unable to track the quarantine period on potential rabies cases.

August 29, 2000

8:16 a.m.

From: Colin Leitch [county Information Technology Contracts manager]

To: David Chung; Gary Clarke; Kristin Zanavich

Subject: Fwd: Washington Technology's Weekly E-Letter

Chaps

Some interesting stories: It would appear that CSC is chasing the National Security Agency's outsourcing program for non-mission information technology. $5 billion over 10 years. We may wish to track how they perform on this procurement. I assume we will not be giving references?

Colin

September 6, 2000

12:25 p.m.

From: staff

To: Thomas Boardman

It has become more apparent in our meetings with the Pennant Alliance the past couple of days that the focus of our vendor with regard to the...project is not on the quality of the final product, but rather on solely fulfilling their contractual requirements. I believe that there is a major disconnect between the County's goals for the end product versus the PA's goals. While the County teams are focused on delivering a quality, well-designed, easy to use, single-point-of-entry system, the Pennant Alliance is only focused on meeting the list of requirements in the contract as they are written word for word. While I completely understand their contractual obligations, I am concerned that valuable information is being withheld from the County because the PA is not contractually obligated to provide it.

Without the mutual agreement on how to work toward a common vision, I believe that the quality of this...implementation is in severe jeopardy. It is becoming impossible to find ways of moving the County toward its ultimate goal of 'e-county' without access to information that will allow us to move in that direction.

If you would like to further discuss any of my concerns, I would be happy to meet with you.

September 14, 2000

From: Colin Leitch

Please find attached my first draft of the CSC default letter. We should get together tomorrow to fill in the blanks and soften the language. This is a serious issue, and we should not take this course of action lightly and without considering all the options. Please advise when you would like to meet.

Also as this may be your first "Default" I thought I should offer some insight.

One of the key factors in successfully supporting a termination for default action is for the government to act quickly once a breach of the contract is anticipated or recognized. The government is granted a period of time (normally 10 days, or longer as necessary) called "forbearance," in which the decision to terminate for default is made. Once that period expires, the government risks waiving its rights to pursue the termination for default.

Again, we should not take this course of action without considering all the possibilities and ramifications. Personally, I believe we have a contractual obligation and right to issue a cure notice to CSC. We must also protect our future rights and remedies under the Service Agreement. However, I believe that it should be drafted in such a way to show that we will work with CSC to resolve the problem. CSC has been in business for many years, and this will not be the first time they have defaulted and been issued a cure notice. I have no doubt that they will receive the cure notice like the professionals they are.

I invite your comments.

Regards, Colin.

Colin Leitch

IT Contracts Manager

Chief Technology Office

County of San Diego

1600 Pacific Highway

San Diego, CA 92101

September 29, 2000

From: Kathy Wilder

To: Capt. Donna Collier

I received a call late yesterday afternoon from Naval Investigative Services. They reported a conversation with a PA help desk employee that concerned them greatly. I have asked for a formal report in writing regarding this incident. The Navy reported a person from the help desk advised them that he had Criminal History Access and was asking the Navy about many of the SUN transactions. He bragged that he had given this access to a female employee at the help desk.

I immediately called Karen Hess, the help desk manager at PA, and advised her of this issue. I asked her to check this person's access and requested an audit on his employee ID. The help desk personnel should not have any access to SUN databases. After Karen talked with the employee she advised me that he was telling the woman from the Navy about a "joke" website on the Internet. When I receive the documentation from the Navy and the results of the audit, Karen and I will discuss this further.

Since G.J. Wade was brought on board to assist with training the help desk personnel, things are starting to improve.

These are the outstanding issues that I am concerned about. Each week is a great adventure!

Kathy

October 6, 2000

To: Dianne Jacob, San Diego County Board of Supervisors

From: Roy D. Nedrow, Agent in Charge

San Diego Office, Social Security Office of Inspector General

This is to bring to your attention an unacceptable lapse in county service which you and the Board of Supervisors have direct control. I am referring to the service, or lack thereof, being provided by Pennant Alliance in your information technology area. This group is responsible for providing services connecting law enforcement to the California Law Enforcement Telecommunications System (CLETS) in San Diego County.

In 1999, the Inspector General for Social Security determined that San Diego County was at considerable risk for Social Security fraud and other related criminal operations. He therefore ordered, forthwith, an office to be opened in San Diego to combat the threat to the National Trust represented by such criminal operations. All the necessary work required to obtain office space was done and a staff was hired. We moved into our new permanent space in Symphony Towers in October of 1999.

As part of this startup operation, we went through a lengthy and exhaustive process to obtain access to the CLETS database. We were approved by the various Federal, State, and County authorities on January 27, 1999. All that remained was the technical connection of our computers to the County's system; surely a small matter compared to the bureaucratic and administrative hurdles we had previously overcome. We couldn't have been more wrong.

Since January of 2000, I have had a technology knowledgeable special agent assigned to the task to complete the CLETS connection. Month after month, our calls went unanswered. When your subcontractor did start working the issue, it was immediately clear the Pennant Alliance did not have the remotest idea on how to complete this simple connection. They had us check software, telephone lines, and other equipment repeatedly to find the cause of their inability to do this job. Outside technicians were tasked to provide services that your contractors should have provided. We checked and rechecked, but to no avail.

As we speak, the system is still not up and running. The contractor, quite frankly, does not have a clue how to complete this simple task.

Ten months' wait for a procedure that should only take a couple of days is outrageous.

Please take whatever action you can to resolve this situation so that we might provide the most efficient and safe service we can to the citizens of San Diego County.

If you have any questions regarding this matter or wish access to our extensive file on this issue, I would be happy to provide you with all our information.

October 16, 2000

From: Nick Macchione

To: Jerald Coleman

Subject: RISK ALERT

Jerry, phones in my children's services departments (both regions) have not been working properly for the past 2 weeks+. Pennant has been working on fixing the problem, but it's now reached such serious problems that staff have brought in their own personal cell phones in order to take the IRS calls from Polinsky's hotline...since the regional hotline phone designated for this function is often busy (yet, no one is using it) or not working at all, this is AN EXTREMELY SERIOUS RISK TO THE SAFETY OF OUR KIDS in how we get notified and the time it takes to make immediate investigations. Again, Pennant Alliance has been working on this, but as of yet no resolution...believe it's a capacity issue with the phone lines...Wanted you to be aware of this. Let me know when we can expect a final resolution. thx.

October 18, 2000

FROM: Ron Lane

TO: Donna

We are working on the issues we discussed yesterday. One key aspect of this is we need to know definitively if the Sheriff is willing to allow Joe Marzo to essentially treat all new (not break-fix) SUN-CLETS as service requests and can provide cost proposals to them, and if they do the work, seek reimbursement. For example, if the Border Patrol calls in and asks to add another SUN station, and Joe determines it will take 25 hours of staff time and $400 of equipment to meet this request, he will notify the Border Patrol that the request will cost 25 hours X the MOU [memo of understanding] analyst rate (about $60) plus $400.

The key is that we need to pull out of the death spiral of providing services to agencies at no cost, having the agencies complaining because their free service isn't good or fast enough, and having limited ability of improving the service because we aren't paying the PA anything for it. One option is charging the agencies; a second is establishing a baseline budget (complete with service-level requirements), funded within the Sheriff allocation, for the SUN application; and a third is referring the agencies to a qualified vendor to provide end user support.

December 12, 2000

To: Andrew Carrico, Pennant Alliance

From: Gracelia Smith, Chief Deputy Recorder

Subject: DEC Automated Recording System Status Report

The stability of the DEC Automated Recording System is our ongoing concern, as a result of end-of-day system slowdowns and frequent satellite office equipment failures. These ongoing occurrences raise our level of anxiety as we anticipate the possibility that they are a precursor to a total system collapse. Again, failure of the Automated Recording System would be a catastrophic event for the County of San Diego.