Information Services Department
Stagnation or Migration?
Santa Cruz County Information Services Department (ISD) provides centralized information technology services, telephone, printing and other support services to county departments. In the area of information technology, the department manages the county’s wide-area network. It also operates and supports a broad range of data processing applications for county departments including public safety applications, which require reliability twenty-four hours a day, seven days per week.
One of the major costs of ISD, and other county departments, is software development and maintenance. In years past ISD directors focused on in-house development to maintain control and customize applications to county needs. Much of this software was based on using a mainframe computer system popular in the 1980s and 1990s. With the advent of server-based computer systems in the late 1990s it was determined that newer “commercial off the shelf” (COTS) programs operating on modern information technology architecture would be more efficient for the individual departments. These new systems are also cheaper for ISD to maintain and much easier to backup for data integrity and disaster recovery.
The 2002/2003 Grand Jury examined the way ISD used software and hardware and recommended, “migrating to current hardware and software technology” and off the obsolete mainframe. The Board of Supervisors agreed. Shortly after the report was published the County committed to an ambitious timeline of four years to transition all applications and retire the mainframe.
This Grand Jury revisited ISD to see what progress had been made towards the stated goal to end county dependence on obsolete technology and found that years passed without any significant progress towards eliminating the old mainframe system. In particular, much time and energy was spent unsuccessfully attempting to implement the Planning Department’s new Hansen® software. However, recent managerial changes in ISD appear to have broken the logjam and restored interdepartmental cooperation and customer satisfaction.
Within the last few months the County has approved, and ISD has begun, numerous major migration projects with another very ambitious timeline. The County appears committed to finally retiring the mainframe that is the backbone of many critical county functions. The Grand Jury commends the current progress and hopes the aggressive timeline represents recognition of the danger to the integrity of county operations posed by reliance on obsolete technology. However, the Grand Jury has concerns about whether the County is truly committed to providing ISD with the resources necessary to successfully complete this critical mission in an era of budget crisis.
This report examines Information Services Department issues that can have a substantial impact on the County and its residents. Specifically whether ISD has made progress towards converting to newer hardware and software technologies, and whether the current ISD service level is resulting in satisfied customers.
Mainframe: Computer hardware most often used by large organizations for critical applications, typically bulk data processing such as census, industry and consumer statistics, enterprise resource planning, and financial transaction processing. The term usually refers to computers compatible with the IBM System/360 line, first introduced in 1965.
Migration: A change from one hardware or software technology to
another or moving data from one storage system to another (data migration).
Application: In computer science, an application is a computer program
designed to help people perform a certain type of work. Depending on the work
for which it was designed, an application can manipulate text, numbers,
graphics, or a combination of these elements. Some application packages offer
considerable computing power by focusing on a single task, such as word
processing; others, called integrated software, offer somewhat less power but
include several functions.
1. The 2002-2003 Grand Jury made several findings regarding the expense, practicality, and obsolescence of the county’s mainframe system. The County agreed in substance with this assessment. These findings from 2002-2003[RG1] included:
· Mainframe architectures are expensive to sustain and difficult to evolve to current industry best practices such as web access.
· The County continues to use a mainframe internal billing structure to allocate ISD expenses across county operations. This requires administrative personnel involved in determining and allocating costs of mainframe usage for the purpose of interdepartmental billing.
· ISD understands the ultimate need to eliminate the mainframe computer operations by migrating to less expensive current technology, but does not have a formal plan or time line for accomplishing this.
No Response Required
2. On more than one occasion prior Grand Juries have found that outdated software applications are impacting the performance of county departments, such as the Sheriff’s Office and Planning. The County has agreed that the systems need updating.
Response: County of Santa Cruz Board of Supervisors – AGREES
3. In 2009 the County relies on substantially the same hardware, although software upgrades and revisions have been implemented in the intervening years.
Response: County of Santa Cruz Board of Supervisors – PARTIALLY DISAGREES
The County has utilized the same IBM Mainframe hardware and software platform for the last 8 years. However, other hardware (Windows servers) has continued to be replaced on a cycle based on manufacturer end of life support (normally every 4 – 5 years). Furthermore the County has moved towards virtual servers (VMWare) on
IBM blades which lower the cost of ownership and ease replacement cycle impacts on existing software applications.
4. Currently only two ISD employees are thoroughly trained and familiar with operating and supporting the mainframe. Both employees are eligible for retirement.
Response: County of Santa Cruz Board of Supervisors – DISAGREES
ISD has 6 mainframe programmers and 6 operators that support the mainframe environment. Each of those mainframe programmers is skilled in different business applications, and some cross-support is in place. While half of the mainframe programmers are eligible for retirement, none have indicated they plan to retire before migrating their primary application off the mainframe.
5. On April 1, 2007, IBM stopped offering support for mainframes such as the County’s 7060-H50.
Response: County of Santa Cruz Board of Supervisors – DISAGREES
IBM ended support of the MVS operating system for 7060_H50 on April 1, 2007.
The County has never run MVS on their mainframe systems. Currently, the County is fully licensed and supported for both hardware and software for the existing mainframe platform that is in place.
6. In the fiscal 2008 year-end report submitted to the Board of Supervisors, the county’s independent auditors, Caporicci & Larson, noted that: “The County’s Disaster Recovery Plan was developed 25 years ago and was not updated to address the new technologies. The Disaster Recovery Plan details the actions required should a disaster occur that affects the computer operations of the County.”
Response: County of Santa Cruz Board of Supervisors – AGREES
ISD has started work on a Continuity of Government plan with a tentative schedule of completion in three years. This is much larger than an IT disaster recovery plan and will involve most departments at the County.
7. In 2002, ISD began a project to transition the Planning Department’s code enforcement system from the mainframe-based Automated Land Use System (ALUS) to a non-mainframe Hansen® system. As of today Planning is still relying on ALUS for permits and land use planning, while running certain code enforcement processes on a dual-entry basis (entering information into both ALUS and Hansen®).
Response: County of Santa Cruz Board of Supervisors – PARTIALLY AGREES
In 2007, Planning began using the Code Enforcement module of the Hansen application. All new Code Enforcement cases were entered into Hansen exclusively.
Old cases remained in ALUS as did Planning and Building Permits. So, while two systems are in use currently, there is no dual-entry and data is only stored in one system. When Hansen goes into production with the remaining modules, data from
ALUS will be migrated to Hansen. At that time, ALUS will be retired and Hansen will become the one and only system used. The production release date is currently scheduled for the end of calendar year 2009.
8. In 2009 the following county departments still rely on applications running on the mainframe: County Administrative Office, Treasurer/Tax Collector, Assessors Office, Auditor Controller, Purchasing, Payroll/Personnel, Planning Department, Probation Department, Recorder’s Office, Sheriff’s Office and the District Attorney’s Office.
Response: County of Santa Cruz Board of Supervisors – PARTIALLY AGREES
In fiscal years 2008-09 and 2009-10, the County’s IT mainframe migration strategy called for 8 mainframe business applications to be migrated off of the mainframe.
The District Attorney’s system was migrated off the mainframe in 2003. In 2009, the
Budget system, owned by the CAO but used by all departments, was migrated to a web-based application. The Payroll/Personnel system was migrated to a web-based application in 2009. The Sheriff’s Records and Jail Management systems are currently slated to be migrated off the mainframe in fall of 2009. The Probation Department’s Case Management System is also planned to migrate off the mainframe by the end of 2009. The Planning Department will complete its migration in 2009. For more details on the Planning Department’s applications, see also the answer to question 7. The new Property Tax system, used by the Auditor, Assessor and Tax Collector, is the third and final phase of the project plan and will be moved into production by 2011. The General Ledger and Purchasing systems will be the last applications to be migrated off the mainframe, dependent on County resources to pay for and implement the project.
9. Within the last year, the County has approved projects that will move the following operations off the mainframe:
Response: County of Santa Cruz Board of Supervisors – PARTIALLY AGREES
The Recorder’s system was replaced on July 20, 2009. It was not a mainframe application.
10. As applications are shifted off, the cost of maintaining the mainframe has been allocated among departments still using it. The number of users has decreased, so charges to each remaining department have increased, although in the 2009-2010 department budget enough cost reduction has been achieved so that there is no need to increase charges to users.
Response: County of Santa Cruz Board of Supervisors – AGREES
11. ISD is working toward being mainframe-independent within four years.
Response: County of Santa Cruz Board of Supervisors – PARTIALLY AGREES
The County’s mainframe migration plan currently has Board-approved plans and funding for migrating eight applications off of the mainframe. In fiscal year 2008-09, two applications were migrated off the mainframe to web-based solutions. In 2009-
10, an additional six migration projects have been funded and five applications are currently scheduled to be replaced. The remaining project is entering its third and final phase and is targeted to be migrated off the mainframe in 2011. The few remaining applications on the mainframe do not currently have project plans or funding sources to replace them. Nonetheless, ISD and the County remained committed to the goal of mainframe independence.
12. A new ISD Director was appointed in 2007. Prior to his appointment, the department operated with a mindset of internally developing and supporting custom applications based on the mainframe architecture.
No Response Required
13. All the county employees this Grand Jury met with thought that ISD communication and responsiveness had improved over the last couple of years.
No Response Required
1. The County has made inadequate progress towards migrating departmental applications off the mainframe in the more-than-five years since the 2002-2003 Grand Jury report.
2. The age of the mainframe, the lack of manufacturer support, and the limited number and status of qualified personnel, pose real risks of catastrophic interruption of service and/or loss of critical data.
3. The appointment of a new ISD Director in 2007 has proven to be a positive step toward elimination of both internal software development and mainframe reliance.
4. The shift in ISD culture, combined with apparent progress after a period of stagnation has resulted in higher levels of customer satisfaction among the departments utilizing ISD services.
1. The Board of Supervisors should provide the resources necessary to allow Information Services Department to complete migration off the mainframe on schedule.
Response: County of Santa Cruz Board of Supervisors – HAS BEEN IMPLEMENTED
The current mainframe migration projects fully utilize the available ISD staff. While
ISD would appreciate more staffing we are fully aware of the financial exigencies facing the County.
2. The Board of Supervisors should monitor ISD progress to ensure adherence to the migration schedule.
Response: County of Santa Cruz Board of Supervisors – HAS BEEN IMPLEMENTED
ISD has been reporting status on the Property Tax System to the Board of Supervisors and currently reports status on all projects to department heads every quarter.
The Grand Jury commends the Information Services Department and director Kevin Bowling for changing the focus and the culture of the department over the past two years. The changes have improved user confidence and resulted in considerable progress towards the goal of eliminating reliance on the mainframe.
|
Respondent |
Findings |
Recommendations |
Respond Within / Respond By |
|
County of Santa Cruz Board of Supervisors |
2, 3, 6-11 |
1 – 2 |
60 Days September 1, 2009 |
|
County of Santa Cruz Information Services |
2-4, 6-11 |
1 – 2 |
90 Days October 1, 2009 |
Officials from Santa Cruz County Departments
Information Services
Planning
Probation
Child Support Services
County Administrative Office
Site Inspection
Information Services Department Server Room, 701 Ocean St., Santa Cruz, CA
Santa Cruz County Board of Supervisors Minutes: August 5, 2008; January 13, 2009; March 3, 2009
Santa Cruz County’s Technology Strategic Directions and Consolidation Plan, Cuong Nguyen, March 10, 2004
County of Santa Cruz Planning Department, Planning News, Spring Issues, April 2006
County of Santa Cruz, Information Services Quarterly Reports
First Quarter 2003-2004 through First Quarter 2008-2009
Santa Cruz County Grand Jury Reports
Review of the Santa Cruz County Planning Department, 2000-2001
Operations of the Santa Cruz County Information Services Department, 2002-2003
Santa Cruz County Jails Review, 2003-2004
Code Compliance Enforcement - Does it pass inspection?, 2007-2008
The end of the IBM System/390, Isham Research, 2003
Who’ll mind the mainframes? Few students are learning to run the decidedly unsexy, but vital, systems, Hiawatha Gray, Boston Globe, August 26, 2005
When to Murder Your Mainframe, Peter Nulty, CNN Money.com, November 1, 1993
IBM Ends 31-bit z/OS Mainframe Support, Mark Fontecchio, Data Center News, April 11, 2007
http://www.wikipedia.org/
http://www.co.santa-cruz.ca.us/
http://www.pcmag.com/encyclopedia/
[RG1]Added to ensure clarity