Statewide Data and Information Systems Committee

Introduction

Members of TRB Committee A1D09, Statewide Transportation Data and Information Systems, convened a Peer Exchange Meeting on the 25^th and 26^th of August 2000, in Madison, Wisconsin.  The main theme of the peer exchange was performance measures in transportation, and in particular the data and data systems that underlie performance measures in a multimodal system planning and management context.  Members representing nine state transportation departments provided descriptions of how their agencies developed and use performance measures, and how this has shaped data collection, processing, and analysis activities.

Ron Tweedie, Chairman of Committee A1D09, opened the peer exchange with references to the origins of the meeting.  Other TRB committees have taken advantage of the opportunity to meet in conjunction with their summer meetings to promote additional dialog, to coordinate related activities, and to share a wide variety of information on topics of interest to members.  By nature, the peer exchange format emphasizes open sharing of practical information, for example, how new ideas are being tried and tested, what is motivating states and others to try new methods, what is working, and what problems have developed.  This was the first such meeting specifically involving Committee A1D09.  The focus on performance measurement systems was in part intended to help provide useful information in advance of several relevant upcoming conferences and workshops, including the North American Travel Monitoring Exhibition and Conference (NATMEC), the TRB Conference on Performance Measures to Improve Transportation Systems and Agency Operations, and the Conference on Remote Sensing and Spatial Information Technologies for Transportation.^[1]

In addition to the live exchange of information, this written summary was proposed as a way of extending the discussion to others.  The following section provides a synthesis of recurrent issues and themes that emerged during the meeting.  This synthesis is followed by a description of each of the State's presentations including related discussion.  Participants in the peer exchange also expressed a desire to generate a list of issues and challenges to help guide a research agenda for both short- and long-term data-oriented projects.  A summary of this list appears as an appendix to this circular.

Section 1:  Summary of Issues

Presentations included discussion of several important topics related to use of performance measures, including data requirements and difficulties associated with data collection and analysis, public and political acceptance of performance data, and the overall agency strategy for using performance data.  The following major issues surfaced during the weekend workshop.

Data Quality and Integrity

Data quality is a significant concern of every agency that participated in the exchange.  There are several aspects to data quality:  currency (or timeliness), consistency (do two sources agree) between different published data sources, precision (level of detail or "grain") and accuracy (is it correct or not).  Expectations for public access to high-quality transportation data have grown as more and more data has been made accessible over the Internet.  This includes demand for accurate, real-time data by those interested in assessing current traffic conditions, and demand for comprehensive traffic counts and long-term forecasts, often for project or development-related analysis.  Agencies need to anticipate and budget for this need in order to avoid public relations problems.

Particular concerns expressed by meeting participants include:

• Data accuracy is a significant concern.  Agency staff acknowledge that they may sometimes have to make available data that they know is imperfect.  A good DOT staff is one that is constantly correcting or improving its data sets.  One way to get better data accuracy is to "expose it to the light of day."  As more data is made available and put to use, there will be more public pressure and hence management support for data quality, timeliness and accuracy.
• "How good is good enough?"  Some data uses have financial impacts for the public or private sector.  While most DOTs continuously strive for better data quality, it is also advisable to put appropriate disclaimers in place.
• The quality of data made available to managers, decision-makers and the public continues to be a significant concern to most DOTs.  The level of attention focused on data quality may be growing due to the greater accessibility to data that has been made possible by posting data on public agency web sites.  Workshop participants cited examples of external data users finding different values and different formats for the same data in different places within the agency's web site.  In other cases, data made accessible through electronic formats such as web sites may differ from traditionally published data sources.  While there will always be the possibility that some data sources are more recent than another, or may be formatted for different purposes, there is a need to ensure that the data users obtain is as consistent across the board with what is available from all sources within the DOT.
• The currency of data is also a concern, again at least partly due to the increased accessibility to data via the web and the availability of more frequently-updated information made possible by increased use of automated data collection and reduction techniques.  Problems were reported with two users tapping into the same data source and obtaining data from different time periods.  DOTS need to establish guidelines and protocol which will determine when and how frequently their data sources are updated.  Users who access agency data need to know what period is represented by the data.  Agencies should establish a procedure for periodically updating published data and ensuring that users understand the time period represented by the latest update.  PennDOT has addressed this problem by restricting public access to data from the most recently-completed and published collection period.  In this approach, a DOT may update internal databases continuously but only release the new data summaries periodically (e.g., monthly, quarterly) to reduce the proliferation of conflicting data sets.
• Some DOTs expressed the concern that the inherent inaccuracies and imperfections in their data limit the useful application of performance measures in critical decisions.  We need to be careful to avoid creating a false sense of perfection, or greater expectations for data quality and accuracy than we can deliver.  For example, if specific project decisions with clear financial or safety implications are based too heavily on performance measures, the outcome can be undesirable.  Decision-makers should use data as indicators of condition and performance trend, for example, rather than as yes-or-no decision points.  The opinion was expressed that we should "let people make decisions, using indicators to guide them, but that we should not absolve them of the responsibility that comes with decision-making authority."
• There is concern that DOTs may be using relatively coarse and imprecise data to drive highly specific decisions, for example, to select one project or design solution from among several alternatives.  While often there is no better alternative method, decision-makers need to be made aware of the quality of the data they are presented.  We should be able to do a better job of defining the variability of a given set of performance data, or the range of valid estimates, rather than a provide a single data point for a given measure.  It should be the role of the data community to ensure that managers and decision-makers understand these data characteristics, as well as and consequences of data variability.  It was suggested that this is something the data community can actually do something about, whereas we may not be able to change the way in which decision-makers use the data in their decision process.

Customer perspectives

Most agencies acknowledge the importance of understanding the audience for performance data.  This audience can span a great variety from internal group managers and staff, to executive management teams, elected decision-makers, private sector system users (e.g., freight carriers, land developers) the motoring public, and the general public including "non-users."  Some recurrent themes included:

• Defining the audience for performance measurement is desirable, and will help shape data collection and reporting programs.  It is not always easy to reach agreement on who the customers for a particular data product are, however.  Careful thought needs to be given in advance to this question before data is developed and released.
• Market research, while conducted by many DOTs, is not necessarily accepted by all managers or decision-makers as accurate or representative.  Due to lack of standardized methods or established regular survey programs, DOTs have often found it difficult to combine or even compare results from different surveys.
• Market research can be used successfully to identify the key concerns, needs and expectations of different customer groups.  Segmenting the broad base of "customers" into more specific market segments will help to refine understanding of needs and to focus solutions.

Use of Performance Measures

• After several years of discussion and debate over the merit of "output" vs. "outcome" measures, agencies seem to be settling on a blend of the two, balancing the customers' expectation for outcome-related performance information with the agency's need for useful output data that has meaning to managers and decision-makers.  While outcome measures may more accurately reflect what the customer sees and expects in the way of performance, the role of the DOT as long-term conservator and owner of the system demands a broad range of measure types.  In addition to system outcome expressed in terms with which the customer can readily identify, DOTs need to measure the efficiency of their programs as well as their progress on fronts that are not readily visible to the public.  For example, the public may be more likely to notice and relate to outcome measures such as system-wide accident rates, or generalized pavement condition ratings, whereas the agency needs to know something about specific high-accident locations, bridge structural condition, or "predictive" pavement condition indicators such as cracking and rutting.  Customers are perceived to be focused on programs with near-term, visible impact and to under-value long-term investments in underlying system integrity and maintenance.  Whether this is true in a certain state or region should be confirmed through surveys, but regardless, planners and engineers do recognize the importance of balancing the customer imperatives with longer-term system preservation requirements.
• Use of performance data requires an evolutionary approach to the process:  There appears to be a natural progression or trend within some agencies using performance data to proceed from detailed, technical or system management applications to more strategic, business-planning applications.
• Several DOTs with significant experience in application of performance measures are moving towards true multimodal applications, which creates added challenges to data collection and maintenance, since different owners and operators are involved than just the state DOT.
• Performance targets are becoming more prevalent, that is, identifying a desired level of achievement on a specific performance measure by a particular date or period of elapsed time.  Unlike the use of continuous but non-specific improvement goals over time, the use of performance targets helps to identify "performance gaps" which can then be linked to business planning and budgeting.
• There is particular interest in measures of mobility, but also more limited hands-on experience in developing and using such measures.  For one, agencies need to be sure to define what mobility means to their managers, decision-makers, and/or customers.  Agencies report that mobility takes on many different meanings depending upon the audience, and may incorporate distinct adjunct characteristics such as accessibility and reliability.  If mobility is to be measured across multiple modes, this further complicates both the questions and the answers, since the DOT is often not be the direct operator of some of the modes, and thus will not be the owner of the data.  The Florida DOT provides one particularly structured approach to defining and quantifying mobility, which is broken into the four dimensions of quantity, quality, accessibility, and utilization (see detailed discussion in Section 3 below).
• There is considerable interest in "Mobility Indices," that is, aggregated measures that consider several different components of mobility rolled-up into a single measure or index.  Opinions were divided, however, as to whether single indices are appropriate for describing mobility, and perhaps for other system focus areas as well.  Mobility may be particularly resistant to meaningful indexing, because it is a complex concept and one that is measured in such a wide variety of ways in different states.  More states seem to have pursued development of indexes in other system areas such as safety, system condition, and maintenance, than in mobility.

Data Integration

Data integration is the next major undertaking for many agencies that have brought performance and condition data into their everyday planning and management strategies.

• An important feature of data integration will be the transfer of data from legacy systems to integrated systems.  Dealing with legacy systems, non-standardized reporting dates, data formats, collection methods, etc. is a problem at one level or another for most agencies.  It is too easy to inadvertently draw different conclusions about a condition or trend if the results depend too much on when a data request is filled or the particular source of data.  This can lead to credibility problems with legislators and other outside stakeholders.
• In order to effectively and efficiently use various data sources and types for decision support, different databases and analysis tools need to be brought together and made to appear to the user as a single, well-integrated information center.  Data integration promises to significantly increase the value of data from the DOT and other transportation agencies to a broad base of internal and external users.  For large and small agencies, true data integration is a significant undertaking with major resource implications.

New Data Needs

• Changes have been made to pre-existing data collection programs and processes in order to address new the data requirements of performance measures.  However, DOTs report that additional funding for enhanced data collection is uncommon.  Some larger DOTs did report increased spending in strategic market research or economic analysis, but this was generally not coupled with additional funding for traditional data collection and analysis programs.
• New sources of data need to be tapped to support new measures and augment limited data collection budgets.  Some DOTs report teaming with university research centers to improve data collection and analysis capabilities.
• Standardized methods for collecting new types of data are desirable.  Some states report using variable collection methodology for collecting data such as travel time, with the result that even within an agency different data sets cannot be directly compared or merged.  This is probably more the case in areas such as mobility and accessibility, where agencies are collecting a type of data that was not previously collected on a wide-spread or systematic basis.
• Freight-related data is in growing demand by agencies, as the public-agency role in providing freight mobility and reliability becomes more important.  A wide variety of data on freight movements at the commodity and vehicle level are of interest to agencies, as well as safety-related information.  Currently, many DOTs find it difficult and/or expensive to acquire this data, nor are they well-equipped to forecast future truck volumes or freight movement patterns.

Section 2: Research Agenda

On the second day of the Peer Exchange meeting the group worked on summarizing key challenges in the area of data to support performance measures.  Nine areas of interest and concern were discussed in detail.  It was agreed that follow-up is desirable in each area.  Following is an outline of these areas of interest, indicating the persons identified from among the group to initiate the appropriate follow-up actions.

It was agreed that the research statements referenced above would be completed and combined for review by the Statewide Data Committee at the January, 2001 TRB Annual Meeting.  Additionally, the individuals assigned agreed to perform research and report at the annul meeting on these topics:  Mobility, Freight, Sustainability, and Safety.

1.  Market Research

Market research is desirable to obtain customer feedback (internal and external) with respect to the selection and use of performance measures.  This type of activity is beneficial to agencies in prioritizing workload.  The group agreed that the following points would be worth researching:  How to use existing surveys; Lessons learned from existing surveys; Understanding when market research is the right tool and when not to use it; and How to develop satisfaction indicators.  An outcome of the research could be standardized indicators, methods and questions.  It was agreed that Ed Christopher would prepare a research proposal in this area.

2.  Mobility

In the area of mobility measures, it was agreed that some standardized approaches might be appropriate.  The Texas Transportation Institute (TTI) Urban Mobility study is looking into this.  It was agreed that Anita Vandervalk will be the liaison between the Data Committee and the TTI team and report back as appropriate.  She will be assisted by Tremain Downey and Rob Bostrom.

3.  Freight Mobility

The development of Freight Performance measures was discussed and it was discovered that there are several studies on going in this area.  The question, "What is/should/could be used and what are the implications for data collection?" was posed.  The following points were considered to be important:  Travel times; Ton miles; Market segmentation; Truck type; Business travelers; Different perception of measures (e.g., pavement condition for cars vs. trucks); and Connectivity between terminals and corridors.  It was agreed Rob Bostrom, Mark Larson and Ron Tweedie would bring the issues before the TRB Freight Data committee.

4.  Intelligent Transportation System (ITS)

Data from ITS sources was discussed.  It was agreed that we need a consensus on the types of planning data that are needed from ITS.  Questions such as how to use the data, and "What can ITS do?" were asked.  Issues such as data capture, formats, software for processing, dictionaries, reliability of data, and integration with planning data all need to be explored.  It was agreed that Bill Walsek and Jonette Kreideweis would prepare a research proposal.

5.  Where are Performance Measures being used in State Government?

It is not clear where or how performance measures are being used in State Government.  The following questions need to be answered:  Have they had an impact on policy decisions? What is the payoff for policy level measurement vs. engineering level measurement? Jonette Kreideweis was tasked with preparing a research statement.

6.  Safety Measures

It was agreed that another level of measures beyond fatality/crash rates is necessary.  The example of high accident locations was discussed.  A research statement outlining how safety measures are used in the states and what the data issues are.  Roger Petzold was tasked with this.  He will also work with the TRB Safety Task Force and Tremain Downey will assist.

7.  Sustainability Measures

Measures of environmental justice, equity, and growth management were determined to be important.  Questions such as:  What are the data implications; and What new measures do we need to support the programs; were asked.  Ed Christopher and Tremain Downey were tasked with bringing the issue to the TRB Urban Data Committee.

8.  Summary of State's Goals, Objectives, and Performance Measures

The group discussed state goals and how they relate to Federal goals and measures.  The question of aligning goals from 50 states, approx. 350 MPOs and the federal level was raised.  In discussing the need for common indicators, it was decided that we should learn from other states rather than attempting to standardize goals and measures.  Ed Christopher, Roger Petzold and Tony Esteve were tasked with investigating this from the federal perspective and reporting back to the Data Committee.

9.  Quality Assurance of Data

The discussion here started out being focused on data quality issues and quickly evolved into several other important data issues such as data integration, data standardization, and privatization of data collection.  The following questions were raised:  How do you set up the relationship for data collection; How do you relate the data; Who is the data owner; Who coordinates integration of data; How are partnerships with MPOs, localities and private sector formed; and What are the effects of data integration on data quality and integrity.  The discussion centered around the changing role of state DOTs with respect to data quality due to new performance measures and the need to compare methods of quality assurance.  It was agreed that Anita Vandervalk and Ron Tweedie would prepare a research statement on this topic.

Section 3:  Summary of States' Activities

MnDOT - Performance Measurement Directions and Issues

Mark Larson of the Minnesota Department of Transportation (MnDOT) described the direction that performance measurement has taken at MnDOT, and some of the key issues the agency has been required to address.  As an early leader in the field of performance measurement in general and customer-based measures in particular, MnDOT's experience is relevant and useful even to those agencies who have already made considerable progress in performance measurement.

Evolution of MnDOT Performance Measurement

The MnDOT Department-level performance effort was introduced in 1993 following pioneer work in the maintenance area.  It has since developed and begun to fold together a number of focus issues and structural concepts, including the "family of measures" concept, a relatively sophisticated market-research orientation, targets, and now business-planning applications.

The original "Family of Measures" focused on the Transportation System, Public Values, and Organizational Performance.  The program is now adding strategic emphasis to areas including inter-regional corridors and program delivery.  According to Mark Larson, the attention to the new program emphasis thus far has been mostly internal to the DOT, and has not yet raised significant attention among the general public or the state legislature.

The following components of MnDOT's performance measurement effort were presented:

MnDOT's Current Performance Planning Process

Some of the tools, techniques, and applications now being pursued by MnDOT include:

• Strategic Objectives
• Customer Market SegmentationBusiness Planning
• Activity-based Costing
• Targets and Measures
• Application of Performance Targets to Budget and Investment Decisions

Business Planning

MnDOT's application of performance measurement to the business planning process was described as a means to deploy assets and resources to create value for their customers.  Steps include:

• Identify customers and their needs
• Establish target Levels of Service (LOS)
• Identify products and services to meet needs
• Determine resources needed
• Request resources through the budget process
• Measure results and adjust strategies and resources

Customer Needs

The five highest level key customer needs identified by MnDOT through substantial market research are:

• Predictability of travel time
• Smooth, uninterrupted trips
• Safety
• Timely and accurate traveler information
• Responsible use of resources

Long-Term Outcomes and Strategic Objectives

MnDOT has also identified desired long-term outcomes and strategic objectives for the program.  Interestingly, these are not all necessarily couched in customer-used terms, but rather speak to the special role and responsibilities of the DOT as owner and conservator of the transportation system.  These include desired long-term outcomes such as economic vitality, adequacy of infrastructure, and travel options.  Strategic objectives of the current program include progress towards a more multimodal orientation, special focus on significant interregional corridors, and delivery of the construction and maintenance programs.  These objectives represent a leadership thrust by the current administration to focus on strategic transportation needs into the future and on streamlining organizational performance.

Alignment of Needs, Objectives, and Targets

MnDOT recognizes the need for alignment between these Customer Need, Outcomes, and Strategic Objectives.  Targets and measures are then developed to provide this alignment.  Relative to past years, emphasis at MnDOT has shifted from measures to specific targets, e.g., a defined LOS for a specific date or period of time.  Use of targets helps to identify the "performance gaps" which need to be addressed in the business plan.  Targets need to be aligned internally as well:  Department, groups, districts/offices, work units all have targets that must speak to common objectives, yet must recognize the different levels of detail or aggregation of information that is required to meet the differing needs of these groups.

Interregional Corridor (IRC)

MnDOT's IRC program was singled out to provide current examples of how they are using performance measures and targets.  Performance targets have been set for three different levels of corridor.  Average speed is the measures, being derived from estimated travel time over route distance.  Higher target speeds are established for the higher priority IRCs.

Strategies to meet IRC targets include highway design, access management, growth management, and integration of ITS techniques, freight and transit planning into system planning and operation.

New Directions

One important new direction at MnDOT is increased segmentation of the customer market.  By disaggregating their market research and focusing on distinct groups of users, more can be learned about the unique needs of these segments, and this better solutions can be devised.  Example customer market segments include commuters, farmers, emergency vehicles, freight carriers and shippers, and intermodal trip makers.

An emerging area of emphasis is freight movement.  MnDOT's private sector Minnesota Freight Advisory Committee has proposed several performance measures.  Measures focus on several important outcome areas, such as economics, IRC travel time, safety, removal of impediments, and reliability of travel time.  Freight measures are divided into groups according to use of the measure, for example, MnDOT's use in investment prioritization and design, vs. the freight community and policy makers, e.g.:

• Examples of Proposed freight measures for project selection and design:
  • Economic benefit costs ratio
  • Shipper point-to-point travel time on IRCs
  • Number of design impediments slowing access to modal/intermodal terminals
• Proposed freight measures for policy makers:
  • Public transpiration investment as percent of the state economy (GSP)
  • Travel time to global markets
  • Congestion compared to other major metropolitan area

Data Management Issues

The following data management issues were identified:

• Need standard methods for collecting new data for new measures such as speed and travel time.
• Dealing with legacy systems, non-standard reporting dates, etc.  It is too easy to inadvertently draw different results depending on when a data request is filled.  This can lead to credibility problems with legislators and other outside stakeholders.

• Current vs. consistent:  data customers want it current, yet they want it comparable to other data sources.  This is a data archiving and synchronizing issue.

• Different data sources need to be better integrated, so different users pulling data at different sites are drawing on a single, consistent data source.

• Quality assurance:  Data users want assurance and validation that the data provided is accurate.  A growing importance is placed on data accuracy.  The more the data is reported and used, the more the accuracy is scrutinized.  Need to move out of data reporting and into data analysis.
• Similarly, more accurate real-time data on speed and travel time is requested by commercial customers, and needs to be accurate to be marketable.

Additional Discussion of MnDOT Presentation

MnDOT has not received more money for data collection, generally speaking.  Increased resources have been put into strategic market research, economic analysis, and strategic planning, but not necessarily traditional data collection and analysis.  There is more emphasis, as well on organizational performance, efficiency of projects, and program delivery.

Some states noted they have relied more on University research centers to improve data collection and analysis.

"Defining the customer" is not as obvious as it seems.  Market research is being used to better identify the customer market segments, to identify what is important to them.  Tennessee DOT also is reported to be applying market segmentation techniques.

There is some concern over whether market research results are accurate, and actually speak for the population as a whole.  Combining or even comparing results from different surveys has proven problematic.

Travel speed in MnDOT is an estimate arrived at via travel time and distance estimates.  Travel time was measured by probe vehicles, but collection methodology is variable.  Some data was from in-ground monitoring devices as well.

There may be a concern or lack of familiarity at the legislative level over the DOTs move into business methods and tools.  One perspective is that the process bypasses elected officials and the political process.  This may also be true of market research, as elected officials see themselves as the "pulse-takers" of public need and opinion.

There is a legitimate concern over getting too technical with decision-makers.  They want measures that tell the necessary story as clearly as possible:  Are we doing well?  Do people think we're doing a good job?.  Engineering and planning departments have tendency to want to make measurement very precise and specific, resulting in too much data potentially being heaped on decision-makers.

Illinois DOT

Jim Hall, formerly of Illinois DOT and now with the University of Illinois at Springfield, gave an overview of the status of performance measurement at IDOT.  According to Jim, IDOT has not yet pursued performance measurement to the same extent as some other states.  He focused the presentation and discussion on some of the concerns at IDOT that may serve as barriers to adoption of the concept of performance measurement, both in decision making applications as well as in providing the public with detailed system performance data.

Concerns of Management and Others

• There is concern over increased public expectations for access to accurate, real-time data, particularly via the Internet and Web.  Currently, IDOT gets most requests for information about snow-related lane closures.  Greater public usage of this data raises the expectation and requirement for data quality and timeliness, and the DOT would have to anticipate and budget for that to avoid public relations problems.
• Even more in demand is traffic data for use by developers (presumably in preparation of impact analyses, etc.).  Rather than real-time data, these users want comprehensive traffic counts and forecasts.  This raises similar concerns about currency of counts, and credibility of and assumptions behind traffic forecasts.
• Some measures can be a two-edged sword, for example, congestion:  Over time it will gradually become worse, but there is the possibility that customer satisfaction with congestion levels will remain generally constant as expectations are adjusted.  The DOT needs to be able to reconcile these kind of discrepancies for decision-makers.

Data Needs and Integration

• GIS is the main data integration tool at IDOT, as most databases are tied to their GIS framework.  Once the system is better integrated internally, they expect to see more public demand for data over the Internet.
  • Per Jim, the technology is available, it is just a question of how much data the DOT chooses to make readily available.
  • Truck-related data is needed for more than just the Highway Performance Monitoring System (HPMS).  They are interested in gathering more truck-related crash data, by area and corridor.  IDOT is investigating sampling methods that focus on what the state needs rather than just HPMS reporting requirements alone.

Data Quality Issues

A discussion emerged over issues of data quality and the public use of DOT data.

• One way to get better data accuracy is to "expose it to the light of day."  As more data is made available and put to use, there will be more public pressure and hence management support for data quality, timeliness and accuracy.
• Sometimes you have to put data out even if you know it is imperfect.  This too may bring about increased pressure for improved data.
• How good is good enough?  Some data uses have financial impacts for the public or private sector.  Need to have appropriate disclaimers, in addition to striving for better data quality.
• Due to inherent inaccuracies and imperfections in data, data should be used for indicators, rather than as hard-and-fast decision points or determinants of funding.  Let people make decisions, using data indicators to guide them.  Even a very good data system should not absolve people of the responsibility that comes with decision-making authority.
• You're constantly uncovering errors in data or areas that need improvement.  How do you address this when data is being used to support critical decisions with financial or safety implications?  We need to be careful to avoid creating a false sense of perfection, or greater expectations for data quality and accuracy than we can deliver.
• Are DOTs and decision-makers using relatively rough indicators to drive specific decisions?  For example, are indicators based on estimates or rough measurements used to make fine design decisions?  Sometimes there is no better alternative, in which case the best approach is to make sure the decision-makers are presented with and understand the variability in the data, range of valid estimates, etc., as opposed to taking one data point and over-applying it.  This is something the data community can do something about, whereas we may not be able to change the way in which decision-makers may use or misuse data.

Florida - Mobility Performance Measures Program

Anita Vandervalk, Manager of the Florida Department of Transportation (FDOT) Transportation Statistics Office, presented an overview of the Mobility Performance Measures Program.  Florida is another state with substantial experience in the development and application of performance measures to transportation system planning, evaluation and management.  Among the motivating factors for FDOT is the State's Performance-Based Budgeting process, which requires each agency in Florida to identify and apply measurable performance objectives.  Performance measures are used to link planning and budgeting, with the goal of improved accountability.

Mobility Measurement at FDOT

FDOT has developed numerous measures related to goals and objectives in four main areas:  Safety, Preservation, Interconnectivity of the System, and Mobility.  Mobility in particular is of concern to the department because it is so difficult to measure.  External factors such as economic growth, land use, and personal preferences of travelers, combine with internal factors such as infrastructure investment and management, to determine mobility performance for the user.  One of the difficulties is the fact that FDOT does not have direct control over mobility results.

Mobility Defined

FDOT has defined mobility as "The ease with which people and goods move through their community, state and world."  They have identified several distinct components or attributes of travel and the transportation system which contribute to mobility The following represents areas for which FDOT has a responsibility to provide for it's customers:

• Modal choice
• Accessibility
• Adequate travel time and speed
• Affordability
• Equity
• Knowledge of the system
• Reliability

Dimensions of Mobility Performance Measures

In order to permit objective, structured quantification of mobility, FDOT had identified four dimensions for measurement:  Quantity, Quality, Accessibility and Utilization.  Specific measures are identified for each dimension.

• Quantity reflects the magnitude of the use of the facility or service, and thus the demand for the service as well as the utility of the service.  Example measures consider quantities such as person miles traveled, vehicle miles traveled, and ridership.  Unlike some states or MPOs, these measures are not indexed to population or other normalizing descriptors, and thus PMT or VMT would be expected to rise along with growth in population, employment, etc.
• Quality of travel is related to the conditions of travel from the users' perspective.  Depending upon the mode or system, different measures are applied.  On state highways, average speed and delay are measured.  In corridors, average travel time and reliability of trip time are used.  In transit modes, the ratio of auto travel time to transit travel time for the same trip is estimated.  Reliability of travel in corridors is expressed as the percentage of travelers whose travel time is significantly worse than should reasonably be expected.  Reliability is therefore defined as "the percent of travel that takes longer than the average time by no more than a certain acceptable additional time."  The average time is the median travel time during the time period being analyzed. is a percentage of the average travel time during the period being analyzed.  An initial value is 5 percent but surveys of users will determine an acceptable percentage.  Notably, travel time data that were influenced by traffic accidents were retained in the sample when determining performance.  The component of travel time variability that is attributable to non-recurring incidents is considered to be part of the reliability performance.  One drawback to reliability measures is the data requirement.  Corridor-specific data is required for the measures to be meaningful, and the State is currently focusing on key urban corridors.
• Accessibility describes the ease with which people can connect to the multimodal transportation system.  For state highways, measures include dwelling unit proximity, employment proximity, and percent of route miles with bike or pedestrian facilities.  Accessibility of the Florida Interstate Highway System (FIHS) is of particular importance because it links the state's commerce and tourism destinations.  Nearly 90 percent of significant intermodal facilities, two-thirds of population and jobs, and 80 percent of industrial warehouse facilities are within 5 miles of the FIHS.  For metropolitan transit systems, accessibility measures include coverage, frequency, and "span" of service.  These data are generally available from existing sources.
• Utilization measures relate demand to capacity, and describe the efficiency with which the system provides mobility.  These measures tend to reflect the Department's perspective rather than those of the customer, and include percent of system that is heavily congested, vehicles per lane mile (a density measure), duration of congestion, and transit load factors.

Data Issues

Data Collection

FDOT applies a combination of estimated and observed data at the system level, and primarily observed data at the corridor level.  The Department focuses on data which is already being collected, with refinement as necessary to meet the needs of the performance measures.  Some special data collection is required to support the measures, such as vehicle occupancy, as well as speed and volume data to support reliability measures.

Changes to FDOT's data collection program required to support the mobility program measures include implementation of a short-term traffic monitoring system to extend the time period for which traffic data is collected, and eventually a permanent TMS which can provide 15-minute interval polling on demand.

Data Modeling Issues

Consistency across the statewide system is deemed more important than great precision on local segments.  To help manage expectations, FDOT advises against mapping data at a level that would give rise to suspicion or complaint.  This helps to ensure that data is used at the statewide system level for which it is intended.

Traffic congestion "spill-backs" are not yet considered by FDOT's travel models, thus congestion statewide may be underestimated by the models.  Nonetheless, trends rather than spot values are of most value.

ITS Data Integration

An effort to integrate data collected through the State's ITS infrastructure is underway.  FDOT is developing statewide data architecture, addressing the following needs:

• Data retrieval:  There is a proliferation of detectors and sensors, but can they provide needed data?
• Data storage:  What data should be kept, by whom, and how should it be stored, for how long?

• Data reporting:  who are the users, and how does FDOT need to package the data for users?

 

FDOT acknowledges the need to identify both architecture and protocols for data sampling and manipulation, in order to distill the vast pool of ITS data for planning and monitoring activities.

Data Quality

Florida has a well-established QA/QC program, and uses other sources and validated models to supplement and check its data quality.

Audience for Performance Measures and Reporting Mechanisms

FDOT's audience includes the public, department staff, elected officials, legislative staff, and other decision-makers.  Legislative staff are taking more interest in the program and starting to appreciate the results and benefits.

At the district level, staff want more detailed information than is generated for the Department level.  There are three areas of reporting:

• Entire state high system
• Florida Intrastate Highway System (FIHS)
• Corridors - in particular, those connecting major urban areas

Future Directions

FDOT will continue to improve and evolve its performance measurement program.  Among those areas identified for development are:

• Person-trip based measures
• Dynamic display of measures
• Refinement of reliability measures
• Corridor-level reporting
• Incorporating ITS data and analysis
• Wider distribution of results

Additional Discussion of FDOT Presentation

Other members of the committee expressed substantial interest in FDOT's substantial reliance on outside consultants to help design and implement highly technical and labor-intensive programs such as this.  It was apparent from the discussion that some states do not do this to the same extent as FDOT, or aren't even able to do use consultants in this fashion..  FDOT condones the use of on-going consultant support as extension of staff.  The current "privatization" initiative at the state level in Florida further supports this approach.

California Department of Transportation System - Performance Measures

Tremain Downey, title, presented Caltrans' system performance measurement effort, focusing on the accomplishments to date and status of implementation.  The Program's objectives were established via committee with broad representation from outside the department.  They acknowledge that performance measurement can focus on Agency performance, modal system performance or multimodal system, and Caltrans has chosen to focus on the multimodal system.

Uses of Performance Data

Caltrans intends to monitor and evaluate system performance, in a way that is complementary to that done by already underway at several major MPOs in the state.  They intend to share existing data and future forecasts of performance.  Objectives include development of "mode-neutral" customer and decision-support information.  Caltrans hopes to improve consensus-building efforts in the transportation planning context by providing objective, understandable information.  While part of the motivation is to improve accountability of the Department, there is also an attempt to improve information about those elements of performance for which Caltrans is not directly responsible.

Key Elements of the Caltrans System

Caltrans' performance measurement system builds on five important concepts.  It should be

• Outcome-based,
• Multimodal in scope,
• Easy to understand,
• Supportable with existing data, and
• Useable to both monitor and forecast performance.

Caltrans is well-along with its three phase project, spanning Design, Proof-of-Concept Testing, and Incremental Implementation. 

Performance Outcomes and Current Measures

Caltrans has defined nine "outcome areas" which guide selection of measures.  These include outcomes such as mobility and accessibility, reliability, cost effectiveness, safety, equity, and economic well-being.  Candidate measures or indicators have been defined in each area.

Thus far, Caltrans has fully tested three measures.  These include measures of mobility/accessibility, reliability, and environmental quality.  Other measures are undergoing testing, or are yet in the testing phase.

The measures of mobility include average point-to-point travel time and travel delay.  Because of the data-intensive nature of this travel time measure, the focus is currently on travel delay.  Delay is estimated as the difference between free-flow travel time and average observed (or projected) travel time.

Caltrans' reliability measure is based upon variability in service between the expected and actual travel times.  Testing of the reliability measure in major metropolitan areas of the state reveals that peak period variability ranges from 10 percent to 50 percent, with most segments experiencing variability that is within 30 percent of expected travel time.  An interesting observation is that reliability is not necessarily correlated with delay.  Some segments or corridors may have high delay but relatively low variability in travel time.  Factors identified by Caltrans that contribute to reliability and variability include distance between interchanges and highway geometrics, among other factors.

Data Issues

Caltrans has embarked on an aggressive data integration program.  This system will provide integrated data on performance, physical characteristics, boundaries, environmental, and financial aspects of the transportation infrastructure.  Data collected by Traffic Management Centers (TMCs) will be routed to a single centralized location, rather than to district offices as at present.  Through this centralized clearinghouse, users and customers will have access to a broad array of data via user-friendly interfaces.

Several steps remain to design and implement the system, including making decisions on what information should be collected and stored, what technology will be used to collect and manage the data (particularly in expansion areas not currently served by TMCs) how to convert data to useful information, and how to best provide user access to the data.

Relationship to Decision-Making

Caltrans believes integrating data as described above is the biggest challenge and most important task to undertake at this point.  Integrating the data into the planning and programming processes would follow.  California is not yet using data to prioritize or program system projects and services, but rather to evaluate existing projects and provide information to local decision-makers and customers to use as they wish.

Kentucky Transportation Cabinet

Rob Bostrom, Transportation Engineering Specialist with the Kentucky Transportation Cabinet (the State's DOT) presented information on Kentucky's use of performance measures in the planning process and their relation to the KTC Quality Initiatives.  The main goals of the Quality Initiatives are to ensure mobility and access, support economic development, continually improve organizational performance, and to strengthen customer and stakeholder relationships.  Overall emphasis is placed on improved performance in the areas of project delivery, safety, human resources, and system preservation.

The presentation of Kentucky's performance measures efforts focused on three units of the KTC:  The Division of Multimodal Programs, the Division of Planning, and the Kentucky Transportation Center, a research unit affiliated with the University of Kentucky.

Division of Multimodal Programs

Performance measures are used in a range of planning and analysis support functions, including:

• Travel demand modeling
• Air quality (measures include emission reduction and SIP compliance)
• Traffic forecasting (timeliness, charging costs to projects)
• Mobility (cost of congestion)
• Small urban area studies

Pilot mobility evaluation programs are underway, testing measures including travel time, travel reliability and congestion.

Division of Planning

The Planning Division at KTC encompasses numerous analytical support functions including Geographic Information systems (GIS), Highway Information Systems (HIS), and Traffic Monitoring Systems (TMS).  Performance measures are used in each of these groups to improve performance and quality of data provided for use by others.  These include organizational performance and program quality measures such as degree of accuracy of data sets, timeliness of inventory, number of mistakes, etc.

Kentucky Transportation Center Research Unit

The Transportation Center has numerous planning related studies underway or ongoing, such as vehicle classification analysis, load spectra development, cost of construction delays, etc.  As an example, the Growth Rate Analysis study will improve VMT estimation capabilities and bring more sophisticated forecasting and analysis techniques into play.  Studies such as this are measured in terms of their timeliness and usefulness.

Pennsylvania Department of Transportation "Strategic Agenda"

Barbara Mason Haines of the Pennsylvania Department of Transportation (PennDOT) presented information on the Department's "Strategic Agenda," a Scorecard of Measures Information System, and state data currency issues.

Strategic Focus Areas

The Strategic Agenda is based upon Vision, Mission, and Value statements developed through a typical strategic planning process.  The eight Strategic Focus Areas encompass most of the issues of interest tackled by other state DOTs, such as mobility, customer focus, and safety.  Of special note in the PennDOT effort are strategies such as a "maintenance first" agenda, singling out the importance of innovation and technology, and building stronger relationships with partners including customers and suppliers.

Each of these Strategic Focus Areas is associated with one or more goals and corresponding objectives.  These goals and objectives tend to be "high level" or "strategic" in nature, that is, focused on relatively broad results and aimed at department-wide processes and activities.

Also of note in the PennDOT Strategic Agenda is that each strategic focus area has an "owner" or leader who is responsible for results.

Scorecard of Measures Information System

The PennDOT Strategic Agenda includes a well-organized "scorecard of measures" that clearly lays out strategic focus areas, goals, measures, measurement tools (the actual metric, e.g., IRI, or ISO 14001 environmental criteria) and targets.  Distinction is made as to whether a goal supports external customer needs or internal (e.g., Department support) needs.

PennDOT has developed an electronic version of this scorecard of measures providing a significant amount of structure, detailed performance data.  This CD-based program stores and provides links to information on a wide variety of performance fronts.  The user can focus in on specific measure, e.g., IRI, observe performance by engineering district and county, or find out which particular pavement type or traffic routes are contributing to a problem.

Data Issues

Traffic volume and other highway statistics are available on the Internet, and draw from same database as the state HPMS.  PennDOT has resolved the data currency vs. reporting cycle issue.  All users draw from same database, which is updated quarterly.  Actual data in the database is updated continuously, but only "owners" can access the latest data that has not yet been summarized for the most recent quarter.  "Non-owners" have access only to latest quarter, eliminating or minimizing the problem of different users citing data from disparate time periods.

Maryland Department of Transportation, State Highway Administration:  System Performance Report

William Walsek, Division Chief of the Highway Information Services Division at the Maryland State Highway Administration, presented the State of Maryland's Highway System Performance Measurements program as well as the SHA's Four-Year Business Plan.  The Business Plan identifies eight key performance areas on which the Department is focused.  These are Mobility, Highway Safety, System Preservation, Economic Development, Community Enhancement, Environmental Responsibility, Customer Service, and Managing our Resources.

Business Plan

Within the Mobility performance area, several goals and corresponding measures have been identified.  While some of the measures are similar to those in use in several other states, there are some notable innovations.  These include, for example,

• Reduce the time required to restore normal traffic flow following an incident, and
• Provide timely and reliable mobility information to the travelling public

These goals suggest an unusual degree of emphasis on incident management and real-time information collection/distribution.

As at PennDOT, each goal is assigned one or two senior managers whose job it is to establish measures and monitor progress.  The performance measures are tied to specific numeric targets that quantify the desired level of service or improvement, and identify a specific target date as well.

The MdSHA objectives (targets) are a blend of external measures of outcome (results) such as "improve average clearing time of incidents by 5 percent by June 2001" and internal measures of agency output, such as "provide more real-time information on the Web by FY 2002."

Highway System Performance Measurements Program

A work currently under development, the Maryland Highway System Performance Measurements program will provide an annual report on system performance.  The report will include data on system coverage, utilization, capital investment, demand, condition, and community enhancement.  The system performance measures and indicators are generally time-based, that is, system condition or performance is expressed as trends over time.  Among current measures in use are track capital investment, lane miles, vehicle miles of travel, and relevant contextual information such as population, number of licensed drivers, size of labor force, etc.  Providing these context measures gives readers a quick way of assessing how transportation system trends (e.g., VMT) relate to general state trends (population or labor force participation).

Most of the data is derived from existing sources, and to date the report focuses largely on historic trends rather than projecting future trends.

Discussion of the Maryland DOT Program

As it is a work in progress, to date the annual performance report has not seen a great deal of external exposure or review.  The agency is concerned about consistency with other data distributed to the public by MSHA., Participants discussed the pros and cons of delivering a single annual report covering all topics, as opposed to, e.g., quarterly reports covering a percentage of the topic areas.  There is concern about the need to manage information flow and prevent "data overload" particularly at the level of senior management and elected officials.  There are also related benefits to a more distributed approach to dissemination, such as managing peak workload at the staff level, improving currency of the reported data, etc.

Texas Department of Transportation

Kim Hajek, Director of Data Management in the Transportation Planning and Programming Division, presented information on performance measurement at the Texas Department of Transportation (TxDOT).

Background on Performance Measures in Texas

Who is the audience and how is it reported?

The performance measurement program serves the Governors Office of Budget and Planning as well as the Legislative Budget Office.  Reporting is relatively formal and structured, as a result, and is fed directly into the budgeting process.

TxDOT reports regularly to legislative finance committees on financial performance.  Overall, the performance measurement process in Texas has a strong financial component to it, relative to other states.  The financial staff and committees are among the main target audience for the data and measures.

A balanced scorecard approach is used, including both internal and external measures, process and results.  Measures attempt to be both explanatory as well as to gauge efficiency of the system.  Department output is measured as well as outcome or results "on the street."

Strategic Plan

TxDOT's Strategic Plan incorporates performance measurement to determine whether specific goals are met.  The plan has a five year horizon but is updated every two years.  The new plan defines 4 goals, 16 strategies, and approximately 120 measures.  Each performance measure reported will have corresponding fiscal analysis, describing what resources will be required to deliver the targeted level of performance.  This is an important component of the plan that other DOTs may wish to review.

Also of interest are objectives and measures that are on the fringe of TxDOT's ability to influence outcome, e.g., reduction in auto theft.

Discussion of TxDOT Presentation

There was a spirited discussion over whether it is appropriate to measure rates (e.g., number of fatalities per million vehicle miles) or absolute values, (e.g., total number of fatalities)  Proponents of absolute values argue that indexed rates mask poor performance.  For example, even though a state's population and VMT are growing, a modest reduction in fatality rate will still result in more absolute fatalities per year, which would be considered poor or unacceptable performance by some..  The corresponding counter-argument is that measures should focus on things the department can more directly influence, such as the accident rate, and not total accidents, since they cannot really influence population growth or even VMT growth to a large extent.  Proponents of the "rate" approach would point to a n improved (reduced) fatality rate.  People say that measures should be "attainable" and it is not likely that absolute fatalities can reasonably be expected to hold or drop in a high growth state, such as Texas.  The opposing argument is that the rate hides the truth, which is that more people are killed per year, and that is what the public and elected officials focus on.

This was an interesting debate because it calls into question the notion that you should measure what you can influence, and instead measure what customers think is relevant or important.  Compromise/consensus opinion is that you need more than one measure for most important items, and that it also helps to have diagnostic measures, as well as indicators (e.g., what are the causes and particulars behind the fatalities, not just the totals and rates).

New York State DOT Performance Measures

Ron Tweedie of the New York State DOT described the state's use of performance measures in its capital programming efforts.  Performance measures are used in development and monitoring of the Capital Program.  They are used for project selection and monitoring, and to gauge the customer satisfaction with results of the program.  Performance measures are applied in four traditional areas of system programming:  Pavement, bridge, mobility and safety.

Pavement measures include pavement condition, with priority given to NHS and major truck routes; paving cycle (how long it takes to return to a specific segment), treatment life, and percent single course overlays..  They also compute the ratio of treatment life to paving cycle.

Bridges:  Per Ron, New York "has lots of old ones!"  A bridge condition index is used to track the 7,600 state highway system bridges.  Primary concerns are safety, preservation and serviceability.  Safety measures include vulnerability to erosion and structural condition.

Mobility.  NYS DOT defines mobility as the ability to move people and goods conveniently, reliably, safely and at reasonable cost.  Measures in use include those related to congestion, and NYS DOT is interested in developing measures based on the value of time in addition to simply travel time itself.  They are also very interested in ton miles of delay due to the importance of shipping time, but acknowledge the difficulty in obtaining good data about shipment travel times.

Safety issues revolve around reducing the number of High Accident Locations (HAL), the number of HSIs conducted, and inclusion of all HALs in the annual work program.

Discussion of NYS DOT Presentation

A discussion ensued regarding the use of "Mobility Indices," that is, aggregated measures that consider several different components of mobility rolled-up into a single measure or index.  Opinions were divided, with a slight majority of group seeming to agree that single indices are inappropriate for describing mobility, and perhaps for other system focus areas as well.  Mobility may be particularly resistant to meaningful indexing, because it is a complex concept and one that is measured in such a wide variety of ways in different states.  More states seem to have pursued development of indexes in other system areas such as safety, system condition, and maintenance, than in mobility.