Sample Methodology for Analyzing Non-Standard (xx) Public Comme

SAMPLE METHODOLOGY FOR ANALYZING VARIABLE PUBLIC COMMENT

An essential but often neglected component of public involvement processes is the sponsoring agency’s summary of public consensus, or lack of consensus, about the project or plan in question. A “report back” should be provided on a timely basis following every comment period or significant public hearing/meeting. The methodology for documenting surveys and charettes is better established than is the methodology for documenting disparate comment in variable formats. The latter challenge is addressed here.

A neutral analysis provides validation to the public that: 1) their effort was seriously considered as part of a body of data contributing to a final decision; and 2) their opinion is or is not shared by many others who took the time to comment. It also symbolizes the reciprocal nature of the democratic process and builds trust and support for future interactions between an agency, the public and elected officials. A climate of suspicion can be created when public comments are unread or inadequately understood at the time decisions are made. Public hearings that are just “window dressing”—an oft-repeated claim made by citizens, often accurately—create animosity between the public and the staff of the sponsoring agencies, who commit great effort to the design and implementation of public hearings and meetings.

Transportation agencies in Illinois are very conscientious about collecting and storing public comments. The challenge to making use of the comments is that decision-makers, public advocates and others who want to know what the comment “meant” must read through pounds of paper – 15 pounds for comment on the 2030 Regional Transportation Plan in Northeastern Illinois, according to the Chicago Area Transportation Study. This burden is not one that most decision-makers have the luxury to assume.

The purpose of this report is to show that neutral analysis of public comment is feasible, that it requires minimal time and that the time is well-spent. The Center for Neighborhood Technology’s prototype analysis^¹ was conducted on over 1000 disparate comments about a potential alignment of the “Prairie Parkway.” The documentation was over two years old at the time of review; however, a review of the comment was deemed timely for two reasons. A $15 million federal allocation to study the feasibility of the highway obligated the state to begin Phase I engineering. In addition, a Context Sensitive Design law was enacted in Illinois and the Illinois Department of Transportation (IDOT) identified the Prairie Parkway as one of its first attempts to implement public involvement requirements of the law.

Data collection can be as elaborate or as simple as the analyst and the agency prefer but the cost of developing a highly sophisticated database of information may not be justified. The prototype analysis described here provided an ample level of detail for analysis, but did not create a large data entry burden. Based on past experience in reviewing and summarizing public comment, CNT set up a spread sheet to capture the following baseline information:

The last name, first initial within one field proved to be useful for sorting for duplicates; the use of separate fields for first and last name would have increased the accuracy only marginally. It was determined that for this analysis of a regional project, the municipality of the commenter, not the full address, was sufficient.

Opinion categories were developed at the outset to capture the broadest level of content, with subcategories added when strong trends became apparent in the documentation.

“Other” was a column used to store phrases or themes that were repeated frequently (or occasionally to capture very unique comment that was worth recording). Depending on the frequency, some of that information was transferred to new columns relatively early in the review. For instance, two of the campaigns that were clear from the start of the analysis were transferred into a column, for speedier coding, although two others that were less numerous or discovered later were coded by typing their key words. Handling the data by different coding methods does not impact one’s ability to analyze data since they can be extracted easily in either format. As long as key words are typed consistently, only the speed of data entry is impacted by using a separate column for a variable.

In this review, two themes rose to prominence. One was the number of people who suggested one particular roadway as an alternative (other roads were mentioned occasionally as alternatives, but not with the same frequency). A second theme was concern for negative impacts on the comprehensive plans of the two counties the proposed highway bisects. In this analysis the former theme was noticed early and often enough to trigger a separate column; the latter was not.

The question of what is a meaningful “theme” is somewhat subjective. A transparent description of themes and accounting for the frequency of a theme increases the objectivity of the analysis or at minimum enables discussion of what is a reasonable threshold for “frequency.” Display of the totality of the data allows others to verify that themes were not omitted.

Duplicates were addressed during data entry by three methods. The spreadsheet program automatically completes a field when the first several letters are uniform to a previous entry; often this helped the operator identify duplications. Operator memory was a second line of defense against duplication as the data was collected. A third safeguard preceding analysis was a quick scan of the complete list sorted in alphabetical order by name to verify that duplicates had been fairly well omitted. Judgment has to be applied because sometimes the same person submitted a letter at different points in time, or submitted comment by a different medium. In other cases IDOT had inadvertently photocopied the same letter more than once. While the system for identifying duplicates may be imperfect, allowing a small number to reside in the data, the way to alleviate concerns that they skewed the results is by full disclosure of the names in the analysis.

The analysis was conducted using common spreadsheet and database software. Data were imported back and forth between programs depending on the analysis required. For instance, a database allows the segregation of records that contain a key word, whether or not that key word is the lead word in the cell. The database also allows more flexibility for printing reports. The spreadsheet was useful for adding dissimilar information (in this case, one useful function of the spreadsheet was to calculate cumulative totals by municipality and sort into columns by proximity to the project - it was apparent from reviewing the data that geography greatly influenced opinion). The spreadsheet was also useful for creating charts and graphs of the data.

The grouping by corridor was achieved by manually sorting municipalities into three “corridors” and one “outside the area” category.^² It could have been done more rapidly by a mapping program, but that is not an essential component for the analysis.

The reports and charts created for the analysis were attached to the report, providing transparency. In addition, a full copy of the dataset was included, printed in the order in which the comment is arranged within three bound volumes. This allows any member of the public or the sponsoring agency to assess the accuracy and fairness of the characterization of opinion as favorable or opposed.

The analysis was accomplished in slightly under one work week. One staff person committed the following time to the project:

To put this time in context, IDOT committed the time of nine staff and seven consultants to a three-and-one-half hour public meeting on December 11, 2001^⁴ (56 hours). IDOT staff and consultants met with additional groups during and after the public comment period. The openness and accessibility of IDOT staff and consultants is commendable; however, it is not unreasonable to question whether a comprehensive evaluation of the comments elicited by their outreach would have been equally or more valuable.

At the completion of this project there are a couple of enhancements we believe could have improved the report. IDOT staff carefully numbered each submission as it arrived. In future analyses we recommend creating a field to capture that number, which may be useful in weeding out duplicated data.

The question of whether or not to code certain secondary themes requires subjective judgment—judgment that is often influenced by how early in the analysis a particular theme first becomes evident. In the analysis used as an example here, CNT chose not to code agricultural preservation as a separate category, given that we did code for the Kane and Kendall County comprehensive plans, which promote agricultural preservation. When it becomes clear at the end of data entry that an additional theme should have been coded, resolution can take one of two forms; the reviewer can return to the documentation and code the existing records for that single variable or can note in the written analysis that the theme was prominent, but not as prominent as other themes.

An analysis of public comment on a specific project is by nature more contained and manageable than analysis of public comments on a long-range plan, which will be multi-faceted and may cover an extensive geographic area. We believe analysis of the official comment on regional transportation plans is equally important but acknowledge that the time commitment to summarize the data is somewhat greater than described here because of the variety of projects and policies people respond to.

However, many of the same principles outlined in this methodology however, can be applied to a broader analysis. The baseline data is fairly similar, but general opinion and themes have to be defined more broadly. CNT has some experience with this kind of analysis. In 2003, in the absence of an analysis by the Chicago Area Transportation Study (CATS) of comment collected on the 2030 Regional Transportation Plan, CNT conducted a partial analysis to try to stimulate CATS to perform a complete analysis (included as an appendix to this report). CNT did not attempt to quantify reaction to every project mentioned by the public but to compute reaction to major themes (transit coordination, lack of project selection criteria, the need for grade separations, etc.) and mode choice (transit, highway, pedestrian and bicycle). We tabulated responses to the five major projects that were most frequently mentioned in the comment. Those projects were: the extensions of Route 53 and I-355, the Prairie Parkway, the widening of I-290 and proposals for an east-west transit alternative to I-290 widening. We believe that while a transportation planning agency may not have the capacity to compute the response on every project mentioned by the public, it should have the responsibility to calculate the response on high profile or controversial projects that consume public funds amounting to hundreds of millions of dollars, as was the case with the five projects mentioned above.

1 The analysis is included as an appendix.

2 The categories were: a corridor approximately ten miles wide with the proposed alignment in the center; a ten mile wide corridor to the east and a ten mile wide corridor to the west.

3 Travel time to the IDOT office is excluded; we propose that this be done by IDOT staff as an integral component of all public involvement processes.

4 The Prairie Parkway: Public Hearing Summary, Illinois Department of Transportation, 2001

The Center for Neighborhood Technology

July 2004