2. PROCEDURAL ISSUES IN THE EVALUATION PROCESS

2.1 Administration of the Project Evaluation

Although the focus of the Residential Area-Based Offices (RABO) Project is to investigate the organizational acceptability and travel impacts of telecommuting centers, some interesting lessons were learned about the project evaluation process. Measurement of telecenter operation, as required by the project, necessitates constant communication with site administrators, who provide information on the status of project participants. As a result, a member of the evaluation team (referred to here as the Evaluation Manager) was assigned to keep in contact with the site administrators to monitor the progress of telecenter use. The duties of the Evaluation Manager included tracking the status of telecommuters, providing survey materials to the sites, collecting returned survey materials, and conducting exit interviews by phone.

A number of issues arose in the process of conducting the evaluation of the RABO Project, and these issues are described more fully in the later sections of this chapter. These issues included both policy decisions as well as unforeseen events that increased the effort required to conduct the evaluation properly. These issues of evaluation procedure can be divided into compliance with contractual obligations (Section 2.2), measurement of site usage (Section 2.3), and modifications to the survey process (Section 2.4).

2.2 Contract Administration

As part of the RABO Project, the university sought applications from developers interested in establishing telecommuting centers. The site developers were required to sign contracts with the university which had provisions requiring the implementation of telecenters and the collection of data from the telecenter users. In addition, site developers were required to have employers interested in utilizing the center sign a memorandum of understanding (MOU) that described telecenter procedures and the evaluation requirements of the university (see Appendix I). Unfortunately, problems arose with the collection of data from project participants that necessitated changes to the site developer contracts. These problems included indirect contractual arrangements, insufficient motivation for users to participate in the evaluation, and inefficient survey administration.

2.2.1 Contractual Arrangements

For some site developers, contractual arrangements hampered the communication link between the evaluation team and the site administrator. Lower-level subcontractors and independently established telecenters were not tied to the evaluation as closely as the majority of the centers, and as a result, data collection from these sites was adversely affected.

In one particular case, a site developer, who was subcontracted under the RABO Project to open and operate a telecommuting center, subcontracted, in turn, with a private company to manage the site. The lack of direct leverage with the company managing the center and interacting with the telecommuters made the evaluation process more difficult. Several months passed before the private firm responded to the data collection needs of the project. Consequently, the baseline surveys were administered after telecommuting had begun (instead of prior to the start of telecommuting) which resulted in poor quality data. For future projects, it is recommended that the contractual relationship be directly with the entity managing the site and performing the evaluation services. Where this is not possible, special contract provisions need to be made to ensure proper compliance with evaluation procedures.

As part of the RABO Project, five telecommuting centers were included in the evaluation that were established outside of the project (see Table 2-1). While three of the five centers also received funding from the California Department of Transportation (Caltrans), they received no direct funding from the RABO Project. The other two sites, Concord and San Jose, were evaluated under an MOU between Caltrans and the Metropolitan Transportation Commission, but yielded very little usable data. As above, all of these sites had no direct contractual relationship with the university and, therefore, lacked an immediate incentive to provide data. It was not until Caltrans used its funding leverage that cooperation from the non-RABO sites was obtained.

¹ The opening date corresponds to either the date of the first telecommuting occasion or the official opening date if sign-in data from the center is not available.
² These sites left the RABO project before June 30, 1995.

Even after cooperation of the sites was assured, the telecommuters themselves were not always willing to participate in the evaluation. Unlike the RABO sites, the center users at non-RABO sites did not sign any type of MOU. Thus, survey completion was not a requirement for telecenter use. While the non-RABO sites have been included in the analysis in this report to the extent possible, the quality of the data collected and the response rate of the telecommuters is poorer than that for the RABO sites.

2.2.2 Ensuring Compliance

Even with a direct link established between the Evaluation Manager and the site administrator, problems arose in the measurement of center activity. Although all but one developer included a provision in their MOU specifying user participation in the evaluation process, at least two developers described this process as "voluntary". This was, in part, a reflection of a faulty understanding of university policy regarding the use of human subjects in behavioral studies, and in part, an attempt on the part of the developer to secure as many users as possible by minimizing program requirements. Developers were concerned that too rigorous an application of evaluation requirements would further jeopardize the already-difficult marketing situation: employers could perceive the evaluation process as overly burdensome and refuse to use the center altogether. As a result, telecenter users and their managers who were originally unaware of the need to complete surveys were less willing to participate in the evaluation in exchange for their use of the telecommuting center. Since compliance with the evaluation process relied only on the goodwill of the participants to return their surveys and travel diaries, the response rate was much less than one hundred percent.

In an effort to address this problem, the university policy regarding behavioral studies was re-examined. State law requires that participation in studies such as the RABO Project must be voluntary. After consulting with the university's Human Subjects Review Committee, the evaluation team determined that while participation in the project is voluntary, once signed up for the project, the completion of survey materials could be made mandatory in order to remain a participant. Hence, for telecenter users and their managers, references describing the completion of surveys as voluntary was removed from survey cover letters and other materials. Those references were retained for the home-based telecommuter and non-telecommuter groups.

The decision to exclude participants based on failure to complete surveys prompted a number of changes to the project evaluation. Perhaps most importantly, the funding of telecenter sites was linked to survey response rate. One provision of an amendment to the project contract stated that if a survey were more than 30 days delinquent, the participant would no longer be counted in the center's occupancy rate in terms of contract compliance (attendance and other evaluation data already collected were retained and analyzed for such a participant, however). The telecenter would not be able to recover the money even if the surveys were received at a later date. After the implementation of this provision, a regular procedure was set up whereby site administrators would be informed on a bimonthly basis of outstanding survey instruments.

2.2.3 Survey Administration

Originally, the site administrator at each center was to handle the distribution and collection of survey materials provided by the Evaluation Manager. The rationale behind having the center as intermediary is ease of contact for the participating employees. With the start of telecommuting, it soon proved difficult to track the distribution of survey materials by the site administrators. Much of this problem was due to the turnover among site personnel that many sites experienced (while some site administrators were temporary employees only, other site personnel left due to low pay for the work involved and/or inadequate job performance). Replacement administrators often received little, if any, training from the site developer. Much time was spent by the evaluation staff in acquainting the new site administrators with the evaluation process. Occasionally, materials and valuable data were lost due to an inexperienced administrator.

In an attempt to solve the problems with survey administration, site administrators were offered the option of distributing materials themselves or allowing the Evaluation Manager to handle the process directly. Although the latter method makes it more difficult for participants to return materials or ask questions, the surveys are much more likely to be distributed and collected efficiently. Meanwhile, a form for all sites was developed to help the Evaluation Manager keep track of new telecommuters. For the sites that continued to administer the surveys directly, the new form ensured that the evaluation team was aware that surveys were being distributed. In the cases where the Evaluation Manager handled the process (which happened at a majority of the sites), the tracking data provided the information needed to distribute survey materials.

2.3 Measuring Site Occupancy and Usage

In order to measure the use of telecommuting centers for both funding and evaluation purposes, certain criteria were developed. The analysis presented in this report distinguishes between site occupancy and site usage. The formula for the monthly site occupancy rate is the number of telecommuting occasions by registered telecommuters that lasted at least four hours divided by the product of the number of workstations and the number of work days in the month. The monthly site usage rate is similar but considers the number of all telecommuting occasions of any length in the calculation, instead (see Section 4.3.1).

Compliance with contractually-set occupancy targets was based on the occupancy rate rather than the usage rate. The criterion for the time spent at the center was arbitrarily set at four hours in an attempt to screen out occasions where the user spent most of the day at the regular workplace. If the telecommuter visits both the center and the main office on the same day, there would be no reduction in vehicle-miles traveled (VMT) for that telecommuting occasion. Nevertheless, it is important to see how usage of the center evolves naturally. If the telecenter is most often utilized on a part-time basis, either on the way to or from work locations or as an occasional supplement to home-based telecommuting, that knowledge would affect how the center is operated, priced, and marketed. Accordingly, both site usage and occupancy rates are analyzed in the evaluation presented in Chapter 4.

However, in one special case, telecommuting occasions of less than four hours were included in the site occupancy rate. Some telecenter users with field jobs found it more convenient to use either of two centers in the area depending on which was closer to the field location. In some cases, the total telecommuting time at both centers may have exceeded four hours on a given day. More importantly, these participants were working at the telecenter instead of returning to the regular workplace to complete paperwork. Therefore, they reduced their daily VMT and for this reason are included in the occupancy rate.

In fact, the centers were utilized in many ways that fell outside the strict definition of telecommuting, some of which had potentially important travel-reduction benefits. Uses that were not formally evaluated included drop-in use by non-participants and leasing of space to large companies. Although some sites excluded those who would have used the telecenter as a primary place of business for small businesses, other sites allowed these workers to work from their centers. Videoconferencing, a non-telecommuting use that was formally tracked, is discussed in Section 2.4.2.

2.3.1 Drop-in Use and Single-company Leasing

In order to bring in additional revenue, all telecenters experimented with drop-in programs. Drop-in use allows customers to walk into the center and rent a workstation for their use. These telecenter users are not participants in the RABO Project since they do not sign a MOU or complete any surveys, and therefore, they are not counted as part of the occupancy rate. Of course, the registered telecommuters have priority for use of the workstations. However, conflicts between project participants and drop-in users were rarely a problem because most centers had unoccupied workstations.

In addition to drop-in programs, some telecenters leased workstations to companies looking for supplemental office space. Since the company itself was in charge of the use of the space, it was unclear whether the workstations would be used for telecommuting or as a branch office (in fact, some of the rented center space was used as classrooms and teaching facilities). Consequently, these tenants were not required to participate in the evaluation and also not counted in the occupancy rate. Although these participants were not involved in the evaluation, some centers measured the travel by the employees of these companies and found significant savings.

2.3.2 Telecenter as a Primary Place of Business

Defining a true telecommuter for the purpose of this study proved to be difficult because many different types of employees used the centers. Additionally, telecommuting centers used different criteria for the admission of prospective telecenter users. While some telecenter users were employees at a large firm who had a main office elsewhere, others were self-employed or employees of small businesses and used the telecenter as their primary place of business. Both non-RABO and RABO sites had telecommuters who fell into the latter category of telecenter users, some of whom were included in the evaluation process. Importantly, small business employees have an advantage for the evaluation program in comparison to large company employees. Small businesses have fewer layers of authority to go through, so it is usually easier to obtain participation in the telecommuting study. The desire to fill telecommuting centers will likely lead to more participation by these types of employees. However, the use of the telecommuting center as a primary place of business has some negative effects on the evaluation process which are further discussed below.

First of all, small business workers may have trouble complying with all elements in the survey process. While some small business employees have a manager, sole proprietors or independent contractors do not, by definition. Additionally, these workers may lack any co-workers or may have only a few. Both manager and co-worker surveys are part of the evaluation process. Using small business employees reduces the size of these comparison groups.

Secondly, one of the goals of the RABO Project was to demonstrate the advantages of an alternative work location for employees who would otherwise have no choice over their work location. Sole proprietors already have the choice to work at whatever location best suits them. For small businesses that use the center as a primary place of business, the employees do not have any other alternative besides the center. Thus, in these cases, the advantages of alternative work locations cannot be evaluated.

Thirdly, the transportation benefits of those who use the telecenter as the primary place of business are indeterminate. For other employees, it is easy to see that travel is reduced if the commute trip is longer to the regular workplace than to the center (assuming that only one commute trip is made that day). But for small business workers, it is not clear what the alternative work location is. There may be no alternative location, or the alternative location may be home, in which case travel is generated, not reduced.

Finally, equity issues are also a concern for small business participation in the telecenter study. As part of the RABO Project, telecommuters are typically provided with workstations at below-market rates in order to stimulate interest in the concept. As a result, sole proprietors and small businesses who do not have an office other than the center could be unfairly subsidized in relation to their competitors who must pay full value for work space. Nevertheless, some level of experimentation is appropriate for a demonstration project, and it may be expected that a rent subsidy would not continue indefinitely for any user.

Despite the problems with those who use the center as a primary place of business, the complete exclusion of these users is not recommended. In fact, they may ensure stability for center operation and, therefore, be a key factor in the long-run success of the telecommuting center concept. A diversity of users and uses appears to be a critical element in the success of telecenters (Bagley, et al., 1994). However, in order to control for the negative effects on the evaluation process, the following recommendations are suggested for future demonstration projects of this nature. First, since these users cannot provide the full data requirements and have no demonstrable transportation benefits, the proportion of workers of this type should be limited (for example, a limit of 25% of occupied workspace-days). Second, site developers should charge the businesses using the center as a primary workplace the full-market rent in order to prevent inequitable subsidization of these businesses.

2.4 Changes to the Survey Process

Although the survey process as described in Section 1.2 generally follows the original evaluation plan, some modifications were made during the course of the RABO Project evaluation. The two most important changes are discussed in this section. Perhaps the most obvious change is the removal of focus groups from the evaluation plan (Section 2.4.1). Also, a survey was developed to measure the use of videoconference equipment that was available at some centers (Section 2.4.2). These changes and the reasons for them are further discussed below.

2.4.1 Focus Groups

Originally, focus groups were slated to be held in order to identify problems in the early stages of telecenter use. For all sites, both telecenter users and their managers would have had a separate session to discuss early experiences with the telecommuting center concept and any difficulties they might be having. Once the problems were exposed, they could be effectively addressed by the university and the site developer. Although focus groups have been used effectively in evaluating home-based telecommuting programs, there are some important differences that hindered their use in the RABO Project.

The first key difference is timing the start of telecommuting. In previous studies at a single organization, telecommuting was scheduled to begin for all participants at roughly the same time. Thus, the participants experienced the same problems together. For the RABO Project, the entry of participants into the program is staggered according to when each signs up. So, if focus groups were held too soon after a center opens, there would not be enough participants. On the other hand, waiting too long after the start of telecommuting may let problems remain unsolved, thereby damaging the telecommuting experience.

The other important change from previous telecommuting projects involves the scheduling of the focus groups. In the previous programs, managers and telecommuters usually participated in separate but consecutive sessions held at the work site during normal business hours. In the case of telecommuting centers, multiple employers are involved which makes it difficult to identify a suitable time and place. Although the telecenter may be a natural location for the employees, work schedules are likely to differ considerably: not everyone telecommutes on common days. For the managers, the somewhat remotely located center is probably not ideal, and scheduling the time of the meeting is likely to be at least as difficult as for the employees.

Due to the problems with timing and scheduling, the alternatives for exposing early problems were investigated. Although professionally-conducted focus groups are costly in terms of money and time, they provide impartial moderators and foster a certain synergy through the group discussion. Telephone interviews conducted by the university are less impartial but are more affordable and allow for asynchronous administration, such that everyone would not need to be in the same place at the same time. Non-aggressive monitoring would let minor problems be resolved at the site level and have only major problems reported to the university. The drawback here is that certain problems may go unreported without direct contact with the telecenter users. Finally, although complete site administrator monitoring would allow for personalized contact, the impartiality of the site personnel is compromised since they may be part of the problem.

After careful consideration, the final decision was to have the Evaluation Manager conduct non-aggressive monitoring by contacting project participants three months after telecommuting began in order to discover any difficulties they might be having. It was felt that since the Evaluation Manager had established contact with most of the participants and was seen as a neutral party, this method of contact would provide the most effective responses.

2.4.2 Videoconference Use

A number of the telecommuting centers provided videoconference rooms to both project participants and outside users as part of their facilities. Since the use of videoconferencing equipment has the potential to reduce the travel required to attend meetings, a survey form was developed to measure the impacts of each use. The form was designed to track the usage of the facility and includes blanks for the names of those attending, the location of other sites participating in the meeting, the reasons behind selecting videoconference technology, and the most likely alternative to the videoconference. Since the survey was developed midway through the evaluation process, results are not yet available for this interim report. The results of the videoconference usage form, however, will be included in the final report.

2.5 Summary of Procedural Issues

The RABO Project not only provides information on the practice of telecommuting center use, it also provides valuable lessons in the process of evaluating the use of telecommuting centers. The procedural issues related in this chapter dealt with contractual compliance, site usage measurement, and modification of the survey process. Since telecommuting centers are a relatively new concept, the lessons learned here will help later evaluation programs be more effective.

Changes to the contracts with site developers were necessitated by problems with data collection. Some contracts did not directly tie the university to the administrators of the telecenter which resulted in poor communication and inadequate survey response rates. At all centers, survey response rates that were lower than desired led to modifications of the center funding policies. Telecommuters were only counted in funding invoices if they had completed the required surveys. This new policy also emphasized that, while participation itself was voluntary, survey completion was a mandatory element of participation. Additional methods for improving survey collection included shifting the duties of survey distribution and collection from the site administrator to the Evaluation Manager for those centers who did not want to handle these activities.

A strict definition of telecenter occupancy was developed to ensure that the measured rate of telecommuting reflected the goals of the study. The monthly site occupancy rate calculated to assess compliance with contractual targets includes only the telecommuting occasions by project participants that lasted at least four hours. However, there were other meaningful (in terms of travel reduction) uses of the center that fell outside this narrow definition. As a result, uses by telecommuters for any length of time were documented and evaluated. Further, most sites set up at least some of their workstations for use by drop-in customers, and others specifically leased work space to particular companies (neither of those types of center users participated in the evaluation). Some participants who use the center as their primary place of business were included as project participants even though their situation is quite different from the typical telecommuter from a large organization. These participants may not have managers or co-workers to participate in surveys, may not have travel reduction benefits, and during the demonstration period, may be unfairly subsidized in relation to other similar businesses in terms of office space costs. It is recommended that a screening process be used to give highest priority to those who are truly telecommuting while still allowing other uses to continue at the site since a diversity of clients is a key to long-term operation.

Other issues involved in the evaluation process included changes to the survey procedure. Originally, focus groups were to be used to identify any initial problems and concerns of both the telecommuters and their managers with the use of telecommuting centers. However, problems with timing and scheduling caused the focus groups to be dropped in favor of individual telephone interviews. Additionally, the availability of a videoconferencing room at some telecommuting centers led to the development of a usage log for these facilities. The information collected from the videoconference logs will be summarized for the final report.

Previous
Next
Table of Contents

Return to the Abstracts of Non-Refereed Publications page.
Return to the Telecommunications and Travel Main page.
Return to the Teleworking Reports page.