Indiana University is widely renowned as a leader in creating and providing cyberinfrastructure, both within and beyond the virtual walls of its academic community. Cyberinfrastructure generally refers to computing and data storage systems, advanced instruments and data repositories, visualization environments, and people, as well as the networks that connect them to make research and innovation possible. As part of the IU Pervasive Technology Institute (PTI), Cyberinfrastructure Assessment and Evaluation (CAE) plays the integral role of assessing how well PTI carries out its mission and serves its local and national constituencies.
Following clear goals and key performance indicators, CAE collects data using various tools, scaling large and small, administering interviews, focus groups, microsurveys, surveys, and longitudinal studies. CAE tailors assessments to gather the data each organization needs, and to comply with various university policies and federal regulations (FERPA, HIPAA, etc.) around data privacy and other user agreements. The group then provides analysis of the data—and offers recommendations based on that analysis—to assist organizations in delivering outstanding value to their various stakeholders and improving their growth.
Within the Indiana University community, CAE is responsible for deploying and analyzing results for the UITS Biennial User Survey. Since 1991, the study has surveyed a random sample comprising 25% of all students, faculty, and staff on all of IU’s campuses. The study is used to determine costing, ABC metrics, setting rates, and usage, and its results are publicly available. As part of IU PTI, CAE also conducts user surveys for its various centers and labs, like workshop satisfaction surveys and most recently overseeing the data collection and analysis for the eLearning Research and Practice Lab’s full-census study of the Spring 2020 shift to remote learning.
CAE also plays an important role in scholarly studies of Return on Investment (ROI) for cyberinfrastructure; within this nascent conversation, the work done through IU PTI provides some of the most robust analyses available to date. The team’s work on ROI occurs at multiple levels and involves evaluating the NSF’s funding of the XSEDE (eXtreme Science and Engineering Discovery Environment) project, and at the local level, measuring the value of IU’s investment in its own cyberinfrastructure. The XSEDE Return on Investment study started as a way to reassure the NFS that their investment in XSEDE was sound. It grew a little bit each year and Indiana University (IU) received formal XSEDE funding in 2018 in recognition of its expertise in evaluating high-performance computing (HPC) projects and initiatives. This type of ROI evaluation is essential, given the state of national budgets and those of research institutions more generally, as facilities must be supported both effectively and sustainably.
As organizations look to make data-driven decisions, we help them do that by envisioning the studies that will get them the data they need, and then analyzing that data.
Julie Wernert, Manager, Cyberinfrastructure Assessment and Evaluation
The XSEDE project has also administered an internal climate study over the last six years, asking its staff about topics ranging from communication tools, to decision making, to equity and inclusion. Based on the findings of these studies, XSEDE has made a series of positive changes, including creating more transparency around decision-making strategies. It also brought to light the tendency for employees to feel disconnected within largely virtual organizations, revealing the importance of connection for employees and their local supervisors, and with those whom they saw primarily via virtual meetings. Another important assessment, the XSEDE User Survey, is set to deploy for the tenth time. One of the project’s lasting legacies is its use of data to guide the project, both operationally and strategically; this has now become expected of other NSF-funded projects.
Program evaluation—backed by data—has become the norm for grand-scale NSF projects, stemming from the belief that a data-driven organization is a growing organization. In the current climate, when university budgets and national funding for the sciences are under constant scrutiny, evaluation is about far more than checking a box. The data is used well to help organizations make necessary changes.
About the IU Pervasive Technology Institute
The Pervasive Technology Institute at Indiana University is a collaborative organization designed to marshal IU’s computational experts and resources quickly in response to societal, research and educational needs. In partnership with UITS, the Pervasive Technology Institute also led the original Jetstream award. The institute was established in 1999 by a generous grant from the Lilly Endowment and has continued to lead productive uses and applications of research technologies for over 20 years.
