If you’re reading this, you’re likely doing so on some kind of computer (be it a smart phone you keep in your pocket, the large kind you keep on your desk, or something in between). To get to this article, you may have gone into an app, or maybe you typed something into a search. But, have you ever used a command line interface? In this age of intuitive, graphical interfaces, many people have not, and find the command line clunky and intimidating. For the IU-affiliated researchers who want to make use of the university’s high performance computing (HPC) resources, mastering the command line is an essential skill…or it used to be. Complementing the popular Research Desktop (RED) system, the Scalable Compute Archive (SCA) team works to make HPC resources, which generally require a special skill set, more broadly accessible to more people.
Arvind Gopu, Manager of SCA in Research Technologies
Let’s say you have a lot of data to process, and nothing but a laptop with a fairly slow processor and limited memory and storage capacity. IU’s HPC resources would likely come in handy here, but without assistance, you may or may not be able to figure out how to get an account and log in, and you likely wouldn’t know which resource would work best for you. Enter HPC everywhere, a quick-access web portal for all things HPC-related at IU. Developed by the SCA team, HPC everywhere has been in beta since August of 2018, and has helped around 700 new users discover, access, and monitor IU’s HPC systems.
When you log into HPC everywhere, it shows how many HPC jobs you have submitted, how many are running, and your utilization of high-performance storage resources. If you’re just getting started, it offers the Pathfinder app that will help you determine the optimal compute cluster (between Big Red II, Carbonate, Research Desktop, or Karst) for your research. The team recognized the necessity of a tool like Pathfinder after finding that many users hear about IU’s HPC resources, and then search for and discover one of them (typically called a “cluster”), figure out how to use it…and then months down the road, discover that another cluster would have been a better fit. The vast majority of those who take the Pathfinder quiz are new to using supercomputers, and benefit from the quiz’s recommendation.
If you already have an idea of what you’re trying to do, you might jump ahead to the many other apps HPC everywhere offers: HPC@IU to view information about and the status of the HPC clusters; HPC Applications (Software Browser) to help you figure out if the software you need is already installed on any of the clusters; Waittime Visualizer to show how your job might be scheduled; MyHPC to monitor your jobs; and finally Job Script Generator to assist you in putting together the arcane “scripts” that make the cluster do what you want it to do. The portal also offers snapshots of the resources, links to creating HPC accounts and requesting software installations, a glimpse of UITS status, and access to your research data through Globus Online.
Mike Young, SCA team member
The Job Script Generator comes in particularly handy for many users, given how easy it is to mess up a job. The HPC resources will accept and even queue jobs that contain errors in their scripts; but, some such jobs won’t run. Ever. Users often don’t understand that their jobs aren’t going to run unless they look at when it’s scheduled to begin - it will say “never.” The script generator helps users ensure that the jobs they submit are free of common syntax and resource request errors and will run as planned.
HPC everywhere, through these various apps, takes information that is typically only accessible to an advanced Unix user, and puts it all in one place within an easy-to-use portal. It takes the burden off researchers & scientists, while supporting internal Research Technologies teams as well as the larger IU research community.
Robert Henschel, Director of Science Community Tools
The SCA team plans to continue developing HPC everywhere. Future iterations will integrate job submission and management features, and merge or link the three related ‘apps’ MyHPC, Job Script Generator, and Wait Time Visualizer. The team also intends to explore further integration between Research Desktop and HPC everywhere. If you have any specific feature requests or suggestions, please contact Arvind Gopu <agopu@iu.edu>; the team is open to considering such requests as long as they are technologically feasible.
