Google Cloud Monitoring

Research Questions

The Google Cloud Monitoring (GCM) service (formerly Stackdriver, acquired by Google in 2014) provides dashboards and alerts as well as other services for users to monitor different cloud-based resources such as databases, servers and virtual machines. When we started this Google sponsored project, our main questions were:

"Is there any usability issue with GCM? If so, what are they?"

"How do we improve the user experience for GCM? "

* Due to Non-Disclosure Agreement, project details will not be discussed.

Assumption

As a Google sponsored course project team, we received background information from a user experience researcher at Google. Our sponsor shared some of her previous research findings, including the main user type. We held the user type assumption throughout this project, which determined our participant recruitment direction.

Based on the user type assumption, we also assumed our target users have the needs to monitor their cloud resources, and the ability to understand cloud-related terminologies. These assumptions turned out to be partially incorrect as the research proceeded.

Research Process

User Profile -- Persona

To verify our user type assumption, I designed a survey and collected data from 13 people through personal connection, and a weekend hackathon. We then created two user profiles based on the survey data as well as job descriptions I found on Linkedin.com for typical job titles our target users have. Our assumption was mostly verified, except that we realized our target users don't have to hold certain job title. It is more about their job function rather than their titles. The user profiles included user demographics, skill set, motivations, goals, pain points and a scenario.

Creating user profiles helped us understand who are using or likely to use the GCM, and also guided the usability testing protocol and design suggestions in later phases.

Define Evaluation Tasks

After we understood the users and their needs, we start to define the testing tasks. The competitive products with Google Cloud Monitoring are Amazon Web Service and Microsoft Azure, we then created the testing tasks based on using process.

Based on face to face interview and our survey, we divided platform's functions as two main parts: Alter and Dashboard

User Testing

Heuristic Evaluation

A heuristic evaluation is a usability inspection method for computer software that helps to identify usability problems in the user interface (UI) design.

We recruited 4 participants who passed our screener, which was provided by our sponsor. Participants were asked to use GCM to monitor the health and performance of a sample web application that was already set up for them. We held 1-hour test sessions using Morae in the LUTE lab at University of Washington including pre-task interview, 8 tasks, and post-task interview. The tasks were designed and prioritized through a task analysis session so that they could cover the functionalities we would like to test. We recorded all the sessions.

We use Morae software to record users activities when they did the 8 tasks. In the user testing, I designed the task protocols, interviewed users and recorded their action log and feeling. 

We used Google Sheet to organize the qualitative data and then asked users to give their scores for each tasks. Then based on the user descriptions and scores, I analyzed the usability of functions in each task.

Findings

From user interview, we found out that users consider Google Cloud Monitoring very useful. They love the intuitive, clean design UI and powerful functions. But they also pointed out some usability issues when they did the testing.

1. Relation between Altering and Dashboard

"Dashboard drop-down contains "Create..." and in other drop-downs are options to "Create New Alerting Policy", "Create" (without ellipses), ...., I am confused." -- Participant X

Different from Amazon AWS and Microsoft Azure, Google Cloud Monitoring designed Altering as an individual section. Thus, some experienced-developers were very confused about this part.

2. Alerting Policy

Altering Policy is the most confused part for most of participants (3 out of 4). The problems that make them confused are related to design and implementation.

Finals

Report

After we finished the research, we wrote up an usability report to present to our stakeholder, Google Cloud Monitoring Research team. The report contains qualitative data and videos from the user testing with our findings and suggestions. For more GCM introduction, you can watch Google I/O 2016 demo in Youtube! :-)

Google I/O 2016 Google Cloud Monitoring Showcase Video