Project Two
IBM Red Hat Marketplace Usage Based Products: Usability Study
Red Hat Marketplace Background
The Red Hat Marketplace is a built-in partnership of Red Hat and IBM that facilitates on an open cloud marketplace, which creates a driven force to access software in a container-based environment among public clouds. It is a straightforward approach to purchase and direct enterprise software with automated deployment to any cloud.
Meeting with Stakeholders
After being contacted by one of the lead designers and OMs, Red Hat Marketplace had a couple of situations that needed to be addressed. Here are the objectives and problems that we agreed upon together in order to solve a case.
We wanted to:
Investigate if users understand the clarity and comprehension of what and how much they are being charged with when purchasing from RHM
Review how users understand the functionality of "current balances" and what it means to them
Observe what users want and expect to see from the drop-down menu of one of the screens
Allow users to differentiate and set apart products in the list-view from the "pending charges"
Distinguish user's preference for receiving their billing terms and exporting data
What is Usage-Based Products (UBP)?
Most of the users that we have recruited have experience purchasing UBP products. UBP is commonly purchased around most companies, which charges users based on usage of that service or product on a monthly basis. If users stop using the product, companies will still add additional charges or fees unless cancelled ahead of time.
Advantages: users can get discounts, minimum fees, or have control of their pricing.
UBP allow users to understand what parts of the product or service they spend the most time on and which they use less of
Disadvantages: users can get a continuation of charges even if a product is not or has stopped being used. Users also have a hard to monitor or track the number of usage their company have been using.
Research Plan
Methodology
With designs already in place, we took the hi-fidelity prototypes and walked users through a 1-on-1 key informant interview (usability testing), while users were asked to think aloud and express any comments, questions, or concerns.
Participation Details
We recruited 11 customers and ISVs with experience purchasing or selling software from the marketplace. The users were recruited through Respondent.io and UserInterview.com using a screener I created.
Their backgrounds consist of:
Full-time employees
Within 1-6 months (of selling or purchasing from the marketplace)
Company size: +250
Titles/Roles: Information Security Specialist | Manager | Analyst/Programmer | Chief Technology Officer | Software Test Engineer | Director of Engineer | Developer | Computer Engineer
User Testing Results
Payment and Charges Screen
Users enjoyed the user interface and concept
Most users thought the screen was clean and straightforward when coming across the payment method.
Users also loved the idea of monthly payments.
Misinterpretations in the payment
Some of the users were unable to identify if usage was billed during the date of purchase or after using it.
One of those users believed you had to commit to a certain number of hours before purchasing. This uncertainty was linked to the heading "Applied rate,” under Hourly usage from the sidebar.
One user thought it was an additional fee to the number of users and hourly usage.
Another user thought it didn't separate the user and hourly cost very well
Payment and Charges Screen Continue
Applied rate causes uncertainty
A few users suggested having a better explanation in regards to the wording of applied rate and hourly usage.
Another user suggested taking the applied rate out if it's not going to be charged immediately from the date of purchase.
Takeover from the payment screen
Majority of users did understand the billing payment and how hourly usage will be processed each month and not during the time of purchase.
Users did appreciate the organization and structure of the UI.
However, the "applied rate" heading puzzled a few of the users.
Recommendations for Stakeholders:
Recommend having two steps for a summary and final checkout view page
Consider Incorporating an icon or hover to show a thorough explanation of what applied rate, hourly rate, and pricing table is
Examine other features. For example, some users suggested using an estimation field. What this does is allow users to put in numbers of users, hours, etc and play around to give them an estimation charge, like IBM automation. Google has a similar structure, but they call it Provision.
User's feedback
"I would not want that [Applied rate] there. If you've done this before, you're paying a license or the user, which is the $10 times 100 right and then you're based upon your metered usage, and that's this [applied rate] as well. And I understand that RedHat is trying to show those two different components here, but I don't think it's coming across that easy to understand.”
"I think applied rate is a headline and the pricing table is another headline. What it means is that it basically divides the cart into two."
Pending Charges
Understanding current balances
Most users loved the design and layout, especially the pie chart that separated each product on the monthly charge overview.
After having users analyze the screen for 1 minute, each were asked what current balance represents. An amalgamation of user’s reviews consisted of expected charges, software purchased and currently in use, and a blend of both answers.
Adjusting monthly view
Next, users were asked if they could change or add anything else to the drop-down menu at the top right.
Most of the users liked the monthly view. Other suggestions were adding quarterly, weekly, bi-annually, and annually views.
Pending Charges Continuation
Product Differentiation
Users were able to identify and differentiate products apart. For example, users spotted the "type" for each product to identify if it is usage, fee, or subscription-based.
The cost for each item was well-received and quickly identified. Users could determine the unit price, charge, frequency, type, and total cost of that product.
Recommendation:
Allow users to see a selection for weekly, bi-annually, annually, and quarterly views
Invoice List
Exporting and Downloading Data
In both instances, users would like to be able to export and download their data onto a PDF or CSV file. Users would like to have the option to select which parts of the data to export, as well.
Additionally, one user was unsure of the differentiation between the “Export data” and the download button.
Invoices
One user suggested changing the Invoice heading to “past or previous invoices.” Also, another user recommended turning the heading “Timeframe” to months.
Invoices
Invoice content
Overall, users thought the invoice was straightforward and and nicely done. The overall display was very similar to what a lot of invoices most users have seen.
Subscription and One-Time Charges
The only downfall to the invoice screen was the headings “Subscription and One-Time Charges.” One user thought it didn’t necessarily match the description because there were a blend of fees, subscription, and usage-based charges.
Notifications
Users would like to get notifications of payment and other alerts through both email and text.
Recommendations
Evaluate "Subscription and One-Time Charges" and change it to “Charges” or “Detailed Charges”
What have I learned from this project?
One of my biggest fears was obtaining users that didn't relate to our project or had knowledge about UBP. There were many users on respondent.io and UserInterview that can be imposters. Luckily, my screener that I created helped me identify who I needed and who would fit right into this project. Because usage-based products is so specific and many people don't understand the term, I thought this would be a huge problem finding users with this knowledge and experience.
A problem that I ran upon this experiment was scheduling users one after the other, giving me absolutely no rest in between. This affected my mood and gave me quite a bit of exhaustion. I ended up not eating, drinking, or using the bathroom until I finished work. For future purposes, I'll make sure to have at least 30 minutes of rest in between each session.
There were a few problems when conducting this experiment. One, I recruited at least three or four users that never came across usage-based products or have never heard of the term. However, their participation and feedback on the interviews and prototype provided valuable responses and reactions. This allowed us to also concentrate on first timers or novices exposed to usage-based products for the very first time.
Another issue that I had was not having a note-taker during some of the sessions. Using otter.io and recording my audio and video, I was able to go back and synthesize on each session, especially the ones where I wasn't able to grab a note-taker.
Overall, I enjoyed researching for this amazing team and playing a role in helping the Red Hat Marketplace become a true competitor against the likes of
AWS and Azure.
What People Said
“I really appreciate that you're putting time in for this. Sometimes, it's difficult to understand some of the invoices we get out when we're doing these offers, and there are just so many different tiers and little bit of surprises, so I'm glad that someone like your team are working on this. I really appreciate it!"
— Participant via respondent.io
"I want to thank you for spending time and running with these reviews with users. It's really valuable to get feedback, very helpful to make sure that we're making the right decisions, and I think it's clear from just the feedback that you've got that. At least my takeaway is that we've made a lot of initial good first decisions, but there's a lot we can do to refine the design, copy, labels, titles, and things. I feel positive about the future of this. So I'm looking to dig more into the recommendations that you have, and see what kind of changes, we can do in the short term and things we need to do a little bit more long term, follow up on other research opportunities."
— Aaron Sickler, Design Lead | IBM Cloud
"This is nice. It's a validation of the work we did that we're not far off. You know something like this where IBM has never built anything like this before where we've got usage based charges. It's new. It validates the work that we did and there's some kind of minor tweaks that's involved, but it's good to hear that we weren't off base with anything"