Goals

1. Review previous testing protocols and existing regulations for usability testing across different national documentations (formative, summative, usability, medical products within UI)
2. Create a new testing procedure and documentation outlining each stage of testing required for a passable, usable UI product
3. Conduct user testing interviews with customers, to evaluate AI-automated products for CT and MR scan reading
4. Build supportive documentation and procedures for evaluating future products developed by the company

‍

Roles& Responsibilities

Stakeholders

Throughout this project, the key stakeholders included Product Manages (for the CT and MR products), Quality Assurance teams, AI & ML developers and engineers, Regulatory Affairs and Quality Management Systems. Each stakeholder was key throughout the development of the test plan, gathering insights and metrics from previous testing, and in discussing the future of UX design and/or research within the company.

‍

Collaborators

Because there were no dedicated UX resources in the company, I was the sole UX researcher/expert in charge of creating and managing usability practices and procedures. I was responsible for bridging business needs and regulatory standards with a user-centric approach to conducting and executing UX practices.

Key collaborators on this project included:

• PM's for each product (CT and MR) who walked me through user flows, previous usability tests, and new features to test within each product/module for the 2024 release
• Marketing and Clinical Application Specialists who gave me training and additional resources for each product, and who helped in recruitment of clinicians and radiologists for testing
• RA & QMS who provided me with previous SOP's and regulatory reports (where applicable) to build from, when establishing new usability and UX practices at the company

‍

Process

This project began without having  UX resources or processes established for the company previously. A main goal was to define usability practices that could be used repeatedly for future products in addition to conducting usability testing on new product releases for 2024. 

‍

Contextual Inquiry

The first step of this project was contextual inquiry, where I gathered any previous usability tests and documentation that was done on cvi42 products. I was able to base my understanding of what had been done on this documentation, and use it as a guide to build a more robust test protocol and supplementary documentation around usability testing.

Another goal of this stage of research was to review the ISO and IEC standards and regulations for Canada and the US regarding medical products and equipment with UI or AI features, formative and summative testing protocols, and other relevant reports. I was able to translate this information into:

• the test protocol and success criteria/metrics
• use specification documents
• a more robust design file emphasizing user-centric principles
• additional documentation to guide future research best practices for this industry

‍

Standard Operating Procedure for Usability Testing

Following a review of the standards and previous usability reports, I was tasked with creating a standard operating procedure (SOP) for usability testing on a medical digital product for the company. This document defined the steps required throughout a product's development cycle to make a usable product that would pass audits and formal reviews. This document included sections like:

• defining product attributes and features
• defining updates or changes to the product and/or module
• identifying use specifications and user flows
• defining what methods can be used for formative and/or summative usability testing based on ISO and IEC standards
• template for creating the usability test protocol and script
• success/fail test criteria and formulas
• defining usability heuristics and user satisfaction scores during testing
• and more.

Once this document was finalized, it was shared with respective stakeholders and used as a guide for the testing and reporting of the 2024 product release process.

‍

Supportive documentation and processes for UX

Additional documentation was developed to support the design file and SOP, usability testing methodologies, and to promote the value of UX and UX strategy within the company.

Documentation included use specifications guide and checklist, heuristic evaluation checklist, workflow processes for major tasks within each module, and success/fail criteria for usability testing. The main goals of these documents was to provide a repeatable and intuitive process for PMs and ML teams to use when developing new user flows or features to existing modules or new product releases.

‍

Usability Testing

Usability testing was done on version6.0 of cvi42 and its newest product launch. This consisted of interviews with cardiac specialists and physicians to evaluate different user flows and the overall usability of each product that was designed and/or updated for the 2024 release.

Participants were guided through a series of tasks in the MR and CT modules, with the purposes of identifying any user or UI failures, challenges or outlying behaviours of the application. User testing and task completion was recorded as a pass or fail according to a number of metrics including:

• task completion (success or not)
• User Satisfaction Scores
• deviations from the user flows identified by PMs and ML teams
• metrics identified in ISO and IEC standards for medical products, UI, and formative and summative evaluations

User insights and test scores were synthesized and organized into recommendations that required immediate fixing and/or could be added to future product roadmaps. The report and findings were shared with PMs, engineers, and the ML team.

A formal report was send for regulation approval before the products reached the market for 2024.

‍

Deliverables

There were multiple share-outs throughout this project with VP’s, project managers, Regulation, Machine Learning, and Quality Assurance teams. These share-outs included updates on the test plan, introducing new regulatory and standards documents relevant to usability testing and UI, updates to SOPs, introduction of new methods and processes (heuristic analyses, use specification reporting, etc.).

A workshop was also conducted with key members of management to discuss UX strategy and approaches to roadmapping for a more user-centric approach and to plan for UX resources within the company. This was done with the CEO, VPs, directors, and Product Management.

‍

Outcomes

A final set of documents were created and stored for future use by PM's and/or any future UX teams

• Usability test plan template with updated metrics and methodologies
• SOP to outline usability procedures within the company
• Checklists and templates for user-centric practices (e.g., heuristic evaluation)
• Training deck and recording for employees to reference in how to conduct and evaluate usability testing

Any errors or challenges identified throughout the user testing was given to the ML and development teams to update before product launch.  Both CT and MR products were updated for enhanced usability prior to the full 2024 release.

Procedure and usability process documentation will continue to be used for future feature updates and/or design launches. 

UX resources and strategy will be roadmapped into future budgets for a more user-centric approach to the company’s processes.

‍

Learnings & Takeaways

The nature of this project was very fast-paced with short timelines. Because I was the first and only UX researcher to come to the company, there was a large learning curve and resource gap. This meant that I had to both learn current processes and update/re-imagine UX processes within the company before conducting any usability testing. Creating new processes and procedures gave me insight into how I can lead a practice and demonstrate the value of UX throughout my career -- and if I find myself in a similar situation with limited UX resources, I am more equipped with best practices in developing similar tools and resources for a company.

‍

Due to the limited time of this contract and slow lead times for recruitment, I was unable to conduct ALL of the clinician usability testing. In order to plan for and mitigate this issue, I created training documents and a recording for PMs who would be responsible for completing the user testing and report. The training walked them through the entirety of the UX process I mapped out:

• how to conduct a usability test
• what to look for during participant feedback sessions
• how to analyze and synthesize the data quantitatively and qualitatively, according to the metrics defined in the SOP
• how to capture key insights or recommendations for product development.

This challenge provided me with the skills of how to train teams and provide documentation that will lead to project success.