Sensitivity and specificity tell you what a test is able to detect. Only usability tells you whether people can actually reach that result.
A diagnostic test can have flawless analytical performance and still fail in real life. The reason rarely lies in the chemistry. It lies in the gesture: the drop of blood that did not fill the capillary, the waiting time estimated in one's head, the faint line someone read as no signal at all. A usability study exists to find these points before the product reaches the people who will depend on it.
Performance that exists only on paper
Usability studies, also called human factors studies, place representative users in front of the complete task, from opening the package to interpreting the result, with only the instructions for use supplied by the manufacturer. Nobody asks whether the user liked the product. What is observed is where they hesitate, what they skip, what they misunderstand and whether that error could lead to harm.
The ABNT NBR IEC 62366-1 standard organizes this discipline for medical devices using a risk management logic: identify the critical tasks, anticipate foreseeable use errors and demonstrate, with data, that the product's user interface keeps them at an acceptable level.
What 1,400 people showed about a self-test
A study conducted in Johannesburg and published in PLOS ONE evaluated seven HIV self-tests with 1,400 adults who had never performed the procedure. With no training, only the instructions for use, each participant carried out the steps under observation. The overall result was good, with mean usability of 92.8% and reported confidence close to 96%. It is the details that make the data useful. Failures concentrated on sample collection and transfer, such as an insufficient blood drop, a poorly filled capillary and inadequate swab technique in oral fluid tests.
Reading the result produced the most uncomfortable finding. Negatives and positives were interpreted correctly by more than 96% of participants, but the weak positive was read correctly by only 74.9%, and was often mistaken for a negative. In an HIV test, a positive perceived as negative is exactly the kind of error that no analytical sensitivity figure reveals.
Formative and summative: two studies, two questions
IEC 62366-1 distinguishes two types of usability evaluation. Formative evaluation takes place during development and asks what still needs to improve. Summative evaluation takes place at the end, with the product and instructions for use in their final version, and asks whether use is safe enough to go to market.
| Formative | Summative | |
|---|---|---|
| Timing | During development, in one or more rounds | At the end, with final product and instructions for use |
| Objective | Find use problems and guide corrections | Demonstrate that critical tasks are performed safely |
| Sample | Small and flexible | Representative of real users, defined in the protocol |
| Role in the dossier | Recorded in the usability engineering file | Validation evidence submitted to the regulator |
The formative study is not the one the regulatory authority reviews to approve the product. That decision rests on the summative study. Even so, skipping the formative study is the most expensive saving in the project.
A summative study is a pass/fail test: if a critical task fails, the developer must correct the product or the instructions for use and repeat the validation, with new recruitment, a new timeline and new cost. The formative study is where mistakes carry no consequence. With few participants, it reveals the ambiguous illustration, the step out of order, the term a lay user does not understand, in time to fix them before the final version.
There is also a less obvious gain. Formative rounds document why the interface and the instructions for use ended up as they did, and this traceability supports the usability engineering file that accompanies the product. Those who reach the summative study after a good formative one arrive to confirm, not to discover.
From observed error to corrected product
The strength of this kind of study lies in turning impressions into actionable evidence. When a step is found to concentrate errors, the developer can redesign the illustrations in the instructions for use, add reference images of weak positives to the instructions and the packaging, adjust the reading time or change the collection device. Doing this before registration costs a design review. Discovering the problem in the field, after launch, costs a recall, reputation and, in the worst case, a wrong clinical decision.
There is also the distinction between an error that gets in the way and an error that causes harm. Getting the waiting time wrong by two minutes may be irrelevant for one product and critical for another. It is up to the study, based on the risk analysis, to separate what is a nuisance from what compromises the safety of the result, and to focus attention where harm is possible.
When the user changes, the problem changes too
Self-tests and PCR tests illustrate two sides of the same challenge. In a self-test, the user is a lay person, of any age and education level, alone at home, with no one to correct a step. In PCR, the operator is trained but handles reagent preparation, pipetting and curve reading, and variation between routines and training levels can produce errors that do not appear in performance studies run by the developer's own experienced team. In both cases the question is the same: does the product work in the hands of the people who will actually use it?
Demand for these answers is growing because tests are leaving the central laboratory and entering routines ever further from it. The further from the bench, the more evidence of use needs to be demonstrated rather than assumed.
The standards that organize usability
Usability does not rest on a single standard. IEC 62366-1 defines the process, but it only holds together with risk management, the quality management system and labelling rules.
| Standard | What it covers |
|---|---|
| ABNT NBR IEC 62366-1 | Usability engineering process for medical devices, including formative and summative evaluations |
| IEC/TR 62366-2 | Practical guidance on applying 62366-1, with methods and examples |
| ABNT NBR ISO 14971 | Risk management, the source of use-related hazards and critical tasks |
| ISO/TR 24971 | Guidance on applying ISO 14971 |
| ABNT NBR ISO 13485 | Quality management system, with design control and product validation |
| ISO 18113 (parts 1, 4 and 5) | Information supplied by the IVD manufacturer, including labels and instructions for self-testing reagents and instruments |
| ABNT NBR ISO 15223-1 | Symbols used on labels and instructions for use |
| ISO 20916 | Good study practice for clinical performance studies of IVDs using human specimens, a reference for the ethical conduct of studies with participants |
In practice, ISO 14971 says where harm can occur, IEC 62366-1 says how to test whether the user gets there, and ISO 18113 says what the instructions for use must state to prevent the error.
What Brazilian regulation requires
Yes, Brazil requires usability evidence, and explicitly so for products intended for lay users. ANVISA's RDC No. 830/2023, in force since June 1, 2024, includes the usability report for products intended for lay users in the IVD technical dossier (art. 58, item VI, subitem I). Annex II of the resolution applies this item to risk classes II, III and IV.
The same resolution requires the language of labels and instructions for use to be compatible with the knowledge, experience and education level of the intended user. Self-tests are, as a rule, class III, which places them under the registration regime, with technical review of the dossier by ANVISA.
For professional-use IVDs, including PCR and rapid tests used by professionals, there is no dossier item requesting a usability study. What exists is the general obligation of RDC No. 848/2024, which covers the essential safety and performance requirements and applies to all medical devices and IVDs. It requires the design to allow safe and accurate use according to the instructions and to reduce the risk of user error in handling and interpreting results. In practice, the summative study is not required for registration of a professional product.
The absence of a requirement does not eliminate the risk. Instructions for use poorly suited to professionals also affect the final result: an ambiguous pipetting volume, a preparation step out of order or a poorly explained run validation criterion produce errors even in trained hands. Experienced operators tend to trust their own routine and skim the instructions, which makes clarity of the text even more decisive. That is why a usability study also makes sense for professional products, even when the regulator does not ask for it.
Together with the risk management report, which is mandatory in the dossier, this makes IEC 62366-1 the recognized path to demonstrate compliance. Neither resolution mentions the terms formative and summative. What ANVISA receives is the final usability report, which corresponds to the summative study.
How to recognize a convincing study
A few design choices determine whether the data will be taken seriously. The sample must reflect the real users, with variation in age, education, health literacy and prior experience. Critical tasks must be defined beforehand, based on risk management, and not after seeing the results. Observation must take place without help from the evaluator, because help hides the error.
The study should include interpretation of simulated results, including ambiguous ones, because that is where the decision is lost. And whatever is found must feed back into the design of the product and the instructions for use, with a new verification round when the change is relevant.
A focus on real-world use
At Base Científica, usability is an established line of work. We conduct formative and summative studies ranging from self-tests, such as HIV tests, to molecular tests, such as PCR, with lay users and trained operators, always starting from the instructions for use and the critical tasks of each product.
Our experience is not limited to ANVISA registration. We also conduct usability tests for CE marking, in the context of the European in vitro diagnostic regulation, the IVDR (Regulation (EU) 2017/746). For manufacturers targeting both markets, this makes it possible to plan a single usability program, with a protocol and report aligned with IEC 62366-1, that serves both submissions.
In every study the principle is the same: observe real people performing the complete procedure, record where use departs from what was intended and turn these observations into clear evidence of what the product demands from those who use it.
References
- Usability assessment of seven HIV self-test devices conducted with lay-users in Johannesburg, South Africa. PLOS ONE.
- ABNT NBR IEC 62366-1. Medical devices, Part 1: Application of usability engineering to medical devices.
- IEC/TR 62366-2. Medical devices, Part 2: Guidance on the application of usability engineering to medical devices.
- ABNT NBR ISO 14971. Medical devices: Application of risk management to medical devices.
- ISO 18113. In vitro diagnostic medical devices: Information supplied by the manufacturer (labelling).
- ANVISA. Resolution RDC No. 830 of December 6, 2023. Risk classification, notification, registration, labelling and instructions for use of in vitro diagnostic medical devices.
- ANVISA. Resolution RDC No. 848 of March 6, 2024. Essential safety and performance requirements for medical devices and IVDs.
- Regulation (EU) 2017/746 of the European Parliament and of the Council on in vitro diagnostic medical devices (IVDR).