Usability Testing · Enterprise UX · Mixed Methods · 2024

The portal worked. Every single time.

So why did nobody trust it?

What

End-to-end usability testing on STC Hub

Where

Saudi Telecom's internal HR portal - One of the largest telecoms in the Middle East & Africa

How

5 real employees · 3 high-stakes tasks · live system · Zoom

The contradiction

100% completion rate · −40 NPS · the gap was the entire story

RoleUX Researcher Team2 researchers MethodsThink-Aloud · SUS · NPS · ASQ Sessions5 participants · Zoom Timeline6 weeks

Task Completion Rate

100%

Every task completed — but completion isn't the whole story.

Zero failures across 5 users and 3 tasks on a live portal.

but

Net Promoter Score

−40

60% of users would actively warn others away.

20% passives · 20% promoters · 60% detractors.

The Context

STC Hub — built by departments, not designers.

Every service was built for internal operations — not for the employee using it. We ran end-to-end usability testing to find out exactly where it broke down.

The company

Saudi Telecom Company

One of the largest telecoms in the Middle East & Africa. STC Hub is used by every employee, every day.

The portal

Internal HR portal

Vacation requests, KPI submissions, internal recruitment — high-stakes tasks with no room for confusion.

The problem

56 violations before testing even started

Heuristic eval + cognitive walkthrough flagged a 25.6% failure rate. The sessions confirmed every prediction.

Sessions

5 users · 1hr each

Tasks tested

3 HR workflows

Method

Think-aloud · live portal

Outcome

15+ prioritized fixes

Before Testing

We already knew something was wrong.

Two evaluations ran before users sat down. Both pointed to the same gap — a system built without the user in mind.

heuristic violations across Nielsen's 10 principles

25.6%

cognitive walkthrough failure rate before any user touched it

①

Heuristic Evaluation

Assessed the interface against Nielsen's 10 usability principles. Found violations across all 10 — most severely in Aesthetic & Minimalist Design.

②

Cognitive Walkthrough

Applied Lewis & Rieman's 4 evaluation questions to each task flow. 1 in 4 steps was predicted to fail before a real user ever sat down.

③

3 Tasks Selected

Vacation request, KPI submission, internal recruitment — prioritized based on frequency, stakes, and walkthrough failure rate.

Violation breakdown — all 10 heuristics

Nielsen's 10 Heuristics 56 total violations

Aesthetic & Minimalist Design 17

Error Prevention 8

Visibility of System Status 5

Recognition rather than Recall 5

User Control and Freedom 4

Help and Documentation 4

Flexibility and Efficiency of Use 3

Match — System & Real World 2

Consistency and Standards 2

Help Users Recover from Errors 1

Submit Vacation Request

Ideal: 1 min · 19 clicks

Add KPIs for the Year

Ideal: 4 min · 23 clicks

Submit Internal Recruitment

Ideal: 3 min · 35 clicks

The Core Finding

The system worked. The experience didn't. And those are not the same thing.

This tension — between technical success and user dissatisfaction — was the defining insight of the entire study.

100%

Task completion rate across all 5 participants and all 3 tasks

−40

Net Promoter Score — 60% detractors, 20% passives, 20% promoters

SUS Score — just below the 68 "acceptable" industry threshold

5.0

Average ASQ on Task 2 — not at all satisfied (scale of 1–7)

"Users could technically do their jobs — but every interaction cost them time, trust, and confidence."

Task 2 (KPI) took 5.5 min on average vs ideal 4 min. Task 3 (Recruitment) averaged 3.8 min vs ideal 3. Not catastrophic — death by a thousand cuts. A system that makes employees feel incompetent doing routine tasks is a system that has already failed, even when the logs show 100% completion.

How We Researched

5 real employees. 3 tasks. One think-aloud protocol.

All participants were active STC employees from the procurement department — identified through usage data as the highest-frequency users of these specific services.

→

🔍

Heuristic Evaluation

56 violations found across all 10 Nielsen principles — before any users were involved.

→

🗺️

Cognitive Walkthrough

Lewis & Rieman's 4 questions applied to each task. 25.6% failure rate identified.

→

🎯

Participant Recruitment

Highest-volume HR service users identified via usage data. 5 procurement employees in Saudi Arabia.

→

🎤

Think-Aloud Sessions

5 × 1-hour Zoom sessions. Participants verbalized every thought completing 3 live tasks.

📊

ASQ · SUS · NPS

Post-task ASQ after each scenario. SUS and NPS collected once all tasks were complete.

Participants

User 01 · Age 41 · Male

Procurement Expert

User 02 · Age 30 · Male

Vendor Management

User 03 · Age 35 · Female

Procurement Support

User 04 · Age 25 · Female

Lead Procurement Analyst

User 05 · Age 34 · Male

Vendor Support

Three Tasks Tested

Submit Vacation Request

Submit annual leave while delegating a colleague to cover during the absence.

1 min

Ideal time

Add KPIs for the Year

Navigate to the KPI portal and add annual performance objectives as a new employee.

4 min

Ideal time

Submit Internal Recruitment

Find and apply for a managerial position in a different department.

3 min

Ideal time

Quantitative Results

What the data said

Three measurement instruments. One consistent signal — tolerable for familiar tasks, deeply frustrating for everything else.

SUS Score

Above average — barely

Industry "acceptable" threshold is 68. STC Hub sits just below — usable but not comfortable.

Net Promoter Score

−40

Critical dissatisfaction

60% detractors · 20% passives · 20% promoters. Systemic trust erosion, not isolated friction.

Task Completion

100%

Every task, every user

But 40% deviated from optimal path on Tasks 2 & 3 — completion came at a real cost.

ASQ by Task — 1 = Satisfied · 7 = Not Satisfied

Task 1 — Vacation Request

2.1

Nearly satisfied. Frequently used, well understood.

Task 2 — KPI Submission

5.0

Not at all satisfied. Worst task by every metric.

Task 3 — Internal Recruitment

4.2

Not satisfied. Unclear workflow, complex form fields.

Completion Time vs Ideal

Participant	Task 1 (ideal: 1 min)	Task 2 (ideal: 4 min)	Task 3 (ideal: 3 min)
User 1	2:15	5:30	4:12
User 2	2:34	5:57	4:09
User 3	1:44	4:50	3:48
User 4	3:10	6:12	4:53
User 5	2:29	6:22	3:20

Think-aloud protocol adds verbalization time. Any completion exceeding ideal + 1:50 flagged as an efficiency failure.

What We Found

Four categories of failure. All pointing to the same root cause.

Every problem we identified across five sessions fell into one of four categories. None were random — all were symptoms of the same underlying issue: services built by departments for departments, not for the people who use them every day.

We used Jakob Nielsen's 0–4 severity scale to rate each issue, from cosmetic problems to usability catastrophes requiring immediate attention.

Nielsen Severity Scale

Severity 4 — Usability Catastrophe

Imperative to fix before release

Severity 3 — Major Issue

High priority fix

Severity 2 — Minor Issue

Low priority fix

Severity 1 — Cosmetic

Fix if time allows

Workflow Opacity

No visibility into what happens after submission. No approval chain, SLA, or next steps.

Vacation request shows no workflow — users don't know if their manager will be notified

→ Add expandable workflow preview before submission

Delegated employee has no approve/reject option — only notified after the fact

→ Add approve/reject before request reaches management

KPI workflow changes annually with no visible current-year flow

→ Auto-updated flow display synced to HR updates

Recruitment confidentiality policy never communicated — causes anxiety, deters use

→ Surface HR policy link directly in the service

Navigation & Discoverability

40% of users went off happy path on Tasks 2 & 3. Services buried or split across platforms.

KPI submission lives on a separate platform (Add'e) — users lose context immediately

→ Consolidate into a single portal interface

Internal recruitment redirects to an external site — users feel lost and disoriented

→ Embed or create clear wayfinding back

Users took up to 35 sec to locate the KPI service vs 25-sec ideal — no navigation aids

→ Search and shortcuts on the main dashboard

Automation & Integration Gaps

The system repeatedly asked users to re-enter information it already had access to.

After selecting vacation dates, users had to re-enter the same dates for delegation

→ Auto-populate delegation dates from vacation selection

KPI question sets must be re-selected for every KPI added, even when identical

→ Auto-carry questions with option to override

Uploading a CV doesn't populate any fields — users re-enter everything in the HR system

→ Parse CV and pull known data from HR database

User Knowledge Gaps

Users knew what to do. They didn't know how the system expected them to do it.

Tasks 2 and 3 required researcher assistance — no in-service guidance exists anywhere

→ Contextual service manuals embedded within each workflow

KPI fields like "target weight" and "target type" confused every participant

→ Inline field descriptions with examples

After submitting an internal application, users receive no feedback unless called for interview

→ Application status tracker with visible timeline

What We Recommended

15+

Prioritized recommendations across four categories

Every recommendation maps directly to a finding, rated by severity. Scoped, justified, and ordered by impact.

Workflow

Severity 3

Submit Vacation · No workflow visibility

Add a workflow preview before submission

Let users see the approval chain before they commit to submitting.

Workflow

Severity 4 — Critical

Submit Vacation · Delegated employee has no agency

Give delegated employee approve/reject control

They're notified only after the fact. They need agency before request reaches management.

Workflow

Severity 3

Submit KPI · Flow changes annually, no visibility

Auto-updated KPI workflow display

The flow changes every year when HR updates it. Reflect the current state automatically.

Workflow

Severity 4 — Critical

Recruitment · Confidentiality not communicated

Surface HR policy directly in-service

Confidentiality is required by policy but never told to users — causing anxiety and deterring applications.

Navigation

Severity 2

Submit KPI · Lives on a separate platform

Consolidate KPI into one portal

KPI redirects to Add'e — a separate system. Consolidating removes context-switching confusion.

Navigation

Severity 2

Recruitment · Redirects to external site

Embed recruitment within the portal

Employees feel lost the moment they leave the main portal. Build it in or create clear wayfinding back.

Automation

Severity 2

Vacation · Delegation dates must be re-entered

Auto-populate delegation from vacation dates

The delegation period is almost always identical to vacation. Auto-fill with option to change.

Automation

KPI questions — confidential, cannot be shown

Severity 2

KPI · Question sets duplicated per entry

Auto-carry KPI question selection

Select once, apply to all KPIs — with option to override per entry.

Automation

Severity 3

Recruitment · CV upload doesn't auto-fill

Parse CV and pull from HR system

After uploading a CV, users re-enter everything. Profile data already exists in the HR database.

Knowledge

In-service contextual guidance

Severity 3

All Tasks · No in-service guidance available

Build service manuals for each workflow

Users needed researcher help for Tasks 2 and 3. Contextual guidance would prevent this entirely.

Knowledge

Severity 2

KPI · Field labels are unclear

Add inline field descriptions and examples

"Target weight" and "target type" confused every participant. Tooltips eliminate the guesswork.

Knowledge

Severity 3

Recruitment · No post-submission feedback

Application status tracker with timeline

After submitting, users hear nothing unless called for interview. A status tracker removes the ambiguity.

Reflection

"A 100% completion rate isn't success. It's the floor. Real UX success is whether people come back — and whether they trust the system enough to use it without anxiety."

This project taught me that usability metrics don't tell the full story. SUS was 65 — technically functional. NPS was −40 — actively damaging. Both were true at the same time. The gap between those two numbers is where the real design problem lives.

Running sessions across time zones with real employees on a live enterprise system pushed our rigor in ways that a lab study never would have. We couldn't break anything. We couldn't fabricate tasks. We had to work with what the system actually was — and that constraint forced us to be precise about what we were measuring and why.

The biggest lesson: when a system makes employees feel incompetent doing tasks they've done dozens of times, that's not a usability problem. That's a design philosophy problem. The portal was built for departments, not people — and every finding traced back to that original sin.

Academic project · HCI 460 · DePaul University · Research conducted with real STC employees under institutional consent protocols.

Next Project

MoodMode — Emotion-adaptive iOS app for ADHD users

View case study →

What we'd do differently

→Run a follow-up round of testing after implementing high-severity fixes

→Include participants from more departments beyond procurement

→Build lightweight wireframe solutions to test alongside the issues

→Present findings directly to the STC product team for implementation feedback