Key takeaways
- A/B testing compares two or more versions of something by randomly assigning people to each and measuring differences in behavior.
- Messages, processes, service scripts, appointment reminders and program designs can all be tested outside digital products.
- When volume is low or randomization is impractical, structured pilots and staggered rollouts still provide useful evidence.
- Ethical review, clear hypotheses and behavior-based outcome measures make experiments credible and safe in sensitive settings.
What A/B testing is and why it matters
A/B testing is a controlled experiment. Two or more versions of something, such as a message, a form, a process or an offer, are presented to comparable groups of people, usually through random assignment, and the resulting behavior is compared. Because the groups are otherwise similar, differences in outcome can be attributed to the difference in version with reasonable confidence. This is the same logic that underpins clinical trials, applied to everyday business decisions.
Technology companies adopted A/B testing early because digital products make it easy to show different versions to different users and to measure what happens. The underlying principle is not specific to software, however. Any organization that makes decisions about how to communicate, sell, serve or support people is making implicit predictions about behavior. A/B testing makes those predictions explicit and checks them. Without testing, organizations tend to adopt whatever the most senior or persuasive person prefers, which is not the same as what works.
What can be tested outside the tech industry
The range of testable decisions is much wider than most organizations assume. Communications are the most obvious example: letters, emails, text reminders, brochures and call scripts can all be varied and compared. Processes can be tested too, such as the order of steps in an application, the timing of a follow-up call or the default option in an enrollment form. Service designs, including how appointments are scheduled or how a sales conversation is structured, are also candidates.
In healthcare, organizations can test reminder timing, the framing of health information or the design of a patient onboarding sequence. In financial services and lending, they can test disclosure formats, application flows or outreach scripts. Associations and nonprofits can test membership renewal messages or donor appeals. Inside organizations, leaders can test variations in training formats, meeting structures or internal communications. In each case, the principle is the same: compare alternatives and measure behavior.
- Reminder timing and channel for appointments, payments or renewals.
- Framing and format of letters, disclosures and educational materials.
- Order of steps and defaults in applications and enrollment forms.
- Call scripts and conversation structures in sales and service.
- Onboarding sequences for patients, members, customers or new employees.
- Training formats and internal communication approaches.
Adapting experiments to real-world constraints
Organizations outside the technology sector often face constraints that make classic A/B testing harder. Volumes may be too low to detect small differences quickly. Randomizing individuals may be impractical when people interact with the same staff or share information with each other. Regulations, union agreements or professional norms may limit what can be varied. These constraints are real, but they rarely make experimentation impossible. They call for design choices rather than abandonment.
Several alternatives are available. Cluster randomization assigns whole locations, teams or time periods to different versions. Staggered rollouts introduce a change in some sites before others, creating a natural comparison. Structured pilots with carefully chosen comparison groups provide weaker but still useful evidence. Longer test periods can compensate for low volume. The key is to decide on the design, measures and decision rule before the test begins, so that results are interpreted honestly.
Ethics and trust in experimentation
Experimenting with how people are treated raises legitimate ethical questions, especially in healthcare, education, finance and public services. The guiding principle is that every version tested should be acceptable on its own terms. Testing two reasonable reminder messages is very different from withholding a beneficial service from one group. Where an intervention is clearly better, it should simply be offered. Experiments are for genuine uncertainty about which reasonable option works best.
Good practice includes review by an appropriate body, such as an institutional review board in research settings or an internal ethics or compliance group elsewhere. Consent and transparency should match the setting and the risk. Data should be handled with care and minimized where possible. Organizations should also consider fairness, checking whether a version helps some groups while disadvantaging others. Done this way, A/B testing strengthens trust, because decisions rest on evidence rather than on preference or assumption.
- Test only options that are acceptable on their own.
- Seek appropriate ethics or compliance review.
- Match transparency and consent to the level of risk.
- Check effects across groups for fairness.
Designing a good test
A good A/B test starts with a hypothesis, not a hunch. The hypothesis should name a behavior, a population and a predicted change, and it should be grounded in research about why the current behavior occurs. The outcome measure should be behavioral and meaningful, such as completed applications, kept appointments or renewed memberships, rather than opens or clicks alone. The test should vary one thing at a time where possible, so the result can be interpreted clearly.
Sample size and duration should be estimated in advance, and the decision rule should be written down: what result will lead to adopting the new version, keeping the old one or testing further. Results should be examined by segment, because an average effect can hide very different responses among different behavioral types. Finally, every result should be documented, including null and negative results. A library of tested hypotheses becomes an organizational asset that prevents teams from repeating experiments or relearning old lessons.
How Datalidy applies A/B testing beyond tech
Experimentation is the Test phase of Datalidy's Observe, Signal, Design, Test, Scale method, and it is where the firm's Behavioral Signals Design methodology proves its value. Predictive KPIs defined in the Signal phase become outcome measures, and interventions created in the Design phase become the versions under test. Datalidy designs A/B tests where volume allows and structured pilots where it does not, drawing on research methods and clinical research practice to keep tests rigorous and ethical.
The firm has supported experimentation for organizations in healthcare and digital health, automotive, mortgage lending, financial services, associations and education. It also helps teams build internal capability, so that testing becomes routine rather than a special project. That includes training staff to write hypotheses, choose behavioral outcome measures and interpret results without overreading them. In the Scale phase, Datalidy helps embed successful interventions and the habits of experimentation through dashboards, process design and leadership coaching, so learning continues after an engagement ends.
Frequently asked questions
How many people do we need for an A/B test?
It depends on how large a difference you expect and how variable the outcome is. Detecting small effects requires larger groups, while large effects can be seen with fewer people. A sample size estimate before the test prevents ending it too early or running it longer than necessary. When volume is limited, longer test periods, cluster designs or structured pilots can still produce useful evidence.
Is A/B testing appropriate in healthcare?
Yes, when designed responsibly. Many health organizations test reminder messages, educational materials, scheduling processes and patient onboarding. The key conditions are that every version tested is acceptable care or communication, that appropriate ethics or compliance review is in place, and that data is protected. Experiments should address genuine uncertainty, never withhold something known to be beneficial.
What if a test shows no difference?
A null result is still information. It may mean the change did not affect behavior, that the test lacked enough volume to detect a modest effect, or that different groups responded in opposite ways. Examine results by segment, check whether the intervention was delivered as intended, and document the finding. Knowing what does not work prevents wasted investment and sharpens the next hypothesis.
