SMS OTP Testing in CI Pipelines — Automate Sign-Up and 2FA

You can test real SMS sign-up and 2FA flows in CI by giving each test run its own temporary verification number: the test creates an activation through the API, enters the number in your app, waits for the OTP, and submits it. Because activations that never receive an SMS are refunded automatically, failed runs do not burn budget — but you still need explicit spend caps, timeouts and cancellation to keep the pipeline fast and predictable.

This guide shows a complete setup: a GitHub Actions workflow, a reusable pytest fixture, a Playwright test, and the guard rails that make SMS tests trustworthy enough to gate a merge.

Why test SMS OTP end to end

Unit tests can mock your SMS provider, and they should. But mocks cannot tell you that the verification screen still accepts international formats, that your rate limiter does not block the second attempt, or that your SMS template still contains a parseable code after a copy change. Those regressions only appear when a real message travels from your sender to a real number and back into the browser.

End-to-end OTP tests typically cover:

• Sign-up with phone verification — the happy path, including number formatting and the code entry UI.
• Login with SMS 2FA — the second factor prompt, resend behaviour and lockout thresholds.
• Phone number change — re-verification on an existing account.
• Localisation — country-specific number formats and templates, using the country parameter.

The QA testing solution page covers manual QA; this article focuses on unattended runs in CI/CD.

Architecture of an OTP test

Every test follows the same five steps:

1. Create an activation with POST /v1/activations and a {service, country} body. The response contains the activation ID and the allocated number.
2. Drive the UI — enter the number in the sign-up form and request the code.
3. Wait for the SMS — either poll GET /v1/activations/{id} until the status is received, or listen for the activation.sms_received webhook.
4. Submit the code and assert the post-verification state.
5. Clean up — if the test failed before the SMS arrived, call POST /v1/activations/{id}/cancel so the wallet is credited immediately.

Polling or webhooks in CI?

Webhooks are the better choice in production because they are push-based and signed (see securing OTP webhooks with HMAC). In CI, however, ephemeral runners usually have no stable public URL, so polling is the pragmatic default. A good compromise:

The helper below polls, with an interval short enough to keep tests fast and long enough to be polite to the API.

A reusable Python helper

Keep all OTP logic in one module so tests stay readable. Field names in API responses are shown illustratively; check the API docs for the exact schema.

# tests/otp_helper.py
import os
import time
import requests

API = "https://otproute.com/api/v1"
SESSION = requests.Session()
SESSION.headers["Authorization"] = f"Bearer {os.environ['OTPROUTE_API_KEY']}"
# requests has no session-wide timeout; pass timeout=15 per call in production code.


class OTPTimeout(Exception):
    pass


def wallet_balance() -> float:
    r = SESSION.get(f"{API}/balance")
    r.raise_for_status()
    return float(r.json()["balance"])


def create_activation(service: str, country: str) -> dict:
    r = SESSION.post(f"{API}/activations", json={"service": service, "country": country})
    r.raise_for_status()
    return r.json()


def wait_for_code(activation_id: str, timeout_s: int = 180, interval_s: float = 3.0) -> str:
    deadline = time.monotonic() + timeout_s
    while time.monotonic() < deadline:
        r = SESSION.get(f"{API}/activations/{activation_id}")
        r.raise_for_status()
        data = r.json()
        if data["status"] == "received":
            return data["code"]
        if data["status"] in ("expired", "cancelled"):
            raise OTPTimeout(f"activation {activation_id} ended as {data['status']}")
        time.sleep(interval_s)
    raise OTPTimeout(f"no SMS for {activation_id} within {timeout_s}s")


def cancel(activation_id: str) -> None:
    SESSION.post(f"{API}/activations/{activation_id}/cancel")

Note that the helper's timeout_s is deliberately much shorter than the typical 20-minute activation window. A CI job should fail fast and cancel rather than hold a runner for the full window.

A pytest fixture with guaranteed cleanup

Wrap the activation in a fixture so cancellation happens even when an assertion fails mid-test:

# tests/conftest.py
import os
import pytest
from otp_helper import create_activation, cancel, wallet_balance

MIN_BALANCE = float(os.environ.get("OTP_MIN_BALANCE", "5"))
MAX_ACTIVATIONS = int(os.environ.get("OTP_MAX_ACTIVATIONS_PER_RUN", "20"))
_created = 0


@pytest.fixture(scope="session", autouse=True)
def budget_guard():
    if wallet_balance() < MIN_BALANCE:
        pytest.exit("OTP wallet below minimum balance; skipping SMS suite", returncode=0)


@pytest.fixture
def otp_number(request):
    global _created
    if _created >= MAX_ACTIVATIONS:
        pytest.skip("per-run activation cap reached")
    country = getattr(request, "param", "us")
    act = create_activation(service=os.environ["OTP_SERVICE"], country=country)
    _created += 1
    yield act
    if act.get("_consumed") is not True:
        cancel(act["id"])

A Playwright test

With the fixture in place, the test itself reads like a user story:

# tests/test_signup_sms.py
from playwright.sync_api import Page, expect
from otp_helper import wait_for_code


def test_signup_with_sms(page: Page, otp_number):
    page.goto("https://staging.example.com/signup")
    page.fill("#email", "[email protected]")
    page.fill("#phone", otp_number["phone_number"])
    page.click("button:has-text('Send code')")

    code = wait_for_code(otp_number["id"], timeout_s=180)
    otp_number["_consumed"] = True

    page.fill("#otp", code)
    page.click("button:has-text('Verify')")
    expect(page.locator("[data-test=welcome]")).to_be_visible()

If you prefer the Node.js Playwright runner, the same polling loop translates directly:

// tests/otp.js
const API = "https://otproute.com/api/v1";
const headers = { Authorization: `Bearer ${process.env.OTPROUTE_API_KEY}`, "Content-Type": "application/json" };

export async function createActivation(service, country) {
  const r = await fetch(`${API}/activations`, { method: "POST", headers, body: JSON.stringify({ service, country }) });
  if (!r.ok) throw new Error(`create failed: ${r.status}`);
  return r.json();
}

export async function waitForCode(id, timeoutMs = 180_000) {
  const deadline = Date.now() + timeoutMs;
  while (Date.now() < deadline) {
    const data = await (await fetch(`${API}/activations/${id}`, { headers })).json();
    if (data.status === "received") return data.code;
    if (["expired", "cancelled"].includes(data.status)) throw new Error(`activation ${data.status}`);
    await new Promise((res) => setTimeout(res, 3000));
  }
  await fetch(`${API}/activations/${id}/cancel`, { method: "POST", headers });
  throw new Error("OTP timeout");
}

The GitHub Actions workflow

Run SMS tests as a separate job so they can have their own schedule, concurrency limits and secrets:

# .github/workflows/sms-e2e.yml
name: sms-e2e

on:
  pull_request:
    paths: ["src/auth/**", "src/signup/**"]
  schedule:
    - cron: "0 3 * * *"
  workflow_dispatch:

concurrency:
  group: sms-e2e-${{ github.ref }}
  cancel-in-progress: true

jobs:
  sms-e2e:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    env:
      OTPROUTE_API_KEY: ${{ secrets.OTPROUTE_CI_API_KEY }}
      OTP_SERVICE: example_app
      OTP_MIN_BALANCE: "5"
      OTP_MAX_ACTIVATIONS_PER_RUN: "20"
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install pytest pytest-playwright requests pytest-rerunfailures
      - run: python -m playwright install --with-deps chromium
      - run: pytest tests/ -m sms --reruns 1 --reruns-delay 10 -p no:randomly

A few deliberate choices here:

• Path filters run the suite only when authentication code changes, plus a nightly schedule for drift detection.
• concurrency with cancel-in-progress stops superseded runs, and the fixture's teardown cancels their open activations.
• A dedicated CI API key. The developer console supports multiple API keys, so give CI its own key. Usage logs then show CI activity separately, and you can revoke it without touching production integrations.

Budget guards

SMS tests cost real money, so treat spend as a first-class test constraint:

Because unreceived activations are refunded automatically, the main cost risk is not failed SMS — it is successful activations created by runaway retries or parallel jobs. The per-run cap is your most important control.

Controlling flakiness

SMS tests involve a third-party network, so some variance is unavoidable. The goal is to make failures meaningful:

• Separate infrastructure failures from product failures. Raise a distinct OTPTimeout when no SMS arrives, and let assertion errors in the UI fail normally. Report them differently in CI so an SMS delay does not look like a broken sign-up page.
• Retry once, not forever. A single rerun with a fresh activation absorbs transient delivery issues without hiding real regressions.
• Fall back to a second country. Parametrise the fixture with an ordered country list; the country routing guide explains how to pick fallbacks.
• Record time-to-OTP. Log the duration from activation creation to code receipt on every run. Trends tell you whether timeouts need adjusting; see measuring OTP delivery latency.
• Isolate accounts. Use unique emails per run so a half-finished sign-up from an earlier run never collides with the current one.
• Keep SMS tests few. Cover each distinct flow once end to end; test edge cases with mocks.

Responsible use

Point automated OTP tests at environments and applications you own or are authorised to test. OTP Route is intended for QA, app verification flows and operational use — respect the terms of any third-party platform involved and never use automation for fraud or abuse.

For the full API surface, see the developer hub and the OTP API integration guide.

Frequently asked questions

Topics: SMS OTP testing · CI OTP automation · end-to-end 2FA testing · Playwright OTP · pytest SMS verification · GitHub Actions OTP · test phone number API