What Directory Reviews and Secret Shoppers Actually Test (And How to Prepare)

Who Is Actually Calling

Who checks your directory depends on the line of business, and it gets misquoted all the time. Here is what is on the record:

• Medicare Advantage: CMS has run online directory reviews. In round three, it found that 48.74% of 10,504 provider locations had at least one inaccuracy. In 2023, Senate Finance Committee majority staff ran a secret shopper study of mental health listings at 12 MA plans. Staff got a possible appointment on 22 of 120 calls.
• Medicaid managed care: Under 42 CFR 438.68(f), states must use an independent entity to run annual secret shopper surveys, for rating periods beginning on or after July 9, 2028.
• Exchange plans: Issuers on the federally facilitated Exchange hire a third party to survey appointment wait times, under the Letter to Issuers.

The details differ. The four things being tested are much the same, and they are what a member runs into when using your directory.

What Gets Tested

A caller starts from your published directory, the same one your members use, and calls the listed number. Four things can go wrong:

• Reachability: Does the phone number connect to a live person or a working scheduling system? In the Senate study, 39 of 120 calls reached non-working numbers, incorrect numbers or calls that were never returned.
• Location accuracy: Is the provider still practicing at the listed address?
• Plan participation: Does the provider actually accept your plan? "We don't take that plan" means your directory lists someone with no active contract, or someone whose contract has lapsed.
• Appointment availability: Is the provider accepting new patients, and how soon? In MA, 42 CFR 422.112(a)(6)(i) sets 7 business days for primary care and behavioral health that needs attention but isn't urgent, and 30 business days for routine and preventive care. Medicaid's standards under 438.68(e) are 10 business days for outpatient mental health and substance use disorder services and 15 for primary care and OB/GYN, for rating periods on or after July 9, 2027.

Nobody is trying to catch you out. A caller is testing what your enrollee would run into trying to use your plan. That's the standard.

Why Each Failure Happens

Disconnected or non-functional numbers: A provider who moved practices left the old number behind. A solo practitioner retired and the number was reassigned. A group practice changed its scheduling system and the published number is now a fax line. Each of these looks like a valid listing until someone actually dials it.

Wrong address: Providers move, and rosters often carry a billing or corporate address instead of the office where the provider sees patients. A wrong address fails your members, which is the real problem.

"We don't take that plan": This points to a systemic data problem. The provider's contract has terminated and your directory hasn't been updated, or the provider was listed by mistake in the first place.

Appointment timeframe failures are tricky because they reflect real capacity, not just data quality. A provider who is technically accepting new patients but can't offer an appointment in time is still a problem. In thin specialties this is often the hardest one to solve, and it connects most directly to whether your network actually serves your members.

What's at Stake

It depends on the program. MA plans must update directory data within 30 days of learning of a change and attest at least annually under 42 CFR 422.111(m), and from plan year 2028 the REAL Health Providers Act requires an annual accuracy score that CMS publishes starting with plan years on or after January 1, 2029. In Medicaid, shopper-found directory errors reach the state within 3 business days and the plan within 3 more, and the state must post results. More detail is in ghost networks and directory accuracy and Medicaid wait-time standards.

How to Prepare Without Waiting for Someone Else to Call First

The readiness playbook is the same as the playbook for running a good network: regular verification, follow-up on providers with data anomalies, and appointment checks in your highest-risk counties and specialties.

• Call a random sample of your own listings every quarter, oversampling behavioral health.
• Prioritize providers who haven't responded to a directory verification: these are your highest-risk listings.
• Track appointment availability, not just network participation.
• Make a contract termination an event that updates the directory: the window between termination and update is where "we don't take that plan" failures start.

A caller doesn't find surprises. A caller finds the gaps that already exist in your directory. Ongoing monitoring beats two weeks of emergency outreach.