How to Evaluate AI Tools Before You Buy: A 7-Day Protocol

Evaluate an AI tool in seven days, not seven weeks. Write the job it must do as one testable sentence, find the vendor's own definition of its billing unit before you compare prices, then run the export on day one — while canceling is still free. Score the result out of 40. Buy above 32, walk below 24, and negotiate the specific weak line in between.

What most AI tool evaluations get wrong#

Most buying guides give you two instructions: compare the pricing, and run a pilot. Neither is executable as written.

"Compare the pricing" assumes the numbers measure the same thing. They usually do not. "Run a pilot" has no start date, no end date and no failure condition, so it drifts until someone signs anyway.

Zylo's 2026 SaaS Management Index, which surveyed 218 IT leaders as part of a dataset covering more than 40 million licenses, reported that 78% had seen unexpected charges from consumption-based or AI pricing and that 61% had cut projects because of unplanned SaaS cost increases. The same index put average unused licenses at 36%.

Step 1: Write the job to be done as one testable sentence#

Before you open a single pricing page, write the job in a sentence a colleague could test without asking what you meant.

Bad: "improve the support workflow." Good: "classify every inbound support email into one of six queues and draft a first reply, within 60 seconds of arrival, for the 400 emails that arrive each weekday."

The good version contains a volume, a latency, a unit of output and a count of categories. Every one becomes a test you can run in the trial. The bad version can only be judged by feel, which is how tools get bought and then abandoned.

Put the sentence at the top of your scoring sheet. A feature that does not serve it earns no points.

Step 2: Find the billing unit before you compare a single price#

Do this before you build the spreadsheet, not after. The unit is what you are actually buying — the price is only its label.

Zapier's pricing page defines a task as a unit of work Zapier successfully completes for you, and states that failed actions are not counted. It also states that triggers and polling for new data do not count, and neither do its own built-in data tools — Formatter, Paths, Filter, Delay, Looping, Sub-Zap, Digest, Zapier Manager, Storage, Tables and Forms are all named as free of task cost. Some actions consume multiple tasks depending on the step type and the AI model tier used.

n8n's pricing page defines its unit differently: an execution is a single run of your entire workflow, and the number of steps or the volume of data inside that run does not change the count.

So a 2,000-task allowance and a 2,000-execution allowance are not the same product at different prices. A workflow with eight billable action steps burns eight units on one platform and one on the other. Putting both into a spreadsheet column headed "included volume" is arithmetic on incompatible units.

The rule: open the vendor's own definition of its unit, paste it into your notes verbatim, and only then write down a price. If you are sizing an API-based build rather than a no-code platform, the same discipline applies to tokens — estimating LLM API costs before you build walks through that version. For a side-by-side of the automation platforms themselves, see n8n vs Make vs Zapier.

The 7-day trial protocol#

Seven working days, one owner, a fixed decision date. The point of the clock is that it forces the awkward tests to happen while cancellation still costs nothing.

1. Day 1 — run the export

   Put a small amount of real data in, then immediately export it. Details in the next section. If the export fails, stop the trial here and save yourself six days.

2. Day 2 — your hardest real input

   Not the demo data. Take the input that your current process handles worst — the malformed invoice, the 40-page contract, the support ticket in three languages — and run it. Record the output verbatim.

3. Day 3 — volume and latency

   Run one full day of realistic volume in a single sitting. Time it. Note every rate limit, queue delay or manual step you hit, because those are what the demo hid.

4. Day 4 — integrations

   Connect it to the two systems it must talk to in production. If either requires a higher plan, a partner, or an engineer you do not have, write down that cost now.

5. Day 5 — the second user

   Hand it to someone who was not in the sales call and give them the one-sentence job with no training. What they get stuck on is your onboarding cost, in hours.

6. Day 6 — the paperwork

   Request the DPA, the sub-processor list and the security report for the exact plan on your quote. A vendor that cannot produce these in 24 hours will not produce them faster after you have paid.

7. Day 7 — score and decide

   Fill in the scoring sheet below. Decide the same day. A trial that gets extended without a new question to answer has already told you the answer.

Day 1: test the exit before you test the features#

Run the export on day one. Your data volume is smallest then, cancellation is still free, and a bad answer saves you the six days you would otherwise spend falling in love with the interface.

Here is what "run the export" means in practice. Create three real records. Export them. Open the exported file in software the vendor did not write — a spreadsheet, a text editor, a database import. Then check whether the relationships between records survived the trip.

Exportability and lock-in: what to actually check#

Export completeness is not one question, it is five. Ask them in this order.

• Format. CSV, JSON or a documented API — not PDF, not HTML, not a proprietary archive that only reimports into the same product.
• Coverage. Does the export include your configuration and prompts, or only the output rows? Rebuilding 200 prompts by hand is a real migration cost.
• Self-service. Can you trigger the export yourself, or does it need a support request? Support-gated exports get slow exactly when the relationship sours.
• Deletion. GDPR Article 28(3)(g) requires the processor contract to have the processor delete or return all personal data at the end of the service, at the controller's choice, and delete existing copies unless law requires retention. Ask which of the two you get, and how deletion is confirmed.
• Sub-processors. Article 28(2) requires prior specific or general written authorisation before a processor engages another processor. Where that authorisation is general, the processor must inform the controller of intended additions or replacements, giving the controller the opportunity to object. Article 28(4) requires the same data protection obligations to be imposed on that sub-processor. Ask for the current list in writing.

The number you actually want is not "can you get the data out" but "how many hours to be running on something else in month four". If nobody can answer that in a number, the lock-in is worse than you think.

Hidden costs that never appear on the pricing page#

The advertised number is usually the annual-prepay number. Three patterns account for most of the gap between the quote and the invoice.

Annual-only pricing dressed as a monthly rate. Zapier states that annual subscriptions are billed yearly at 33% off the monthly rate. Microsoft lists its Microsoft 365 Copilot add-on at $18.00 per user/month paid yearly against a $21.00 list price (checked August 2026), and at $25.20 per user/month on monthly billing. The $18.00 figure carries three conditions Microsoft states on the same page: an annual commitment is required, the promotional rate runs through 30 September 2026, and it applies to the first year only.

Bundled seats and mandatory services. HubSpot's Marketing Hub pricing page lists Professional as including 3 core seats with additional core seats at $45/mo each, and Enterprise as including 5 core seats with additional seats at $75/mo (checked August 2026). Both carry a required one-time onboarding fee — $3,000 for Professional, $7,000 for Enterprise. HubSpot also states that once you select a contact tier you cannot downgrade until your contract renews.

Overage that is on by default. Zapier's help documentation states that exceeding your task limit automatically switches you to pay-per-task billing, enabled by default for accounts created after January 2024. Usage stops at three times your plan limit, and the per-task rate depends on your plan and billing cycle — Zapier directs you to your own Billing settings for the figure rather than publishing it. It also notes that pay-per-task rates change for monthly plans from the first billing cycle occurring on or after 15 July 2026. That is a real limitation: you cannot price your worst month from the public pricing page.

n8n carries the same exposure from the other direction. Its pricing page states that overage charges may apply if you do not upgrade to the next usage tier, prices additional Business-plan capacity at €4,000 per 300,000 executions, and invoices unpaid overages 45 days after they occur — so the cheap-looking unit still has a step function behind it, and the plan prices are quoted in euros rather than dollars.

Security and data handling: the questions worth asking#

Six questions, all answerable from documents a vendor can send you.

1. Is customer content used to train models by default, and for which plan? OpenAI's API documentation states that data sent to the API has not been used to train or improve its models since 1 March 2023 unless you explicitly opt in. Anthropic states that by default it does not use inputs or outputs from commercial products such as Claude for Work and the Anthropic API to train models.
2. How long is content retained, and what changes that? OpenAI states abuse-monitoring logs are generated by default for API usage and retained up to 30 days, with a Zero Data Retention option that excludes customer content from those logs. Anthropic states that where a user submits feedback — a thumbs up or down, for example — it stores the related conversation for up to five years.
3. Which SOC 2 categories does the report cover? The AICPA Trust Services Criteria name five: Security, Availability, Processing Integrity, Confidentiality and Privacy. A badge does not tell you which were in scope. Read the report's scope section and the period it covers.
4. Which risk framework does the vendor map to? NIST's AI Risk Management Framework is explicitly voluntary guidance organized around four functions — Govern, Map, Measure and Manage — with a separate Generative AI Profile published in July 2024. Asking which controls map to those functions beats asking whether a vendor is "compliant".
5. What transparency rules touch your use case? The EU AI Act's general applicability date was 2 August 2026. Its transparency rules require people to be told when they are interacting with a machine and require generative AI output to be identifiable, with deepfakes clearly labeled. The Commission places the vast majority of AI applications currently used in the EU in the minimal-or-no-risk category. Put this to your own counsel — it is a procurement question, not legal advice.
6. Who are the sub-processors, and how will you be told when they change? Get the answer as a contract clause rather than a sales email.

For the deeper version of this list, including the questions to ask about admin controls and audit logs, see the AI tool security checklist.

The tier trap: the terms you read are not always the terms you buy#

This is the most common evaluation error, and it runs in both directions.

Anthropic's August 2025 announcement about consumer terms shows how sharp the split can be inside one vendor. It applies to Claude Free, Pro and Max plans: users who allow their data to be used for model improvement get a five-year retention window on new or resumed chats, while users who do not keep the existing 30-day retention. The announcement explicitly excludes Claude for Work, Claude for Government, Claude for Education and API use, including through Amazon Bedrock and Google Cloud's Vertex AI.

So a team that trials on a personal plan and reads the consumer terms may reject a product over a policy that would never have governed its purchase. A team that reads the enterprise page and then trials on a consumer login has the opposite problem: real company data sitting under terms nobody reviewed.

The rule: get the data policy and DPA for the exact SKU on your quote. Not the vendor's name, not the product family — the SKU.

The "who is this bad for" test#

Ask the vendor directly: who should not buy this? A rep who cannot name a single bad-fit customer either does not know the product or is not going to tell you the truth about it.

Then run the test on yourself. This 7-day protocol is the wrong method if:

• The tool's value is cumulative. A knowledge base that gets useful after 5,000 indexed documents cannot show you anything in a week. Extend to a 30-day trial with a documented mid-point checkpoint.
• Your buying decision needs a formal security review. Start that on day one in parallel, and expect the review, not the trial, to set your decision date.
• You are replacing a system of record. Migration risk dominates feature fit. Score export and rollback at double weight and rehearse a full parallel run before committing.
• You have no baseline. If you cannot say what the current process costs in hours or errors, you cannot tell whether the tool helped. Measure it for a week first, then start the trial.

The products named in this article appear because their public documentation states billing or data terms clearly enough to quote. That is not a recommendation to buy any of them.

The scoring sheet#

Eight criteria, zero to five each, 40 points total. Score on day 7 in one sitting, with the one-sentence job in front of you.

The decision rule, and it commits:

• 32 and above. Buy. Negotiate the monthly-billing option anyway so year two is not automatic.
• 24 to 31. Not yet. Take the single lowest-scoring line to the vendor in writing as a condition. If they fix it, rescore that line only.
• Below 24. Walk. Two low scores on one tool almost always share a root cause you cannot negotiate away.
• Any zero on criterion 3 or 6. Automatic walk, whatever the total. A failed export test or an unverifiable data policy is not something the other seven criteria offset.

The copy-paste evaluation checklist#

JOB TO BE DONE (one sentence, must contain a number):
_____________________________________________________

BEFORE PRICING
[ ] Vendor's own definition of its billing unit, pasted verbatim
[ ] What that unit does NOT count (triggers, retries, built-in steps)
[ ] Estimated units per month at real volume

DAY 1 - EXPORT
[ ] Export triggered without contacting support
[ ] File opens in non-vendor software
[ ] Config and prompts included, not just output rows
[ ] Written answer on deletion vs return at end of service

DAYS 2-5 - USE
[ ] Hardest real input run, output recorded verbatim
[ ] One full day of volume run in one sitting; rate limits noted
[ ] Both required integrations connected
[ ] Untrained second user attempted the job unaided

DAY 6 - PAPERWORK
[ ] DPA for the exact SKU on the quote
[ ] Data-training and retention policy for that same SKU
[ ] Sub-processor list plus change-notification mechanism
[ ] Security report scope and period read, not just the badge

COST
[ ] Monthly-billing rate without annual commitment
[ ] Included seats and price per additional seat
[ ] One-time onboarding or implementation fees
[ ] Overage behavior on the day you exceed the limit
[ ] Downgrade rules before renewal

DAY 7 - DECIDE
[ ] Eight criteria scored out of 5
[ ] Total ___/40   Decision: buy / condition / walk
[ ] Decision recorded with date and owner

What to do on day 7#

Score it, decide, and write the decision down with a date and a name against it. An evaluation with no recorded owner turns into an auto-renewal in eleven months.

If you buy, do three things in week one: set a calendar reminder 45 days before renewal, put the export procedure in your runbook, and record the baseline number your one-sentence job was supposed to move. If you walk, keep the scoring sheet — the same eight lines work on the next vendor, and the comparison only means anything if you scored both the same way.

The single highest-leverage change to how most teams buy AI tools is moving the export test from never to day one. Everything else here is a refinement. That one move is the difference between choosing a tool and being chosen by it.

For the process behind these product tests, see the testing method.