Reference Checks: The Questions Vendors Don't Want You to Ask Their Other Customers

A construction equipment company was evaluating two autonomous mobile robot vendors for its parts distribution center. Both vendors provided three reference contacts. All six references were positive. All six had purchased within the last 12 months. All six were still in deployment.

After signing with the lower-scoring vendor — partly because of a slightly better price — the procurement director ran into the operations manager at a competitor's facility at a trade show. That manager had deployed the same vendor's system 18 months earlier and had not renewed. The vendor's support response had degraded after the first year. Software updates had introduced regressions. The integration with their WMS had broken twice and taken three weeks each time to restore.

This information would have changed the procurement decision. The vendor didn't offer this contact. The buyer didn't know to look for him.

The reference check process exists to surface information the sales process conceals. Done correctly, it does that. Done the way most buyers do it — by calling the names on the list and asking "are you happy with the product?" — it adds no signal at all.

The Structural Problem with Vendor Reference Lists

When a vendor provides a reference list, they have already made every selection decision for you. They chose which customers to include, when to add them, and how to frame the relationship. The list will reliably contain:

• Customers who are enthusiastic adopters
• Customers who are still in the honeymoon phase of deployment (typically under 12 months)
• Customers who are in ongoing conversations with the vendor about expansion or renewal — customers who have an incentive to maintain the relationship
• Customers who are not in contexts comparable to yours

The list will reliably not contain:

• Customers who did not renew
• Customers who had significant implementation problems
• Customers who are in litigation or active dispute
• Customers who are comparable to you but whose deployment revealed product limitations

A thorough reference check process supplements the vendor list with contacts you find independently.

Finding Contacts the Vendor Didn't Give You

Method 1: Ask for the full customer list, then make independent selections

The most effective method: ask the vendor for a complete list of all current and former customers (or at minimum, a list of all customers in your industry segment), and make your own selections rather than accepting the vendor's five curated names. Most vendors will push back. That pushback is informative.

A vendor who is genuinely confident in their operations will share a broader list and let you pick. A vendor who insists on controlling the reference list is telling you something about what an unfiltered conversation would produce.

If the vendor refuses to provide a broader list, note it in your evaluation. It is not disqualifying on its own — there may be confidentiality constraints — but it shifts the burden to you to find independent references.

Method 2: Trade association networks and peer communities

Procurement forums, industry association networks, and LinkedIn groups for operations directors in your sector often contain people who have deployed the same vendors you are evaluating. A post in the right group — "Has anyone deployed [Vendor X] in a distribution environment? Looking to speak with operators, not the vendor" — will typically generate several responses within a week.

Trade shows are particularly effective. The people working the floor at an operations or logistics conference are exactly the population you want to reach, and many are willing to have candid conversations outside of any formal reference process.

Method 3: Search review platforms for the vendor's product

B2B review platforms (G2, Capterra, TrustRadius) and industry-specific equivalents carry unfiltered buyer reviews. These are not a substitute for direct conversation, but they surface names of past buyers and reveal patterns that would not appear in vendor-curated references. Look for reviews that mention implementation challenges — the reviews that gave 3 stars after initially giving 5 are often the most informative.

Method 4: Ask the vendor directly for a non-renew reference

Say: "In addition to the references you've provided, we'd like to speak with one customer who chose not to renew their contract with you." Watch the reaction.

A vendor with strong operations and a genuine confidence in their product will either provide the reference or give you a specific honest reason why they cannot (e.g., "we only have two non-renewals in five years; one is under NDA for unrelated reasons and one is a former customer in litigation with their own operations contractor — I can tell you both situations"). That response is reassuring.

A vendor who deflects the question, changes the subject, or says they don't have non-renewals will not give you a non-renew reference. Make a note of how they handled the question.

The Reference Call Framework

Structure reference calls in three phases: warm up, probe depth, and break the frame.

Phase 1: Warm Up (5 minutes)

Start with context-setting questions that don't feel adversarial and establish the reference's level of operational involvement:

• "Can you describe how your deployment is configured — fleet size, task types, volume?"
• "How long have you been operational with this system?"
• "What role did you play in the selection and implementation process?"

You are gauging whether this reference has genuine operational depth. A C-level executive who signed the purchase order but never set foot in the facility will give you a different quality of information than an operations manager who troubleshoots issues weekly.

Phase 2: Probe Depth (15 minutes)

These are the substantive questions. Ask them in order — each one builds on the previous:

"What was harder than expected about the implementation?"

This question almost always produces something useful. Almost every implementation has at least one thing that was harder than expected. A reference who says "nothing, it went perfectly" is either in the honeymoon phase or is too invested in the success story to be candid. Follow up: "If you could go back and change one thing about how you approached the implementation, what would it be?"

"What is the most common reason your system is unavailable during scheduled operating hours?"

This cuts through uptime percentages to the operational reality. You want to know whether downtime is predictable and manageable (scheduled maintenance, routine software cycles) or operational (repeated sensor failures, integration instability, network issues). Follow up: "How long does recovery typically take?"

"How has the vendor's support quality changed since the initial implementation period?"

This is the question that most often surfaces useful negative information. Vendor support in the first 90 days of a deployment is typically excellent — the account team is still engaged, the implementation team is responsive, and the vendor is motivated to ensure a successful early period. What happens in month 18, when the implementation team has rotated to another account and your primary contact has changed twice, is the more relevant question.

"Have you had any software updates that changed robot behavior in a way that affected your operations?"

If the answer is yes, follow up on how the vendor handled it: advance notice, rollback capability, support response time, root cause communication.

"What does the vendor do well that you didn't expect?"

Include at least one genuine positive probe. You want the reference to feel that you are seeking a balanced picture, not building a case against the vendor. And vendors often have genuine strengths that the sales process understates — specific industry expertise, strong product management responsiveness, excellent training materials.

Phase 3: Break the Frame (5 minutes)

End with two questions that break the conventional reference conversation:

"If you were selecting this vendor again today, knowing everything you know now, what would you do differently in the procurement process?"

This question surfaces information about where the vendor's proposals were opaque, where contract terms were unfavorable, and where the buyer's own process created problems. It is equally useful for improving your process as for evaluating the vendor.

"Is there anything I haven't asked that you think I should know?"

This is the open-field question. Experienced procurement professionals know that the most important information in a reference call often comes at the end, when the reference has relaxed and is speaking freely.

Reference Call Scoring Matrix

Rate each reference on four dimensions after the call:

A reference who scores 10–12 is providing high-quality signal. A reference who scores 4–6 is providing vendor-filtered marketing information.

Aim for at least three references scoring above 8, with at least one that you identified independently (not from the vendor's list).

What to Do with What You Learn

Reference checks are most valuable when you bring findings back into the contract negotiation. If multiple references report that support quality degrades after year one, negotiate a service level agreement that is measured over a rolling 90-day window and includes financial penalties for sustained degradation — not just a point-in-time uptime metric.

If references consistently describe integration scope disputes ("they said it was included but it wasn't"), add a line-item integration scope annex to your contract before signing.

If references describe software updates that broke their operations, add a minimum advance notice and opt-out requirement for updates to your contract.

Reference data is not just due diligence material. It is a negotiating instrument.

For what to negotiate on price and hidden costs once you have reference data in hand, see the next article in this series: Total Cost of Ownership vs. Sticker Price — What to Add Up.