What a Thorough Technology Evaluation Includes

Beyond "what it does": three more questions

First: how much work will it be to run here? “Supports LTI 1.3” is one line on a matrix. Whether that integration is a configuration screen or a multi-week project pulling in your identity team is a different question entirely. And often the one that ends up setting the timeline.

Second: does it fit how your institution decides things? A university moves consequential decisions through committees, a senate, consultation. A product built around an organization that can simply assign people to a new system will need that gap bridged, and bridging it is real, plannable effort.

Third: what does the agreement say about the parts nobody ever demonstrates? Breach notification, data portability on the way out, how feature changes get communicated. The contract settles all of it, and the contract is usually drafted only after the matrix has already picked a winner.

The rest of this piece covers those three, plus a fourth option worth keeping on the table.

Giving the implementation its own line

Every evaluation compares license cost. Implementation is the bigger number, and the one that shapes the project. Integration with the student information system and the identity provider, data migration, configuration against your academic policies, and the faculty and staff development adoption requires.

Enterprise and student-system projects overrun their planned budget or schedule often enough that treating overrun as the default assumption is the safer bet. Panorama Consulting Group’s annual research has found for years that most enterprise implementations run past budget or timeline (Panorama Consulting Group, ERP Report). The reason is not mysterious. A supplier’s estimate is built for a typical customer, and no institution’s integration environment is typical.

Committees that get this right tend to do the same handful of things. They ask for implementation scope broken out from the license, in hours by role, with a named answer for who supplies those hours. The supplier, a third-party integrator, or your own staff. They write down what the estimate assumes about the environment, from the number of source systems to expected data quality to which integrations are in scope versus billable, because those assumptions are the terms the number rests on. They have their own people, the LMS administrator, the identity team, the instructional designers, size their side independently, and add those numbers up to see what the project will take internally. And they give implementation its own budget line and its own timeline, because two products can carry nearly identical license costs and still be a full year apart in time-to-working.

An implementation planned this way tends to land close to its estimate, because the estimate came from the real environment instead of a reference one.

Reading "built for higher education" correctly

Nearly every product in this market uses the phrase, and it covers a wide range. At one end, a product genuinely designed around how a university operates; at the other, a product where the labels in the interface changed and the underlying model did not. It is a testable claim, so test it. A few things worth asking to see, not just hear about:

A governance workflow. How does the product handle a configuration change that has to clear a curriculum committee before it takes effect? Can it stage the change, hold it for approval, and keep a record of who signed off?

A non-employee user lifecycle. Your users are students this term, alumni next term, applicants the term before that. How do licensing, provisioning, and de-provisioning handle that without someone writing a script to patch it?

A default that meets a requirement. When an out-of-the-box setting has to change to match your accessibility standard or your records policy, is that a supported configuration, or a modification you are now maintaining?

Data portability. Do roster and results data move over the sector’s interoperability standards, LTI, OneRoster, and the rest of what 1EdTech maintains, or through an export someone has to keep up by hand?

Specific answers tell you the claim is real. This is not a knock on suppliers who cannot go deep here. Most of this market is corporate, and products get built for whoever is funding them. It just means “built for higher education” is a sentence worth checking, not a box worth ticking.

Making reference calls that tell you something

Supplier references are useful, and they are most useful when you pick who to call and what to ask.

Choose institutions that resemble yours. Similar Carnegie classification, size, staffing. A doctoral university with forty instructional designers had a different implementation than a regional comprehensive with three. Ask about process, not satisfaction: “How was faculty consultation handled, and how many months did it add to the timeline the supplier originally proposed?” That answer tells you what your own timeline will look like. And talk to at least one institution you found yourself. Post in the relevant EDUCAUSE community group or a professional listserv and ask who is done this migration. Peers tend to be generous with what they learned the hard way.

Planning around the governance calendar

Organizational researchers describe universities as “loosely coupled” systems: the parts connect, but a decision made at the top does not automatically reach every classroom (Weick, 1976). That is why a rollout has to move through several committees and a reading at the senate. It is a normal feature of how academic institutions govern themselves, not a dysfunction to engineer around.

That governance runs on its own calendar. Committees meet monthly. The senate meets a handful of times a term. Summer removes the people whose input the project needs most. A plan built around that calendar from the start stays on schedule. A plan that assumes decisions land on the date printed in the Gantt chart tends to slip.

Before you commit to a go-live date, put the real governance calendar into the plan: which bodies review, when they meet, which decisions can be delegated, and where the quiet stretches of the academic year fall. Choosing a go-live week when faculty have room to absorb it, instead of the second week of term, is one of the higher-leverage calls in the whole project.

Treating the contract as a planning document

Once a product is chosen, the agreement is where the operational details get locked in. The sector has already built shared instruments so you are not drafting each of these from a blank page.

HECVAT 4.1.5 is the Higher Education Community Vendor Assessment Toolkit. A standardized security and risk questionnaire from EDUCAUSE, Internet2, and REN-ISAC. The February 2025 release folded the old Full, Lite, and On-Premise versions into a single assessment and added sections on AI and machine-learning governance and on IT accessibility (EDUCAUSE). Ask every finalist for a completed one, and read the specific answers.

An Accessibility Conformance Report on the current VPAT 2.5 template for every product faculty or students will touch, with the key claims verified rather than filed away. Reading one well is a subject on its own.

SOC 2 Type II for hosted services, scoped to the service you are buying.

A short list of terms is worth locking down before signature: a breach-notification window measured in days; a data-deletion obligation on exit, backed by a certificate; an accessibility conformance commitment with a dated schedule for closing known gaps; audit rights; advance notice before a feature you depend on gets removed or materially changed, with a way out if that change breaks something you cannot work around; and price protection at renewal.

Cooperative purchasing agreements and consortium contracts, through E&I, state master contracts, or a regional consortium, let you buy on terms other institutions already negotiated together. Check whether one already covers the product before drafting from scratch.

The fourth option: extend what you already have

A matrix comparing three products cannot show you a fourth path, keeping the current system, closing its gaps, and bringing back the people who quietly stopped using parts of it.

A platform you have already configured, integrated, and trained your people on carries value you have already paid for: the custom fields, the integrations that finally work, the faculty who know where things live. Replacement is the right call when a platform is genuinely at end of life, when the supplier relationship has broken down, or when the architecture simply cannot do something you now need. When the actual reason is that the newer product demonstrated well, it is worth asking what specifically it does that the current one cannot, whether that is worth restarting the integration and adoption clocks, and what it would cost to close the gap in place instead. The in-place option often delivers most of the benefit with a fraction of the disruption. Whether re-onboarding takes is its own question.

A composite: an evaluation that went smoothly

A regional university of about 6,000 students set out to replace its LMS. Alongside the fifteen-requirement scored matrix, the committee added an integration discovery before the shortlist closed. A working session between the supplier’s technical staff and the university’s LMS and identity teams. That session surfaced a locally built enrollment workflow the supplier’s “standard” integration had not anticipated, and both finalists were asked to price the extra work in writing.

The committee also built the governance calendar into the plan and set go-live for the start of the following summer session, a low-enrollment term, with the full fall cohort on the new system. The contract carried a breach-notification window, a data-deletion clause, and a schedule for fixing the two “partially supports” items flagged in the conformance report.

The implementation came in about a week over its estimate. And the estimate had been a realistic one to begin with. Go-live was quiet. Because the plan had funded a two-term faculty cohort alongside the migration, interactive-feature use in the participating courses was measurably higher a year later than it had been on the old platform. The parts that usually go wrong were exactly the parts the evaluation had looked at.