Accountable AI: Constructor University researchers develop AI-powered, human-centered framework for supplier selection
With global supply chains facing increasingly complex and unpredictable socio-economic, environmental and regulatory conditions, many companies are facing a growing need for quick and effective supplier evaluation. To address this concern, a group of researchers at Constructor University have proposed a new supplier selection framework that combines the interpretive power of large language model (LLM) AI with human oversight to improve supplier outcomes. The article, "A governed framework combining large language models and human oversight for supplier selection," was published this past month in the journal Machine Learning with Applications.
While supplier selection has traditionally relied on core factors like cost, quality and delivery performance, procurement teams today must often factorin many other requirements such as sustainability, regulatory compliance, operational risk, resilience and changing market conditions. The new framework developed by Industrial Engineering Professor Dr. Fatahi Valilai, PhD Researcher Mohammadreza Rezaei and Master's student Sridhar Sreemulnath Iyer at Constructor University's Emerging Technologies in Industrial Engineering (EITIE) research group addresses this complexity by proposing a complementary division of responsibility between AI, rule-based analytical systems and human oversight.
"This research was motivated by a question we see arising more and more across our work in emerging technologies: How can we benefit from the flexibility and reasoning capabilities of AI without giving up transparency, traceability and human responsibility in industrial decision-making?” said Prof. Dr. Omid Fatahi Valilai. “Our work therefore moves toward agentic and declarative decision architectures in which AI can interpret changing conditions and formulate possible actions, while explicit analytical rules and human governance determine what is actually implemented.”
The framework proposes leveraging the LLM throughout the process to interpret procurement requests, structure and analyze evidence provided by suppliers, and explain the results using natural language. This role is complemented by human oversight and intervention at key stages, allowing the speed, scale and processing power of AI to be harnessed while still preserving human accountability for the assumptions, evidence and ultimately decisions that inform the final outcome.
“A central insight behind this research is that the value of AI in procurement depends not only on how much information it can process, but also on whether decision makers can understand, examine and responsibly rely on its outputs,” said Mohammadreza Rezaei. “We therefore designed the framework around a clear division of responsibilities. Where established deterministic methods already exist, we retain them for numerical scoring, constraint enforcement, and supplier ranking.”
The framework incorporates four checkpoints to be completed by human procurement professionals:
- Approving the underlying data
- Reviewing scenarios and weighting
- Assessing the relevance of evidence, and
- Authorize the final decision.
Putting the framework to the test
As part of the project, the research team evaluated the framework by applying it retroactively to records of 22,230 historic European public-procurement cases, to see how often the originally selected supplier appeared in the top-ranked results produced by the framework. They found that rankings became much more closely aligned when additional, procurement-specific context was included in the data, such as supplier experience in the relevant category, geographic suitability and previous relationship with the buyer. In these scenarios, the original successful supplier ranked in the framework's top 10 about 32% of the time, compared to just 5% of the time when a more general set of supplier characteristics was considered. This increase suggests that supplier rankings become far more informative when they reflect the specifics of a procurement situation rather than general activity alone.
The team also assessed the LLM and determined it had a valid structured-output rate of 1.000, meaning it consistently returned information in the required format. It also achieved a mean evidence-extraction F1 score of 0.928, indicating a strong performance at identifying and extracting relevant claims from the supplier documents it analyzed. Finally, the researchers tested whether the LLM could consistently explain rankings without changing them. In every explanation case, the model preserved the fixed rankings established by the framework, which suggests that it could help communicate the reasoning behind completed analysis without taking over the calculation or silently altering the result.
Taken together, the findings support the framework’s division of responsibilities, with the AI efficiently interpreting and organizing information and explaining results, programmed analytical tools producing fixed scores and ranking, and human procurement professionals reviewing the evidence and approving consequential decisions. The experiments also identified limitations that reinforce the need for human review, particularly when evidence is incomplete, conflicting or qualified.
Visit the EITIE research group website to learn more about their work on defining the interaction of agents in manufacturing supply networks and proposing new ways to exchange data between the virtual and physical world.
The full published article is available online: https://doi.org/10.1016/j.mlwa.2026.101028
Receive all updates straight to your inbox.